Dealing With Byte Data

00:00 In the previous lesson, I showed you the built-in functions used to create and manipulate string data. In this lesson, I’ll be showing you the two ways of dealing with byte data.

00:10 Bytes are a generic container holding any kind of binary data. Technically speaking, all data types are bytes, as that’s what your computer deals with, but most of the time you want to treat them as a specific data type to take advantage of their features.

00:25 Yes, the capital letter A is technically a byte with the value 65, but treating it as a string means you can do lots of stringy things with it. Sometimes, though, you just want the straight binary data, and Python has two data types to deal with this situation.

00:41 The aptly named bytes data type is an immutable representation of binary data, similar to how strings are immutable, while the byte array is a mutable representation, so if you need to edit your binary data, the byte array is the way to go.

00:57 Say it with me now, let’s go to the REPL and check these out. Like the rest of the data type built-in functions, you can pass a string to bytes() to convert the content. Note my use of the b prefix on this string.

01:12 When converting text, you have to tell the bytes() function how the text is encoded so it knows how to change it into the equivalent byte values.

01:20 For a simple string, you get back what you put in. Without the prefix,

01:27 you get an error because you didn’t say how it was encoded. If I want to try something a little fancier,

01:36 I also get an error because that b prefix only allows ASCII characters, not any Unicode. The encoding argument to bytes() allows you to use something more than the default ASCII.

01:51 You’ll recall from the last lesson that the little snake gave a code point of 128,013. That code point number uniquely identifies the snake, but there are different ways of encoding that into data. Python’s default Unicode encoding is UTF-8, which is probably the most common.

02:10 It stores that 128,013 as a series of bytes. When I hiss here, the bytes() function is showing exactly how UTF-8 stores that info about the snake.

02:22 I hinted earlier that text and encoding is kind of complicated. If I was too subtle, well, hopefully now you’re getting the idea.

02:30 Strings in Python are immutable. You may think you’re concatenating them together or changing them, but in reality what is happening is Python is creating a new string object based on whatever you did to the old one.

02:42 The byte object is the same. Any operation I do on a byte object really is creating a new object with the change applied. Sometimes that can be expensive, so Python also has a mutable mechanism called a byte array.

02:58 The bytearray() function takes the same arguments as the bytes() function, but returns a byte array object instead. Byte objects tend to be faster and take up a little less memory.

03:10 There’s some overhead involved in something being changeable. That said, if you’re having to edit the content, the constant creation of new byte objects based on your operations can be expensive, so in that case you would want a byte array instead.

03:24 When I exit the REPL, I’ll link to more information on these topics as well. The bytes() and bytearray() functions both do conversion based on encoding.

03:34 Not all data can be decoded. For example, you can’t just group together any four bytes and expect it to be valid UTF-8. The encoding requires certain values in a certain order.

03:46 You already saw some of this conversion failure when I tried to convert the Hiss string as ASCII. The bytes() and bytearray() functions take another argument that specifies what to do when an error occurs.

04:05 Remember, the snake isn’t valid ASCII, so if I tried to convert this without the errors argument, it would fail. Setting errors to ignore, like I’ve done here, just skips past anything that can’t be converted.

04:23 The replace error handler replaces any non-valid characters with a question mark.

04:29 Notice that this set of bytes is longer than the previous one, as before, the snake got cut, but this time it got replaced. Also note that the placeholder is a single value representing a character.

04:41 You don’t get four question marks for the four bytes of UTF-8 because you said this was ASCII. Those two are probably the more common of the error handlers.

04:50 There are others, though.

05:01 That’s actually four words squished together. XML, char, ref, replace. Okay, XML isn’t technically a word, and neither is ref, but you get my meaning. Instead of replacing with a generic question mark, this error handler replaces the problematic character with its equivalent XML entity.

05:19 If that number seems familiar, that’s because for high-value code points, the XML entity is just the code point number.

05:36 Two words squished together this time, or at least my spell checker is telling me that backslash is compound. Similar idea to xmlcharrefreplace here, but instead this time the snake is being replaced by the backslash escape sequence used in Python text to represent the same thing.

05:53 That \U is the Unicode code point in hex. There’s one more error handler called strict, but you’ve seen it already because it’s the default.

06:03 That’s what causes the exceptions if something can’t be converted. Before I move to the next lesson, there is one last thing I want to point out. Unlike the numeric and string functions you’ve seen so far, the bytes() function doesn’t convert numbers.

06:16 If you pass a number in,

06:20 what you get back is an initialized sequence of bytes populated with zeros, the length of the calling argument. Here, by calling with 42, I get 42 bytes of zero back in a sequence. Being immutable, I’m not sure how useful the bytes version of this

06:38 is, but the bytearray equivalent provides a quick way of creating a size-specific buffer that you can manipulate later.

06:49 Of course, there’s more at Real Python if you want to learn about binary data or byte arrays as well. Coming up, even more data type creators, this time for collections.