Exploring Application Protocols

00:00 In the previous lesson, I demonstrated the use of a selector to handle multiple sockets in a single program. In this lesson, I’ll move up the network stack to the application layer.

00:10 So far, you’ve only been sending raw bytes back and forth, but most coders don’t want to think in bytes, but in a more abstract concept. For want of a better word, I’m going to call that a message.

00:21 When writing client and server code, I only want to think about that message and its contents, not how many bytes it was or how many packets it took for it to get to my code.

00:29 So you need a way to transition between the byte level and the message level. To do that, you need to be able to distinguish one message from the next out of a stream of bytes.

00:39 One way to do this would be to hard code the message size, but that’s not particularly flexible and for a lot of messages, you might want to be able to have different kinds of things inside the message.

00:50 Consider how HTTP delivers both text containing HTML and binary data containing the images on a web page. Most application protocols solve a similar kind of problem.

01:02 There are a variety of ways to tackle this, but I’m going to show you a mechanism that is similar to a lot of protocols out there. Let’s start with the meat of the message: the content that you want to send back and forth.

01:14 This content may have different types like text, images or sound, and the size of the content is likely to be different from message to message, but to be able to differentiate one message from the next in a stream, you need to be able to know how many bytes belong to this message as opposed to the next one.

01:31 One way of handling that is to have header information that comes before the content. This is typically a dictionary-like thing, which contains key-value pairs that tell the deserializer how big the body content is and what kind of things it contains.

01:46 Well, great, you’ve added a layer of abstraction, but you’ve still got the same problem. The number and size of headers could be different from message to message, so one more layer of abstraction.

01:57 The proto-header is a fixed size thing, which at bare minimum contains the size of the header block. It might also contain a descriptor byte or a few other things if you want to support multiple types of messages, but our protocol has headers to do that same thing, so you don’t have to do that.

02:14 I explained this from bottom up because that’s the reasoning used to come up with the structure. The code that parses this message from a stream is top down though as that’s the order the data hits the stream.

02:25 It starts with the proto-header, which has a fixed size. It reads that value, which contains the number of bytes in the message header block. The parser then can read that many bytes and interpret the data as a dictionary.

02:37 It can then use the dictionary to determine how big the next block is and what content it has. With that information, it knows how many more bytes to expect and what, if anything, to do with the content.

02:49 Let’s take a look at some code that implements the serialization and deserialization of a message.

02:57 This is message.py where I’ve defined the message object and the code that serializes and deserializes it. JSON is a decent format for dictionaries.

03:07 I’m going to use it for the header block, so I need to import that here. First off, in the message object is the initializer. Inside I set all its attributes to None.

03:19 Note that there are two ways that a message might get constructed. One is from bytes, and the other is as a structure that you want to turn into bytes. As such, __init__() is pretty vanilla.

03:30 I have a factory method for the other situation,

03:33 and this is that factory method. It takes some content and a type indicator. Then inside the factory, I create an empty message and then assign its content and type attributes.

03:46 I then calculate the header based on the content and its type, and then serialize the header into JSON. One caveat with this implementation is nobody should muck with the content or content_type attributes.

03:59 If you do, after calling the factory, the header and the serialized copy of it won’t match. I probably should have denoted content and type as underscore values to indicate they were private, but oh, well, I didn’t.

04:12 Once I’ve got the JSON version of the header, I can then calculate the size of the header block, which I’ll then use in the proto-header. All that gets stored away and the factory returns the newly-constructed message.

04:24 Let me scroll down here, and this is the serialization method that turns a constructed message object into bytes for sending. First I check what type the content is.

04:36 If it’s text, I have to pass in the encoding information in order to change it into bytes properly. Our protocol currently supports two types, text and bin.

04:46 In the bin case, I just convert the content directly into bytes, and with that done, I’m ready to serialize. The struct module’s pack() method is used to create static size chunks.

04:58 The greater than H here indicates to use a 16-bit byte to store our header size. That means our headers can’t be more than 64 kilobytes in size, but that’s a fair amount of space as remember, this is just our headers.

05:12 Speaking of after the proto-header size comes the headers themselves, converting the JSON text to bytes, and then finally, the actual content data.

05:22 A little later, I’m going to want to test this class, so implementing a __eq__() method makes it easier to compare two objects to check if they’re the same, and you do that in tests a lot. Equal for our case means the two messages have the same proto header value, content type, and content.

05:41 Scrolling down some more.

05:47 Eventually, I’m going to want a server that handles these messages. That server should handle multiple sockets, which means a selector. That selector needs some way of taking chunks of bytes and turning them into a message, knowing that any given byte may be any part of the message.

06:04 This class stores the contents of bytes received on a stream until it has the exact right amount to construct a message, then returns that message.

06:13 To do that, I’m going to use a finite state machine. When receiving any given byte, the behavior of the code is different if it’s part of the proto-header, headers, or content blocks.

06:23 So I have some states here to indicate what part the current chunk belongs. START is where things begin, HEADERS is when processing the header block, and CONTENT is when processing the content block.

06:36 The message processor initializer class constructs the state in the start position. It has an empty byte buffer and an empty message object. It also stores the address of the connection so that the server can show that info later.

06:53 The receive bytes method is where the byte handling happens.

06:58 Whatever bytes are sent in get appended to the current byte buffer. Then the state is examined. If it is the START state and there is enough in the buffer to deconstruct the proto-header, then use the opposite of pack(), which is unpack() and get the two-byte proto-header out.

07:16 This then gets stored as the size of the HEADER block. Then I strip the proto-header out of the buffer and move to the next state. If you’re in header mode and the buffer is greater or equal to the header size, then you can parse the headers.

07:31 Note that this is an if clause, not an elif clause. If you receive a large number of bytes, then this second condition can fire immediately after the first.

07:41 In the header handling block, I decode the headers first, converting them from bytes to strings, then using JSON to deserialize them into a dictionary.

07:50 Remember that the message object has an attribute for the content type, so I copy that information out of the header dictionary into the attribute. Then once more, I clear the processed data out of the buffer and move to the next state.

08:05 The last stage in the state machine is the CONTENT step. Here, I check if the size of the buffer is greater than or equal to the expected size of the content.

08:14 Why is this a separate if instead of an and you ask? I don’t know. I suspect at some point I refactored this code to fix a bug and never cleaned up. Welcome to the real world. Code doesn’t always have the best choices in it. Anyhow, if the content type is text, then decode the buffer as a UTF-8 string.

08:33 Otherwise, just set the content.

08:36 Then finally, as I’ve consumed a message, reset the state machine to its initial empty state, and then return the resulting message. Down here at the bottom, if you get to this point, it’s because the message processing wasn’t complete, so None gets returned to indicate that.

08:54 Now that you’ve got a message, in the next lesson, I’ll put this in a server.