Serialize Your Data With Python

Textual vs Binary

At the end of the day, all serialized data becomes a stream of bytes regardless of its original shape or form. But some byte values—or their specific arrangement—may correspond to Unicode code points with a meaningful and human-readable representation. Data serialization formats whose syntax consists purely of characters visible to the naked eye are called textual data formats, as opposed to binary data formats meant for machines to read.

The main benefit of a textual data format is that people like you can read serialized messages, make sense of them, and even edit them by hand when needed. In many cases, these data formats are self-explanatory, with descriptive element or attribute names. For example, take a look at this excerpt from the Real Python web feed with information about the latest tutorials and courses published:

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>Real Python</title>
  <link href="https://realpython.com/atom.xml" rel="self"/>
  <link href="https://realpython.com/"/>
  <updated>2023-09-15T12:00:00+00:00</updated>
  <id>https://realpython.com/</id>
  <author>
    <name>Real Python</name>
  </author>
  <entry>
    <title>Bypassing the GIL for Parallel Processing in Python</title>
    <id>https://realpython.com/python-parallel-processing/</id>
    <link href="https://realpython.com/python-parallel-processing/"/>
    <updated>2023-09-13T14:00:00+00:00</updated>
    <summary>In this tutorial, you&#x27;ll take a deep dive (...)</summary>
    <content type="html">
      &lt;div&gt;&lt;p&gt;Unlocking Python's true potential (...)
    </content>
  </entry>
  (...)
</feed>

The Real Python feed uses the XML-based Atom Syndication Format, which is a form of a data serialization format. You can open the document above in any text editor without needing specialized software or libraries. Furthermore, just by looking at this feed, you can immediately break down its structure and probably guess the meaning of the individual elements without checking the format’s specification.

Now, compare the text-based Atom feed above with an equivalent binary feed. You can generate one by parsing atom.xml with xmltodict and dumping the resulting dictionary to a file using MongoDB’s BSON format. When you do, the result will look something like this:

$ hexdump -C atom.bson | head
00000000  51 d2 01 00 03 66 65 65  64 00 46 d2 01 00 02 40  |Q....feed.F....@|
00000010  78 6d 6c 6e 73 00 1c 00  00 00 68 74 74 70 3a 2f  |xmlns.....http:/|
00000020  2f 77 77 77 2e 77 33 2e  6f 72 67 2f 32 30 30 35  |/www.w3.org/2005|
00000030  2f 41 74 6f 6d 00 02 74  69 74 6c 65 00 0c 00 00  |/Atom..title....|
00000040  00 52 65 61 6c 20 50 79  74 68 6f 6e 00 04 6c 69  |.Real Python..li|
00000050  6e 6b 00 72 00 00 00 03  30 00 3f 00 00 00 02 40  |nk.r....0.?....@|
00000060  68 72 65 66 00 20 00 00  00 68 74 74 70 73 3a 2f  |href. ...https:/|
00000070  2f 72 65 61 6c 70 79 74  68 6f 6e 2e 63 6f 6d 2f  |/realpython.com/|
00000080  61 74 6f 6d 2e 78 6d 6c  00 02 40 72 65 6c 00 05  |atom.xml..@rel..|
00000090  00 00 00 73 65 6c 66 00  00 03 31 00 28 00 00 00  |...self...1.(...|

In this case, you use the command-line tool hexdump to view the contents of your binary file, but it’s rather difficult to understand. While there are bits of text here and there, much of the data looks like gibberish.

Another great advantage of a textual data format is that text has a uniform and mostly unambiguous interpretation. As long as you know the correct character encoding, which is almost always the ubiquitous UTF-8 these days, then you’ll be able to read your serialized messages everywhere on any hardware and system imaginable. No wonder so many popular serialization formats like XML, JSON, YAML, and CSV are all based on text.

Unfortunately, there’s no rose without a thorn. Text-based data serialization formats are slower to process than their binary counterparts because text has to be translated to native data types. They also use the available space less efficiently, which can limit their usefulness in larger datasets. Therefore, textual formats aren’t as common in data science and other fields that have to deal with big data.

Moreover, textual formats tend to become overly verbose and may contain a lot of redundant boilerplate. Notice how each element in the Atom feed above, such as the author of the feed, is wrapped in at least one pair of opening and closing XML tags, adding to the bloat. You can try compressing the serialized data to mitigate that, but it’ll add some overhead without providing predictable message sizes.

If you’re able to describe your data using plain English, then a textual data format is all you need. But what if you need to mix both text and binary assets in one text message? It’s possible. After all, email uses a text-based protocol, yet it lets you include binary attachments like pictures or PDF documents.

Embedding binary data in a text-based format requires expressing arbitrary bytes using a limited number of characters. One such technique involves Base64 encoding, which turns bytes into ASCII characters. The downside is that it increases the size of the binary asset by about thirty percent. So, sharing your vacation photos with colleagues at work through email can quickly bring your department’s network down!

There’s also the issue of security risks associated with leaking sensitive information due to the ease of reading textual data formats. In contrast, making sense of a message serialized to a binary data format is more challenging but certainly not impossible. Therefore, serializing private information like passwords or credit card numbers always requires strong encryption regardless of the data format!

To sum up, here’s how textual and binary data formats stack up against each other:

With this table, you can understand the key differences between textual and binary data serialization formats before deciding which one fits your specific needs. Once you know the answer, the next question should be whether to use a schema or not.

Schemaless vs Schema-Based

Regardless of whether they’re textual or binary, many data serialization formats require a schema document, which is a formal description of the expected structure of serialized data. At the same time, some formats are schemaless, while others can work with or without a schema:

Depending on the format at hand, you can express the corresponding schema differently. For example, it’s common to provide an XML-based XSD schema for XML documents, while the binary Avro format relies on JSON for its schemas. Protocol Buffers use their own interface definition language (IDL), on the other hand.

The schema for the Atom feed that you saw earlier leverages a somewhat dated RELAX NG format, which stands for regular language for XML next generation. Unlike the more widespread XML Schema (XSD), it’s not based on XML itself:

# -*- rnc -*-
# RELAX NG Compact Syntax Grammar for the
# Atom Format Specification Version 11

namespace atom = "http://www.w3.org/2005/Atom"
namespace xhtml = "http://www.w3.org/1999/xhtml"
namespace s = "http://www.ascc.net/xml/schematron"
namespace local = ""

start = atomFeed | atomEntry

# Common attributes

atomCommonAttributes =
   attribute xml:base { atomUri }?,
   attribute xml:lang { atomLanguageTag }?,
   undefinedAttribute*

# ...

A schema typically defines the allowed set of elements and attributes, their arrangement, relationships, and the associated constraints, such as whether an element is required or what range of values it can take. You can think of it as the vocabulary and grammar of a data serialization language.

This concept is analogous to a relational database schema, which specifies the tables, their column types, foreign keys, and so on. It’s a blueprint for re-creating the database from scratch, which can also act as a form of documentation. At runtime, the database schema governs your data’s referential integrity. Lastly, it helps facilitate object-relational mapping (ORM) in frameworks like Django.

The benefits of using a schema in a data serialization format include:

• Automation: The formal specification of data allows you to generate code stubs in different programming languages to handle automatic serialization and deserialization of each language’s native data types. This is also known as data binding when you use the XML format.
• Consistency: The schema enforces a standard structure for serialized data, ensuring its integrity and consistency across different systems.
• Documentation: The schema provides a clear definition of the structure of the data, which can help you quickly understand how the information is organized.
• Efficiency: Referencing the schema instead of including explicit field names reduces the size of serialized data. The schema can be known ahead of time or be embedded in the serialized message.
• Interoperability: Sharing the schema between different applications or services can facilitate their integration by allowing them to communicate with each other.
• Validation: The schema can be used to validate serialized data in an automated way, catching potential errors early on.

While there are many benefits to employing a schema, it also comes with some drawbacks, which you should weigh before deciding whether or not to use one. The highest price to pay for adopting a schema is the limited flexibility of the serialized data. The schema enforces a rigid structure, which may not be desirable if your data evolves over time or is dynamic in the sense that it doesn’t have a fixed layout.

Moreover, it can be difficult to alter a schema once you commit to one. Even though some schema-based formats, such as Avro, allow for schema versioning, this may generally break the consistency of data that you previously serialized. To mitigate this problem, some tools in the relational database domain offer schema migration, which gradually transforms data from an old schema to a new one.

For rapid prototyping or when working with unstructured data with unpredictable layouts, a schemaless data serialization format may be more suitable. Conceptually, this is like having a NoSQL database that can accept and process data from multiple sources. New types of elements or unknown attributes would be ignored without breaking the system instead of failing the schema validation.

All in all, these are the most important pros and cons of schemaless and schema-based data serialization formats:

All right, you’ve successfully narrowed down your options regarding the numerous types of data serialization formats. You’ve determined whether choosing a binary format over a textual one is more suitable in your case. Additionally, you understand when to use a schema-based format. Nonetheless, there are still a few choices left on the table, so you must ask yourself some more questions.

General-Purpose vs Specialized

Some data serialization formats are expressive enough to represent arbitrary data, making them universal formats. For example, JSON has become the prevailing data serialization format, especially in web development and REST API design. This format has a minimalistic syntax with only a few essential building blocks that are straightforward to map onto numerous data types in the vast landscape of programming languages.

On the opposite end of the spectrum, you’ll find specialized formats that can only represent a particular type of data. For example, XML continues to be an excellent format for describing deeply nested hierarchical data like user interfaces. After all, XML belongs to the same family of markup languages as HTML, which is widely used to structure content on the Internet.

Another example of a specialized data serialization format is the comma-separated values (CSV) format. It works best with flat tabular data like spreadsheets, database tables, or DataFrames. While you might be able to serialize a data record by mapping its attributes to table columns, modeling hierarchical data with CSV is less convenient than with XML. But, as a textual format, CSV can reach its limits even when handling tabular data.

In data science, you often have to process enormous amounts of data. To optimize performance and reduce storage costs, it’s usually preferable to choose a binary data serialization format dedicated to such large datasets.

These days, Parquet and Feather are gaining popularity in the data science space. They’re both compatible with Arrow, which is an in-memory specification that allows data-sharing between different libraries and even different programming languages. A couple of older but still popular ones are HDF5 and NetCDF. Their newer counterpart, Zarr, offers better support for distributed data storage and computation.

Special data serialization formats emerged in other domains, as well. Some examples include the following:

• DICOM: A binary format for storing and transmitting medical images
• GeoJSON: A specialized flavor of JSON for serializing geographic features
• GPX: An XML-based format for exchanging GPS coordinates
• MusicXML: An XML-based format for storing musical notation
• OBJ: A textual format for storing three-dimensional models

Whether you can use a textual or binary format, with or without a schema, may actually depend on your use case. Some specialized data serialization formats give you little choice in that regard. But there’s one final question that you must ask yourself before choosing the right data serialization format for you. You’ll read about it in the next section.

Portable vs Python-Specific

Another criterion to consider when choosing a data serialization format for your use case is where you’re going to use it. If you wish to exchange information between foreign systems, then opt for a popular data serialization format that’s globally understood. For example, JSON and Protocol Buffers are widely adopted across different programming languages and platforms.

On the other hand, if you only intend to serialize and deserialize data within Python, then you may choose a Python-specific format for practicality reasons. It’ll be a more efficient and convenient option, provided that you’re not planning to share the serialized data with other systems.

Python ships with the following modules in the standard library, which provide binary data serialization formats for different purposes:

• pickle: Python object serialization
• marshal: Internal object serialization
• shelve: Python object persistence
• dbm: An interface to Unix databases

In practice, you’ll almost always want to serialize your objects with pickle, which is the standard data serialization format in Python. The rest on the list are either low-level formats used internally by the interpreter or legacy formats kept for compatibility. You’ll review them now to get a complete picture of the available options.

Python uses marshal behind the scenes to read and write special files containing the bytecode of the imported modules. When you import a module for the first time, the interpreter builds a corresponding .pyc file with compiled instructions to speed up subsequent imports. Here’s a short code snippet that roughly demonstrates what happens under the hood when you import a module:

>>> import marshal

>>> def import_pyc(path):
...     with path.open(mode="rb") as file:
...         _ = file.read(16)  # Skip the file header
...         code = marshal.loads(file.read())
...         exec(code, globals())
...
>>> import sysconfig
>>> from pathlib import Path

>>> cache_dir = Path(sysconfig.get_path("stdlib")) / "__pycache__"
>>> module_path = cache_dir / "decimal.cpython-312.pyc"

>>> import_pyc(module_path)
>>> Decimal(3.14)
Decimal('3.140000000000000124344978758017532527446746826171875')

First, you locate the compiled .pyc file of the decimal module in Python’s standard library and then use marshal to execute its bytecode. As a result, you can access the Decimal class in the highlighted line without having to import it explicitly.

The marshal module is fairly limited in functionality. It can only serialize a few primitive values like integers and strings, but not user-defined or built-in classes. The implementation details of the format are left intentionally undocumented to deter you from using it. The module authors don’t give any guarantees about its backward compatibility. Your marshaled data could become unreadable or incompatible with newer Python versions one day.

The official documentation makes it clear that you’re not supposed to use marshal in your code. Again, you should generally prefer pickle to transmit or persistently store Python objects. But sometimes, you want a simple key-value store for your objects, which might be the case for caching purposes. In such a case, you can take advantage of shelve, which combines pickle and a lower-level dbm module:

>>> import shelve
>>> with shelve.open("/tmp/cache.db") as shelf:
...     shelf["last_updated"] = 1696846049.8469703
...     shelf["user_sessions"] = {
...         "jdoe@domain.com": {
...             "user_id": 4185395169,
...             "roles": {"admin", "editor"},
...             "preferences": {
...                 "language": "en_US",
...                 "dark_theme": False
...             }
...         }
...     }
...

The resulting shelf is a dictionary-like object or, more specifically, a hash table whose keys must be Python strings and whose values must be picklable objects. The module automatically calls .encode() on the keys to apply the default character encoding and uses pickle to convert the corresponding values to byte sequences.

Because shelve is a convenience module that builds on top of pickle and dbm, you can use the latter to take a peek at what’s inside the created file:

>>> import dbm
>>> with dbm.open("/tmp/cache.db") as db:
...     for key in db.keys():
...         print(f"{key} = {db[key]}")
...
b'user_sessions' = b'\x80\x04\x95{\x00\x00\x00\x00\x00\x00\x00}…'
b'last_updated' = b'\x80\x04\x95\n\x00\x00\x00\x00\x00\x00\x00G…'

Both keys and values in the file are stored as raw bytes. Additionally, the values comply with the pickle protocol, which shelve uses internally to serialize Python objects.

Python’s dbm module is an interface to DBM databases popular on Unix. For instance, the manual pages on Unix-like operating systems use this format to index the individual documentation pages. Note that a DBM instance only supports a few basic operations of the regular Python dictionary, so it might not expose methods like .copy() or support the union operators and iteration.

If you have special needs that the standard pickle format can’t fulfill, then consider installing a third-party library like dill, which extends its capabilities while remaining compatible with pickle. It can handle more sophisticated data types, including lambda expressions and Python modules.

One key point to keep in mind when choosing a Python-specific serialization format, regardless of which one, is the security ramifications. Because serialized data may contain malicious code that could compromise or damage your system, you should only deserialize data from trusted sources! This applies to pickle and other formats that allow arbitrary code execution during deserialization.

As you can see, there’s no one-size-fits-all data serialization format, as each comes with its own pros and cons. While this section gave you a comprehensive overview of the various categories, it’s now time dive deeper into concrete formats to understand them better.

Pickle Your Python Objects

As you’ve learned, pickle is the standard data serialization format in Python, and it’s capable of representing a wide range of native data types as well as user-defined classes. It’s a binary format specific to Python, which requires no schema definition and can handle data in almost any shape or form. If you need to persist your Python objects for later use or send them to another Python interpreter, then this should be your preferred choice.

Getting started with pickle is fairly straightforward:

>>> import pickle

>>> data = 255
>>> with open("filename.pkl", mode="wb") as file:
...     pickle.dump(data, file)
...

>>> pickle.dumps(data)
b'\x80\x04K\xff.'

First, you need to import the pickle module and then call either dump() or dumps() to turn an arbitrary Python object into a bytes instance. The first function expects a file-like object open in binary mode for writing, and the latter returns the sequence of bytes to the caller. Both functions allow you to choose the underlying protocol version, which might produce slightly different output:

>>> for protocol in range(pickle.HIGHEST_PROTOCOL + 1):
...     print(f"v{protocol}:", pickle.dumps(data, protocol))
...
v0: b'I255\n.'
v1: b'K\xff.'
v2: b'\x80\x02K\xff.'
v3: b'\x80\x03K\xff.'
v4: b'\x80\x04K\xff.'
v5: b'\x80\x05K\xff.'

>>> pickle.DEFAULT_PROTOCOL
4

At the time of writing, protocol version 5 was the highest, and version 4 was the default. In general, higher versions are more efficient but may not work in older Python releases. Therefore, it’s usually good practice to stick with the relatively conservative defaults if you want your code to remain portable across Python versions. You can always try compressing the resulting byte stream to reduce its size.

Deserializing pickled data boils down to calling the module’s load() or loads(), which are counterparts of the two functions mentioned earlier:

>>> with open("filename.pkl", mode="rb") as file:
...     pickle.load(file)
...
255

>>> pickle.loads(b"\x80\x04K\xff.")
255

When unpickling data, you don’t get the option to specify the protocol version because it gets baked into the pickled byte sequence during serialization. This ensures that you can correctly unpickle your data using the right protocol.

>>> import array
>>> import pickle
>>> pickle.loads(array.array("B", [128, 4, 75, 255, 46]))
255

The pickle module can deal with most data types that you’ll ever work with. Moreover, it can handle objects with reference cycles, such as graphs, including recursive data structures and even deeply nested ones up to the recursion limit in Python:

>>> cycle = {}
>>> cycle["sibling"] = {"sibling": cycle}
>>> pickle.loads(pickle.dumps(cycle))
{'sibling': {'sibling': {...}}}

>>> recursive = []
>>> recursive.append(recursive)
>>> pickle.loads(pickle.dumps(recursive))
[[...]]

>>> import sys
>>> deeply_nested = []
>>> for _ in range(sys.getrecursionlimit() // 2):
...     deeply_nested = [deeply_nested]
...
>>> pickle.loads(pickle.dumps(deeply_nested))
[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[...]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]

The variable cycle in the code snippet above is a dictionary whose only key-value pair holds another dictionary, which in turn contains a reference back to the original dictionary, forming a cycle. The recursive list above has only one element, which is a reference to itself. Finally, the deeply_nested list uses half of Python’s maximum recursion limit to wrap lists inside one another like a Matryoshka doll.

However, there are a few notable exceptions that will cause an error when you try to pickle an object. In particular, instances of the following data types aren’t picklable by design:

• Very deeply nested data structures approaching Python’s recursion limit
• Lambda expressions
• Generator objects
• Python modules
• File objects
• Network sockets
• Database connections
• Threads
• Stack frames

For example, these are the kinds of errors that you’ll get when you try to pickle a lambda expression, a Python module, and a very deeply nested list:

>>> pickle.dumps(lambda: None)
Traceback (most recent call last):
  ...
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7fdd14107920>:
⮑ attribute lookup <lambda> on __main__ failed

>>> pickle.dumps(pickle)
Traceback (most recent call last):
  ...
TypeError: cannot pickle 'module' object

>>> very_deeply_nested = []
>>> for _ in range(sys.getrecursionlimit()):
...     very_deeply_nested = [very_deeply_nested]
...
>>> pickle.dumps(very_deeply_nested)
Traceback (most recent call last):
  ...
RecursionError: maximum recursion depth exceeded while pickling an object

This limitation particularly affects inter-process communication (IPC) in Python, which uses pickle to serialize and deserialize data exchanged between the running interpreter processes:

from concurrent.futures import ProcessPoolExecutor

def worker(function, parameter):
    return function(parameter)

if __name__ == "__main__":
    with ProcessPoolExecutor() as pool:
        future = pool.submit(worker, lambda x: x**2, 10)
        future.result()

This code will result in a pickling error because of the attempt to send a lambda expression to another system process. When you use a process pool from concurrent.futures or the equivalent pool from the multiprocessing module, then you won’t be able to share certain types of data between your workers running in parallel.

Similarly, copying objects in Python relies on the pickle module, which suffers from the same deficiencies:

>>> def generator():
...     yield
...

>>> import copy
>>> copy.copy(generator())
Traceback (most recent call last):
  ...
TypeError: cannot pickle 'generator' object

If that’s too prohibitive for your needs, then consider using the third-party dill package, which acts as a drop-in replacement for pickle. It has the same interface as pickle and adheres to its underlying protocol while extending it to some extent:

>>> import pickle
>>> pickle.dumps(42)
b'\x80\x04K*.'

>>> import dill
>>> dill.dumps(42)
b'\x80\x04K*.'

>>> dill.loads(dill.dumps(lambda: None))
<function <lambda> at 0x7f78e793e660>

>>> dill.dumps(generator())
Traceback (most recent call last):
  ...
TypeError: cannot pickle 'generator' object

For basic data types, dill behaves similarly to pickle but lets you serialize some of the more exotic types, such as lambda expressions. At the same time, it can’t serialize everything—including generator objects.

try:
    import dill as pickle
except ModuleNotFoundError:
    import pickle

Anyway, as long as you stay within most of the native data types in Python, classes from the standard library, or even classes of your own, then pickle is all you need. However, in some cases, you’ll want more control over the serialization and deserialization processes.

Customize the Pickling Process

When you serialize a custom object, Python uses type introspection to discover the underlying class attributes and instance attributes. Whether these attributes are stored in .__dict__ or .__slots__, the pickle module can correctly recognize them. Their names and the corresponding values will be encoded on the target byte stream along with the fully qualified name of your class, which includes the module name where you defined the class.

Deserializing a pickled class instance resembles cloning an object, as it bypasses the initializer method completely. This may not be ideal when your initializer method contains important setup code for the object.

Other situations that may require customizing the pickling or unpickling process include:

• Adding metadata: During pickling, you can introduce extra attributes that aren’t part of your object to augment the serialized byte stream with additional information, like timestamps.
• Hiding sensitive information: To avoid serializing passwords and other secrets, you can exclude one or more attributes from pickling.
• Unpickling stateful objects: If your object has a non-picklable state, such as a database connection or an open file that can’t be directly serialized, then you might need to include some extra logic to handle the state during pickling and unpickling.
• Providing default values: When the structure of your class evolves, it may gain new attributes that old instances lack. In such cases, you can provide default values for missing attributes when unpickling old objects.
• Reducing pickled size: During unpickling, you can recalculate some attributes instead of persisting them on the byte stream if they take a lot of space in serialized form.

To hook into the pickling process, you can specify these two special methods in your class:

1. .__getstate__()
2. .__setstate__()

Python will call the first one before pickling an instance of your class, expecting the method to return a dictionary of attribute names and their values. Conversely, during unpickling, Python will call the second method and pass a dictionary of attributes to let you properly initialize your object after its state has been deserialized.

Say that you have the following data class representing a user in your system, which has the .name and .password attributes:

 1import time
 2from dataclasses import dataclass
 3
 4@dataclass
 5class User:
 6    name: str
 7    password: str
 8
 9    def __getstate__(self):
10        state = self.__dict__.copy()
11        state["timestamp"] = int(time.time())
12        del state["password"]
13        return state
14
15    def __setstate__(self, state):
16        self.__dict__.update(state)
17        with open("/dev/random", mode="rb") as file:
18            self.password = file.read(8).decode("ascii", errors="ignore")

To control the pickling and unpickling of your instances, you defined the two special methods above. Here’s how they work line by line:

• Line 10 makes a copy of the object’s internal dictionary of attributes comprising the user’s name and password.
• Line 11 injects an extra attribute named timestamp with the current Unix timestamp into the copied dictionary.
• Line 12 removes the key-value pair corresponding to the user’s password from the new dictionary.
• Line 13 returns the new dictionary with the modified state of your object to the pickle module.
• Line 16 populates the object’s internal dictionary using the deserialized state provided by the pickle module.
• Lines 17 and 18 set a new password based on random bytes loaded from the /dev/random file on a Unix-like operating system.

As you can see, pickle leaves the door open to overriding its default behavior, giving you plenty of flexibility. If you need even more freedom, then consider defining a custom Pickler object to serialize references of objects persisted externally—for example, in a database. You can also create a private dispatch table to decide when to use a custom serialization code and when to fall back to the default one.

With the protocol version 5 proposed and described in PEP 574, you can now efficiently pickle out-of-bound data. This can help serialize very large datasets, such as NumPy arrays or pandas DataFrames, meant for transferring between processes or distributed machines rather than persisting on disk. By leveraging buffers managed outside the usual pickling process, data engineers can avoid unnecessary copies and significantly reduce memory use.

So far, you’ve seen examples of serializing Python objects to a byte stream using the standard pickle module. It’s the most versatile, efficient, and battle-proven method for object serialization in Python. But it won’t always be the best choice due to its lack of cross-language compatibility and potential security risks, which you’ll learn how to mitigate later.

And now, it’s time to look into serializing Python objects using the text-based JSON format, which enjoys widespread use beyond the Python ecosystem.

Encode Objects Using JSON

Unlike the binary protocols that the pickle module uses, JSON is a textual serialization format readable by humans. Thanks to its huge popularity and simplicity, it’s become a universally accepted standard for data exchange across various programming languages and platforms.

Unfortunately, you can only represent a few Python data types with pure JSON, whereas pickle is able to handle nearly all Python data types out of the box, including more advanced cases like user-defined classes. That said, you’ll learn how to customize the serialization of various types with JSON in this section.

The interface of the json module looks similar to pickle and other serialization-related modules like PyYAML. You get the familiar dump() and dumps() methods, along with their load() and loads() counterparts.

In the following example, you dump a Python object to a JSON string:

>>> import json

>>> data = {
...     "email": None,
...     "name": "John Doe",
...     "age": 42.5,
...     "married": True,
...     "children": ["Alice", "Bob"],
... }

>>> print(json.dumps(data, indent=4, sort_keys=True))
{
    "age": 42.5,
    "children": [
        "Alice",
        "Bob"
    ],
    "email": null,
    "married": true,
    "name": "John Doe"
}

In this case, you serialize a Python dictionary whose keys are strings mapped to a few different data types. What you get as a result is a Python string formatted according to the grammar rules of JSON. Notice that you can optionally request to pretty-print the output and sort the keys alphabetically to improve the readability of larger objects.

The JSON format supports only six native data types:

1. Array: [1, 2, 3]
2. Boolean: true, false
3. Null: null
4. Number: 42, 3.14
5. Object: {"key1": "value", "key2": 42}
6. String: "Hello, World!"

In some cases, the json module will be able to convert your Python values to their closest JSON equivalents. At other times, it’ll fail with the following error:

>>> json.dumps({"Saturday", "Sunday"})
Traceback (most recent call last):
  ...
TypeError: Object of type set is not JSON serializable

The error message above tells you that a Python set isn’t serializable using the JSON format. However, you can teach the json module how to deal with such non-standard data types or even custom classes.

There are two ways to do so. You can provide callback functions to the dumping and loading functions, or you can extend the encoder and decoder classes from the json module.

To serialize a Python set as a JSON string using the first approach, you could define the following callback function and pass its reference to json.dump() or json.dumps() through the default parameter:

>>> def serialize_custom(value):
...     if isinstance(value, set):
...         return {
...             "type": "set",
...             "elements": list(value)
...         }
...

>>> data = {"weekend_days": {"Saturday", "Sunday"}}
>>> json.dumps(data, default=serialize_custom)
'{"weekend_days": {"type": "set", "elements": ["Sunday", "Saturday"]}}'

Python will now call serialize_custom() for every object that it can’t serialize as JSON by itself. It’ll pass down that object as an argument to your function, which should return one of the known data types. In this case, the function represents a Python set as a dictionary with elements encoded as a list and a type field denoting the intended Python data type.

If you’re happy with this new representation, then you’re essentially done. Otherwise, you can implement the corresponding deserialization callback to retrieve the original Python set from its current JSON representation:

>>> def deserialize_custom(value):
...     match value:
...         case {"type": "set", "elements": elements}:
...             return set(elements)
...         case _:
...             return value
...
>>> json_string = """
...     {
...         "weekend_days": {
...             "type": "set",
...             "elements": ["Sunday", "Saturday"]
...         }
...     }
... """

>>> json.loads(json_string, object_hook=deserialize_custom)
{'weekend_days': {'Sunday', 'Saturday'}}

Here, you pass a reference to deserialize_custom() through the object_hook parameter of json.loads(). Your custom function takes a Python dictionary as an argument, letting you convert it into an object of your choice. Using structural pattern matching, you identify if the dictionary describes a set, and if so, return a Python set instance. Otherwise, you return the dictionary as is.

In addition to these generic callbacks, you can specify a few specialized ones to override how to deserialize integer and floating-point number literals in JSON:

>>> json.loads("42", parse_int=float)
42.0

>>> from decimal import Decimal
>>> json.loads("3.14", parse_float=Decimal)
Decimal('3.14')

>>> json.loads("[NaN, Infinity, -Infinity]", parse_constant=str)
['NaN', 'Infinity', '-Infinity']

In the code snippet above, you convert all integers to Python floats and all floating-point number literals to decimal numbers. You also treat the special constants defined in the IEEE 754 standard, such as NaN, as regular strings.

If you prefer a more object-oriented approach to customizing the logic behind JSON serialization in Python, then extend the JSONEncoder and JSONDecoder classes. They’ll let you keep your reusable code in one place while giving you even more fine-grained control, including a streaming API. However, the general principles are the same as with the callback-based approach.

When you decide to use JSON over the binary pickle format, remember that you’ll have to take care of many corner cases manually. One such area is the handling of reference cycles, which pickle accounts for automatically:

>>> cycle = {}
>>> cycle["sibling"] = {"sibling": cycle}

>>> json.dumps(cycle)
Traceback (most recent call last):
  ...
ValueError: Circular reference detected

>>> json.dumps(cycle, check_circular=False)
Traceback (most recent call last):
  ...
RecursionError: maximum recursion depth exceeded while encoding a JSON object

When an object contains a cycle, the pure json module is unable to serialize it. You must implement some mechanism of tracking the objects that you’ve already serialized and reconstruct the original object graph in memory during deserialization.

This exhausts the list of data serialization formats supported natively by Python. Next, you’ll look into some third-party alternatives.

Use Formats Foreign to Python

Both pickle and JSON have their pros and cons. The first one can automatically serialize many types of Python objects out of the box but isn’t portable or human-readable. On the other hand, the text-based JSON is widely supported outside of Python but comes with only a few basic data types. Fortunately, you can combine the best of both worlds with the help of the jsonpickle library.

Contrary to what the name suggests, jsonpickle doesn’t actually use pickle under the hood to serialize Python objects. That would defeat the purpose of creating a more portable and human-readable format. Instead, it uses a custom serialization format on top of JSON to allow you to serialize complex Python objects.

If you still have your User data class around, then this is how you can serialize it to a JSON-formatted string with jsonpickle:

>>> import jsonpickle
>>> from models import User
>>> user = User(name="John", password="*%!U8n9erx@GdqK(@J")
>>> user_json = jsonpickle.dumps(user, indent=4)
>>> print(user_json)
{
    "py/object": "models.User",
    "py/state": {
        "name": "John",
        "timestamp": 1699805990
    }
}

>>> jsonpickle.loads(user_json)
User(name='John', password='\\\x06,\x19')

As you can see, the library respects the standard .__getstate__() and .__setstate__() methods that pickle uses. Moreover, it retains the type’s fully qualified name as a plain string in the resulting JSON under the "py/object" key. This means that the corresponding class must be importable when you try deserializing one of its instances.

That idea sounds remarkably similar to application-specific tags that some YAML parsers use to extend their capabilities. For example, this is what the same user object will look like when you serialize it using the PyYAML library:

>>> print(yaml.dump(user))
!!python/object:models.User
name: John
timestamp: 1699807790

The first line, which starts with a double exclamation mark (!!), uses a custom tag to let you store a User object. YAML is an excellent data serialization format because it supports a large number of data types that you’ll find in many programming languages. Even so, you can extend it with custom data types like the User class above for your specific needs.

Now that you know how to serialize Python objects representing data, you can tackle the task of serializing executable Python code.

Pickle Importable Symbols

In Python, modules, functions, and classes are the most fundamental building blocks of any program, encapsulating its logic. The pickle module can serialize functions and classes but with certain limitations.

If you followed along with the earlier section on customizing the pickling process, then you might recall that pickle only retains the names of your functions and classes. Later, it uses those names to look up the corresponding source code in your virtual environment.

Say you have a Python module with the following definitions, which you’d like to serialize using pickle:

def create_plus(x):
    def plus(y):
        return x + y

    return plus

plus_one = create_plus(1)
plus_two = lambda x: x + 2

The create_plus() function is an example of a higher-order function, or more specifically, a factory function that returns a brand-new function closure created dynamically at runtime. The outer function takes a number, x, as input and returns an inner function, which takes another number, y, as an argument and adds that number to the first one.

The two variables at the bottom, plus_one and plus_two, represent callable objects. The first is a function closure returned by the factory function defined earlier, while the other one is a lambda expression.

Now, because Python modules aren’t picklable, you can’t serialize the entire module using pickle:

>>> import pickle
>>> import plus
>>> pickle.dumps(plus)
Traceback (most recent call last):
  ...
TypeError: cannot pickle 'module' object

By the same token, you can’t serialize inner functions and lambda expressions imported from your module, as well as code objects of regular functions. To get the code object associated with your function, you can access its .__code__ attribute, as shown at the bottom of the following code snippet:

>>> pickle.dumps(plus.plus_one)
Traceback (most recent call last):
  ...
AttributeError: Can't pickle local object 'create_plus.<locals>.plus'

>>> pickle.dumps(plus.plus_two)
Traceback (most recent call last):
  ...
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7f6f32f2a480>:
⮑ attribute lookup <lambda> on plus failed

>>> pickle.dumps(plus.create_plus.__code__)
Traceback (most recent call last):
  ...
TypeError: cannot pickle code objects

The first error message tells you that you can’t pickle a local object, which is a function defined inside another function in this case. Python must be able to reference your object from the global namespace to serialize it. Similarly, a lambda is an anonymous function defined on the fly as an expression that doesn’t belong to the global scope.

You can only pickle top-level functions, such as create_plus(), that you defined with the def keyword in the global namespace of a module:

>>> pickle.dumps(plus.create_plus)
b'\x80\x04\x95\x18\x00\x00\x00\x00\x00\x00\x00\x8c
⮑ \x04plus\x94\x8c\x0bcreate_plus\x94\x93\x94.'

Notice that the resulting byte stream is fairly short and contains the fully qualified name of your function, including the enclosing module name, plus. You can now store this byte sequence on disk or transfer it to another Python interpreter over a network. However, if the receiving end has no access to the corresponding functions or class definitions, then you still won’t be able to unpickle the serialized code:

>>> # This runs on a remote computer without the "plus" module
>>> import pickle
>>> pickle.loads(
...     b"\x80\x04\x95\x18\x00\x00\x00\x00\x00\x00\x00\x8c"
...     b"\x04plus\x94\x8c\x0bcreate_plus\x94\x93\x94."
... )
Traceback (most recent call last):
  ...
ModuleNotFoundError: No module named 'plus'

As you can see, Python is able to deserialize your byte stream correctly, knowing which module to import, but it fails to find the missing plus module.

In contrast, when the enclosing module of your serialized function or class is on the import path, making the module importable, then you’ll be able to retrieve the original object from the byte stream:

>>> # This runs where the "plus" module is importable
>>> import pickle
>>> pickle.loads(
...     b"\x80\x04\x95\x18\x00\x00\x00\x00\x00\x00\x00\x8c"
...     b"\x04plus\x94\x8c\x0bcreate_plus\x94\x93\x94."
... )
<function create_plus at 0x7f89d84a2700>

Here, you get a reference to the create_plus() function, which Python imported for you behind the scenes. It’s as if you had imported it yourself.

The lack of a built-in mechanism for serializing Python source code is a bit disappointing. A quick and dirty way to work around this limitation of the pickle module is to share the code in literal form for the exec() function to execute later:

>>> import importlib
>>> import inspect

>>> def get_module_source(module_name):
...     module = importlib.import_module(module_name)
...     return inspect.getsource(module)
...
>>> source_code = get_module_source("plus")
>>> print(source_code)
def create_plus(x):
    def plus(y):
        return x + y

    return plus

plus_one = create_plus(1)
plus_two = lambda x: x + 2

>>> exec(source_code)
>>> plus_two(3)
5

You define get_module_source() for dynamically importing a Python module by name with the help of importlib. Additionally, you rely on inspect to return the Python source code of the imported module as a string. Then, you use your new helper function to obtain the source code of the plus module that you defined earlier and execute the associated code with its function definitions. Finally, you call one of those functions without importing it.

This trick avoids serializing the code altogether, but it comes with its own set of problems. What if you only wanted to share a specific function or class without revealing the rest of your module? Perhaps executing the module causes unwanted side effects. Maybe you don’t have access to the high-level Python source code because it’s a compiled C extension module. Do you trust that the code will remain intact and secure when you execute it again?

>>> import dill.source
>>> from plus import plus_two
>>> dill.source.getsource(plus_two)
'plus_two = lambda x: x + 2\n'

The next few sections will address these concerns and provide alternative solutions to serializing executable code in Python using the third-party dill library.

Serialize Python Modules

You’ve already heard of the third-party dill library a few times in this tutorial. One of its main advantages over the standard library’s pickle module is the ability to serialize Python source code without requiring an importable enclosing module. However, there’s a catch that you need to be aware of, which you’ll learn about soon.

First of all, you can call dump_module(), which is specific to dill, in order to retain and later restore the state of global variables defined in a module. Unless you specify otherwise, dill serializes the special __main__ module by default. This can be useful when you’re experimenting in the interactive Python REPL and would like to continue the session where you left off, potentially on another computer:

>>> from dataclasses import dataclass

>>> @dataclass
... class Person:
...     first_name: str
...     last_name: str
...
>>> jdoe = Person("John", "Doe")

>>> import dill
>>> dill.dump_module("session.pkl")

Here, you define a data class, create an instance of it, and assign the resulting object to a global variable. Then, you dump the current session to a binary file, which you can load in a new Python REPL session later:

>>> import dill
>>> dill.load_module("session.pkl")

>>> jdoe
Person(first_name='John', last_name='Doe')

>>> Person("Jack", "Ryan")
Person(first_name='Jack', last_name='Ryan')

After loading the serialized module, you continue working with the object that you created before and the class that you defined earlier as if you’d never left the original session.

>>> import dill
>>> dill.load_module("plus.pkl")
Traceback (most recent call last):
  ...
ModuleNotFoundError: No module named 'plus'

Okay. Dumping Python modules with dill allows you to share your functions and classes between Python interpreters, but what if you wanted to be more selective about what you serialize? What about a situation where your recipient doesn’t have access to your source code? In such cases, you can use dill to dump a specific code object associated with a function or class.

Serialize Code Objects

Somewhat surprisingly, when you try serializing a function imported from another module, such as your create_plus() factory function from the plus module, dill and pickle may sometimes produce an identical binary sequence:

>>> import pickle
>>> import dill
>>> import plus

>>> pickle.dumps(plus.create_plus)
b'\x80\x04\x95\x18\x00\x00\x00\x00\x00\x00\x00\x8c
⮑ \x04plus\x94\x8c\x0bcreate_plus\x94\x93\x94.'

>>> dill.dumps(plus.create_plus)
b'\x80\x04\x95\x18\x00\x00\x00\x00\x00\x00\x00\x8c
⮑ \x04plus\x94\x8c\x0bcreate_plus\x94\x93\x94.'

This seems to break the promise made by dill to be capable of serializing such code without needing the enclosing module.

However, if you dump your function from within that module, then you’ll get the expected result. To do so, you can run your module through the Python interpreter with the -i option, which lets you inspect the finished program’s state in an interactive Python shell:

$ python -i plus.py
>>> import dill
>>> dill.dumps(create_plus)
b'\x80\x04\x95U\x01\x00\x00\x00\x00\x00\x00\x8c
⮑ \ndill._dill\x94\x8c\x10_create_function\x94
⮑ \x93\x94(h\x00\x8c\x0c_create_code\x94\x93
⮑ \x94(C\x04\x04\x01\n\x03\x94K\x01K\x00K\x00K
⮑ \x02K\x02K\x03C\x12\x87\x00\x97\x00\x88\x00f
⮑ \x01d\x01\x84\x08}\x01|\x01S\x00\x94Nh\x04(C
⮑ \x02\x04\x01\x94K\x01K\x00K\x00K\x01K\x02K
⮑ \x13C\x0e\x95\x01\x97\x00\x89\x01|\x00z\x00
⮑ \x00\x00S\x00\x94N\x85\x94)\x8c\x01y\x94\x85
⮑ \x94\x8c\x0c/tmp/plus.py\x94\x8c\x04plus\x94
⮑ \x8c\x19create_plus.<locals>.plus\x94K\x02C
⮑ \x0c\xf8\x80\x00\xd8\x0f\x10\x901\x89u\x88
⮑ \x0c\x94C\x00\x94\x8c\x01x\x94\x85\x94)t\x94R
⮑ \x94\x86\x94)h\x11h\r\x86\x94h\x0c\x8c
⮑ \x0bcreate_plus\x94h\x17K\x01C\x0f\xf8\x80\x00
⮑ \xf4\x02\x01\x05\x15\xf0\x06\x00\x0c\x10\x80K
⮑ \x94h\x10)h\x11\x85\x94t\x94R\x94c__builtin__
⮑ \n__main__\nh\x17NNt\x94R\x94}\x94}\x94\x8c
⮑ \x0f__annotations__\x94}\x94s\x86\x94b.'

With this option, Python executes the code in your module, including the function definitions, while letting you stay in the interactive shell to inspect global variables and other symbols afterward.

Because create_plus() is now a part of the global namespace, dill can serialize the function properly. It finally outputs a much longer byte sequence, which includes the serialized code object that you can deserialize without having the plus module in your virtual environment:

>>> # This runs on a remote computer without the "plus" module
>>> import dill
>>> create_plus = dill.loads(
...     b"\x80\x04\x95U\x01\x00\x00\x00\x00\x00\x00\x8c"
...     b"\ndill._dill\x94\x8c\x10_create_function\x94"
...     b"\x93\x94(h\x00\x8c\x0c_create_code\x94\x93"
...     b"\x94(C\x04\x04\x01\n\x03\x94K\x01K\x00K\x00K"
...     b"\x02K\x02K\x03C\x12\x87\x00\x97\x00\x88\x00f"
...     b"\x01d\x01\x84\x08}\x01|\x01S\x00\x94Nh\x04(C"
...     b"\x02\x04\x01\x94K\x01K\x00K\x00K\x01K\x02K"
...     b"\x13C\x0e\x95\x01\x97\x00\x89\x01|\x00z\x00"
...     b"\x00\x00S\x00\x94N\x85\x94)\x8c\x01y\x94\x85"
...     b"\x94\x8c\x0c/tmp/plus.py\x94\x8c\x04plus\x94"
...     b"\x8c\x19create_plus.<locals>.plus\x94K\x02C"
...     b"\x0c\xf8\x80\x00\xd8\x0f\x10\x901\x89u\x88"
...     b"\x0c\x94C\x00\x94\x8c\x01x\x94\x85\x94)t\x94R"
...     b"\x94\x86\x94)h\x11h\r\x86\x94h\x0c\x8c"
...     b"\x0bcreate_plus\x94h\x17K\x01C\x0f\xf8\x80\x00"
...     b"\xf4\x02\x01\x05\x15\xf0\x06\x00\x0c\x10\x80K"
...     b"\x94h\x10)h\x11\x85\x94t\x94R\x94c__builtin__"
...     b"\n__main__\nh\x17NNt\x94R\x94}\x94}\x94\x8c"
...     b"\x0f__annotations__\x94}\x94s\x86\x94b."
... )
>>> create_plus
<function create_plus at 0x7f0f88324220>
>>> plus_three = create_plus(3)
>>> plus_three
<function create_plus.<locals>.plus at 0x7f0f883242c0>
>>> plus_three(2)
5

When you pass the same byte sequence to dill.loads(), it returns a new function object that works exactly like your original create_plus() factory function. Even if the plus module with the corresponding Python source code can’t be found, the deserialized function still works as expected.

Inspecting a module to serialize one of its functions or classes is a lot of hassle. Sure, you could always slap the call to dill.dumps() at the bottom of your module and call it a day:

import dill

def create_plus(x):
    def plus(y):
        return x + y

    return plus

plus_one = create_plus(1)
plus_two = lambda x: x + 2

print(dill.dumps(create_plus))

This might work in some cases, but you generally prefer your code to be flexible so that you can decide later which function or class to serialize. Fortunately, you have another trick up your sleeve. You can manually substitute the special .__module__ attribute and instruct dill to recursively traverse the globals() dictionary:

>>> import dill
>>> import plus

>>> plus.create_plus.__module__
'plus'
>>> dill.dumps(plus.create_plus)
b'\x80\x04\x95\x18\x00\x00\x00\x00\x00\x00\x00\x8c
⮑ \x04plus\x94\x8c\x0bcreate_plus\x94\x93\x94.'

>>> plus.create_plus.__module__ = None
>>> dill.dumps(plus.create_plus, recurse=True)
b'\x80\x04\x95N\x01\x00\x00\x00\x00\x00\x00\x8c
⮑ \ndill._dill\x94\x8c\x10_create_function\x94
⮑ \x93\x94(h\x00\x8c\x0c_create_code\x94\x93\x94
⮑ (C\x04\x04\x01\n\x03\x94K\x01K\x00K\x00K\x02K
⮑ \x02K\x03C\x12\x87\x00\x97\x00\x88\x00f\x01d
⮑ \x01\x84\x08}\x01|\x01S\x00\x94Nh\x04(C\x02
⮑ \x04\x01\x94K\x01K\x00K\x00K\x01K\x02K\x13C
⮑ \x0e\x95\x01\x97\x00\x89\x01|\x00z\x00\x00\x00S
⮑ \x00\x94N\x85\x94)\x8c\x01y\x94\x85\x94\x8c
⮑ \x0c/tmp/plus.py\x94\x8c\x04plus\x94\x8c\x19
⮑ create_plus.<locals>.plus\x94K\x02C\x0c\xf8\x80
⮑ \x00\xd8\x0f\x10\x901\x89u\x88\x0c\x94C\x00\x94
⮑ \x8c\x01x\x94\x85\x94)t\x94R\x94\x86\x94)h\x11h
⮑ \r\x86\x94h\x0c\x8c\x0bcreate_plus\x94h\x17K
⮑ \x01C\x0f\xf8\x80\x00\xf4\x02\x01\x05\x15\xf0
⮑ \x06\x00\x0c\x10\x80K\x94h\x10)h\x11\x85\x94t
⮑ \x94R\x94}\x94\x8c\x08__name__\x94Nsh\x17NNt
⮑ \x94R\x94}\x94}\x94\x8c\x0f__annotations__\x94}
⮑ \x94s\x86\x94b.'

The .__module__ attribute of a function or a class indicates the name of the Python module it belongs to. As you might expect, the initial value of the create_plus() function’s .__module__ attribute is plus. This triggers the fallback mechanism, which delegates the serialization to pickle, resulting in a short byte sequence comprising only the fully qualified name of your function instead of the code object.

By artificially changing this attribute’s value to None, you detach the function from its module. However, callable objects have another special attribute, .__globals__, which is a dictionary representing the global namespace where you defined them. Some of its elements may be redundant, forcing dill to serialize more than necessary and potentially making the enclosing module required for deserialization.

The .__globals__ attribute is read-only, so you can’t easily override it. What you can do, though, is instruct dill to recursively traverse that namespace and cherry-pick objects to serialize as needed. In contrast, when you remove the recurse=True parameter from one of the highlighted lines above, the resulting byte sequence will be a few times longer!

Finally, it’s worth noting that recursive functions need special attention when you deserialize them. Here’s a sample function, which calculates the nth Fibonacci number, represented as a byte stream produced by dill:

>>> import dill
>>> fn = dill.loads(
...     b"\x80\x04\x95-\x01\x00\x00\x00\x00\x00\x00\x8c\ndill._dill"
...     b"\x94\x8c\x10_create_function\x94\x93\x94(h\x00\x8c\x0c"
...     b"_create_code\x94\x93\x94(C\x02\x02\x01\x94K\x01K\x00K\x00K"
...     b"\x01K\x05K\x03CJ\x97\x00|\x00d\x01k\x02\x00\x00r\x02d\x02S"
...     b"\x00t\x01\x00\x00\x00\x00\x00\x00\x00\x00|\x00d\x01z\n\x00"
...     b"\x00\xab\x01\x00\x00\x00\x00\x00\x00t\x01\x00\x00\x00\x00"
...     b"\x00\x00\x00\x00|\x00d\x02z\n\x00\x00\xab\x01\x00\x00\x00"
...     b"\x00\x00\x00z\x00\x00\x00S\x00\x94NK\x02K\x01\x87\x94\x8c"
...     b"\x03fib\x94\x85\x94\x8c\x01n\x94\x85\x94\x8c\x0b/tmp/fib.py"
...     b"\x94h\x08h\x08K\x01C(\x80\x00\xd8\x10\x11\x90A\x92\x05\x881"
...     b"\xd0\x042\x9c3\x98q\xa01\x99u\x9b:\xac\x03\xa8A\xb0\x01\xa9E"
...     b"\xab\n\xd1\x1b2\xd0\x042\x94C\x00\x94))t\x94R\x94c__builtin__"
...     b"\n__main__\nh\x08NNt\x94R\x94}\x94}\x94\x8c\x0f__annotations__"
...     b"\x94}\x94s\x86\x94b."
... )
>>> fn
<function fib at 0x7fd6b68dc220>
>>> fn(5)
Traceback (most recent call last):
  ...
NameError: name 'fib' is not defined

You successfully load the function from that byte stream and assign it to a variable named fn. Later, you use your new variable to call the deserialized function, which fails with a NameError due to referencing an undefined symbol, fib. What’s going on?

A recursive function calls itself until the base condition is met. However, to call itself, the function must know its own name. In the serialized version, the function was named fib, as indicated by the code object that you evaluated in the Python REPL. But there’s no such name in your current scope.

To fix this problem, you can create an alias for the function, like so:

>>> fib = fn
>>> [fib(n) for n in range(10)]
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

Great! With the function assigned to fib, the recursive calls now succeed because the correct function name exists in your scope.

You’ve covered a lot of ground when it comes to serializing executable code. Before closing this topic, you should address the elephant in the room, which is the potential security risk associated with deserializing code that can run on your computer.

Protect From Malicious Code

When you browse Python’s documentation of the various data serialization formats that allow you to serialize code, you’ll notice prominent warnings about the potential security risks involved. These can enable the execution of arbitrary code during deserialization, exposing you to hacker attacks.

The official documentation page of pickle demonstrates an example of a handmade byte stream, which results in running a system command when unpickled:

>>> import pickle
>>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
hello world
0

After unpickling this cleverly crafted byte sequence, Python automatically imports the os module and calls the system() function, passing a command to run in your shell. It merely executes the innocent echo command, which prints hello world on the screen. However, it could potentially do anything, including deleting files from your system, stealing sensitive data, installing malware, or even giving unauthorized access to your computer.

As a rule of thumb, you should always double-check that you can trust the source of the code before allowing it to run. But even that may not be enough when someone in the middle tampers with your code. The recommended practice is to either use a safer serialization format that requires input validation or implement countermeasures to prevent malicious code injection.

When using pickle, you can restrict which functions and classes are allowed to be deserialized by extending the Unpickler class and overriding its .find_class() method:

import importlib
import io
import pickle

class SafeUnpickler(pickle.Unpickler):
    ALLOWED = {
        "builtins": ["print"],
        "sysconfig": ["get_python_version"],
    }

    @classmethod
    def safe_loads(cls, serialized_data):
        file = io.BytesIO(serialized_data)
        return cls(file).load()

    def find_class(self, module_name, name):
        if module_name in self.ALLOWED:
            if name in self.ALLOWED[module_name]:
                module = importlib.import_module(module_name)
                return getattr(module, name)
        raise pickle.UnpicklingError(f"{module_name}.{name} is unsafe")

The SafeUnpickler class defined above contains a custom whitelist consisting of the only functions that you can deserialize. The .safe_loads() class method conveniently wraps the input byte sequence into a file-like object, which the unpickler expects. Finally, the overridden .find_class() method either calls one of the approved functions—returning its result to the caller—or raises an exception.

Any attempt to unpickle a function or class that’s not on your whitelist will fail. At the same time, you can execute deserialized code that’s been marked as safe:

>>> from safe_unpickler import SafeUnpickler

>>> SafeUnpickler.safe_loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
Traceback (most recent call last):
  ...
_pickle.UnpicklingError: os.system is unsafe

>>> SafeUnpickler.safe_loads(b"cbuiltins\nprint\n(S'hello world'\ntR.")
hello world

>>> SafeUnpickler.safe_loads(b"csysconfig\nget_python_version\n(tR.")
'3.12'

By limiting what the pickle module can process, you provide a more secure way of deserializing executable code. But what if that’s too restrictive? Perhaps you really want to be able to call certain functions when the data is coming from a trusted source.

A common method of establishing trust is to digitally sign the serialized code with a shared secret key that only you and your recipient know. In Python, you can take advantage of the standard library module named hmac, which implements the HMAC algorithm used in cryptography for checking the authenticity and integrity of your messages.

Here’s how you can combine dill, hmac, and hashlib to make the serialization of executable code in Python trustworthy:

import hashlib
import hmac
import dill

def safe_dump(obj, file, secret_key):
    serialized_data = dill.dumps(obj)
    signature = sign(serialized_data, secret_key)
    dill.dump(signature, file)
    file.write(serialized_data)

def safe_load(file, secret_key):
    signature = dill.load(file)
    serialized_data = file.read()
    if signature == sign(serialized_data, secret_key):
        return dill.loads(serialized_data)
    raise dill.UnpicklingError("invalid digital signature")

def sign(message, secret_key, algorithm=hashlib.sha256):
    return hmac.new(secret_key, message, algorithm).digest()

The safe_dump() function takes an arbitrary Python object as the first argument, followed by a file-like object, which must be open in binary mode for writing, and a secret byte sequence used for signing. It then dumps the signature of your pickled data to the file using dill and writes the pickled data in the same file right after the signature.

Conversely, the safe_load() function expects a file-like object open for reading in binary mode and a secret byte sequence as arguments. It loads the signature from the file along with pickled data and calculates the signature for that data using the secret key supplied as an argument. If the two signatures compare equal, then the function unpickles and returns your data.

Both functions call the sign() helper to calculate the digital signature of your executable code. By default, the function uses the SHA-256 algorithm, which is popular for storing hashed passwords. However, you can replace it with any available cryptographic algorithm that fits your specific needs.

To verify if this works as expected, import and use your trustworthy module in a new Python shell:

>>> from pathlib import Path
>>> from trustworthy import safe_dump, safe_load

>>> path = Path("code.pkl")
>>> code = lambda a, b: a + b

>>> with path.open(mode="wb") as file:
...     safe_dump(code, file, b"top-secret")
...
>>> with path.open(mode="rb") as file:
...     safe_load(file, b"top-secret")
...
<function <lambda> at 0x7fa1248d0180>

You start by defining a path object to indicate the location of a binary file for your serialized code. The code that you’re going to serialize is a lambda expression stored in a variable named code. Then, you open the indicated file for writing and safely dump your code using top-secret as the key to sign the data with.

Later, you open the same file for reading and safely load its content using the correct secret key. As a result, you get a new callable object, which is a faithful replica of your original lambda expression.

Now, if someone tries to alter the code that you serialized, they’ll need to update the corresponding signature, which they can’t do without knowing the secret key. That’s why it’s important to keep the key secret! If you do, then you’ll immediately know when something’s wrong. The same is true when you supply an incorrect secret key:

>>> with path.open(mode="rb") as file:
...     safe_load(file, b"wrong-key")
...
Traceback (most recent call last):
  ...
_pickle.UnpicklingError: invalid digital signature

The key used to digitally sign the serialized data is different from the one used for deserialization. Therefore, the signatures don’t match, which should raise your suspicion.

Deserializing executable code often carries the risk of introducing potential security threats. If you don’t trust the source where the serialized data is coming from, then it’s best to use a safer format or introduce measures to prevent unauthorized code execution.

Build a REST API Using Flask

Flask is a popular web framework that gives you plenty of flexibility in terms of structuring your project and choosing its internal components. It’s quick to start with, as long as you only need the bare-bones functionality or you know exactly what you’re doing and want fine-grained control over every aspect of your application.

This framework is ideal for creating simple REST APIs because it supports JSON out of the box, building on top of Python’s json module. When your view function returns a dictionary or list, then you don’t need to take any extra steps. Flask serializes your data into a JSON-formatted string behind the scenes, wraps the payload in an HTTP response, and sets the Content-Type header to application/json before sending it back to the client.

Unlike the pure json module, the default serializer in Flask can handle a few additional data types, including data classes, dates and times, decimal numbers, and UUIDs. Below is a sample Flask application demonstrating these capabilities by implementing a basic REST API and using JSON for data serialization:

 1from dataclasses import dataclass
 2from datetime import datetime
 3from uuid import UUID, uuid4
 4
 5from flask import Flask, jsonify, request
 6
 7app = Flask(__name__)
 8
 9@dataclass
10class User:
11    id: UUID
12    name: str
13    created_at: datetime
14
15    @classmethod
16    def create(cls, name):
17        return cls(uuid4(), name, datetime.now())
18
19users = [
20    User.create("Alice"),
21    User.create("Bob"),
22]
23
24@app.route("/users", methods=["GET", "POST"])
25def view_users():
26    if request.method == "GET":
27        return users
28    elif request.method == "POST":
29        if request.is_json:
30            payload = request.get_json()
31            user = User.create(payload["name"])
32            users.append(user)
33            return jsonify(user), 201

This code takes on a few responsibilities that you’d typically organize better in separate Python modules. However, they’re combined here for the sake of brevity. So, to make sense of it, you can scrutinize the code line by line:

• Lines 9 to 17 define a User model as a Python data class with three properties. The named constructor, .create(), sets two of those properties for you using a random UUID and the current date and time.
• Lines 19 to 22 specify a collection of users with two sample users named Alice and Bob.
• Lines 24 to 33 define the only REST endpoint, /users, which responds to the HTTP GET and HTTP POST methods.
• Line 27 returns a list of all users, which Flask serializes as a JSON array.
• Lines 30 to 32 create a new user based on JSON deserialized from the payload and append it to the collection of all users.
• Line 33 returns a tuple comprising the newly created user and an HTTP 201 status code. To help Flask with serializing an instance of your data class, you call jsonify() on the user object to return.

When you run this server in debug mode, then Flask will automatically format your JSON responses using extra indentation, making the payload easier to read. Otherwise, the framework will remove all unnecessary whitespace to make your HTTP responses smaller, conserving the network bandwidth.

As you can see, choosing Flask as your web framework can help you quickly build a REST API with JSON-formatted responses and requests. Frequently, you’ll want to combine Flask with SQLAlchemy or another object-relational mapper (ORM) library. In cases like this, you’ll need an extra tool, such as marshmallow, to serialize the modeled data into JSON.

What if you’re already working with another web framework, such as Django, and would like to use your existing Django models as payload for REST API endpoints? You’ll find out next.

Serialize Django Models With DRF

The Django web framework alone is meant for creating conventional web applications, which typically query a relational database for data that gets weaved into an HTML page template, such as Jinja2. If you’d like to build a REST API on top of your existing Django models, then you’ll need to install a Django extension that provides an additional layer of functionality.

Django REST Framework (DRF) is among the most popular choices, offering numerous features for building robust REST APIs in Django. Assuming you have a working Django project, you can conveniently turn your models into REST API resources by defining serializers, which are strikingly similar to model forms in standard Django.

Say you have the following model class representing a user in your Django application:

import uuid

from django.db import models
from django.utils import timezone

class User(models.Model):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4)
    name = models.CharField(max_length=200)
    created_at = models.DateTimeField(default=timezone.now)

This model is analogous to the data class that you saw earlier in the Flask example. But, because it extends the Model class from Django, you can take advantage of the Django ORM and other features built into the framework to make your life easier.

There are many different ways in which you can leverage the Django REST Framework to expose your models through an API. One approach to replicate the same REST API as before is to define two separate serializers for the incoming and outgoing messages:

from rest_framework import serializers

from . import models

class UserSerializerOut(serializers.ModelSerializer):
    class Meta:
        model = models.User
        fields = "__all__"

class UserSerializerIn(serializers.ModelSerializer):
    class Meta:
        model = models.User
        fields = ["name"]

The first serializer includes all fields in your model, as indicated by the special string "__all__". On the other hand, the second serializer only takes the .name property into account while disregarding the others, including unknown properties.

You can use these serializers to either serialize model instances into JSON or deserialize JSON the other way around. Additionally, you can attach one or more validators to your serializers, just as you would with form validation in Django.

You can now define a view function for your REST API endpoint:

from rest_framework import status
from rest_framework.decorators import api_view
from rest_framework.response import Response

from .models import User
from .serializers import UserSerializerIn, UserSerializerOut

@api_view(["GET", "POST"])
def handle_users(request):
    if request.method == "GET":
        users = User.objects.all()
        serializer = UserSerializerOut(users, many=True)
        return Response(serializer.data)
    elif request.method == "POST":
        serializer_in = UserSerializerIn(data=request.data)
        if serializer_in.is_valid():
            user = serializer_in.save()
            serializer_out = UserSerializerOut(user)
            return Response(serializer_out.data, status.HTTP_201_CREATED)
        return Response(serializer_in.errors, status.HTTP_400_BAD_REQUEST)

In this case, you use the @api_view decorator to define a function-based view, which handles both the HTTP GET and HTTP POST methods.

When a client requests your resource with a GET request, you query the database for all users and dump them to JSON using the first serializer, which retains all the attributes of each user.

On a POST request, you try deserializing the incoming payload using the other serializer. If it succeeds, then you save the new user into your database, which sets the next ID of a user. Finally, you return the newly created user to the client by serializing all of its attributes using the first serializer again.

This is barely scratching the surface, as the Django REST Framework has much more to offer. Still, there are cases when you’ll need to look elsewhere. For instance, if you want full support for Python’s asynchronous processing, then your best bet would be to explore frameworks like FastAPI.

Leverage FastAPI for Performance

As the name implies, the FastAPI framework promises high performance and quick development time by combining the best features of existing components. It stands on the shoulders of giants, namely the Starlette web framework and Pydantic data validation library.

Thanks to being asynchronous from the ground up, FastAPI claims to be one of the fastest Python web frameworks today, making it suitable for high-traffic web applications that must handle real-time data. It’s also quick to start with because it resembles Flask, which should make you feel at home if you’re familiar with the latter. Finally, FastAPI lets you use modern Python syntax to write concise and expressive code that’s cheap to maintain.

As a bonus, compatibility with the OpenAPI specification and the JSON Schema standard allows the framework to automatically generate interactive documentation of your API endpoints, as well as serialize and validate their JSON payloads. By default, FastAPI generates two alternative documentation pages based on Swagger UI and Redoc, but it’s not limited to those.

As with Flask, you can return a dictionary or list in your view functions to have them automatically serialized to their JSON counterparts. In addition to this, FastAPI supports the conversion between Pydantic models and JSON-formatted payloads, so you don’t need to write custom serializers like in the Django REST Framework.

If you wish to rewrite the sample REST API that you saw earlier into FastAPI, then this is what it could look like:

 1from datetime import datetime
 2from uuid import UUID, uuid4
 3
 4from fastapi import FastAPI
 5from pydantic import BaseModel, Field
 6
 7app = FastAPI()
 8
 9class UserIn(BaseModel):
10    name: str
11
12class UserOut(UserIn):
13    id: UUID = Field(default_factory=uuid4)
14    created_at: datetime = Field(default_factory=datetime.now)
15
16users = [
17    UserOut(name="Alice"),
18    UserOut(name="Bob"),
19]
20
21@app.get("/users")
22async def get_users():
23    return users
24
25@app.post("/users", status_code=201)
26async def create_user(user_in: UserIn):
27    user_out = UserOut(name=user_in.name)
28    users.append(user_out)
29    return user_out

Again, it helps to break this code down line by line:

• Lines 9 and 10 define a user model comprising the .name property, which you’ll receive as payload on incoming HTTP POST requests.
• Lines 12 to 14 define another user model, which extends the former through inheritance by adding two extra properties initialized automatically at creation time. This is what you’ll send back to the client.
• Lines 16 to 19 specify a collection of users populated with two sample users named Alice and Bob.
• Lines 21 to 23 define an API endpoint triggered by an HTTP GET request, which returns the list of users.
• Lines 25 to 29 define another API endpoint, which responds to HTTP POST requests by adding a new user to the collection and returning it to the client.

Although you could’ve used the regular def keyword, you implemented both endpoints using async def, which is the preferred approach in FastAPI. The underlying web server runs those functions in an asynchronous event loop. When you use plain old synchronous functions in FastAPI, the framework wraps them in a pool of threads to avoid blocking the event loop.

Another noteworthy point is that FastAPI heavily relies on type hinting to serialize, validate, and document your models. Using type hints can make your code more readable and less prone to defects. Your code editor can leverage them, too, by warning you about errors and even providing suggestions as you write your code. Most importantly, FastAPI lets you specify additional constraints for your data right in your models.

So far, you’ve built a REST API using several Python web frameworks. In the next section, you’ll switch gears and explore how to consume one.

Parse JSON Responses With Pydantic

Once you know Pydantic—for example, through FastAPI—you can start using it in your REST API consumers to deserialize the JSON content into Python objects.

Say you’re building a client for the /users endpoint that you saw earlier in this tutorial. To build such an API consumer from scratch, you can use the venerable requests library. Alternatively, you can install the slightly newer HTTPX library, which remains mostly compatible with the latter but supports an asynchronous interface if needed.

This is what a basic API consumer might look like when you incorporate Pydantic and HTTPX:

from datetime import datetime
from uuid import UUID

import httpx
from pydantic import BaseModel, Field, field_validator

class Metadata(BaseModel):
    uuid: UUID = Field(alias="id")
    created_at: datetime

class User(Metadata):
    name: str

    @field_validator("name")
    def check_user_name(cls, name):
        if name[0].isupper():
            return name
        raise ValueError("name must start with an uppercase letter")

if __name__ == "__main__":
    response = httpx.get("http://localhost:8000/users")
    for item in response.json():
        user = User(**item)
        print(repr(user))

The Metadata model defines a base class with two properties that your User model inherits. Notice that you rename the original "id" attribute received from the server as JSON into a .uuid counterpart on the client side using an alias. You may sometimes need this to adapt the responses of a REST API to your internal models.

Additionally, you specify a custom validation rule for the user’s name so that only names starting with a capital letter can get through. Your .check_user_name() method acts as a class method, taking the class instance as the first positional argument. Using the explicit @classmethod decorator isn’t necessary here, while its incorrect placement could cause the validation to fail silently.

Then, you make an HTTP request, asking the server for all users serialized as a JSON-formatted string, which you convert to a Python dictionary by calling response.json(). At this point, you only deal with Python’s primitive data types, such as strings. To turn them into an instance of your Pydantic model, you pass the corresponding keys and values to the User model’s constructor, which triggers the validation.

Although this works, there’s an even better way. Pydantic can deserialize the payload directly, simultaneously parsing and validating a JSON-formatted string in one go using high-performance code. To take advantage of this, you must specify a so-called root model and call .model_validate_json() on your new model class, like so:

from datetime import datetime
from uuid import UUID

import httpx
from pydantic import BaseModel, Field, RootModel, field_validator

# ...

Users = RootModel[list[User]]

if __name__ == "__main__":
    response = httpx.get("http://localhost:8000/users")
    users = Users.model_validate_json(response.text)
    print(users.root)

As a bonus, your code becomes more concise. The root model differs from a regular model in that it operates on the root of your JSON data, which may not have named attributes.

Nowadays, the vast majority of REST APIs out there use JSON as the underlying data serialization format, allowing everyone to quickly integrate with them. Python libraries like Pydantic and web frameworks such as Flask, Django REST Framework, and FastAPI make working with these APIs and JSON data even more convenient and efficient.

Textual: XML, YAML, and JSON

As you learned before, hierarchical data is the perfect use case for XML representation, so you’ll start with that. Expand the collapsible section below to reveal what a sample training program might look like when you express it using this format:

<?xml version="1.0" encoding="utf-8" ?>
<program>
    <metadata>
        <author>John Doe</author>
        <goals>
            <goal>health improvement</goal>
            <goal>fat loss</goal>
        </goals>
    </metadata>
    <exercises>
        <exercise id="plank">
            <muscles>
                <muscle>abs</muscle>
                <muscle>core</muscle>
                <muscle>shoulders</muscle>
            </muscles>
        </exercise>
        <exercise id="push-ups">
            <muscles>
                <muscle>chest</muscle>
                <muscle>biceps</muscle>
                <muscle>triceps</muscle>
            </muscles>
        </exercise>
    </exercises>
    <days>
        <day id="rest-day">
            <type>rest</type>
        </day>
        <day id="workout-1">
            <type>workout</type>
            <routine>
                <segment>
                    <type ref="plank"/>
                    <duration seconds="60"/>
                </segment>
                <segment>
                    <type>rest</type>
                    <duration seconds="10"/>
                </segment>
                <segment>
                    <type ref="push-ups"/>
                    <duration seconds="60"/>
                </segment>
            </routine>
        </day>
    </days>
    <schedule>
        <day ref="workout-1"/>
        <day ref="rest-day"/>
        <day ref="rest-day"/>
        <day ref="workout-1"/>
        <day ref="rest-day"/>
    </schedule>
</program>

You have several options for deserializing such an XML document in Python, using either the standard library or a number of third-party packages.

Here are the XML parsers that ship with Python:

• xml.dom.minidom
• xml.sax
• xml.dom.pulldom
• xml.etree.ElementTree

Additionally, you can install external Python libraries to parse or dump XML documents, including the following:

• untangle
• xmltodict
• lxml
• lxml.objectify
• BeautifulSoup

If you’d like to learn more about them, then head over to the Roadmap to XML Parsers in Python, which takes you on a deep dive into various interesting ways of dealing with XML content in Python. Choose the best approach that meets your requirements.

The downside of using XML as a data serialization format is its verbosity, which often results in large documents even when there isn’t that much data in them. This also affects readability and performance. Despite being a textual format, XML is arguably challenging for a human to read. Finally, the elements in an XML document don’t translate well into common data structures that you’d find in popular programming languages.

That’s where YAML enters the picture. Below, you’ll find an equivalent training program written as a YAML document:

program:
  metadata:
    author: John Doe
    goals:
      - health improvement
      - fat loss
  exercises:
    - &plank
      muscles:
        - abs
        - core
        - shoulders
    - &pushups
      muscles:
        - chest
        - biceps
        - triceps
  days:
    - &restday
      type: rest
    - &workout1
      type: workout
      segments:
        - type: *plank
          duration:
            seconds: 60
        - type: rest
          duration:
            seconds: 10
        - type: *pushups
          duration:
            seconds: 60
  schedule:
   - day: *workout1
   - day: *restday
   - day: *restday
   - day: *workout1
   - day: *restday

When you load this YAML document into Python using, for example, the PyYAML library, you’ll receive a dictionary that you can traverse using the square brackets syntax:

>>> import yaml
>>> with open("training.yaml", encoding="utf-8") as file:
...     document = yaml.safe_load(file)
...
>>> document["program"]["schedule"][0]["day"]
{
    'type': 'workout',
    'segments': [
        {
            'type': {'muscles': ['abs', 'core', 'shoulders']},
            'duration': {'seconds': 60}
        },
        {
            'type': 'rest',
            'duration': {'seconds': 10}
        },
        {
            'type': {'muscles': ['chest', 'biceps', 'triceps']},
            'duration': {'seconds': 60}
        }
    ]
}

Notice how the individual segments of a workout routine on a specified day end up populated with the exercises defined at the top of the document.

YAML is an excellent data serialization format, which fits this particular scenario perfectly. Unfortunately, it’s a missing battery that Python doesn’t support out of the box. Therefore, you’ll need to choose and install an external serializer to start using this format in your programs.

At the same time, YAML has a fairly complicated syntax, which can make serializing data a relatively slow process. If you seek the ultimate performance and portability without sacrificing too much readability, then you should stick to more widespread formats.

Here’s a hypothetical JSON document representing the same training program as before:

{
    "program": {
        "metadata": {
            "author": "John Doe",
            "goals": ["health improvement", "fat loss"]
        },
        "exercises": {
            "plank": {"muscles": ["abs", "core", "shoulders"]},
            "push-ups": {"muscles": ["chest", "biceps", "triceps"]}
        },
        "days": {
            "rest-day": {"type": "rest"},
            "workout-1": {
                "type": "workout",
                "segments": [
                    {"type": "@plank", "seconds": 60},
                    {"type": "rest", "seconds": 10},
                    {"type": "@push-ups", "seconds": 60}
                ]
            }
        },
        "schedule": [
            "@workout-1",
            "@rest-day",
            "@rest-day",
            "@workout-1",
            "@rest-day"
        ]
    }
}

For more tips and tricks on working with JSON data in Python, check out the corresponding tutorial.

All three data serialization formats that you’ve explored so far in this section were textual. Next up, you’ll look into binary formats suitable for serializing hierarchical data.

Binary: BSON

Choosing a binary data serialization format can often be beneficial for improving speed and storage efficiency. Perhaps the most straightforward binary format that you can use to represent a hierarchy of objects is BSON, which stands for Binary JSON. It was invented for the sake of representing JSON-like documents in MongoDB, a popular NoSQL database.

The main idea behind BSON was to make the serialized documents traversable as efficiently as possible to allow quick querying. This was achieved by encoding length and terminator fields, eliminating the need to parse the entire document when looking for a specific piece of information.

BSON is one example of schemaless formats, which are often preferable to schema-based ones in NoSQL databases. These databases typically store unstructured data, unlike their relational counterparts. The lack of a schema gives you more flexibility. On the other hand, it adds overhead to BSON documents, which must encode the field names within the serialized byte stream.

A notable limitation of BSON is that it can only serialize JSON objects or Python dictionaries. So, if you want to represent scalar values or lists, then you must wrap them in a dictionary with at least one key. On the flip side, you have a few additional data types in BSON at your disposal, such as a 128-bit decimal floating point or a UTC date and time, that aren’t available in plain JSON.

To work with BSON in Python, you must first install the official MongoDB driver for Python called PyMongo. Installing it brings a few top-level packages into your virtual environment, including bson.

To serialize the sample training program that you saw earlier, you can take its JSON representation from the previous section and dump it into BSON like so:

>>> import json
>>> import bson

>>> with open("training.json", encoding="utf-8") as file:
...     document = json.load(file)
...     bson.encode(document)
...
b'9\x02\x00\x00\x03program\x00+\x02\x00\x00\x03metadata
⮑ \x00P\x00\x00\x00\x02author\x00\t\x00\x00\x00John Doe
⮑ \x00\x04goals\x00/\x00\x00\x00\x020\x00\x13\x00\x00\x00
⮑ health improvement\x00\x021\x00\t\x00\x00\x00fat loss
⮑ \x00\x00\x00\x03exercises\x00\x8e\x00\x00\x00\x03plank
⮑ \x00;\x00\x00\x00\x04muscles\x00-\x00\x00\x00\x020\x00
⮑ \x04\x00\x00\x00abs\x00\x021\x00\x05\x00\x00\x00core\x00
⮑ \x022\x00\n\x00\x00\x00shoulders\x00\x00\x00\x03push-ups
⮑ \x00=\x00\x00\x00\x04muscles\x00/\x00\x00\x00\x020\x00\x06
⮑ \x00\x00\x00chest\x00\x021\x00\x07\x00\x00\x00biceps\x00
⮑ \x022\x00\x08\x00\x00\x00triceps\x00\x00\x00\x00\x03days
⮑ \x00\xc7\x00\x00\x00\x03rest-day\x00\x14\x00\x00\x00\x02
⮑ type\x00\x05\x00\x00\x00rest\x00\x00\x03workout-1\x00\x99
⮑ \x00\x00\x00\x02type\x00\x08\x00\x00\x00workout\x00\x04
⮑ segments\x00x\x00\x00\x00\x030\x00#\x00\x00\x00\x02type
⮑ \x00\x07\x00\x00\x00@plank\x00\x10seconds\x00<\x00\x00
⮑ \x00\x00\x031\x00!\x00\x00\x00\x02type\x00\x05\x00\x00\x00
⮑ rest\x00\x10seconds\x00\n\x00\x00\x00\x00\x032\x00&\x00\x00
⮑ \x00\x02type\x00\n\x00\x00\x00@push-ups\x00\x10seconds\x00<
⮑ \x00\x00\x00\x00\x00\x00\x00\x04schedule\x00\\\x00\x00\x00
⮑ \x020\x00\x0b\x00\x00\x00@workout-1\x00\x021\x00\n\x00\x00
⮑ \x00@rest-day\x00\x022\x00\n\x00\x00\x00@rest-day\x00\x023
⮑ \x00\x0b\x00\x00\x00@workout-1\x00\x024\x00\n\x00\x00\x00
⮑ @rest-day\x00\x00\x00\x00'

This produces a long byte sequence conforming to the BSON specification. In fact, when you compare it to the original JSON string, you’ll conclude that it’s actually longer! Despite using a compact binary representation, the serialized BSON data contains the extra fields mentioned earlier, which help speed up document queries. Unfortunately, they take up additional space.

While you can use BSON to serialize hierarchical data, you’re limited to a dozen basic data types that it supports. If you want to go beyond that—for example, to serialize a hierarchy of custom objects—then you need to use something else. Protocol Buffers is a good choice, which you’ll explore later when you delve into schema-based data serialization.

Textual Spreadsheets: CSV

For starters, why don’t you try the comma-separated values (CSV) file format? It’s arguably the most common data format out there. It’s a text-based, schemaless, and row-oriented serialization format suitable for tabular data like spreadsheets or database tables.

Although this format seems pretty straightforward, don’t be tempted to implement it yourself! It has a few dialects that hide surprising edge cases. Usually, you’re better off using either Python’s csv module or third-party packages like pandas or polars to read and write CSV files.

Say you wanted to dump a collection of users into a CSV file with the help of the csv module from the standard library in Python. Because the writer object exposed by that module expects each row to be a sequence of values, you can conveniently define your user as a named tuple. This will make it possible for you to pass an instance of the User class directly to the CSV writer without further fiddling.

Here’s what your User class might look like:

from datetime import datetime
from enum import StrEnum
from typing import NamedTuple

class Language(StrEnum):
    DE = "de"
    EN = "en"
    ES = "es"
    FR = "fr"
    IT = "it"

class User(NamedTuple):
    id: int
    name: str
    email: str
    language: Language
    registered_at: datetime

A user has five properties, which are a mix of built-in types, a standard library class, and a user-defined class. One of them is a datetime instance, and another one is a custom enumeration of language codes that you defined just above.

Next, you can populate a list of fake users while taking advantage of the Faker library to generate plausible values for testing:

import random
from datetime import datetime
from enum import StrEnum
from typing import NamedTuple

from faker import Faker

# ...

class User(NamedTuple):
    id: int
    name: str
    email: str
    language: Language
    registered_at: datetime

    @classmethod
    def fake(cls):
        language = random.choice(list(Language))
        generator = Faker(language)
        return cls(
            generator.pyint(),
            generator.name(),
            generator.email(),
            language,
            generator.date_time_this_year(),
        )

You’ve added a named constructor called .fake() to your class, which creates and returns a new instance with fake attribute values filled in. In it, you randomly choose the language to initialize the generator with and then use it to produce the respective values.

Later, you can import the User class in a Python REPL and call its new method in a list comprehension to generate fifty fake users:

>>> from models import User
>>> users = [User.fake() for _ in range(50)]

Each should look similar to this one:

User(
    id=3042,
    name='Anna Arellano Trujillo',
    email='almansaamarilis@example.org',
    language=<Language.ES: 'es'>,
    registered_at=datetime.datetime(2023, 5, 25, 9, 5, 42),
)

Notice how you populated the properties for each user with credible data reflecting the chosen language. For example, a user with 'ES' as their language code has a Spanish name. However, the generated numeric identifiers may not be globally unique.

Now, you can dump these fake users to a CSV file in one go by providing the corresponding list to the writer object’s .writerows() method:

>>> import csv

>>> with open("users.csv", mode="w", encoding="utf-8", newline="") as file:
...     writer = csv.writer(file)
...     writer.writerows(users)
...

Whenever you open a text file in Python, it’s a good practice to explicitly set the character encoding to ensure portability across the operating systems. Additionally, when working with CSV files, it’s recommended to disable the universal newline translation mechanism to avoid issues with quoting field values.

When you run this code, you should see a file named users.csv appear in your current working directory with content that looks like this:

1261,Anna Scott,jessica11@example.org,en,2023-04-04 05:13:12
2821,Steve Townsend,pamela65@example.net,en,2023-05-10 12:53:32
1449,Alderano Pagnotto,ermes07@example.org,it,2023-09-18 09:20:50
1825,Hortense Rocher,usalmon@example.org,fr,2023-06-11 14:17:17
6618,Adolfine Wernecke,salzliane@example.net,de,2023-04-24 15:11:38
4729,Horacio Egea Sarmiento,piriarte@example.com,es,2023-04-15 03:32:02
 ⋮

Great! You can share this file with your colleagues or import it into other software that supports the CSV format.

How about loading those users back into Python from the file that you’ve just created? To do so, you can use the corresponding reader object:

>>> with open("users.csv", mode="r", encoding="utf-8", newline="") as file:
...     reader = csv.reader(file)
...     next(reader)
...
['1261', 'Anna Scott', 'jessica11@example.org', 'en', '2023-04-04 05:13:12']

Here, you call the built-in next() function to take a peek at the first row in the CSV file by advancing the reader object, which happens to be a Python iterator. As you can see, each row is a list of strings, regardless of what the original data types were. Additionally, the order of fields in each row determines their meaning.

Alternatively, you can use a more specific DictReader instance to convert each row into a Python dictionary, where the keys correspond to the field names:

>>> with open("users.csv", mode="r", encoding="utf-8", newline="") as file:
...     reader = csv.DictReader(file, fieldnames=User._fields)
...     next(reader)
...
{
    'id': '1261',
    'name': 'Anna Scott',
    'email': 'jessica11@example.org',
    'language': 'en',
    'registered_at': '2023-04-04 05:13:12'
}

Because you haven’t included a column header in your CSV file, you must specify the field names by hand. In this case, you use the ._fields attribute of your named tuple.

To make your life just a little bit easier, you can write another class method in your User class to perform the necessary type conversions from strings:

# ...

class User(NamedTuple):
    # ...

    @classmethod
    def from_dict(cls, row_dict):
        transforms = {
            "id": int,
            "name": str.title,
            "language": Language,
            "registered_at": datetime.fromisoformat,
        }
        return cls(**{
            key: transforms.get(key, lambda x: x)(value)
            for key, value in row_dict.items()
        })

This method takes a dictionary of fields provided by DictReader as an argument, defines the transformation functions for select fields, and applies them in a dictionary comprehension before returning a new User instance. The lambda expression represents an identity function used as a default value in case there’s no transformation for the given key.

This is how you can use this new class method to load your users from a CSV file:

>>> with open("users.csv", mode="r", encoding="utf-8", newline="") as file:
...     reader = csv.DictReader(file, fieldnames=User._fields)
...     users = [User.from_dict(row_dict) for row_dict in reader]
...

>>> users[0]
User(
    id=1261,
    name='Anna Scott',
    email='jessica11@example.org',
    language=<Language.EN: 'en'>,
    registered_at=datetime.datetime(2023, 4, 4, 5, 13, 12),
)

You get a list of User instances filled with the expected types of values. Additionally, one of the transformations, str.title(), ensures that user names will always appear in the title case even when they aren’t stored that way in the file.

With that out of the way, it’s now time to look at serializing and deserializing your tabular data using a binary format. This can be especially helpful when you have a massive dataset and want to save disk space while improving the processing time.

Binary DataFrames: Parquet

Apache Parquet is yet another serialization format that’s widely used for representing tabular data. It’s binary, schema-based, and columnar, making it blazingly fast and space-efficient compared to CSV files. It enjoys native support in many cloud providers and libraries, including pandas and Dask. Plus, it integrates well with popular frameworks for distributed computing like Apache Spark and Apache Hadoop.

To work with Parquet files in Python, you must install the necessary libraries. For example, when using pandas, you have two options on the table:

1. pyarrow
2. fastparquet

While pandas comes with the high-level functions to interface with Parquet files, it lacks the underlying implementation by default. It’s one of the optional dependencies that provides the necessary logic. If you have both dependencies installed, then pandas prefers pyarrow’s implementation, but you can override this by specifying the engine parameter when reading or writing a DataFrame:

>>> import pandas as pd
>>> df1 = pd.read_parquet("/path/to/file")
>>> df2 = pd.read_parquet("/path/to/file", engine="fastparquet")
>>> df2.to_parquet("/path/to/another_file", engine="pyarrow")

In addition to using these libraries in conjunction with pandas, you can use them on their own:

>>> import pyarrow.parquet as pq
>>> table = pq.read_table("/path/to/file")
>>> df1 = table.to_pandas()

>>> import fastparquet
>>> pf = fastparquet.ParquetFile("/path/to/file")
>>> df2 = pf.to_pandas()

>>> df1.equals(df2)
True

In most cases, you should expect identical or similar results regardless of which engine you use. However, pyarrow has a few tricks up its sleeve that fastparquet currently lacks.

One of the most powerful features of Parquet that pyarrow exposes is row filtering, also known as predicate pushdown filtering, which you can apply when reading the file. This has a significant impact on performance, memory use, and network bandwidth, as it lets you request only a small subset of rows from a large dataset before reading them into memory for further processing.

For example, assuming you have a Parquet file equivalent to the CSV counterpart that you worked with in the last section, you can cherry-pick users whose language is French:

>>> import pandas as pd
>>> df = pd.read_parquet(
...     "users.parquet",
...     filters=[("language", "=", "fr")],
...     engine="pyarrow"
... )
>>> df.head()
     id                      name  ... language        registered_at
0  9875           Aurélie Foucher  ...       fr  2023-06-27 00:47:34
1  7269  Jeannine-Josette Clément  ...       fr  2023-05-10 13:23:11
2  4887   Richard Albert-Ollivier  ...       fr  2023-03-08 17:29:38
3  8494    Bertrand Boulay-Brunel  ...       fr  2023-05-10 23:13:19
4  7133            Hélène Guillet  ...       fr  2023-02-27 23:05:35

[5 rows x 5 columns]

By specifying the predicate expression through the filters parameter, you can skip the majority of data stored in your file. Note that this only works as intended with pyarrow configured as the Parquet engine in pandas. Otherwise, the filter may fail to apply, or it may only work partially.

Another great feature of any columnar data format, including Parquet, is column pruning or projection. To further improve your performance, you can specify which columns to read, ignoring all the rest:

>>> df = pd.read_parquet(
...     "users.parquet",
...     filters=[("language", "=", "fr")],
...     columns=["id", "name"],
...     engine="pyarrow"
... )
>>> df.head()
     id                      name
0  9875           Aurélie Foucher
1  7269  Jeannine-Josette Clément
2  4887   Richard Albert-Ollivier
3  8494    Bertrand Boulay-Brunel
4  7133            Hélène Guillet

[5 rows x 2 columns]

Because the data is stored in a column-oriented way, skipping uninteresting columns becomes straightforward. It’s not uncommon to deal with really big datasets that contain hundreds of columns or even more. Focusing on the relevant ones can drastically reduce the computational load.

As you learned earlier, columnar formats compress better than their row-based counterparts, and Parquet is no exception. When you dump a DataFrame to a Parquet file, pandas applies compression by default. You can optionally choose one of several compression algorithms that pandas supports.

Parquet is a schema-based data serialization format because it retains the column names and their types. This eliminates the risk of incorrect type inference or guessing, which often plagues schemaless data serialization formats like CSV. While Parquet supports only a handful of primitive types, which are mostly numerical, it builds more sophisticated logical types on top of them. For instance, strings are merely annotated byte arrays.

To view the schema of a Parquet file, you can use either of the engine libraries:

>>> import pyarrow.parquet as pq
>>> pf = pq.ParquetFile("users.parquet")
>>> pf.schema
<pyarrow._parquet.ParquetSchema object at 0x7f212cba42c0>
required group field_id=-1 schema {
  optional int64 field_id=-1 id;
  optional binary field_id=-1 name (String);
  optional binary field_id=-1 email (String);
  optional binary field_id=-1 language (String);
  optional int64 field_id=-1 registered_at (
    Timestamp(
        isAdjustedToUTC=false,
        timeUnit=nanoseconds,
        is_from_converted_type=false,
        force_set_converted_type=false
    ));
}
>>> pf.schema.to_arrow_schema()
id: int64
name: string
email: string
language: dictionary<values=string, indices=int32, ordered=0>
registered_at: timestamp[ns]
-- schema metadata --
pandas: '{"index_columns":
⮑ [{"kind": "range", "name": null, "start": 0, "' + 867

>>> import fastparquet
>>> pf = fastparquet.ParquetFile("users.parquet")
>>> print(pf.schema)
- schema: REQUIRED
| - id: INT64, OPTIONAL
| - name: BYTE_ARRAY, STRING, UTF8, OPTIONAL
| - email: BYTE_ARRAY, STRING, UTF8, OPTIONAL
| - language: BYTE_ARRAY, STRING, UTF8, OPTIONAL
  - registered_at: INT64, TIMESTAMP[NANOS], OPTIONAL

In any case, you can see the column names followed by their types, optional annotations, and whether they can take empty values.

As a column-oriented data format, Parquet makes schema evolution possible. You can add, update, or delete columns without having to rewrite the entire file from scratch. At the same time, your data remains immutable, as you can’t easily change a single value in an existing file without saving a new one. This can help prevent you from introducing sneaky bugs by accidentally modifying data without realizing it.

While pandas itself doesn’t give you direct control over the schema of a Parquet file, you can manage it to some extent by using one of the underlying engine libraries. For example, you may convert your DataFrame to a pyarrow table and configure its schema accordingly.

In the next section, you’ll learn about other data serialization formats that give you such control.

Big Data: Apache Avro

Apache Avro is one of the most common data serialization formats in Apache Hadoop and Apache Kafka, among many other contexts. The first is a framework used for big data analytics, while the second is a distributed event streaming platform originally developed at LinkedIn.

The format ticks all the boxes for large-scale data processing and real-time communication systems, including:

• Efficiency: As a binary format, it’s compact and quick to process, whether you need to persist your data or send it over the wire.
• Data Consistency: Thanks to always using a schema for serializing and deserializing data, it reduces the risk of data corruption or misinterpretation.
• Self-Explanation: Every message embeds the corresponding schema, so consumers don’t need to know it ahead of time to correctly interpret the payload. At the same time, they can optionally receive the schema separately while processing schemaless messages to improve performance further.
• Backward Compatibility: Because the format has excellent support for schema evolution, you can safely and seamlessly change the structure of your data over time without breaking anything.
• Language Independence: The format supports a range of programming languages, making it a good fit for distributed multi-language computing environments.

In a way, Avro is similar to Parquet. They’re both schema-based binary formats used in the Hadoop ecosystem, but they serve slightly different purposes and have different strengths and weaknesses. The biggest difference is how they organize data.

Parquet is column-oriented, while Avro is row-oriented, making it faster for write-centric operations where you must append data in real time. On the other hand, Parquet is more suitable for query-heavy analytics due to the efficient vectorization that it enables. Additionally, Avro supports complex data types, requiring you to define a detailed schema yourself, while Parquet usually infers its few basic column types directly from your data.

Avro closely resembles other binary serialization formats, including Protocol Buffers, which you’ll learn about in the next section. They all achieve similar goals. However, Avro differentiates itself by offering unique features, like:

• Support for Dynamic Languages: You can consume an Avro message at runtime without having to generate the serialization code tailored to the specific schema beforehand, although you can if you want to. It’s an optional optimization step, which makes sense in compiled and statically typed languages like Java. Because of that, Avro’s official binding for Python doesn’t support code generation from a schema.
• Symbolic Field Names: Avro schemas contain symbolic field names, whereas some other formats require that you manually assign numeric identifiers to those fields. Operating on numbers can get in the way when you try to resolve differences between the evolved versions of your schema.

Apart from the data serialization and storage scenarios, you can use Avro to facilitate remote procedure calls (RPCs) by specifying a custom protocol and messages of your remote interface. Check out the Avro RPC Quick Start guide to learn how you can use Avro to overcome programming language barriers for distributed communication. Note that this project hasn’t received updates for several years, so it may not reflect the current practices.

When defining a schema or an RPC interface, you typically reach for the widely used JSON data format, which must adhere to the Avro specification. Additionally, if you’re only interested in specifying an RPC interface, then you can use Avro IDL. It’s an alternative format with a syntax resembling a programming language, which may appeal to users experienced with traditional interface description languages in frameworks like Protocol Buffers.

To serialize Python dictionaries as Avro records or the other way around, you need a schema and a third-party library. The official avro package is implemented in pure Python, making it relatively slow. If you’re looking for something snappier, then consider installing the fastavro library, which offers speeds comparable to its Java counterpart. It leverages Cython to generate C extension modules for much better performance.

Here’s an Avro schema corresponding to the User class that you defined earlier when working with the CSV format:

{
    "name": "User",
    "type": "record",
    "fields": [
        {"name": "id", "type": "long"},
        {"name": "name", "type": "string"},
        {"name": "email", "type": "string"},
        {
            "name": "language",
            "type": {
                "name": "Language",
                "type": "enum",
                "symbols": ["de", "en", "es", "fr", "it"]
            }
        },
        {
            "name": "registered_at",
            "type": {"type": "long", "logicalType": "timestamp-millis"}
        }
    ]
}

This JSON document defines an Avro record consisting of a 64-bit integer identifier followed by two string fields, a custom enumeration of language codes, and a timestamp. Assuming that you can load this schema from a local file named user.avsc and you can import the models.User class, this is how you can dump a few fake users to an object container file:

>>> from fastavro.schema import load_schema
>>> from fastavro import writer
>>> from models import User

>>> users = [User.fake() for _ in range(5)]
>>> with open("users.avro", mode="wb") as file:
...     schema = load_schema("user.avsc")
...     writer(file, schema, [user._asdict() for user in users])
...

Notice that writer() expects an iterable of Python dictionaries, so you call ._asdict() in a list comprehension to convert your named tuples into plain dictionaries.

Reading the resulting users.avro file is even more straightforward. Because the file already contains the schema, you don’t need to load it yourself. Each binary record gets deserialized into a Python dictionary, which you can unpack into the User class constructor with a double asterisk (**):

>>> from fastavro import reader
>>> from models import User

>>> with open("users.avro", mode="rb") as file:
...     for record in reader(file):
...         print(User(**record))
...
User(
    id=9103,
    name='Barbara Barsanti',
    email='paulinatedesco@example.org',
    language='it',
    registered_at=datetime.datetime(2023, 1, 24, 14, 54, 31,
tzinfo=datetime.timezone.utc)
)
⋮