text encoding
Text encoding is the serialization of text into a sequence of bytes that computers can store and transmit. Python calls the rules for doing this text encodings, a family of text serialization codecs such as UTF-8. Turning a string into bytes is encoding, and turning those bytes back into a string is decoding.
In Python, every string is already a sequence of Unicode code points. Unicode is a character set that assigns a unique code point to every character and symbol in the world’s writing systems, while an encoding is the rule that turns those code points into bytes. The most common encoding used in Python is UTF-8, which can encode every character defined by Unicode using one to four bytes.
Understanding text encoding is essential for working with text data in Python, especially when dealing with multiple languages or special characters. Incorrect handling of encodings can lead to errors and data corruption.
Example
Here’s an example where you encode a string into bytes using UTF-8 and then decode it back to a string:
>>> text = "Hello, World!"
>>> # Encode into bytes
>>> encoded_text = text.encode("utf-8")
>>> encoded_text
b'Hello, World!'
>>> # Decode back to a string
>>> decoded_text = encoded_text.decode("utf-8")
>>> decoded_text
'Hello, World!'
In this example, the .encode() method converts the string to a bytes object using the UTF-8 encoding, while .decode() converts it back to a string.
Related Resources
Tutorial
Unicode & Character Encodings in Python: A Painless Guide
In this tutorial, you'll get a Python-centric introduction to character encodings and unicode. Handling character encodings and numbering systems can at times seem painful and complicated, but this guide is here to help with easy-to-follow Python examples.
For additional information on related topics, take a look at the following resources:
