Skip to content

Unicode

Unicode is a universal character encoding standard that assigns a unique number (code point) to every character in every language, plus symbols, emojis, and control characters. It enables consistent representation and handling of text across different platforms and languages.

It extends ASCII by preserving its first 128 characters while enabling representation of over a million characters.

Key Concepts

Code Points

A code point is a unique number assigned to each character in the Unicode standard, typically written in hexadecimal with a U+ prefix, such as U+0041 for “A” or U+1F600 for 😀.

Encoding Schemes

These encoding schemes determine how Unicode code points are stored as bytes:

  • UTF-8: Variable-length encoding (1–4 bytes), backward compatible with ASCII
  • UTF-16: Variable-length encoding (2 or 4 bytes). Windows commonly uses UTF-16LE
  • UTF-32: Fixed-length encoding (4 bytes), simple but space-inefficient

Type a string or pick an example to see each character’s code point, the bytes it becomes in UTF-8, UTF-16-LE, and UTF-32-LE, and how len() compares with the number of visible characters:

Interactive diagram — enable JavaScript to view.

Python and Unicode

In Python, all strings are Unicode by default. The str type represents Unicode text, while bytes represents encoded text data:

Language: Python
>>> # Unicode string (str type)
>>> text = "Hello, 世界 🌍"

>>> # Encode to bytes
>>> utf8_bytes = text.encode("utf-8")
>>> utf8_bytes
b'Hello, \xe4\xb8\x96\xe7\x95\x8c \xf0\x9f\x8c\x8d'

>>> # Decode back to string
>>> decoded = utf8_bytes.decode("utf-8")
>>> decoded
'Hello, 世界 🌍'

An example of getting code points:

Language: Python
>>> # Get code point of a character
>>> ord("A")
65
>>> ord("世")
19990

>>> # Get character from code point
>>> chr(65)
'A'
>>> chr(19990)
'世'

Different sequences of code points can represent the same character. Unicode normalization ensures consistent representation:

Language: Python
>>> import unicodedata

>>> composed = "\u00ea"  # Single code point for ê
>>> decomposed = "e\u0302"  # Letter e + COMBINING CIRCUMFLEX ACCENT
>>> composed == decomposed
False
>>> len(composed), len(decomposed)
(1, 2)

>>> # Normalize to NFC (composed form)
>>> unicodedata.normalize("NFC", decomposed) == composed
True

Common Pitfalls

  • Mixing encodings: Always know what encoding your data uses.
  • Assuming one character = one code point: Some characters require multiple code points, such as emojis with skin tones.
  • File-encoding issues: Specify encoding when opening files.

Best Practices

  • Always specify encoding explicitly when reading or writing files.
  • Use UTF-8 as your default encoding.
  • Handle encoding errors deliberately by choosing an error handler, such as "replace", "ignore", or "backslashreplace", instead of the default "strict", which raises an exception.
  • Be aware that string length (len()) counts code points, not visual characters.
  • ASCII: 7-bit character encoding that corresponds to the first 128 Unicode code points.
  • Code Unit: Fixed-size storage unit used by an encoding, such as 8 bits in UTF-8.
  • BOM (Byte Order Mark): Optional marker at the start of a text stream indicating encoding.
  • Surrogate Pairs: UTF-16 mechanism for encoding code points beyond U+FFFF.
Unicode & Character Encodings in Python: A Painless Guide

Tutorial

Unicode & Character Encodings in Python: A Painless Guide

In this tutorial, you'll get a Python-centric introduction to character encodings and unicode. Handling character encodings and numbering systems can at times seem painful and complicated, but this guide is here to help with easy-to-follow Python examples.

advanced python

For additional information on related topics, take a look at the following resources:

Have a question about this? Mentor AI can show you examples, compare related terms, and point you to tutorials.


By Dan Bader • Updated Nov. 19, 2026 • Reviewed by Leodanis Pozo Ramos