How letters become numbers
Every file you read, API you call, and database row you store crosses a text-to-bytes boundary. When a customer named José shows up as José in production, or a CSV from a client refuses to parse, this lesson is the one that fixes it.
RAM stores bits, so the letter A must secretly be a number. The oldest agreement is ASCII: A=65, B=66, ..., a=97, and so on, covering English letters, digits, and punctuation in the numbers 0-127. One character fit in one byte and life was simple.
But the world writes in thousands of scripts. Unicode is the modern agreement: a giant table giving every character on Earth a number, called a code point. A is still 65, é is 233, 🐍 is 128013.
Python gives you both directions: ord() turns a character into its number, chr() turns a number back into a character.
Code exercise · python
Run this. ord and chr convert between characters and their Unicode numbers, and .encode turns a string into raw bytes.
UTF-8: fitting big numbers into bytes
A code point like 128013 does not fit in one byte (bytes max out at 255, as you saw in lesson 2-1). An encoding is the rule for packing code points into bytes. The winner, used by ~98% of the web, is UTF-8:
- ASCII characters take 1 byte (unchanged, which is why old files still work)
- é and most European accents take 2 bytes
- most Asian scripts take 3 bytes
- emoji take 4 bytes
So in Python, len(string) counts characters, but len(string.encode("utf-8")) counts bytes, and they differ as soon as text leaves plain English. Files, networks, and disks always carry bytes, never characters.
Code exercise · python
Run this. Watch the character count and the byte count disagree.
Mojibake: decoding with the wrong rule
Bytes do not know what encoding made them. If you decode UTF-8 bytes using a different rule, you get garbage that is famous enough to have a name: mojibake. The two bytes of é, read one-at-a-time by the old latin-1 rule, become é.
If you have ever seen ’ where an apostrophe should be, or café on a menu website, you have watched this exact bug. The fix is always the same: know your encoding, and in modern code, use UTF-8 everywhere.
Code exercise · python
Run this to create mojibake on purpose: encode with UTF-8, then decode the same bytes with the wrong rule (latin-1).
Code exercise · python
Your turn. For the word "naïve", print the number of characters, then the number of UTF-8 bytes.
Quiz
A file contains the 5 bytes b'caf\xc3\xa9'. A program reads it as latin-1 and shows café. What went wrong?