Python Unicode: Identical-Looking Strings That Are Not Equal (tested)

Two Python strings can print as exactly the same text on screen, in the same font, with no visible difference -- and still have different lengths, fail an == check, and silently duplicate in a set. We built this with ordinary accented text and measured every step.

1. The same visible word, built two ways, has two different lengths and fails ==

precomposed = "caf\u00e9"       # single codepoint for accented e
decomposed  = "cafe\u0301"       # plain e followed by a combining acute accent

print(precomposed, "/", decomposed)
print(len(precomposed), len(decomposed))
print(precomposed == decomposed)
café / café
4 5
False

Measured: both strings display identically wherever they're rendered, but precomposed is 4 codepoints (the accented e is one codepoint) and decomposed is 5 (a plain e plus a separate combining-accent codepoint). Python's == compares codepoints, not rendered appearance, so it correctly says they're different strings -- which is exactly the problem if you expected it to say they're the same word.

2. A set silently keeps both as separate entries -- deduplication just fails

precomposed = "caf\u00e9"
decomposed  = "cafe\u0301"
seen = {precomposed, decomposed}
print(len(seen))
2

Measured: a set built from both forms of "the same" word keeps two entries, not one. The same applies to dict keys, in membership checks, and database unique constraints on raw text -- any of them can let in two "duplicate" values that look completely identical to a human. This is realistic any time text comes from different sources: a form field typed on one OS, a value copied from a PDF, or data normalized by a different library than the one that wrote it originally.

3. unicodedata.normalize() fixes it, but you must normalize both sides the same way

import unicodedata

precomposed = "caf\u00e9"
decomposed  = "cafe\u0301"

nfc_of_decomposed = unicodedata.normalize("NFC", decomposed)
nfd_of_precomposed = unicodedata.normalize("NFD", precomposed)

print(nfc_of_decomposed == precomposed)
print(nfd_of_precomposed == decomposed)
True
True

Measured: normalizing either string to the same form as the other makes them equal. NFC ("compose") prefers single combined codepoints and is the right default for storage and comparison; NFD ("decompose") splits them apart and is mainly useful for things like stripping accents. The fix isn't "normalize the incoming string" -- it's normalizing every string the same way before it's stored, compared, or deduplicated, since one unnormalized value anywhere breaks the whole comparison again.

4. Bonus: str.lower() and str.casefold() disagree on German sharp s

german = "straße"  # "strasse" with an eszett

print(german.lower())
print(german.casefold())
print(german.upper().lower() == german)
straße
strasse
False

Measured: .lower() leaves ß (eszett) alone, since it's already lowercase. .casefold() expands it to "ss", which is what you actually want for case-insensitive comparison, because "STRASSE".casefold() also produces "strasse" while "STRASSE".lower() does not produce "straße" -- confirmed by the round-trip check above returning False. Use .casefold(), not .lower(), for case-insensitive matching on any text that might contain non-ASCII letters.

Frequently asked questions

Why do two Python strings that look identical fail an == check?

We measured this with accented text: one string used a single precomposed codepoint for the accented letter, the other used a base letter plus a separate combining accent codepoint. They render identically but == compares codepoints, so it correctly reports them as different.

Can a Python set or dict silently treat two identical-looking strings as different keys?

Yes -- we confirmed it: a set built from both Unicode forms of the same visible word kept two entries instead of one. The same applies to dict keys and database unique constraints on raw, unnormalized text.

What does unicodedata.normalize() actually fix?

We measured normalizing both forms of the same word to NFC and to NFD: both conversions made the previously-unequal strings compare equal. The fix is to normalize every string the same way before storing or comparing it, not just the one that looks wrong.

Should I use NFC or NFD normalization?

NFC (composed, single codepoints) is the usual default for storage and comparison, since it's more compact and what most text already looks like. NFD (decomposed) is mainly useful when you need to strip accents, since it separates the base letter from the combining marks.

What's the difference between str.lower() and str.casefold() in Python?

We measured this on German eszett (ß): .lower() left it unchanged, but .casefold() expanded it to 'ss', matching how an uppercase 'STRASSE' casefolds too. Use casefold() for case-insensitive comparisons on non-ASCII text; lower() alone can miss matches like this.

Written with AI assistance. Measured on 2026-10-04 with Python 3.13.5's built-in unicodedata module: built the same visible word 'cafe' with an accented e two different ways (a precomposed codepoint vs. a base letter plus a combining accent), then compared length, ==, set deduplication, and unicodedata.normalize() across both forms, plus str.lower() vs str.casefold() on a German sharp s. PlainHeaders is our extension; Python's unicodedata module is built in.