difflib Differ Hint-Line Caret Misaligns on Wide Unicode Chars (tested)
difflib.Differ().compare() adds a '?' hint line with a caret under the exact character that changed. We measured what happens to that caret once the line contains double-width Unicode characters, and it stops pointing at the right spot on screen.
1. The caret's string index is correct, but its printed column is not
import difflib
a = ["日本語abc
"] # three CJK chars + "abc"
b = ["日本語abd
"] # last char changed: c -> d
result = list(difflib.Differ().compare(a, b))
for line in result:
print(line, end="")- 日本語abc
? ^
+ 日本語abd
? ^Measured: printed in a terminal, the ^ lands under b, not c/d where the actual change is -- it's visibly three columns too far left.
2. Why: difflib counts characters, terminals count display columns
minus_line = result[0].rstrip("\n")
hint_line = result[1].rstrip("\n")
idx = hint_line.index("^")
print("caret string-index:", idx)
print("character at that index in the content line:", repr(minus_line[idx]))caret string-index: 7
character at that index in the content line: 'c'Measured: by string index, the caret is exactly right -- index 7 is the c/d character. The hint line is built one output character per one input character, with no concept of display width. But each of the three CJK characters prints as 2 terminal columns, not 1, so the content line's c prints 3 columns further right than the hint line's same-index ^ does. The mismatch is purely a rendering problem -- the data difflib produces is correct, only its appearance in a monospace terminal is misleading.
3. Fix: don't rely on the hint line's visual position for wide-character text
print("caret is at character index", idx, "-- read this number, not the on-screen column, for non-Latin text.")caret is at character index 7 -- read this number, not the on-screen column, for non-Latin text.Measured: there's no built-in option to make the hint line display-width-aware. For text that may contain CJK, emoji, or other wide characters, use the caret's string index programmatically (as shown above) rather than trusting where it lines up visually, or switch to difflib.HtmlDiff(), which highlights changes with HTML markup instead of column-based carets.
Frequently asked questions
Does difflib's Differ.compare() hint line account for wide Unicode characters?
No -- we measured it building the caret position purely from character count, with no adjustment for characters like CJK text that print as 2 terminal columns instead of 1.
Is the caret's position actually wrong, or just the on-screen display?
The string index is correct -- we confirmed it points at the exact character that changed. Only the printed/visual column in a monospace terminal is misleading once double-width characters appear earlier in the line.
Does this affect unified_diff() or context_diff() too?
No -- those don't use the character-level '?' hint line at all, so this specific misalignment is unique to Differ.compare()'s output.
How can I reliably point at the changed character for non-Latin text?
Use the hint line's string index (hint_line.index('^')) programmatically against the content line, rather than trusting the visual column when printed to a terminal.
Is there a difflib tool that avoids this problem entirely?
Yes -- difflib.HtmlDiff() marks changes with HTML tags instead of column-based ASCII carets, so it doesn't depend on monospace character width at all.
Written with AI assistance. Measured on 2026-10-04 with Python 3.13.5's built-in difflib module: ran difflib.Differ().compare() on two near-identical lines containing CJK characters, then compared the hint line's string index against where its caret actually prints in a terminal. PlainHeaders is our extension; Python's difflib module is built in.