How to remove duplicate lines (Linux, bash, Python)

sort -u is the classic answer but it re-orders your lines. If the order matters, use awk. Here is what each command does on the same file.

The example file

list.txt (note the trailing space after the second apple):

banana
apple
banana
Cherry
apple 
cherry
apple

1. sort -u (sorted, not order-preserving)

sort -u list.txt

apple
apple 
banana
cherry
Cherry

Exact duplicates are gone, but the lines are sorted, and apple with a trailing space counts as different. (The order of cherry and Cherry depends on your locale; ours was en_GB.UTF-8. LC_ALL=C sort -u puts uppercase first.) Write to a new file with sort -u list.txt > clean.txt (never redirect onto the input file itself, which empties it).

2. awk (keep the original order)

awk '!seen[$0]++' list.txt

banana
apple
Cherry
apple 
cherry

Each line is printed the first time it is seen. To also ignore case and surrounding spaces:

awk '{k=tolower($0); gsub(/^[ \t]+|[ \t]+$/,"",k)} !seen[k]++' list.txt

banana
apple
Cherry

3. Count or list the duplicates

sort list.txt | uniq -c | sort -rn   # how often each line occurs
sort list.txt | uniq -d              # only lines that occur more than once

Our output: 2 banana, 2 apple, then the three single lines; uniq -d printed apple and banana. uniq only collapses adjacent duplicates, which is why the input is sorted first.

4. Python

python3 -c "import sys; print(''.join(dict.fromkeys(open('list.txt'))), end='')"

Same result as the awk command: first occurrences, original order (dictionaries keep insertion order).

5. No terminal

Paste the list into List A of the free compare two lists tool and leave List B empty: "Only in A" shows each distinct line once, in the original order, ignoring case and spaces if you want. To compare against a second list at the same time, paste that into List B.

Frequently asked questions

How do I remove duplicates but keep the order?

Use awk '!seen[$0]++' file, or the Python one-liner. sort -u always sorts.

Why does sort -u keep lines that look the same?

They differ in invisible characters: trailing spaces, tabs or Windows line endings (\r). Use the case- and space-insensitive awk version, or remove \r first with tr -d '\r'.

How do I remove duplicate lines in Excel?

Select the column, then Data → Remove Duplicates. It keeps the first occurrence of each value and ignores case.

Written with AI assistance. Every command was run on 2026-10-01 (GNU coreutils, gawk, Python 3.13 on Linux) on the 7-line example file shown; outputs are copied from the terminal. PlainTables is our extension; the other methods are free and built into Excel, Google Sheets and pandas.