About this tool
Paste a list (emails, URLs, keywords, IDs, log lines, anything with one item per line) and every repeated line is removed, leaving one copy of each. Unlike sorting with sort -u, the original order is kept: each line stays where it first appeared.
Options
- Case-sensitive: off by default, so
Appleandapplecount as duplicates. Turn it on when case matters, such as for passwords or case-sensitive IDs. - Ignore leading/trailing spaces: treats
appleandappleas the same line, which catches most copy-paste duplicates. - Keep first or last occurrence: keep the last copy when later lines are newer, as in an appended log or a list of updates.
- Remove blank lines too: by default blank lines are kept so the layout of your text doesn’t change.
The status line tells you how many lines were removed. Lines are compared exactly as text, so 1 and 01 are different lines.
On the command line
awk '!seen[$0]++' list.txt # keep first, keep order
sort -u list.txt # sorted and unique
Get-Content list.txt | Select-Object -Unique # PowerShell
FAQ
How do I remove duplicate lines from a list?
Paste the list into the box. The result keeps one copy of each line, in the original order. Copy or download it.
Does it keep the original order?
Yes. Each line stays where it first appeared (or where it last appeared, if you choose to keep the last occurrence).
Can I ignore uppercase and lowercase differences?
Yes. Case-sensitive comparison is off by default, so “Apple” and “apple” are treated as duplicates.
How do I remove duplicates and sort at the same time?
Use Sort Lines with “Remove duplicates” turned on.