toolqik

152 · Text

Remove Duplicate Lines

Delete repeated lines from a list and keep the first (or last) of each.

Runs in your browser
Keep
Compare
Unique lines

  

About this tool

Paste a list (emails, URLs, keywords, IDs, log lines, anything with one item per line) and every repeated line is removed, leaving one copy of each. Unlike sorting with sort -u, the original order is kept: each line stays where it first appeared.

Options

  • Case-sensitive: off by default, so Apple and apple count as duplicates. Turn it on when case matters, such as for passwords or case-sensitive IDs.
  • Ignore leading/trailing spaces: treats apple and apple as the same line, which catches most copy-paste duplicates.
  • Keep first or last occurrence: keep the last copy when later lines are newer, as in an appended log or a list of updates.
  • Remove blank lines too: by default blank lines are kept so the layout of your text doesn’t change.

The status line tells you how many lines were removed. Lines are compared exactly as text, so 1 and 01 are different lines.

On the command line

awk '!seen[$0]++' list.txt          # keep first, keep order
sort -u list.txt                     # sorted and unique
Get-Content list.txt | Select-Object -Unique   # PowerShell

FAQ

How do I remove duplicate lines from a list?

Paste the list into the box. The result keeps one copy of each line, in the original order. Copy or download it.

Does it keep the original order?

Yes. Each line stays where it first appeared (or where it last appeared, if you choose to keep the last occurrence).

Can I ignore uppercase and lowercase differences?

Yes. Case-sensitive comparison is off by default, so “Apple” and “apple” are treated as duplicates.

How do I remove duplicates and sort at the same time?

Use Sort Lines with “Remove duplicates” turned on.

esc
↑↓ navigate↵ open