Remove Duplicate Lines From a List

What counts as "the same line"

Deduplication is only as good as the rule you use to decide two lines match. The tool builds a comparison key for each line — trimmed, lowercased and normalised according to the checkboxes — and treats lines with the same key as copies of each other. The line that gets printed is the original, not the key, so turning on Ignore case does not lowercase your output. It only changes what is considered a repeat.

That distinction matters when copies differ. If your list contains Acme Ltd and ACME LTD, ignoring case collapses them into one, and Which copy to keep decides which spelling survives. Keep the first is what you want for a log or an append-only list. Keep the last is what you want when later entries are corrections.

Why two identical-looking lines refuse to match

This is the single most common complaint about any deduplicator, and it is almost never a bug in the tool.

• Trailing whitespace. A space after a word is invisible and makes the line unique. This is why trimming is on by default.
• Non-breaking spaces. Copying from a web page or a word processor frequently brings U+00A0 along instead of a normal space. It looks identical and is a different character. Trimming does remove it, because JavaScript counts it as whitespace, but one sitting in the middle of a line survives.
• Unicode normalisation. The letter é can be stored as a single code point (U+00E9) or as a plain e followed by a combining acute accent (U+0301). They render the same and compare as different strings. macOS filenames historically use the decomposed form, Windows and most web input use the composed one, so a list assembled from both sources will have twins. Normalise accents converts everything to the composed form (NFC) before comparing.
• Line endings. Windows text ends lines with a carriage return plus a line feed. The tool strips the carriage return before anything else, so a mixed-ending list does not produce phantom duplicates.

What normalisation will not fix is lookalike characters from different alphabets. Cyrillic а and Latin a are separate letters that happen to share a shape, and no normalisation form merges them. The same goes for fullwidth Latin characters and for a curly apostrophe versus a straight one. If a line stubbornly refuses to deduplicate, that is usually the reason, and you have to find and replace the character itself.

The command line version, and its trap

On a Unix shell the equivalent is uniq, and it has a gotcha that catches people for years: uniq only removes adjacent duplicates. It reads the file as a stream and compares each line to the previous one, so a list with repeats scattered through it comes out unchanged. The idiom is sort file | uniq, or just sort -u file.

This tool works differently. It keeps every distinct key in a hash map, so repeats are found no matter how far apart they are, and the original order survives. That costs memory proportional to the number of distinct lines, which is the trade uniq avoids by streaming.

Where this tool stops being the right one

Everything runs in your browser tab on a single thread. A few hundred thousand lines is fine; a multi-million-line log will freeze the tab while it works, and pasting a file that large into a text box is unpleasant before you even press anything. For those, sort -u or awk is the answer.

It is also strictly line-based. A CSV where one field contains a quoted value with a newline inside it will be split at that newline, and the two halves will be treated as separate records. Deduplicating CSV by a single column is not something a line tool can do — you need a spreadsheet or a script that understands the format.

Finally, case folding uses your browser's default rules. Those are correct for the overwhelming majority of text and wrong for a couple of known cases: Turkish dotted and dotless i do not fold the way a Turkish reader expects, and German ß uppercases to SS without a route back.

Last updated September 19, 2026