Paste the list into a deduplicator, run sort -u list.txt in a terminal, or select the column in Excel and use Data → Remove Duplicates. Any of the three finishes in seconds. What costs you the afternoon is the pair of lines that look identical and refuse to merge, and the question none of those tools asks out loud: when a line repeats, which copy should survive?
What is the quickest way to dedupe a list?
For anything you could scroll through — a few thousand email addresses, a column of SKUs, a pile of URLs from a crawl — a browser tool is the shortest path, because you get to look at the result before you commit to it. Pasting the list into a duplicate line remover returns every distinct line once, in the order each one first appeared, and tells you how many repeats it dropped. Nothing is uploaded: it is JavaScript running in your tab, which matters when the list is a client roster or a set of internal URLs.
Whatever tool you use, check whether it preserves order. Sorting is the cheapest way to find duplicates, so several of the standard answers — sort -u, PowerShell's Sort-Object -Unique — reorder the list as a side effect. If yours was in a meaningful order, priority or chronology or the order a script wrote it in, that is gone and you cannot recover it from the output.
Which copy should survive?
If the duplicates are byte-for-byte identical, it makes no difference at all. The moment your matching rule is loose — ignoring case, ignoring surrounding spaces — the copies genuinely differ and you are choosing between spellings.
- Keep the first for anything append-only: a log, a signup list, a queue. The earliest entry is the real one and everything after it is a re-submission.
- Keep the last when later entries are corrections. CRM and billing exports often work this way — the same record appears several times and the final row carries the current phone number.
Most tools decide this silently. Excel keeps the top row of each group and deletes the rest. With sort -u the question never arises, because two lines only count as equal when they are identical. It is only worth thinking about when you have deliberately told a tool to be forgiving about case or whitespace.
Why two identical-looking lines will not merge
This is the complaint aimed at every deduplicator ever written, and it is almost never the tool's fault. Something invisible differs.
- Trailing spaces. A single space after a word makes the line unique and is completely invisible. This is why most tools trim by default.
- Non-breaking spaces. Copying out of a web page, Word or a PDF frequently brings U+00A0 along instead of a normal space. Same shape, different character.
- Decomposed accents. The letter é can be one code point (U+00E9) or a plain
efollowed by a combining acute accent (U+0301). They render identically and compare as different strings. macOS filenames have historically used the decomposed form while most web input uses the composed one, so a list assembled from both sources arrives full of twins. Normalising to NFC collapses them. - Carriage returns. A file written on Windows ends its lines with CR + LF. Mix it with a Unix-made file and a naive tool sees a stray carriage return hanging off half your lines.
Normalisation will not save you from lookalikes. Cyrillic а (U+0430) and Latin a (U+0061) are different letters that happen to share a shape; so are a curly apostrophe and a straight one, and the fullwidth Latin letters used in Japanese text. No normalisation form merges them, because merging them would be wrong. Those you fix at the character level, with a find and replace pass over the whole list before you deduplicate anything. If you cannot work out which character is to blame, the guide to invisible characters in text lists the usual suspects and how to spot them.
On the command line, uniq is a trap
uniq only removes adjacent duplicates. It streams the file and compares each line with the previous one, so a list with repeats scattered through it comes out completely unchanged, with no error and no warning. People lose hours to this.
sort -u list.txt— sorts and deduplicates in one pass. Original order is gone. It compares using your locale's collation rules; putLC_ALL=Cin front of it for a plain byte comparison.awk '!seen[$0]++' list.txt— keeps the first occurrence of each line and preserves the original order. It holds every distinct line in memory, which is the price of not sorting.sort list.txt | uniq -d— lists only the lines that appeared more than once.uniq -cprefixes each with its count. Both need sorted input, for the reason above.Get-Content list.txt | Sort-Object -Unique— the PowerShell equivalent. Note that it is case-insensitive by default, unlike the Unix tools; add-CaseSensitiveif that matters.
In Excel and Google Sheets
Excel's Remove Duplicates lives under the Data tab. Select the range, tick the columns to compare, and it deletes the offending rows in place and reports how many went. Two things to know: rows are duplicates only if every ticked column matches, and the deletion is destructive — undo is the only way back, so duplicate the sheet first if the data is not reproducible.
Google Sheets has the same destructive command under Data → Data cleanup → Remove duplicates, but it also has =UNIQUE(A2:A), which writes the deduplicated list into a spare column and leaves the original untouched. That is the better default: you can compare the two side by side, and the result updates when the source does. To count rather than remove, =COUNTIF(A:A, A2) filled down tells you how many times each value appears in the column.
Finding the duplicates instead of deleting them
Often the repeats are the point. Which address signed up twice, which invoice number got issued to two customers, which tracking ID landed in the export more than once — you do not want a clean list, you want the offenders.
The browser tool has a mode for exactly this: switch Show to "Only lines that repeated" and tick the count prefix, and you get each repeated line once with the number of times it occurred. The inverse — only the lines that appeared exactly once — is useful for reconciling two lists that should have matched. On the command line, sort file | uniq -d does the first job and uniq -u the second.
Where line-based deduplication stops working
A tool that treats each line as one opaque string is fast, predictable and wrong for several common jobs.
- CSV with quoted fields. A value containing a newline is legal CSV and will be split into two "lines" by anything that does not parse the format. Deduplicating by one column of many needs a spreadsheet or a script that understands records.
- Near-duplicates. "Acme Ltd." and "Acme Limited" are the same company and no exact-match tool will ever pair them. Nor will "john@example.com" and "John@Example.com " with a trailing space, unless you turn on trimming and case folding. Fuzzy matching is a different problem with a different class of tool.
- Differences you want to inspect rather than remove. If the real question is what changed between two versions of the same list, deduplicating destroys the evidence. Comparing the two texts line by line is the tool for that.
- Very large files. A browser tab handles a few hundred thousand lines comfortably. A multi-million-line log will freeze it, and pasting a 200 MB file into a text box is miserable before you press anything. That is what
sort -uis for.
One last habit worth forming: run the dedupe, then look at the counts. "12,000 lines in, 11,998 distinct" usually means you already had a clean list and something else is wrong. "12,000 in, 340 distinct" means your export ran 35 times. The numbers tell you more about the data than the cleaned list does.
If you have a list open in another window, the duplicate line remover will do it now — trimming and blank-line removal are already on, and the counts underneath show what it found before you copy anything out. Turn on Unicode normalisation if the list came from more than one source.
The other half of tidying a list is putting it in a sensible order, and that is less obvious than it looks once numbers or mixed case are involved. Sorting a list alphabetically and numerically covers why "item10" keeps landing before "item2".
Frequently asked questions
How do I remove duplicate lines from a text file?
Paste it into a browser deduplicator for anything up to a few hundred thousand lines, or run sort -u file.txt in a terminal for larger files. In a spreadsheet, select the column and use Data then Remove Duplicates. All three keep one copy of each distinct line.
How do I remove duplicates but keep the original order?
Use a tool that indexes lines rather than sorting them. On the command line, awk '!seen[$0]++' file.txt keeps the first occurrence of each line in its original position. Plain sort -u cannot do this, because sorting is how it finds the duplicates.
Why are duplicate lines not being removed?
Because they are not actually identical. The usual causes are a trailing space, a non-breaking space pasted from a web page, an accent stored in decomposed form, or a lookalike character from another alphabet. Turn on trimming and Unicode normalisation first; if the lines still refuse to merge, the difference is a character you cannot see and you need to find and replace it directly.
Does uniq remove all duplicate lines?
No. uniq only compares each line with the one immediately before it, so duplicates scattered through a file survive untouched. Sort the input first with sort file | uniq, or use sort -u, which does both in one pass.
How do I find which lines are duplicated instead of deleting them?
Switch the browser tool to "Only lines that repeated" and tick the count prefix, which shows each repeated line once with how many times it appeared. On the command line, sort file | uniq -d lists the repeated lines and uniq -c prefixes every line with its count.
Is it safe to deduplicate a list online?
It depends on whether the site sends your text to a server. Many do, which means a customer list or an internal export sits in someone else’s logs. A tool that runs in the browser never transmits the text at all, and closing the tab discards it.
Last updated September 19, 2026