Remove Duplicate Lines — dedupe a list online
Delete repeated lines from a list, with optional sorting, blank-line trimming and case-insensitive matching. Free, instant, and processed locally.
About the Remove duplicate lines tool
Duplicate lines appear whenever lists get combined. Two exported mailing lists are pasted together and the overlap is invisible. A log file repeats the same error three hundred times. Keyword research from three sources contains the same phrases in each. A CSV column exported twice has every row doubled. In every case the fix is the same operation, and doing it by eye on more than about thirty lines is unreliable as well as slow.
The options matter more than the deduplication itself, because they decide what counts as the same. Case sensitivity is the big one: with it on, "Alice" and "alice" are two different entries, which is right for passwords, identifiers and anything case-bearing. With it off they collapse into one, which is right for email addresses and names, where the difference is almost always a typing inconsistency rather than a real distinction.
Trimming whitespace addresses the same problem from another angle. A line with a trailing space looks identical to one without and is not, which is why lists pasted from spreadsheets or PDFs often refuse to deduplicate properly. Turning trimming on strips leading and trailing spaces before comparing, and it resolves the large majority of cases where a tool insists two visibly identical lines are different.
Sorting and dropping blank lines are conveniences rather than corrections. Sorting alphabetically makes the result easy to scan and easy to compare against another list later, though it discards whatever order the original had — leave it off if the sequence carries meaning, such as chronological log entries. Dropping empty lines tidies up the ragged gaps that paste operations tend to leave behind.
How it works
Paste your list
One item per line. Anything separated by line breaks works — emails, URLs, keywords, log entries, IDs.
Set the matching rules
Decide whether case matters, whether to trim whitespace first, and whether to sort and drop blank lines.
Copy the clean list
The result shows how many duplicates were removed. Copy it out — the original stays for comparison.
Frequently asked questions
- Should I turn case sensitivity on or off?
- Off for email addresses, names, domains and keywords, where a difference in capitalisation is nearly always an inconsistency rather than a real distinction. On for passwords, API keys, file paths on Unix systems, and anything where the case is part of the value. When in doubt, run it both ways and see how many extra duplicates the case-insensitive pass removes.
- Why are two lines that look identical not being merged?
- Almost always invisible whitespace — a trailing space, a tab, or a non-breaking space pasted from a web page or a PDF. Turn on trim whitespace and they will usually collapse. If they still do not, the lines may contain different Unicode characters that render alike, such as a curly apostrophe against a straight one.
- Which duplicate is kept?
- The first occurrence, in the order the lines were pasted. Later copies are the ones removed. This matters when your list is ordered by something meaningful — if the most recent entry should win rather than the earliest, reverse the list before deduplicating and reverse it back afterwards.
- Does it remove lines that are similar but not identical?
- No. Matching is exact, subject to the case and trimming options you chose. Near-duplicates such as "John Smith" and "John A. Smith", or an address written two ways, are treated as different entries because there is no unambiguous rule for how close is close enough. Fuzzy matching is a different problem and needs a different tool.
- Will sorting ruin the order of my list?
- It replaces the original order with alphabetical order, so leave it off whenever the sequence carries information — log entries in time order, steps in a process, a ranked list. Sorting is worth turning on when the list is a set rather than a sequence, since a sorted result is far easier to scan and to diff against another list later.
- How large a list can it handle?
- Tens of thousands of lines process effectively instantly, since deduplication uses a hash set rather than comparing every line against every other. Very large pastes in the megabytes are limited by the browser handling that much text in a textarea rather than by the deduplication itself. Splitting into chunks works if you reach that point.
- Is my list uploaded anywhere?
- No. Everything runs in JavaScript inside your browser tab, so nothing is transmitted, stored or logged. Deduplicated lists are frequently customer emails, subscriber exports or internal identifiers, and those should not be pasted into a website that sends them somewhere. Closing the tab discards the data entirely.