Remove duplicate lines

Remove duplicate lines without reordering the rest

The first time a line appears, it stays exactly where it was. Every later repeat is dropped. This matters more than it sounds: many deduplicators sort the list as a side effect of removing duplicates, which destroys any ordering you cared about — chronological, by priority, by the order someone signed up.

If you want the result sorted as well, that is a separate decision and a separate step. Sorting a list you did not ask to have sorted is the kind of helpfulness that costs an hour of confusion later.

The count is the useful part

Getting a deduplicated list back is easy. Knowing how many duplicates there were is what tells you whether something is wrong upstream. Three duplicates in a thousand-line export is normal. Four hundred duplicates means a join in your query is multiplying rows, and cleaning the output would be treating the symptom.

Case sensitivity is a real decision

By default, Anna and anna are two different lines, because that is what the text literally says.

For email addresses and usernames that is usually the wrong answer. Email local parts are technically case-sensitive but treated as case-insensitive by every major provider, so [email protected] and [email protected] reach the same inbox — and sending to both means someone gets your newsletter twice.

For anything where the text is a label rather than an identifier, keep case sensitivity on. Turning it off can merge two genuinely different values that happen to differ only in capitalisation.

Whitespace is compared after trimming

A line with a trailing space is, byte for byte, a different line from one without. Almost nobody means that. Copying out of a spreadsheet or a web page routinely attaches stray spaces, and comparing raw would leave both copies in your list while insisting they are different.

Lines are trimmed before comparison and the trimmed version is what you get back. Blank lines are dropped rather than deduplicated down to one, since a single stray blank in the middle of a list is rarely wanted either.

Nothing is uploaded

Deduplication runs on lists of email addresses, customer records, and exported identifiers more often than on anything else. All of it stays in this tab — the comparison happens in your browser and there is no server to send it to.