Remove duplicate lines

Remove duplicate lines without reordering the rest

The first time a line appears, it stays exactly where it was. Every later repeat is dropped. This matters more than it sounds: many deduplicators sort the list as a side effect of removing duplicates, which destroys any ordering you cared about — chronological, by priority, by the order someone signed up.

If you want the result sorted as well, that is a separate decision and a separate step — sorting lines lives on its own page for exactly that reason, and you can run it on the deduplicated output afterwards. Sorting a list you did not ask to have sorted is the kind of helpfulness that costs an hour of confusion later.

The count is the useful part

Getting a deduplicated list back is easy. Knowing how many duplicates there were is what tells you whether something is wrong upstream. Three duplicates in a thousand-line export is normal. Four hundred duplicates means a join in your query is multiplying rows, and cleaning the output would be treating the symptom.

When the count itself is the surprise, the next question is usually which entries are the repeats — and that is a comparison rather than a cleanup. Keep the original list, run the deduplicated version against it with a line-by-line comparison, and the removed entries are listed by name instead of summarized as a number.

For an identifier list that will become physical labels, clean only the accidental repeats here and pass the result to the bulk barcode generator. Its duplicate count remains visible because repeated labels can also be intentional.

Repeated vocabulary is a different question from repeated rows. For prose, a word frequency counter ranks the terms inside the lines without deleting any part of the source.

Case sensitivity is a real decision

By default, Anna and anna are two different lines, because that is what the text literally says.

For email addresses and usernames that is usually the wrong answer. Email local parts are technically case-sensitive but treated as case-insensitive by every major provider, so Anna@example.com and anna@example.com reach the same inbox — and sending to both means someone gets your newsletter twice.

For anything where the text is a label rather than an identifier, keep case sensitivity on. Turning it off can merge two genuinely different values that happen to differ only in capitalization.

Spaces and blank lines are compared as written

A line with a trailing space is, byte for byte, a different line from one without, and by default this tool treats it that way: banana and banana both stay. That is deliberate. In a list of codes or fixed-width values the space can be part of the value, and silently merging two different lines would be a change you never asked for.

Copying out of a spreadsheet or a web page routinely attaches stray spaces, though. When that is what happened, turn on Ignore spaces around each line: lines are then trimmed before comparison, and the trimmed version is what you get back.

Blank lines are lines too. Two blank lines become one, like any other repeat, and the one that stays is counted as kept. If you want them gone entirely, the text cleaner can empty and close up the gaps first.

Nothing is uploaded

Deduplication runs on lists of email addresses, customer records, and exported identifiers more often than on anything else. All of it stays in this tab — the comparison happens in your browser and there is no server to send it to.