This tool removes duplicate lines from input text in your browser; nothing leaves your machine. The interesting question is what “duplicate” means and what to do about order.

The two strategies

Order-preserving dedup

Keep the first occurrence of each unique line; remove later duplicates.

Input:        Output:
apple         apple
banana        banana
apple         cherry
cherry
banana

The result has the original ordering of the first occurrence of each line. This is the common request — “make this list of names unique” usually means this.

Sort then dedup

Input:        Output:
apple         apple
banana        banana
apple         cherry
cherry
banana

The output is sorted alphabetically with duplicates removed. The order is destroyed; the result is canonical.

Equivalent to Unix sort -u or sort | uniq.

The tool above is order-preserving. For the sorted form, run the result through sort lines.

Case sensitivity

Input:        Case-sensitive:    Case-insensitive:
APPLE         APPLE              APPLE
apple         apple              banana
APPLE         banana
banana

Case-insensitive dedup treats APPLE and apple as duplicates. The tool above is case-sensitive only; to dedupe case-insensitively, lower-case the text first with text case.

Whitespace handling

Input:                   Strict:               Trim mode:
  apple                    apple                 apple
apple                    apple                 banana
apple

Strict mode treats lines with different leading/trailing whitespace as different. Trim mode normalizes whitespace before comparing — apple, apple, apple all dedup to one line.

The tool above uses strict mode: lines must match exactly, including spaces.

Empty lines

A common ambiguity: should multiple consecutive blank lines collapse to one? Or be removed entirely? Or be preserved as-is?

Option	Behavior
Treat empty as duplicates	Multiple empty lines collapse to one
Preserve all empty lines	Empty lines aren’t deduplicated
Remove all empty lines	Empty lines are stripped entirely

The tool above does the first: empty lines are compared like any other line, so only the first blank line is kept.

What “line” means

A line is a sequence of characters terminated by \n (or \r\n on Windows files). For dedup, the line content is the comparison key — the line terminator itself isn’t compared (otherwise CRLF and LF lines wouldn’t match).

The tool splits on \n and \r\n alike, so a Windows file dedupes the same way as a Unix one (the output uses \n).

Performance

For small inputs (< 10,000 lines), all dedup strategies are fast.

For large inputs:

Hash-based dedup (what the tool uses): O(n) time, O(unique lines) memory. Walks the input once, keeping a Set of seen lines.
Sort-based dedup: O(n log n) time, O(n) memory. The Unix sort -u approach.

Hash-based is faster for moderate inputs but memory-heavy for very large data. Sort-based scales better to multi-GB inputs but is slower in absolute terms.

The tool runs in browser memory; expect issues above ~100 MB.

Use cases

Cleaning a list of email addresses, names, IDs before importing.
Deduplicating log files when grepping for unique error patterns.
Finding the unique lines in two files: combine with sort-based dedup.
Removing duplicate URLs from a list of links.
Deduplicating SQL output that wasn’t DISTINCT-ed at query time.

Lines vs records

For data files where one record spans multiple lines (CSV with embedded newlines, multi-line log entries), line-based dedup gets the wrong answer. Use a record-aware dedup:

For CSV, parse with a CSV-aware tool then dedup on the parsed rows.
For multi-line logs, identify the record boundary (often a timestamp) and dedup on full records.

Common pitfalls

Leading/trailing whitespace silently making “duplicates” different. Use trim mode if you want whitespace-insensitive dedup.
Case sensitivity surprise. Email addresses are case-insensitive in the local part by some servers, case-sensitive by others. For consistent dedup of email lists, lowercase first.
CRLF vs LF. A Windows file deduped against a Unix-origin file won’t dedup if line endings differ. Normalize line endings first.
Sort stability. Order-preserving dedup keeps the first occurrence of each line. If you want the last occurrence (e.g., the most recent log entry), reverse the input first.
Lossy normalization. Aggressive whitespace normalization (collapse multiple spaces, etc.) can mark legitimately-different lines as duplicates. Use the lightest normalization that fits your data.
Sorting numeric lines alphabetically. 1, 10, 2 sort to 1, 10, 2. For numeric sort, use the dedicated sort lines tool with numeric mode.

For sorting (without dedup), see sort lines. For pattern-based filtering, find & replace.

Remove Duplicate Lines