Data deduplication is a fundamental operation in data processing, from cleaning spreadsheet exports to preprocessing log files for analysis. The core challenge is defining "duplicate" — exact byte-for-byte match, case-insensitive match, match after whitespace normalization, or match on a specific field within each line. Each definition produces different results, and the right choice depends on the data source and downstream use.

Case sensitivity is a critical consideration. In many datasets, "John Smith" and "john smith" are the same entity entered differently. Case-insensitive deduplication catches these duplicates but risks merging genuinely distinct entries (the programming variable "count" and a document section "Count"). Whitespace normalization (trimming leading/trailing spaces and collapsing multiple spaces to one) is almost always desirable, since trailing spaces are invisible and inconsistent spacing is a common data quality issue.

Preserving insertion order matters when the sequence of lines carries meaning — chronological logs, ordered lists, or narrative text. Some deduplication algorithms sort the data first (which makes finding duplicates O(n log n) but destroys order) while others use a seen-set (O(n) with order preserved). For most user-facing deduplication needs, preserving the first occurrence's position in the original order is the expected behavior.

At scale, deduplication becomes a systems-level challenge. Deduplicating terabytes of data (as in backup systems, search engine crawling, or data warehouses) uses techniques like MinHash, SimHash, or Bloom filters to identify approximate duplicates efficiently. These probabilistic data structures trade a small false-positive rate for massive memory savings compared to keeping a full set of seen items.