Developer & Data

Clean Duplicate CSV Records Without Losing Cells

Updated

A repeated import can add the same rows twice. Cleaning those duplicates is different from deleting every row that shares a customer name. Two purchases by the same customer may both be legitimate.

CSV Cleaner & Deduplicator works on complete CSV records. You can trim cell edges and remove repeated rows while keeping the remaining header and column structure.

Decide whether spaces are meaningful

Consider a file headed name,code with the rows Ali,001 and Ali,001. With duplicate removal selected, one record remains. If another row says Ali,002, it remains because the whole row is different.

Trimming removes whitespace around each cell. That can help with accidental spaces around city names. It can also remove deliberate spaces in fixed width identifiers or copied text, so inspect the source before enabling it.

The order matters: when trimming is enabled, cells are trimmed before duplicate comparison. Two rows that differed only by edge spaces can then become identical and collapse into one.

A duplicate is not a business rule

The tool does not choose the newest customer record, reconcile different addresses or group orders by ID. If the same order appears with two different amounts, both rows are retained because they are not equal records.

CSV quoting still handles commas and newlines inside a cell. Do not split a multiline note into separate physical rows to make the file look cleaner; doing so changes the data.

The exported CSV uses the selected delimiter. Formula protection remains a separate setting and may add an apostrophe to suspicious text cells. Check that setting before sending the cleaned file to a machine importer.

Keep an untouched source copy and compare the reported number of removed duplicates with your expectation. An unexpectedly large reduction is a reason to inspect the trimming choice, not to assume the original records were all mistakes.

Join the conversation

Your email address will not be published. Required fields are marked *

Explore Whatson tool information