List Cleaning Reference

The order that works

1. Trim each line

removes invisible leading/trailing spaces

2. Remove blank lines

blanks are not items

3. De-duplicate

now that identical items really are identical

4. Sort

on clean text, so spaces don't sort

5. Number

LAST — or you sort by the numbers

⚠ De-duplication compares

RAW strings, byte for byte

⚠ 'apple' vs 'apple '

NOT duplicates without trimming

⚠ De-duplication is

CASE-SENSITIVE — Banana ≠ banana

⚠ Sorting is

locale-aware, so apple comes before Banana

⚠ Numbers sort as TEXT

10 before 9 — use a spreadsheet

Before trusting a de-duplicated list

• Trimmed BEFORE de-duplicating, not after
• Blank lines removed
• Counted before and after — does the difference look right?
• Case differences checked, if they should have merged
• Numbers not being sorted as text
• Original order preserved instead, if the order meant something
• A copy of the original kept — de-duplication is not reversible

Trailing spaces are invisible and decisive

Text pasted from a spreadsheet or a web page carries trailing spaces you cannot see, and de-duplication compares raw strings. So the duplicates are still there, the count looks plausible, and nothing tells you it went wrong. Trim first, every time.

Keep the original

De-duplication is not reversible: once two identical lines have become one, nothing records that there were two. If the count of duplicates was itself information — how many times something appeared — you needed that before you cleaned the list, not after.