Sort & Deduplicate Lines

Sort, deduplicate, trim, number. Any one of these is two lines of code you could write yourself in a minute. Chaining them is where it gets fiddly, because the order changes the answer, so the order is shown above and fixed.

1  whitespace     trim, collapse, drop blanks
2  keep or drop   filter by text or a pattern
3  second list    union, intersection, difference
4  duplicates     keep the first, the last, or only the unique
5  order          sort, reverse, shuffle
6  decorate       prefix, suffix, numbering, joining

Trimming before deduplicating is why "item " and "item"
count as one. Filtering before sorting is why you do not
sort things you are about to throw away.

The order is not arbitrary. Trimming has to happen before deduplicating, or the same value with a trailing space counts as a different one. That is easily the most common reason a "remove duplicates" looked like it worked and had not. Numbering has to happen last, or the numbers describe the order before it was sorted.

Alphabetical is rarely what you meant

Alphabetical        Natural

  item1               item1
  item10              item2
  item2               item10
  item20              item20
  item3               item3 -> item20

Alphabetical compares character by character, and "1" comes
before "2", so item10 lands between item1 and item2.

A plain sort compares character by character, and the character1 sorts before 2, so item10 lands between item1 and item2. Natural ordering reads runs of digits as numbers. It is what file managers do and what people expect.

There is a third option worth knowing about: sorting by the number found anywhere in the line. Handy for a log with timestamps, or a list where the number is not at the start. Lines with no number in them go to the end, whichever direction the sort is, because they are not part of the ordering being asked for.

Three different things called deduplicating

Input          first    last     only unique

  a              a        b        b
  b              b        a
  a
  c              c        c        c

"Keep the first of each" leaves one of everything.
"Only lines that appear once" throws the repeats away entirely.

"Keep the first of each" is what sort -u and most tools mean, and it leaves one of everything. "Only lines that appear once" throws the repeats away entirely. That is uniq -u behaviour, and what people usually want when they are looking for the odd one out. They are different questions and the answers rarely overlap.

Case is a switch on both, and it matters more than it sounds: email addresses are case-insensitive in the domain and case-sensitive in the local part by specification, and every mail server on earth ignores that and treats them as case-insensitive.

Comparing two lists

The second list turns this into a set calculator. Most "which of these are missing" questions actually are:

• Only what is not in the second. What the first list has that the second does not. Usually the answer to "what have we not done yet".
• Only what is in both. The overlap.
• Only what is in one but not both. Differences running in either direction. Use this one when neither list is the authority.
• Everything in either. The two merged, duplicates gone.

The shuffle is deliberately not random

Everything on this site recomputes on every keystroke. A shuffle usingMath.random would therefore reorder itself continuously while you typed, and would never settle.

So the shuffle is seeded from the content: the same list always shuffles the same way, and changing the list reshuffles it. If you want a different order for the same input, add a character and take it away again.

Where this stops and something else starts

This works on whole lines. As soon as the interesting part is a field within a line, say sorting by the third column or deduplicating on one key, you have a table and not a list. The CSV converter on this site is the one that understands columns.

Two hundred thousand lines is the ceiling here, and it is about the browser and not the algorithm: laying out that much text in a textarea takes longer than every operation on this page put together.