sort and uniq cheat sheet

sort flags

Flag	What it does
`-n`	Numeric: `10` after `9`
`-h`	Human sizes: `2K`, `5M`, `1G`
`-V`	Versions: `1.9` before `1.10`
`-r`	Reverse
`-u`	Drop duplicate lines
`-f`	Ignore case
`-k2,2`	Sort by field 2 only
`-t,`	Fields are separated by commas
`-s`	Stable: keep input order for equal keys
`-R`	Shuffle
`-c`	Check if already sorted
`-o file`	Write to a file (safe even when it’s the input)
`-S 2G` / `--parallel=4`	Memory buffer / threads for big files

Sort by column

bash

sort -k2,2 file                         # by the second field
sort -k3,3n file                        # by the third field, numeric
sort -t, -k3,3nr data.csv               # CSV, column 3, numbers, descending
sort -t: -k3,3n /etc/passwd             # users by UID
sort -k2,2 -k1,1n file                  # by field 2, ties by numeric field 1
sort -k1.5,1.8 file                     # by characters 5–8 of field 1

Always write the end of the key (-k2,2). -k2 means “from field 2 to the end of the line”, which is rarely what you want.

Common sorts

bash

du -sh * | sort -h                      # sizes, smallest first
ls -l | sort -k5,5nr | head             # biggest files
sort -V versions.txt                    # 1.2, 1.9, 1.10
sort -u ips.txt > unique-ips.txt
sort -o file file                       # sort a file in place
LC_ALL=C sort big.txt > sorted.txt      # byte order: faster, repeatable

uniq

uniq only compares neighbouring lines, so sort first.

Command	Prints
`sort file \| uniq`	Each line once (same as `sort -u`)
`sort file \| uniq -c`	Each line with its count
`sort file \| uniq -d`	Only lines that appear more than once
`sort file \| uniq -u`	Only lines that appear exactly once
`uniq -i`	Ignore case
`uniq -f 1`	Skip the first field when comparing
`uniq -w 10`	Compare only the first 10 characters

Count and rank

bash

# most common lines first
sort file | uniq -c | sort -rn | head

# top 10 client IPs in an access log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

# most used shell commands
history | awk '{print $2}' | sort | uniq -c | sort -rn | head

# duplicate lines across two files
sort a.txt b.txt | uniq -d

# remove duplicates but keep the original order
awk '!seen[$0]++' file

Keep in mind

sort order depends on the locale. With LC_ALL=C it compares bytes, which is faster and gives the same result on every server.
sort -n reads 1e3 as 1, and sort -g as 1000. Use -g for scientific notation.
sort file > file empties the file before reading it. Use sort -o file file.

Linux pipes cheat sheet: redirects and chaining commands
text processing one-liners: cut, tr, paste, comm and join
head and tail cheat sheet: first and last lines, following logs
awk cheat sheet: columns, sums and reports

sort and uniq cheat sheet

sort flags

Sort by column

Common sorts

uniq

Count and rank

Keep in mind

Related

Comments