uniq — Remove Duplicate Lines
What is uniq?
uniq filters adjacent duplicate lines. It does NOT dedupe a whole unsorted file — duplicate lines must be next to each other. Always run sort first, or use sort -u instead.
Syntax
uniq [OPTIONS] [INPUT [OUTPUT]]
sort file | uniq📌 Flags
| Flag | Description |
|---|---|
-u | Unique only — suppress duplicate lines entirely |
-c | Prefix lines with occurrence count |
-d | Print only duplicate lines |
-D | Print all duplicate lines (not collapsed) |
-i | Case-insensitive comparison |
-f N | Skip first N fields before comparing |
-s N | Skip first N characters |
📌 Examples
# WRONG — won't dedupe non-adjacent dupes
uniq unsorted.txt
# CORRECT
sort file.txt | uniq
sort -u file.txt # easier — same result
# Count occurrences
sort access.log | uniq -c | sort -rn | head
# Output: 142 192.168.1.1
# Show only duplicates
sort file.txt | uniq -d
# Count unique IPs in log
cut -d' ' -f1 access.log | sort | uniq -c | sort -rn📌 When to Use uniq Over sort -u
Use uniq -c when you need counts — that’s its killer feature:
sort users.txt | uniq -c | sort -rnFor simple deduplication → use sort -u.
📌 Quick Cheat Sheet
sort file.txt | uniq
sort file.txt | uniq -c
sort file.txt | uniq -c | sort -rn # most common first
sort -u file.txt # prefer this for dedupe only