Pipelines & Chaining — Combine Text Tools
How pipes work
cmd1 | cmd2 | cmd3 | cmd4 > output.txt| Symbol | Meaning |
|---|---|
| | Send stdout of left command to stdin of right |
> | Redirect final output to file (overwrite) |
>> | Append to file |
2>/dev/null | Hide errors (permission denied, etc.) |
2>&1 | Redirect stderr to stdout |
Design order: filter lines first → extract fields → transform → dedupe → save.
RAW DATA → grep (filter lines) → awk/cut (fields) → sed/tr (clean) → sort -u → file
📌 1) Screenshot Example — XMPP User Extraction (HTB)
Goal: Extract unique usernames from an XMPP log for jab.htb.
grep jab.htb xmpp.txt | awk -F\> '{print $2}' | awk -F@ '{print $1}' | sort -u > users.txtStep-by-step
| # | Command | Input → Output |
|---|---|---|
| 1 | grep jab.htb xmpp.txt | Full log → only lines with domain |
| 2 | awk -F\> '{print $2}' | Line with > tag → text after first > |
| 3 | awk -F@ '{print $1}' | user@jab.htb → user |
| 4 | sort -u | Messy list → sorted unique usernames |
| 5 | > users.txt | Save for Kerbrute, Hydra, AS-REP roast |
Why two awk passes?
First -F\> strips XML/tag prefix. Second -F@ strips domain. Could sometimes be one awk:
grep jab.htb xmpp.txt | awk -F'[@>]' '{for(i=1;i<=NF;i++) if($i~/^[a-z]/) print $i}' | sort -uTwo-pass awk is easier to read under exam pressure.
Use the output
# Kerberos user enum
kerbrute userenum --dc 10.10.10.10 -d jab.htb users.txt
# AS-REP roast
impacket-GetNPUsers jab.htb/ -dc-ip 10.10.10.10 -no-pass -usersfile users.txtGet
xmpp.txtfrom Pidgin (XMPP Console after user search) or export from the client. Full XMPP workflow: UseCases for ports > Port 5222 / 5223 — XMPP / Jabber.
📌 2) Pattern Library
Pattern A — Filter → Extract → Dedupe
grep PATTERN file | awk -FSEP '{print $N}' | sort -u > output.txtUse when: logs, exports, tool output with structured lines.
Pattern B — Find files → Search inside
find PATH -name "*.ext" 2>/dev/null | xargs grep -l "PATTERN" 2>/dev/nullUse when: hunting passwords in configs across filesystem (Linux privesc).
find /var/www -name "*.php" 2>/dev/null | xargs grep -i "password\|passwd\|db_pass" 2>/dev/null
find / -name "*.conf" 2>/dev/null | xargs grep -l "password" 2>/dev/nullPattern C — Extract column → Transform → List
cut -d: -f1 /etc/passwd | grep -v "^#" | sort -uUse when: /etc/passwd, CSV, colon-delimited files.
grep -v "nologin\|false" /etc/passwd | cut -d: -f1Pattern D — Build comma-separated list for tools
grep "^[0-9]" ports.txt | cut -d'/' -f1 | tr '\n' ',' | sed 's/,$//'Use when: feeding port list to Nmap:
PORTS=$(grep "^[0-9]" allports.txt | cut -d'/' -f1 | tr '\n' ',' | sed 's/,$//')
nmap -sC -sV -p$PORTS -oN targeted.txt TARGETPattern E — Count occurrences
cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | headUse when: top IPs, most common errors, frequency analysis.
Pattern F — Find + execute
find / -perm -4000 -type f 2>/dev/null | xargs ls -la 2>/dev/null
find . -name "*.sh" -exec chmod +x {} \;Use when: SUID enum, batch file ops.
📌 3) OSCP Recipe Book
Extract users from /etc/passwd
grep -v "nologin\|false" /etc/passwd | cut -d: -f1 | sort -u
awk -F: '$3 >= 1000 && $3 < 65534 { print $1 }' /etc/passwdHunt credentials in web root
grep -rni "password\|passwd\|secret\|api_key" /var/www/ 2>/dev/null
find /var/www -name "*.php" -o -name "*.config" 2>/dev/null | xargs grep -i "password" 2>/dev/nullParse nmap output
# Open ports only
grep "/open" nmap.txt
# Port numbers for second scan
grep "^[0-9]" allports.txt | cut -d'/' -f1 | tr '\n' ',' | sed 's/,$//'
# Service versions
grep -E "^[0-9]+/(tcp|udp)" nmap.txt | awk '{print $1, $3, $4, $5}'Build username list from multiple sources
# Combine sources, dedupe
cat users1.txt users2.txt emails.txt | awk -F@ '{print $1}' | sort -u > all_users.txtExtract from HTTP access log (IPs)
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20Extract from auth.log (failed SSH)
grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort -uClean wordlist / remove blank lines
grep -v "^$" wordlist.txt | sort -u > clean.txt
sed '/^$/d' wordlist.txt | sort -u > clean.txtStrip Windows line endings
tr -d '\r' < windows.txt > unix.txt
sed -i 's/\r$//' file.txtSUID hunting + strings
find / -perm -4000 -type f 2>/dev/null | while read f; do echo "=== $f ==="; strings "$f" | grep -i "bin/sh"; done📌 4) Debugging Pipelines
Run one stage at a time to see where data breaks:
# Step 1 — how many lines after grep?
grep jab.htb xmpp.txt | wc -l
# Step 2 — what does first awk produce?
grep jab.htb xmpp.txt | awk -F\> '{print $2}' | head
# Step 3 — full chain preview
grep jab.htb xmpp.txt | awk -F\> '{print $2}' | awk -F@ '{print $1}' | head
# Step 4 — final
grep jab.htb xmpp.txt | awk -F\> '{print $2}' | awk -F@ '{print $1}' | sort -u | tee users.txt| Tool | Debug tip |
|---|---|
tee file | Save AND show output mid-pipeline |
head | Preview first 10 lines |
wc -l | Count lines at each stage |
cat -A | Show hidden chars ($, ^M) |
📌 5) Tool Selection Cheat Sheet
| Task | Pipeline start |
|---|---|
| Filter lines by text | grep PATTERN file |
| Filter lines by regex | grep -E 'regex' file |
| Get column N (simple) | cut -d: -fN |
| Get column N (complex) | awk -F: '{print $N}' |
| Replace text | sed 's/a/b/g' |
| Delete char / join lines | tr |
| Remove dupes | sort -u |
| Count dupes | sort | uniq -c | sort -rn |
| Search many files | find ... | xargs grep |
| Save output | > file or tee file |
📌 Quick Copy/Paste — Most Common One-Liners
# XMPP / log → user list
grep DOMAIN file | awk -F@ '{print $1}' | sort -u > users.txt
# Passwd → usernames
grep -v "nologin\|false" /etc/passwd | cut -d: -f1
# Config password hunt
grep -rni "password" /var/www/ 2>/dev/null
# Find + grep configs
find / -name "*.conf" 2>/dev/null | xargs grep -l "password" 2>/dev/null
# Nmap ports → comma list
grep "^[0-9]" ports.txt | cut -d'/' -f1 | tr '\n' ',' | sed 's/,$//'
# Top IPs in log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head
# SUID find
find / -perm -4000 -type f 2>/dev/null