Note: All tools, techniques, and methodologies contained in this repository are intended for authorized security testing and educational purposes only. Use only on systems you own or have explicit written permission to test. Unauthorized use may violate local, state, and federal law. The author assumes no liability for misuse.
- Overview
- Getting started
- Tcpdump basics
- Ngrep basics
- Tcpflow basics
- Connection stats
- DNS digging
- Private IP/MAC address leakage
- Passive OS/app profiling
- Profiling HTTP traffic
- Extracting objects/files
- Content profiling
- Personal contact info
- Email traffic
- Password hunting
- Digging for PII & confidential data
- Parsing SMB/CIFS traffic
- Parsing SQL traffic
- Hardware/mobile device profiling
- Location tracking data
- Mobile app data
- Inspecting HTTPS Traffic
- Decrypting TLS traffic
A collection of Deep Packet Inspection (DPI) techniques, regular expressions, and PCAP test files.
Originally created for a Defcon 26 Packet Mining Workshop, co-presented with Sean Gallagher:
https://defcon.org/html/defcon-26/dc-26-workshops.html#porcello
Some packet capture traffic courtesy of Chris Sanders & wireshark.org:
https://github.com/chrissanders/packets
https://wiki.wireshark.org/SampleCaptures
Install mining tools (Ubuntu/Debian):
apt update && apt install ngrep tcpflow xplico ssldump dsniff tshark p0f pads python-html2text
Install mining tools (macOS via brew):
brew install ngrep tcpflow ssldump p0f
ln -s /Applications/Wireshark.app/Contents/MacOS/tshark /usr/local/bin/tshark ### add tshark to path, assuming Wireshark is already installed
Set a variable for your capture file name:
CAPFILE=CaptureFile.pcap
Basic local capture:
tcpdump -vvv -nn -i eth0 -w output.cap
Reading a capture:
tcpdump -vvv -nn -r output.cap
Remote capture through ssh (Capture on remote host's eth0):
ssh [email protected] 'sudo tcpdump -vUnni eth0 -w -' > output.cap
Show HTTP traffic on port 80:
tcpdump -vvvAnn -i eth0 port 80
Show SMTP/POP3 traffic for specific host:
tcpdump -vvvAnn -i eth0 'host 10.0.0.10 and port (25 or 110)'
Save filtered traffic to a new file (Example: Save only DNS traffic to a new file):
tcpdump -r $CAPFILE -w dns-only.cap port 53
Print live web traffic to console:
ngrep -d eth0 -W byline -q -t port 80
Grep live network traffic for "password"
ngrep -d eth0 -q -t -i 'password'
Grep for HTTP GET/POST requests:
ngrep -d eth0 -W byline -q -t '^(GET|POST)' port 80
Print ASCII packet data to console:
tcpflow -c -s -r $CAPFILE
Extract all flows, objects, & files to output folder:
mkdir tcpflow
tcpflow -a -r $CAPFILE -o tcpflow/
Top 10 source IPs:
tcpdump -nn -r $CAPFILE |grep " IP " | awk '{print$3}' |cut -d. -f -4 |sort |uniq -c |sort -nr |head
Top 10 destination IPs:
tcpdump -nn -r $CAPFILE |grep " IP " | awk '{print$5}' |cut -d. -f -4 |sort |uniq -c |sort -nr |head
Top connection pairs:
tcpdump -nn -r $CAPFILE |grep " IP " | awk '{print$3,$4,$5}' |sort |uniq -c |sort -nr |head
Top IP protocols:
tcpdump -nn -v -r $CAPFILE |grep " IP " |awk -F, '{print$6}' |sort |uniq -c |sort -nr
Top 10 destination ports (based on SYN packets):
tcpdump -nn -r $CAPFILE |grep " IP " |grep "Flags \[S\]" |awk '{print$5}' |cut -d. -f 5- |sort |uniq -c |sort -nr |head
Top domains:
tcpdump -nn -r $CAPFILE port 53 | egrep " A\? " | awk '{print$8}' | egrep -io "[a-z0-9]*\.[a-z]*\.$" | sort | uniq -ic | sort -nr | head
Top subdomains:
tcpdump -nn -r $CAPFILE port 53 | egrep " A\? " | awk '{print$8}' |sort |uniq -c |sort -nr |head
Grep for private IPs in packet data:
ngrep -q -t -W byline -I $CAPFILE '10\.([0-9]{1,3}\.){2}[0-9]{1,3}|192\.168\.[0-9]{1,3}\.[0-9]{1,3}|172\.([0-9]{1,3}\.){2}[0-9]{1,3}'
Grep for MACs in packet data:
ngrep -q -t -W byline -I $CAPFILE '([0-9a-fA-F][0-9a-fA-F]:){5}([0-9a-fA-F][0-9a-fA-F])' not port 5353
OS/app summary via p0f:
p0f -r $CAPFILE |egrep "^\| (os|app)" |sort |uniq
OS/app list via PADS:
pads -v -r $CAPFILE -w assets.csv port 80
Top 10 websites:
ngrep -I $CAPFILE -W byline -q -t '^(GET|POST)' port 80 | grep "^Host:" | sort |uniq -ic |sort -nr |head
Top 10 referrers:
ngrep -I $CAPFILE -W byline -q -t '^(GET|POST)' port 80 | egrep "^Referer: " |sort |uniq -ic | sort -nr |head
Top 10 GET requests (URLs):
ngrep -I $CAPFILE -W byline -q -t '^(GET )' port 80 | grep "^GET " | sort |uniq -ic | sort -nr |head
HTTP POSTs & POST data:
ngrep -I $CAPFILE -W byline -q -t '^(POST )' port 80 | egrep "^POST|^<|^[a-z]"
URL log with timestamps:
ngrep -I $CAPFILE -W byline -q -t '^(GET|POST)' port 80 |egrep "^T |^(GET|POST)|^Host:|^$"
Unique cookies:
tcpflow -r $CAPFILE -c -s port 80 | grep -v "\.\." | grep "^Set-Cookie" | sort |uniq
Unique session IDs/UUIDs:
tcpflow -r $CAPFILE -c -s port 80 | grep -v "\.\." | egrep -i "session.id|sessionid|session.token|SESSID|UUID|oauth|Authorization:" ### Add some --color if needed!
User-Agent profiling:
ngrep -I $CAPFILE -W byline -q -t port 80 | egrep "^User-Agent: " |sort |uniq -ic | sort -nr
Extract all objects/files & decode HTML:
tcpflow -a -r $CAPFILE -o tcpflow/
Breakdown by file type:
find tcpflow/ |egrep -o "\.[a-zA-Z]*$" |sort |uniq -ic |sort -nr
Extracting & decoding with xplico:
xplico -m pcap -f $CAPFILE
Search engine queries:
ngrep -I $CAPFILE -W byline -q -t port 80 | egrep 'GET \/search\?q=' |sort |uniq
URL "keyword" strings:
ngrep -I $CAPFILE -W byline -q -t '^(GET|POST)' port 80 | egrep "^GET |^POST |^Referer: " | egrep -o "[a-z-]*" | egrep "[a-z-]*-[a-z-]*-" | egrep -v "(^-|-$)" |sort | uniq -ic |sort -nr |head
Top words from HTML content:
cat tcpflow/*.html |html2text | egrep -o '\w{4,}' |sort |uniq -c |sort -nr |head -n25
Email addresses with common TLDs:
tcpflow -r $CAPFILE -c -s | egrep -i --color '\w+@[a-zA-Z_]+?\.(com|org|net|gov|mil|edu|co|biz|info)'
Email addresses with any TLD (more false positives):
tcpflow -r $CAPFILE -c -s | egrep -i --color '\w+@[a-zA-Z_]+?\.[a-zA-Z]{2,6}'
"Dashed" phone numbers:
tcpflow -r "$CAPFILE" -c -s port 80 | grep --color -P "\d{3}-\d{3}-\d{4}"
Dashed or dotted phone numbers (more false positives):
tcpflow -r "$CAPFILE" -c -s port 80 | grep --color -P "\d{3}[-.]\d{3}[-.]\d{4}"
Email senders, recipients, & email subjects:
ngrep -q -t -W byline -I $CAPFILE port 25 or port 110 |egrep "^To:|^From:|^Subject"
Email client apps & AV scanners:
tcpflow -c -s -r $CAPFILE port 25 or port 110 |egrep -A1 "^User-Agent:|X-Antivirus" |sort -u
Extract emails to console:
tcpflow -c -s -r $CAPFILE port 25 or port 110
Extract emails to disk:
tcpflow -a -r $CAPFILE port 25 or port 110 -o tcpflow/
Extracting email attachments:
tcpflow -C -0 -r $CAPFILE port 25 or port 110
cat base64.txt | base64 -d > file.xxx
file file.xxx ### Verify file is correct type
FTP, Telnet, SMTP, POP3, HTTP, etc:
ngrep -I $CAPFILE -W byline -q -t | egrep --color "[Pp]assword[=:]|&[Pp]ass=|[Ss]ecret=|pwd=|^PASS|^USER |^AUTH |login:|^Authorization:"
Decoding HTTP Basic auth, SMTP, POP3 (base64):
echo 'QWxhZGRpbjpPcGVuU2VzYW1l' | base64 -d
Finding SNMP community strings:
tcpdump -A -nn -r $CAPFILE port 161
Credit card numbers:
tcpflow -c -s -r $CAPFILE | grep -P --color '(6011|5[1-5]\d{2}|4\d{3}|3\d{3})[- ]\d{4}[- ]\d{4}[- ]\d{4}'
Social security numbers:
tcpflow -c -s -r $CAPFILE | grep -P --color '[ ^]([0-6]\d\d|7[0-256]\d|73[0-3]|77[0-2])[- ]\d{2}[- ]\d{4}'
DOB/License/Passport numbers:
tcpflow -c -s -r $CAPFILE | grep -v Cookie |egrep --color 'DOB[:= ]|[Pp]assport[:= ]|[Ll]icense number'
Classified/tagged documents:
tcpflow -c -s -r $CAPFILE | grep -v Cookie |egrep --color -i 'CONFIDENTIAL|PROTECTED|INTERNAL USE ONLY|TOP SECRET|CLASSIFIED'
SMB users, domains, & password hashes:
tshark -nn -r $CAPFILE -V -Y tcp.port==445 |egrep "Lan Manager Response|NTLM Response|NTLMv2 Response|Domain name|User name|Host name"
SMB share & file access timeline:
tshark -nn -r $CAPFILE -V -Y tcp.port==445 |egrep "Arrival Time: |Tree Id: |\[Account: |\[Domain: |\[Host: |NT Status: |Command: |GUID handle File: "
Carving files out of SMB traffic:
tshark -nn -r $CAPFILE -q --export-objects smb,tmpfolder
MySQL password hashes, queries, & responses:
tshark -nn -r $CAPFILE -V -Y tcp.port==3306 | egrep 'Username:|Password:|Statement:|text:'
MSSQL queries & responses:
tshark -nn -r $CAPFILE -V -Y tcp.port==1433 | egrep "Query:|Data:|Data \[truncated\]:"
Device info via HTTP:
ngrep -I "$CAPFILE" -W byline -q -t port 80 | egrep --color "device_name=|device_type=|os_version=|dev=|X-Device-Info:|Device:|DEVICE:|deviceId=|deviceModel="
Device info via mDNS:
tshark -nn -r $CAPFILE -V -Y udp.port==5353 |egrep "Name: |product=|model=" |sort |uniq |egrep -v "Domain Name:|Name: _"
Windows error reporting: Hardware vendor, model, BIOS/firmware versions, running processes, exe/dll versions, & connected USB devices:
ngrep -I "$CAPFILE" -W byline -q -t '^(GET|POST)' port 80 |egrep "^T |^GET|^Host:" |egrep -B2 "watson.microsoft.com.$"
Cell carrier codes:
ngrep -I "$CAPFILE" -W byline -q -t port 80 | egrep --color "mcc=|mnc=|csc=|mccmnc"
Apple plist files: extract with tcpflow, decode with plistutil:
grep "plist version" tcpflow/*
apt install libplist-utils
plistutil -i <plistfile>
Via Apple default weather app, Wunderground, etc:
ngrep -I "$CAPFILE" -W byline -q -t '^(GET|POST|HTTP/)' port 80 |egrep "%2Clatitude%2|maxlat=|latitude=|latlon"
Via Windows default weather app:
ngrep -I "$CAPFILE" -W byline -q -t 'weather.microsoft.com' port 80 |egrep --color "DisplayName="
Android apps, versions, usage, etc:
ngrep -I $CAPFILE -W byline -q -t '^(GET )' port 80 | egrep "^GET |^Host:" |grep --color -A1 "ap_an="
Android app traffic (via Dalvik agent):
ngrep -I $CAPFILE -W byline -q -t 'User-Agent: Dalvik' port 80
Apple apps/store traffic:
ngrep -I $CAPFILE -W byline -q -t port 80 | egrep -B1 "bundleId=|dpkg.ipa|^[Xx]-[Aa]pple"
iTunes audio downloads:
ngrep -I $CAPFILE -W byline -q -t port 80 | egrep -B6 "User-Agent: AppleCoreMedia"
Kindle app traffic ("key=" indicates ASIN of each ebook)
ngrep -q -t -I $CAPFILE -W byline | grep --color 'type="EBOK" key='
Prime video streaming file downloads:
ngrep -q -t -I $CAPFILE -W byline | grep -B6 'Prime%20Video'
Extract SSL certificates with tcpflow:
tcpflow -a -r $CAPFILE -o tcpflow/ port 443
Extract domain names via certificate Server Name Indication (SNI):
ngrep -I $CAPFILE -q -t -W byline port 443 |egrep -o "[a-z0-9]*\.[a-z0-9]*\.(com|org|net|gov|mil|edu|co|biz|info)" |sort -u
Sessions using weak cipher suites:
ssldump -n -r $CAPFILE | grep "cipherSuite" | egrep -i "RC4|MD5|EXP|NULL|_DES|ANON|64"
Sessions using weak encryption protocols:
ssldump -n -r $CAPFILE | grep Version |sort -u
Decrypting TLS traffic (SSL/TLS 1.2 and prior without PFS) using a known private key:
tshark -r SSL-decryption.pcap -q -o "ssl.keys_list:192.168.56.101,443,http,server.pem" -z "follow,ssl,ascii,2"
Decrypting TLS 1.3 with PFS (or TLS 1.2 and prior with PFS) via SSLKEYLOGFILE environment variable:
# On Windows:
set SSLKEYLOGFILE=%USERPROFILE%\Desktop\keylogfile.txt ### or use setx to set the variable persistently & globally!
start firefox ### or chrome/brave/edge
# On macOS:
export SSLKEYLOGFILE=$HOME/keylogfile.txt
open -a /Applications/Google\ Chrome.app/ ### or firefox/brave/edge
# On Linux:
export SSLKEYLOGFILE=$HOME/keylogfile.txt
firefox ### or chrome/brave/edge
With the environment variable set, you can now decrypt TLS 1.3/PFS traffic generated by firefox/chrome/brave/edge, using the queries below.
Note 1: Some websites (such as Google, Facebook, Instagram) use the UDP-based HTTP2/3 QUIC protocol instead of TCP/HTTPS, which is not as consistently decryptable due to the experimental status of the Google/IETF QUIC protocol spec. As a workaround (until the IETF QUIC standard is more widely adopted and support by Wireshark/tshark), go to chrome://flags (or brave://flags for Brave, or about:config for Firefox), search for "QUIC", and Disable support for the experimental QUIC protocol. Alternately, you can also block UDP ports 80/443 outbound at the perimeter, or via iptables if performing a MITM attack.
Note 2: For consistent reproducability, start packet captures before browsing to any target sites. If needed, clear session keyfile and start over, and try starting the capture before opening browser (browser may cache sessions for some sites, even after tab is closed).
Decrypting GET/POST requests:
tshark -r capture.cap -o tls.keylog_file:~/keylogfile.txt -o tls.desegment_ssl_records:TRUE -o tls.desegment_ssl_application_data:TRUE -V | egrep "GET /|POST /"
Decrypting HTTP header data:
tshark -r capture.cap -o tls.keylog_file:~/keylogfile.txt -o tls.desegment_ssl_records:TRUE -o tls.desegment_ssl_application_data:TRUE -V -Y "http || http2 || http3" | egrep "Header: "
Decrypting authorization headers (possibly containing creds/hashes):
tshark -r capture.cap -o tls.keylog_file:~/keylogfile.txt -o tls.desegment_ssl_records:TRUE -o tls.desegment_ssl_application_data:TRUE -V -Y "http || http2 || http3" | egrep "Header: authorization:"
Decrypting interesting headers (including auth/keys/cookies):
tshark -r capture.cap -o tls.keylog_file:~/keylogfile.txt -o tls.desegment_ssl_records:TRUE -o tls.desegment_ssl_application_data:TRUE -V -Y "http || http2 || http3" | egrep "Header: (:method:|:authority:|:path:|authorization: |cookie: )"
Decrypting HTTP POST Form Data (including credentials) (Note: rarely (i.e. for Apple ID logins), form data may show up under String value: instead of Form item:):
tshark -r capture.cap -o tls.keylog_file:~/keylogfile.txt -o tls.desegment_ssl_records:TRUE -o tls.desegment_ssl_application_data:TRUE -V | egrep "Form item: "
Decrypting all HTTP POST parameter data:
tshark -r capture.cap -o tls.keylog_file:~/keylogfile.txt -o tls.desegment_ssl_records:TRUE -o tls.desegment_ssl_application_data:TRUE -V -Y http2.headers.method==POST -Tfields -e http2.data.data
Determining GQUIC version(s) in use:
tshark -r capture.cap -o tls.keylog_file:~/keylogfile.txt -o tls.desegment_ssl_records:TRUE -o tls.desegment_ssl_application_data:TRUE -V | grep -i google_quic_version
Redirecting traffic through an attacker-controlled proxy server (to capture encrypted traffic remotely):
google-chrome --proxy-server="socks5://YOUR_PROXY:8080"