Passive Gathering with theHarvester and Maltego
Attack Chain Position
graph LR
A[Reconnaissance] --> B[Passive Gathering]
B --> C[theHarvester]
B --> D[Maltego]
C --> E[Emails, IPs, Hosts]
D --> F[Visual Entity Graph]
E --> G[Data Correlation]
F --> G
G --> H[Active Recon]
H --> I[Exploitation]
style B fill:#f9d56e,stroke:#333,stroke-width:2px
style C fill:#a8e6cf,stroke:#333,stroke-width:1px
style D fill:#a8e6cf,stroke:#333,stroke-width:1px
Position: Passive Reconnaissance Phase — theHarvester and Maltego are complementary tools that gather intelligence from public sources without directly interacting with the target, reducing detection risk while building a comprehensive picture of the target’s attack surface.
Table of Contents
- theHarvester Deep Dive
- Maltego Deep Dive
- theHarvester vs Maltego Comparison
- Combining theHarvester and Maltego
- Other Passive Gathering Tools
- Automating Passive Gathering
- Data Correlation Techniques
- Building Passive Recon Pipelines
- Managing Large Result Sets
- OSCP Value Assessment
- OSCP Exam Tips
theHarvester Deep Dive
Overview and Purpose
theHarvester is one of the oldest and most widely-used open-source tools for passive information gathering. Originally created by Luis Martinez, it is now maintained by the TrustedSec team and is part of the OWASP project.
Primary Purpose: Gather emails, subdomains, hosts, employee names, open ports, and banners from various public sources.
Key Characteristics:
- Passive-only: Does not interact directly with the target
- Multi-source: Queries dozens of public data sources
- Fast: Results typically return in seconds to minutes
- Free and open-source: Available on GitHub
- Cross-platform: Works on Linux, macOS, and Windows
Installation
# Method 1: pip install (recommended)
pip3 install theHarvester
# Method 2: From source
git clone https://github.com/laramies/theHarvester.git
cd theHarvester
pip3 install -r requirements.txt
python3 setup.py install
# Method 3: Kali Linux (pre-installed)
# theHarvester comes pre-installed on Kali Linux
theHarvester --help
# Verify installation
theHarvester --version
Command-Line Syntax
theHarvester [-b engines] [-d domain] [-l limit] [-f output_filename] [-n] [-c] [-j] [-v] [-t] [-h]
All Commands and Flags
| Flag | Short | Description | Example |
|---|---|---|---|
-b |
--sources |
Comma-separated list of sources to use | -b bing,google,linkedin |
-d |
--domain |
Target domain name | -d example.com |
-l |
--limit |
Maximum number of results to return | -l 500 |
-f |
--filename |
Output filename (without extension) | -f results |
-n |
--dns-lookup |
Perform DNS resolution on found hosts | -n |
-c |
--dns-reverse |
Perform reverse DNS lookups | -c |
-j |
--dns-brute |
Perform DNS brute force using subdomains list | -j |
-v |
--verbose |
Enable verbose output | -v |
-t |
--takeover |
Check for potential subdomain takeovers | -t |
-h |
--help |
Show help message | -h |
-z |
--subtakeover |
Enable subdomain takeover detection | -z |
--shodan |
Shodan API key for enhanced results | --shodan API_KEY |
|
--hackertarget |
Enable HackerTarget source | --hackertarget |
|
--full |
Run all available sources | --full |
All Available Sources
theHarvester supports dozens of data sources. Here is a comprehensive list:
Search Engine Sources
| Source | Data Types | Description |
|---|---|---|
bing |
Subdomains, Emails, IPs | Microsoft Bing search engine |
google |
Subdomains, Emails, IPs | Google search engine |
yahoo |
Subdomains, Emails | Yahoo search engine |
ask |
Subdomains, Emails | Ask.com search engine |
duckduckgo |
Subdomains, Emails | DuckDuckGo search engine |
yandex |
Subdomains, Emails | Yandex search engine (Russian) |
baidu |
Subdomains, Emails | Baidu search engine (Chinese) |
google-site |
Subdomains | Google site: operator |
google-profiles |
Employee Names | Google+ profiles (deprecated) |
DNS and Certificate Sources
| Source | Data Types | Description |
|---|---|---|
virustotal |
Subdomains, IPs, Emails | VirusTotal passive DNS |
crt |
Subdomains | Certificate Transparency logs |
crtsh |
Subdomains | crt.sh CT log database |
securitytrails |
Subdomains, IPs | SecurityTrails DNS history |
shodan |
Subdomains, IPs, Ports | Shodan internet scanner |
fullhunt |
Subdomains, IPs | FullHunt attack surface |
intelx |
Subdomains, Emails | Intelix.io intelligence |
Social Media and People Sources
| Source | Data Types | Description |
|---|---|---|
linkedin |
Employee Names, Emails | LinkedIn professional network |
github |
Subdomains, Emails, Code | GitHub repositories |
twitter |
Employee Names | Twitter/X social media |
facebook |
Employee Names | Facebook social media |
phonebook |
Employee Names, Emails | Phonebook lookup services |
Specialized Sources
| Source | Data Types | Description |
|---|---|---|
dnsdumpster |
Subdomains, IPs | DNSdumpster.com DNS research |
threatminer |
Subdomains, IPs | ThreatMiner threat intelligence |
passivetotal |
Subdomains, IPs | PassiveTotal DNS data |
bufferover |
Subdomains | BufferOver DNS API |
bevigil |
Subdomains, IPs | BeVigil attack surface |
c99 |
Subdomains | C99.nl DNS API |
whois |
IPs, Emails | WHOIS database records |
hunter |
Emails | Hunter.io email finder |
netlas |
Subdomains, IPs | Netlas.io network data |
redhunt |
Subdomains, IPs | RedHunt Labs API |
subdomaincenter |
Subdomains | Subdomain.center API |
whois |
IPs, Emails | WHOIS database records |
zoomeye |
Subdomains, IPs | ZoomEye search engine |
Output Formats
theHarvester supports multiple output formats:
| Format | Extension | Description |
|---|---|---|
| HTML | .html |
Human-readable report with tables |
| JSON | .json |
Machine-readable structured data |
| CSV | .csv |
Spreadsheet-compatible format |
| XLS | .xls |
Excel spreadsheet format |
| XLSX | .xlsx |
Modern Excel format |
| SQLite | .db |
SQLite database for querying |
Practical Command Examples
Basic Usage
# Basic subdomain and email harvest
theHarvester -d example.com -b bing,google,duckduckgo -l 500
# Harvest with DNS resolution
theHarvester -d example.com -b bing,google,virustotal -l 500 -n
# Harvest with all sources
theHarvester -d example.com --full -l 1000
# Harvest with specific output format
theHarvester -d example.com -b bing,google,virustotal -l 500 -f results_example
Advanced Usage
# Comprehensive harvest with all options
theHarvester -d example.com \
-b bing,google,duckduckgo,virustotal,crtsh,dnsdumpster,github,linkedin \
-l 1000 \
-f comprehensive_results \
-n \
-v \
-t
# Harvest with Shodan integration
theHarvester -d example.com \
-b shodan,virustotal,crtsh \
-l 500 \
--shodan YOUR_SHODAN_API_KEY \
-f shodan_results
# Harvest emails specifically
theHarvester -d example.com \
-b hunter,linkedin,google,bing \
-l 500 \
-f email_results
# Harvest with takeover detection
theHarvester -d example.com \
-b bing,google,virustotal,crtsh \
-l 500 \
-t \
-f takeover_results
DNS Resolution and Reverse Lookup
# Forward DNS resolution
theHarvester -d example.com -b bing,google -l 500 -n
# Reverse DNS lookup on found IPs
theHarvester -d example.com -b bing,google -l 500 -c
# Both forward and reverse
theHarvester -d example.com -b bing,google -l 500 -n -c
DNS Brute Force
# DNS brute force using built-in subdomain list
theHarvester -d example.com -b bing,google -l 500 -j
# Combine passive sources with brute force
theHarvester -d example.com -b bing,google,virustotal -l 500 -j -n
Takeover Detection
theHarvester can detect potential subdomain takeover vulnerabilities:
# Enable takeover detection
theHarvester -d example.com -b bing,google,virustotal,crtsh -l 500 -t
# What takeover detection checks:
# 1. CNAME records pointing to non-existent services
# 2. Subdomains with no A/AAAA records but with CNAME
# 3. CNAMEs to services that no longer host the subdomain
# 4. Common takeover-prone services (GitHub Pages, Heroku, S3, etc.)
Common Takeover-Prone Services:
| Service | Indicator | Risk |
|---|---|---|
| GitHub Pages | CNAME to *.github.io |
High |
| Heroku | CNAME to *.herokuapp.com |
High |
| Amazon S3 | CNAME to *.s3.amazonaws.com |
High |
| Azure Blob | CNAME to *.blob.core.windows.net |
Medium |
| Firebase | CNAME to *.firebaseapp.com |
Medium |
| Netlify | CNAME to *.netlify.app |
Medium |
| GitLab Pages | CNAME to *.gitlab.io |
High |
| WordPress | CNAME to *.wordpress.com |
Medium |
Result Interpretation
HTML Output Structure
┌─────────────────────────────────────────────────────────────┐
│ theHarvester Report: example.com │
├─────────────────────────────────────────────────────────────┤
│ │
│ Emails: │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ admin@example.com │ │
│ │ info@example.com │ │
│ │ john.doe@example.com │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ Hosts: │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ www.example.com -> 93.184.216.34 │ │
│ │ mail.example.com -> 93.184.216.35 │ │
│ │ api.example.com -> 93.184.216.36 │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ IPs: │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ 93.184.216.34 │ │
│ │ 93.184.216.35 │ │
│ │ 93.184.216.36 │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ Virtual Hosts: │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ dev.example.com │ │
│ │ staging.example.com │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
JSON Output Structure
{
"emails": [
"admin@example.com",
"info@example.com",
"john.doe@example.com"
],
"hosts": [
"www.example.com",
"mail.example.com",
"api.example.com",
"dev.example.com",
"staging.example.com"
],
"ips": [
"93.184.216.34",
"93.184.216.35",
"93.184.216.36"
],
"virtual_hosts": [
"dev.example.com",
"staging.example.com"
],
"names": [
"John Doe",
"Jane Smith"
],
"sources": [
"bing",
"google",
"virustotal"
]
}
Source Selection Strategy
| Goal | Recommended Sources | Reason |
|---|---|---|
| Maximum subdomain coverage | bing,google,virustotal,crtsh,dnsdumpster,shodan |
Broadest coverage across different data types |
| Email harvesting | hunter,linkedin,google,bing |
Best sources for email addresses |
| Fast results | bing,google,crtsh |
Quick response times |
| Deep historical data | virustotal,securitytrails,shodan |
Historical DNS and certificate data |
| Code and infrastructure | github,shodan |
Source code and exposed services |
| Employee intelligence | linkedin,google,twitter |
People and organizational data |
Maltego Deep Dive
Overview and Purpose
Maltego is a visual link analysis and open-source intelligence (OSINT) tool developed by Paterva (formerly Paterva Ltd.). It transforms data from various sources into an interactive graph of entities and relationships.
Primary Purpose: Visual mapping of relationships between entities (domains, IPs, people, organizations, etc.) to discover connections and patterns that might not be apparent from raw data.
Key Characteristics:
- Visual interface: Graph-based representation of data
- Entity-based: Works with predefined entity types
- Transformations: Pre-built queries that expand the graph
- Multi-source: Integrates with dozens of data sources
- Commercial and Community editions: Free Community edition available
Installation
# Method 1: Download from Paterva website
# Visit: https://www.maltego.com/products/maltego/
# Download Maltego Community Edition (free)
# Method 2: Linux package
wget https://downloads.maltego.com/maltego/Maltego_4.4.1-1_amd64.deb
sudo dpkg -i Maltego_4.4.1-1_amd64.deb
# Method 3: Windows installer
# Download .exe from Paterva website
# Launch Maltego
maltego
Entity Types
Maltego works with predefined entity types — structured representations of real-world objects:
Core Entity Types
| Entity Type | Icon | Description | Common Transformations |
|---|---|---|---|
| Domain Name | 🌐 | A registered domain | DNS Resolution, WHOIS, Subdomains |
| IP Address | 📍 | IPv4 or IPv6 address | Reverse DNS, Geolocation, Hostnames |
| Hostname | 💻 | A specific host name | DNS Resolution, Port Scan, Services |
| URL | 🔗 | A web address | Content Analysis, Links, Technologies |
| Person | 👤 | An individual | Email, Social Media, Affiliations |
| Email Address | 📧 | An email address | Domain Extraction, Person Lookup |
| Phone Number | 📞 | A telephone number | Geolocation, Owner Lookup |
| Organization | 🏢 | A company or entity | Domains, People, Technologies |
| ASN | 🌍 | Autonomous System Number | IP Ranges, Organization, Peers |
| Certificate | 🔒 | SSL/TLS certificate | Domain Extraction, Issuer Info |
| MX Record | 📬 | Mail exchange record | Mail Server, IP Resolution |
| NS Record | 📋 | Name server record | DNS Server, IP Resolution |
| TXT Record | 📝 | DNS TXT record | SPF, DKIM, DMARC Analysis |
| SSH Key | 🔑 | SSH public key | Fingerprint, Host Association |
| SSL Certificate | 🛡️ | SSL certificate details | Domain, Issuer, Validity |
| Whois Record | 📄 | WHOIS registration data | Registrant, Dates, Nameservers |
Entity Hierarchy
graph TD
A[Domain Name] --> B[IP Address]
A --> C[Hostname]
A --> D[MX Record]
A --> E[NS Record]
A --> F[TXT Record]
A --> G[SSL Certificate]
A --> H[Whois Record]
B --> I[ASN]
B --> J[Geolocation]
B --> K[Reverse DNS]
C --> L[URL]
C --> M[Services]
G --> N[Issuer Organization]
H --> O[Registrant Person]
H --> P[Registrant Organization]
style A fill:#f9d56e,stroke:#333,stroke-width:2px
Key Transformations
Transformations are the core mechanism in Maltego — they query data sources and return new entities linked to the selected entity.
Domain Name Transformations
| Transformation | Source | Returns | Description |
|---|---|---|---|
| DNS: Resolve to IP | DNS | IP Address | Resolve domain to IP addresses |
| DNS: Reverse Lookup | DNS | Hostname | Reverse DNS lookup on IPs |
| DNS: Get MX Records | DNS | MX Record | Get mail exchange records |
| DNS: Get NS Records | DNS | NS Record | Get name server records |
| DNS: Get TXT Records | DNS | TXT Record | Get TXT records (SPF, DKIM) |
| VirusTotal: Get Subdomains | VirusTotal | Domain Name | Get subdomains from VT |
| PassiveTotal: Get Subdomains | PassiveTotal | Domain Name | Get subdomains from PT |
| Certificate: Get Certificates | CT Logs | SSL Certificate | Get SSL certificates |
| Whois: Get Registration | WHOIS | Whois Record | Get domain registration info |
| Shodan: Get Hosts | Shodan | Hostname | Get hosts from Shodan |
| Google: Get Subdomains | Domain Name | Search Google for subdomains | |
| Bing: Get Subdomains | Bing | Domain Name | Search Bing for subdomains |
IP Address Transformations
| Transformation | Source | Returns | Description |
|---|---|---|---|
| DNS: Reverse Lookup | DNS | Hostname | Reverse DNS on IP |
| GeoIP: Get Location | MaxMind | Geolocation | Get geographic location |
| Shodan: Get Info | Shodan | Host details | Get Shodan host details |
| Censys: Get Host | Censys | Host details | Get Censys host details |
| ASN: Get ASN Info | RIPEDB | ASN | Get ASN information |
| VirusTotal: Get Report | VirusTotal | VT Report | Get VirusTotal report |
| AbuseIPDB: Get Report | AbuseIPDB | Report | Get abuse reports |
Email Address Transformations
| Transformation | Source | Returns | Description |
|---|---|---|---|
| Extract Domain | Built-in | Domain Name | Extract domain from email |
| Hunter: Get Person | Hunter.io | Person | Get person details |
| Snovio: Get Person | Snovio | Person | Get person details |
| Phonebook: Lookup | Phonebook | Person | Phonebook lookup |
Maltego Workflow
graph TD
A[Start: Add Domain Entity] --> B[Right-click: Run Transformations]
B --> C[DNS: Resolve to IP]
B --> D[DNS: Get MX Records]
B --> E[VirusTotal: Get Subdomains]
B --> F[Certificate: Get Certificates]
C --> G[Analyze IP Entities]
D --> H[Analyze MX Entities]
E --> I[Analyze Subdomain Entities]
F --> J[Analyze Certificate Entities]
G --> K[Run IP Transformations]
H --> L[Run MX Transformations]
I --> M[Run Subdomain Transformations]
J --> N[Run Certificate Transformations]
K --> O[Build Complete Graph]
L --> O
M --> O
N --> O
O --> P[Identify Patterns]
P --> Q[Export Results]
Q --> R[Use in Further Recon]
style A fill:#f9d56e,stroke:#333,stroke-width:2px
style O fill:#a8e6cf,stroke:#333,stroke-width:2px
Step-by-Step Workflow
Step 1: Create a New Graph
- Open Maltego
- Click File > New Graph
- Give the graph a name (e.g., “example.com Recon”)
Step 2: Add Initial Entities
- Right-click in the canvas
- Select Add Entity
- Choose Domain Name
- Enter the target domain (e.g.,
example.com)
Step 3: Run Transformations
- Right-click the domain entity
- Select Run Transformations
- Choose transformations to run:
- DNS: Resolve to IP
- DNS: Get MX Records
- DNS: Get NS Records
- VirusTotal: Get Subdomains
- Certificate: Get Certificates
Step 4: Analyze Results
- Review the generated graph
- Look for patterns and connections
- Identify high-value targets
- Expand interesting branches with additional transformations
Step 5: Export Results
- Click File > Export
- Choose export format:
- PNG/SVG: Visual graph image
- CSV: Tabular data
- Maltego Package: Full graph for sharing
- Text: Plain text entity list
Graph Analysis Techniques
Visual Pattern Recognition
| Pattern | What It Looks Like | What It Means |
|---|---|---|
| Star Pattern | One central node with many connections | Central infrastructure (main domain) |
| Chain Pattern | Linear sequence of connected nodes | DNS delegation chain |
| Cluster Pattern | Dense group of interconnected nodes | Related services or environments |
| Island Pattern | Isolated group with few external connections | Separate infrastructure segment |
| Hub Pattern | Node with many incoming connections | Shared resource (CDN, cloud provider) |
Graph Navigation Tips
| Action | Shortcut | Description |
|---|---|---|
| Zoom In/Out | Mouse wheel | Zoom the graph view |
| Pan | Middle mouse + drag | Move around the graph |
| Select Multiple | Ctrl + click | Select multiple entities |
| Expand All | Right-click > Expand | Run all available transforms |
| Collapse | Right-click > Collapse | Hide child entities |
| Filter | Filter panel | Show/hide entity types |
| Search | Ctrl + F | Search entities by name |
| Auto-Layout | Right-click > Auto-Layout | Reorganize graph layout |
Export Options
| Format | Use Case | Command |
|---|---|---|
| PNG | Documentation, reports | File > Export > PNG |
| SVG | Scalable graphics, presentations | File > Export > SVG |
| CSV | Data analysis, spreadsheets | File > Export > CSV |
| Maltego Package | Sharing graphs with team | File > Export > Package |
| Text | Quick entity lists | File > Export > Text |
| GEXF | Import to other graph tools | File > Export > GEXF |
Maltego Machine Accounts
Maltego uses Machine Accounts to authenticate with data sources:
| Machine Account | Purpose | Free Tier? |
|---|---|---|
| Maltego Service | Built-in transforms | Yes |
| VirusTotal | VT data access | Yes (limited) |
| PassiveTotal | PT DNS data | Yes (limited) |
| Shodan | Shodan data | Yes (limited) |
| Google search | Yes | |
| Bing | Bing search | Yes |
| DNS | DNS resolution | Yes |
| WHOIS | WHOIS data | Yes |
| Certificate | CT log data | Yes |
Setting Up Machine Accounts:
- Click Configuration > Machine Accounts
- Add new machine account
- Enter credentials (API keys where required)
- Test connection
- Save configuration
theHarvester vs Maltego Comparison
Feature Comparison Table
| Feature | theHarvester | Maltego |
|---|---|---|
| Interface | Command-line | Graphical (GUI) |
| Cost | Free | Free (Community) / Paid (Enterprise) |
| Learning Curve | Low | Medium-High |
| Speed | Fast (seconds-minutes) | Medium (minutes) |
| Visualization | None (text/HTML/JSON) | Interactive graph |
| Data Sources | 30+ sources | 50+ sources (with transforms) |
| Customization | Source selection | Custom transforms, machine accounts |
| Automation | Easy (CLI, scripts) | Medium (Paterva transforms) |
| Export Formats | HTML, JSON, CSV, XLS, SQLite | PNG, SVG, CSV, Package, Text |
| Relationship Mapping | No | Yes (core feature) |
| Team Collaboration | No | Yes (Enterprise) |
| API Access | No | Yes (Enterprise) |
| Resource Usage | Low | Medium-High |
| Installation | pip install | Download installer |
| Best For | Quick subdomain/email harvest | Deep relationship analysis |
When to Use Each Tool
| Scenario | Recommended Tool | Reason |
|---|---|---|
| Quick subdomain list | theHarvester | Fast, simple, CLI-friendly |
| Email harvesting | theHarvester | Dedicated email sources |
| Visual relationship mapping | Maltego | Graph-based visualization |
| Deep infrastructure analysis | Maltego | Multi-hop relationship discovery |
| Automated pipelines | theHarvester | Easy to script and automate |
| Team collaboration | Maltego | Shared graphs and reports |
| Time-constrained (OSCP) | theHarvester | Faster setup and execution |
| Comprehensive OSINT | Both | Complementary strengths |
Strengths and Weaknesses
theHarvester
Strengths:
- Fast and lightweight
- Easy to automate
- No GUI overhead
- Great for quick results
- Integrates well with scripts
Weaknesses:
- No visual representation
- Limited relationship mapping
- Source availability varies
- No built-in correlation
Maltego
Strengths:
- Visual relationship mapping
- Deep multi-hop analysis
- Rich entity types
- Professional reporting
- Team collaboration features
Weaknesses:
- Steeper learning curve
- Slower execution
- Resource-intensive
- Free edition has limitations
- Requires API keys for best results
Combining theHarvester and Maltego
Integrated Workflow
graph LR
A[theHarvester] --> B[Subdomain List]
A --> C[Email List]
A --> D[IP List]
B --> E[Import to Maltego]
C --> E
D --> E
E --> F[Run Maltego Transforms]
F --> G[Visual Graph]
G --> H[Identify Patterns]
H --> I[Export Results]
style A fill:#a8e6cf,stroke:#333,stroke-width:1px
style E fill:#f9d56e,stroke:#333,stroke-width:2px
Step-by-Step Integration
Step 1: Run theHarvester
# Comprehensive theHarvester run
theHarvester -d example.com \
-b bing,google,virustotal,crtsh,dnsdumpster,github \
-l 1000 \
-f harvester_results \
-n \
-v
Step 2: Import Results to Maltego
Method 1: Manual Import
- Open Maltego
- Right-click in canvas
- Select Add Entity
- Add each subdomain as a Domain Name entity
Method 2: CSV Import
# Export theHarvester results as CSV
theHarvester -d example.com -b bing,google,virustotal -l 500 -f results
# The CSV file can be imported into Maltego
# File > Import > CSV
Method 3: Scripted Import
#!/usr/bin/env python3
"""
harvester_to_maltego.py - Import theHarvester results into Maltego
"""
import csv
import json
def harvester_csv_to_maltego(csv_file, output_file):
"""Convert theHarvester CSV to Maltego import format."""
entities = []
with open(csv_file, 'r') as f:
reader = csv.DictReader(f)
for row in reader:
# Add domain entities
if row.get('hosts'):
for host in row['hosts'].split(','):
host = host.strip()
if host:
entities.append({
'Type': 'DomainName',
'RawData': host,
'DisplayName': host
})
# Add IP entities
if row.get('ips'):
for ip in row['ips'].split(','):
ip = ip.strip()
if ip:
entities.append({
'Type': 'IPv4Address',
'RawData': ip,
'DisplayName': ip
})
with open(output_file, 'w') as f:
json.dump(entities, f, indent=2)
print(f"[*] Exported {len(entities)} entities to {output_file}")
if __name__ == "__main__":
harvester_csv_to_maltego("results_example.csv", "maltego_import.json")
Step 3: Run Maltego Transformations
- Import theHarvester results
- Select all domain entities
- Right-click > Run Transformations
- Choose:
- DNS: Resolve to IP
- DNS: Get MX Records
- Certificate: Get Certificates
- VirusTotal: Get Subdomains
Step 4: Analyze and Export
- Review the visual graph
- Identify patterns and relationships
- Export results in desired format
Combined Command Workflow
#!/bin/bash
# combined_recon.sh - theHarvester + Maltego workflow
DOMAIN="${1:-example.com}"
echo "[*] Phase 1: theHarvester passive gathering"
theHarvester -d "$DOMAIN" \
-b bing,google,virustotal,crtsh,dnsdumpster \
-l 1000 \
-f "${DOMAIN}_harvester" \
-n \
-v
echo "[*] Phase 2: Extract subdomains for Maltego"
grep -oP '"hosts"\s*:\s*\[\K[^\]]+' "${DOMAIN}_harvester.json" | \
tr ',' '\n' | \
sed 's/[" ]//g' | \
sort -u > "${DOMAIN}_subdomains.txt"
echo "[*] Phase 3: Prepare Maltego import"
python3 harvester_to_maltego.py "${DOMAIN}_harvester.csv" "${DOMAIN}_maltego.json"
echo "[*] Phase 4: Open Maltego and import ${DOMAIN}_maltego.json"
echo "[*] Then run transformations on imported entities"
Other Passive Gathering Tools
Recon-ng
Recon-ng is a full-featured web reconnaissance framework written by Tim Tomes.
Features:
- Modular framework architecture
- Built-in modules for various sources
- Database-backed results storage
- Custom module development
- Professional reporting
Installation:
git clone https://github.com/lanmaster53/recon-ng.git
cd recon-ng
pip3 install -r requirements.txt
./recon-ng
Basic Usage:
# Start Recon-ng
./recon-ng
# Install modules
marketplace install
# Search for modules
search domain
# Use a module
modules load recon/domains-hosts/google_site_search
options set SOURCE example.com
run
# View results
show hosts
show emails
# Export results
creds export --csv results.csv
Key Modules:
| Module | Purpose | Data Source |
|---|---|---|
recon/domains-hosts/google_site_search |
Subdomain discovery | |
recon/domains-hosts/bing_domain_search |
Subdomain discovery | Bing |
recon/domains-hosts/virustotal_passive_dns |
Passive DNS | VirusTotal |
recon/domains-hosts/crt_sh |
CT log subdomains | crt.sh |
recon/contacts-emails/hunter_io |
Email discovery | Hunter.io |
recon/contacts-names/google_gmail_search |
Name discovery | |
recon/hosts-ports/shodan |
Port discovery | Shodan |
recon/hosts-tech/wappalyzer |
Technology detection | Wappalyzer |
SpiderFoot
SpiderFoot is an automated OSINT and threat intelligence platform.
Features:
- Fully automated scanning
- 200+ modules
- Web interface
- API access
- Real-time results
Installation:
# Docker installation (recommended)
docker pull spiderfoot/spiderfoot
docker run -d -p 5001:5001 spiderfoot/spiderfoot
# From source
git clone https://github.com/smicallef/spiderfoot.git
cd spiderfoot
pip3 install -r requirements.txt
python3 sf.py --web
Basic Usage:
# CLI mode
python3 sf.py --target example.com --modules all --scan myscan
# Web interface
# Navigate to http://localhost:5001
# Create new scan
# Select target and modules
# Start scan
# View results
Amass (Passive Mode)
Amass by OWASP is a comprehensive subdomain enumeration tool.
Passive Mode Usage:
# Passive-only enumeration
amass enum -passive -d example.com -o amass_passive.txt
# Passive with specific sources
amass enum -passive -d example.com -src bing,google,virustotal,crtsh
# Passive with API keys
amass enum -passive -d example.com \
-df /path/to/api.conf \
-o amass_passive.txt
Tool Comparison
| Tool | Interface | Sources | Automation | Best For |
|---|---|---|---|---|
| theHarvester | CLI | 30+ | High | Quick harvest |
| Maltego | GUI | 50+ | Medium | Visual analysis |
| Recon-ng | CLI/Menu | 40+ | High | Modular recon |
| SpiderFoot | Web/GUI | 200+ | Very High | Automated scanning |
| Amass (passive) | CLI | 25+ | High | Subdomain focus |
Automating Passive Gathering
Automated Pipeline Script
#!/usr/bin/env python3
"""
passive_recon_pipeline.py - Automated passive reconnaissance pipeline
"""
import subprocess
import json
import os
import argparse
from datetime import datetime
from pathlib import Path
class PassiveReconPipeline:
def __init__(self, domain, output_dir="passive_results"):
self.domain = domain
self.output_dir = Path(output_dir)
self.output_dir.mkdir(parents=True, exist_ok=True)
self.results = {
'subdomains': set(),
'emails': set(),
'ips': set(),
'names': set(),
}
def run_theharvester(self, sources=None, limit=500):
"""Run theHarvester with specified sources."""
if not sources:
sources = ['bing', 'google', 'virustotal', 'crtsh', 'dnsdumpster']
sources_str = ','.join(sources)
output_file = self.output_dir / f"{self.domain}_theharvester"
cmd = [
'theHarvester',
'-d', self.domain,
'-b', sources_str,
'-l', str(limit),
'-f', str(output_file),
'-n',
'-v'
]
print(f"[*] Running theHarvester with sources: {sources_str}")
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
print(result.stdout)
# Parse JSON output if available
json_file = output_file.with_suffix('.json')
if json_file.exists():
with open(json_file, 'r') as f:
data = json.load(f)
self.results['subdomains'].update(data.get('hosts', []))
self.results['emails'].update(data.get('emails', []))
self.results['ips'].update(data.get('ips', []))
self.results['names'].update(data.get('names', []))
print(f"[+] theHarvester complete. Found {len(self.results['subdomains'])} subdomains")
except subprocess.TimeoutExpired:
print("[!] theHarvester timed out")
except Exception as e:
print(f"[!] theHarvester error: {e}")
def run_amass_passive(self):
"""Run Amass in passive mode."""
output_file = self.output_dir / f"{self.domain}_amass.txt"
cmd = [
'amass', 'enum',
'-passive',
'-d', self.domain,
'-o', str(output_file)
]
print(f"[*] Running Amass passive enumeration")
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
if output_file.exists():
with open(output_file, 'r') as f:
for line in f:
sub = line.strip()
if sub and sub != self.domain:
self.results['subdomains'].add(sub)
print(f"[+] Amass complete. Found {len(self.results['subdomains'])} total subdomains")
except subprocess.TimeoutExpired:
print("[!] Amass timed out")
except Exception as e:
print(f"[!] Amass error: {e}")
def run_subfinder(self):
"""Run Subfinder for passive subdomain discovery."""
output_file = self.output_dir / f"{self.domain}_subfinder.txt"
cmd = [
'subfinder',
'-d', self.domain,
'-o', str(output_file),
'-silent'
]
print(f"[*] Running Subfinder")
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
if output_file.exists():
with open(output_file, 'r') as f:
for line in f:
sub = line.strip()
if sub:
self.results['subdomains'].add(sub)
print(f"[+] Subfinder complete")
except subprocess.TimeoutExpired:
print("[!] Subfinder timed out")
except Exception as e:
print(f"[!] Subfinder error: {e}")
def generate_report(self):
"""Generate comprehensive report."""
report_file = self.output_dir / f"{self.domain}_report.md"
with open(report_file, 'w') as f:
f.write(f"# Passive Reconnaissance Report: {self.domain}\n\n")
f.write(f"**Date:** {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n\n")
f.write("## Summary\n\n")
f.write(f"- **Subdomains:** {len(self.results['subdomains'])}\n")
f.write(f"- **Emails:** {len(self.results['emails'])}\n")
f.write(f"- **IPs:** {len(self.results['ips'])}\n")
f.write(f"- **Names:** {len(self.results['names'])}\n\n")
f.write("## Subdomains\n\n")
f.write("```text\n")
for sub in sorted(self.results['subdomains']):
f.write(f"{sub}\n")
f.write("```\n\n")
f.write("## Emails\n\n")
f.write("```text\n")
for email in sorted(self.results['emails']):
f.write(f"{email}\n")
f.write("```\n\n")
f.write("## IPs\n\n")
f.write("```text\n")
for ip in sorted(self.results['ips']):
f.write(f"{ip}\n")
f.write("```\n\n")
f.write("## Names\n\n")
f.write("```text\n")
for name in sorted(self.results['names']):
f.write(f"{name}\n")
f.write("```\n")
print(f"[+] Report saved to {report_file}")
return report_file
def save_results(self):
"""Save results to individual files."""
# Save subdomains
with open(self.output_dir / "subdomains.txt", 'w') as f:
for sub in sorted(self.results['subdomains']):
f.write(f"{sub}\n")
# Save emails
with open(self.output_dir / "emails.txt", 'w') as f:
for email in sorted(self.results['emails']):
f.write(f"{email}\n")
# Save IPs
with open(self.output_dir / "ips.txt", 'w') as f:
for ip in sorted(self.results['ips']):
f.write(f"{ip}\n")
# Save combined results as JSON
json_results = {
'domain': self.domain,
'date': datetime.now().isoformat(),
'subdomains': sorted(self.results['subdomains']),
'emails': sorted(self.results['emails']),
'ips': sorted(self.results['ips']),
'names': sorted(self.results['names']),
}
with open(self.output_dir / "results.json", 'w') as f:
json.dump(json_results, f, indent=2)
print(f"[+] Results saved to {self.output_dir}/")
def run(self):
"""Run the complete pipeline."""
print(f"{'='*60}")
print(f"Passive Reconnaissance Pipeline: {self.domain}")
print(f"{'='*60}\n")
self.run_theharvester()
self.run_amass_passive()
self.run_subfinder()
self.save_results()
self.generate_report()
print(f"\n{'='*60}")
print(f"Pipeline Complete!")
print(f"{'='*60}")
print(f"Subdomains: {len(self.results['subdomains'])}")
print(f"Emails: {len(self.results['emails'])}")
print(f"IPs: {len(self.results['ips'])}")
print(f"Names: {len(self.results['names'])}")
def main():
parser = argparse.ArgumentParser(description="Passive Recon Pipeline")
parser.add_argument("domain", help="Target domain")
parser.add_argument("-o", "--output", default="passive_results",
help="Output directory")
args = parser.parse_args()
pipeline = PassiveReconPipeline(args.domain, args.output)
pipeline.run()
if __name__ == "__main__":
main()
Cron-Based Automation
# Daily passive recon for multiple domains
0 6 * * * /usr/bin/python3 /opt/tools/passive_recon_pipeline.py example.com -o /opt/results/example.com
# Weekly comprehensive recon
0 6 * * 0 /usr/bin/python3 /opt/tools/passive_recon_pipeline.py target.org -o /opt/results/target.org
Data Correlation Techniques
Cross-Source Correlation
graph TD
A[theHarvester Results] --> D[Correlation Engine]
B[Maltego Graph] --> D
C[Amass Results] --> D
E[Subfinder Results] --> D
F[CT Log Results] --> D
D --> G[Unified Dataset]
G --> H[Priority Scoring]
H --> I[High Priority]
H --> J[Medium Priority]
H --> K[Low Priority]
style D fill:#f9d56e,stroke:#333,stroke-width:2px
Correlation Script
def correlate_passive_results(domain):
"""Correlate results from multiple passive sources."""
sources = {
'theharvester': set(),
'amass': set(),
'subfinder': set(),
'crtsh': set(),
}
# Load results from each source
source_files = {
'theharvester': f"{domain}_theharvester.json",
'amass': f"{domain}_amass.txt",
'subfinder': f"{domain}_subfinder.txt",
'crtsh': f"{domain}_crtsh.txt",
}
for source, filepath in source_files.items():
try:
if filepath.endswith('.json'):
with open(filepath, 'r') as f:
data = json.load(f)
sources[source].update(data.get('hosts', []))
else:
with open(filepath, 'r') as f:
for line in f:
sub = line.strip()
if sub:
sources[source].add(sub)
except FileNotFoundError:
print(f"[!] {filepath} not found")
# Find subdomains found by multiple sources
all_subs = set()
for subs in sources.values():
all_subs.update(subs)
correlation = {}
for sub in all_subs:
found_in = [src for src, subs in sources.items() if sub in subs]
correlation[sub] = {
'sources': found_in,
'source_count': len(found_in),
'priority': 'high' if len(found_in) >= 3 else
'medium' if len(found_in) == 2 else 'low'
}
# Sort by priority
high_priority = [sub for sub, data in correlation.items() if data['priority'] == 'high']
medium_priority = [sub for sub, data in correlation.items() if data['priority'] == 'medium']
low_priority = [sub for sub, data in correlation.items() if data['priority'] == 'low']
return {
'high_priority': sorted(high_priority),
'medium_priority': sorted(medium_priority),
'low_priority': sorted(low_priority),
'total_unique': len(all_subs),
'source_counts': {src: len(subs) for src, subs in sources.items()},
}
Priority Scoring Matrix
| Source Count | Priority | Action |
|---|---|---|
| 4+ sources | Critical | Immediate active scanning |
| 3 sources | High | Priority target for enumeration |
| 2 sources | Medium | Include in standard scanning |
| 1 source | Low | Include in comprehensive scan |
Building Passive Recon Pipelines
Pipeline Architecture
graph LR
A[Input: Domain] --> B[Passive Phase]
B --> C[theHarvester]
B --> D[Amass Passive]
B --> E[Subfinder]
B --> F[CT Logs]
C --> G[Deduplication]
D --> G
E --> G
F --> G
G --> H[Correlation]
H --> I[Priority Scoring]
I --> J[Output: Prioritized List]
J --> K[Active Phase]
K --> L[DNS Resolution]
K --> M[Port Scanning]
K --> N[Service Enumeration]
style B fill:#a8e6cf,stroke:#333,stroke-width:2px
style K fill:#f9d56e,stroke:#333,stroke-width:2px
Pipeline Best Practices
- Start passive, go active later: Always exhaust passive sources before active scanning
- Deduplicate early: Remove duplicates after each source to save processing time
- Correlate results: Cross-reference sources to identify high-confidence targets
- Prioritize targets: Focus active scanning on high-priority subdomains first
- Document everything: Save all results with timestamps and source attribution
- Automate where possible: Use scripts to run multiple tools in sequence
- Set time limits: Don’t let passive recon run indefinitely
Managing Large Result Sets
Deduplication
# Simple deduplication
sort -u all_subdomains.txt > unique_subdomains.txt
# Deduplication with source tracking
python3 -c "
import csv
sources = {}
with open('all_results.csv', 'r') as f:
reader = csv.DictReader(f)
for row in reader:
sub = row['subdomain']
source = row['source']
if sub not in sources:
sources[sub] = set()
sources[sub].add(source)
with open('deduplicated_results.csv', 'w') as f:
writer = csv.writer(f)
writer.writerow(['subdomain', 'sources', 'source_count'])
for sub, subs in sorted(sources.items()):
writer.writerow([sub, ','.join(subs), len(subs)])
"
Result Filtering
# Filter for specific TLDs
grep '\.com$' all_subdomains.txt > com_subdomains.txt
# Filter out known CDNs and cloud providers
grep -v -E '(cloudfront\.net|azurewebsites\.net|amazonaws\.com)' all_subdomains.txt > filtered_subdomains.txt
# Filter for subdomains only (exclude base domain)
grep -v '^example\.com$' all_subdomains.txt > subdomains_only.txt
# Filter for specific patterns
grep -E '(dev|staging|test|admin|api|portal)' all_subdomains.txt > interesting_subdomains.txt
Storage and Organization
passive_results/
├── example.com/
│ ├── raw/
│ │ ├── theharvester.json
│ │ ├── amass.txt
│ │ ├── subfinder.txt
│ │ └── crtsh.txt
│ ├── processed/
│ │ ├── subdomains.txt
│ │ ├── emails.txt
│ │ ├── ips.txt
│ │ └── deduplicated.txt
│ ├── correlated/
│ │ ├── high_priority.txt
│ │ ├── medium_priority.txt
│ │ └── low_priority.txt
│ └── reports/
│ ├── summary.md
│ └── results.json
└── target.org/
└── ...
OSCP Value Assessment
Tool Value for OSCP
| Tool | OSCP Value | Time Investment | Recommendation |
|---|---|---|---|
| theHarvester | Medium-High | 5-10 minutes | Use for quick subdomain harvest |
| Maltego | Medium | 15-30 minutes | Use if time allows for visual analysis |
| Recon-ng | Medium | 10-15 minutes | Good alternative to theHarvester |
| SpiderFoot | Low-Medium | 10-20 minutes | Automated but may be overkill |
| Amass (passive) | High | 5-10 minutes | Excellent subdomain coverage |
OSCP-Specific Considerations
- Time is critical: OSCP exams have strict time limits. Don’t spend too much time on passive recon.
- Lab constraints: Some lab environments may not have internet access for external APIs.
- Focus on results: The goal is to find attackable targets, not to build perfect graphs.
- Documentation matters: Save all results for the final report.
- Have backups: If one tool fails, have alternatives ready.
Recommended OSCP Passive Workflow
# OSCP Passive Recon Quick Workflow (10-15 minutes max)
# Step 1: Quick theHarvester run (3-5 minutes)
theHarvester -d target.lab -b bing,google,virustotal,crtsh -l 500 -f target_passive -n
# Step 2: Quick Amass passive (3-5 minutes)
amass enum -passive -d target.lab -o target_amass.txt
# Step 3: Combine and deduplicate (1-2 minutes)
cat target_passive.txt target_amass.txt | sort -u > target_all_subs.txt
# Step 4: Quick DNS resolution (2-3 minutes)
while read sub; do
host "$sub" 2>/dev/null | grep "has address" | awk '{print $4}'
done < target_all_subs.txt > target_resolved_ips.txt
# Step 5: Move to active scanning
nmap -sV -sC -iL target_resolved_ips.txt -oA target_nmap
OSCP Exam Tips
Key Takeaways
┌─────────────────────────────────────────────────────────────────┐
│ OSCP Passive Gathering Tips │
├─────────────────────────────────────────────────────────────────┤
│ │
│ theHarvester: │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ • Fast and reliable for subdomain discovery │ │
│ │ • Use -b flag to select specific sources │ │
│ │ • Add -n for DNS resolution │ │
│ │ • Save results in multiple formats │ │
│ │ • Quick command: │ │
│ │ theHarvester -d TARGET -b bing,google,virustotal │ │
│ │ -l 500 -f results -n │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ Maltego: │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ • Best for visual relationship mapping │ │
│ │ • Steeper learning curve - practice before exam │ │
│ │ • Community edition is sufficient for OSCP │ │
│ │ • Set up machine accounts before exam day │ │
│ │ • Export graphs for documentation │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ General Tips: │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ • Limit passive recon to 15-20 minutes per target │ │
│ │ • Always deduplicate results from multiple sources │ │
│ │ • Cross-reference sources for high-confidence targets │ │
│ │ • Save all raw results for the final report │ │
│ │ • Have backup tools if primary tool fails │ │
│ │ • Focus on finding attackable targets, not perfection │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
Common Mistakes
| Mistake | Consequence | Fix |
|---|---|---|
| Spending too much time on passive recon | Not enough time for exploitation | Set a 15-20 minute timer |
| Not saving raw results | Missing data for report | Save all outputs immediately |
| Relying on a single source | Missing subdomains | Use at least 2-3 sources |
| Not deduplicating results | Wasted scanning time | Always deduplicate before active scanning |
| Ignoring email results | Missing social engineering targets | Save and review email results |
| Not checking for takeovers | Missing easy wins | Enable takeover detection |
Quick Reference Commands
# theHarvester quick commands
theHarvester -d TARGET -b bing,google,virustotal -l 500 -f results -n
theHarvester -d TARGET --full -l 1000 -f comprehensive -n -t
theHarvester -d TARGET -b hunter,linkedin,google -l 500 -f emails
# Maltego quick workflow
# 1. Open Maltego
# 2. Add Domain entity
# 3. Run: DNS Resolve, Get MX, Get Subdomains, Get Certificates
# 4. Export graph as PNG for documentation
# Amass passive
amass enum -passive -d TARGET -o amass_results.txt
# Subfinder
subfinder -d TARGET -o subfinder_results.txt
# Combine and deduplicate
cat results*.txt | sort -u > all_unique_subs.txt
Summary
theHarvester and Maltego are complementary tools that form a powerful passive reconnaissance toolkit:
theHarvester excels at:
- Fast, command-line subdomain and email harvesting
- Easy automation and scripting
- Quick results with minimal setup
- Integration into automated pipelines
Maltego excels at:
- Visual relationship mapping
- Deep multi-hop analysis
- Professional reporting and documentation
- Team collaboration and sharing
For OSCP:
- Use theHarvester for quick, reliable subdomain discovery
- Use Maltego if you have time for visual analysis
- Always combine multiple sources for comprehensive coverage
- Deduplicate and prioritize results before active scanning
- Document everything for the final report
Next Steps: After passive gathering, proceed to:
- DNS resolution and validation
- Active port scanning with Nmap
- Service enumeration on discovered hosts
- Web application reconnaissance
- Vulnerability identification and exploitation