Domain 1Recon & ReportingCORE

Passive Gathering with theHarvester and Maltego

Comprehensive guide to passive information gathering using theHarvester and Maltego. Covers all commands, flags, sources, output formats, Maltego entity types, transformations, visual mapping techniques, tool comparison, automation strategies, and OSCP-specific workflows for building effective passive reconnaissance pipelines.

Domain: Domain 1: Reconnaissance & Reporting•Updated: 2026-09-01
#theharvester#maltego#passive-gathering#visual-mapping

Passive Gathering with theHarvester and Maltego

Attack Chain Position

graph LR
    A[Reconnaissance] --> B[Passive Gathering]
    B --> C[theHarvester]
    B --> D[Maltego]
    C --> E[Emails, IPs, Hosts]
    D --> F[Visual Entity Graph]
    E --> G[Data Correlation]
    F --> G
    G --> H[Active Recon]
    H --> I[Exploitation]
    
    style B fill:#f9d56e,stroke:#333,stroke-width:2px
    style C fill:#a8e6cf,stroke:#333,stroke-width:1px
    style D fill:#a8e6cf,stroke:#333,stroke-width:1px

Position: Passive Reconnaissance Phase — theHarvester and Maltego are complementary tools that gather intelligence from public sources without directly interacting with the target, reducing detection risk while building a comprehensive picture of the target’s attack surface.


Table of Contents

  1. theHarvester Deep Dive
  2. Maltego Deep Dive
  3. theHarvester vs Maltego Comparison
  4. Combining theHarvester and Maltego
  5. Other Passive Gathering Tools
  6. Automating Passive Gathering
  7. Data Correlation Techniques
  8. Building Passive Recon Pipelines
  9. Managing Large Result Sets
  10. OSCP Value Assessment
  11. OSCP Exam Tips

theHarvester Deep Dive

Overview and Purpose

theHarvester is one of the oldest and most widely-used open-source tools for passive information gathering. Originally created by Luis Martinez, it is now maintained by the TrustedSec team and is part of the OWASP project.

Primary Purpose: Gather emails, subdomains, hosts, employee names, open ports, and banners from various public sources.

Key Characteristics:

  • Passive-only: Does not interact directly with the target
  • Multi-source: Queries dozens of public data sources
  • Fast: Results typically return in seconds to minutes
  • Free and open-source: Available on GitHub
  • Cross-platform: Works on Linux, macOS, and Windows

Installation

# Method 1: pip install (recommended)
pip3 install theHarvester

# Method 2: From source
git clone https://github.com/laramies/theHarvester.git
cd theHarvester
pip3 install -r requirements.txt
python3 setup.py install

# Method 3: Kali Linux (pre-installed)
# theHarvester comes pre-installed on Kali Linux
theHarvester --help

# Verify installation
theHarvester --version

Command-Line Syntax

theHarvester [-b engines] [-d domain] [-l limit] [-f output_filename] [-n] [-c] [-j] [-v] [-t] [-h]

All Commands and Flags

Flag Short Description Example
-b --sources Comma-separated list of sources to use -b bing,google,linkedin
-d --domain Target domain name -d example.com
-l --limit Maximum number of results to return -l 500
-f --filename Output filename (without extension) -f results
-n --dns-lookup Perform DNS resolution on found hosts -n
-c --dns-reverse Perform reverse DNS lookups -c
-j --dns-brute Perform DNS brute force using subdomains list -j
-v --verbose Enable verbose output -v
-t --takeover Check for potential subdomain takeovers -t
-h --help Show help message -h
-z --subtakeover Enable subdomain takeover detection -z
--shodan Shodan API key for enhanced results --shodan API_KEY
--hackertarget Enable HackerTarget source --hackertarget
--full Run all available sources --full

All Available Sources

theHarvester supports dozens of data sources. Here is a comprehensive list:

Search Engine Sources

Source Data Types Description
bing Subdomains, Emails, IPs Microsoft Bing search engine
google Subdomains, Emails, IPs Google search engine
yahoo Subdomains, Emails Yahoo search engine
ask Subdomains, Emails Ask.com search engine
duckduckgo Subdomains, Emails DuckDuckGo search engine
yandex Subdomains, Emails Yandex search engine (Russian)
baidu Subdomains, Emails Baidu search engine (Chinese)
google-site Subdomains Google site: operator
google-profiles Employee Names Google+ profiles (deprecated)

DNS and Certificate Sources

Source Data Types Description
virustotal Subdomains, IPs, Emails VirusTotal passive DNS
crt Subdomains Certificate Transparency logs
crtsh Subdomains crt.sh CT log database
securitytrails Subdomains, IPs SecurityTrails DNS history
shodan Subdomains, IPs, Ports Shodan internet scanner
fullhunt Subdomains, IPs FullHunt attack surface
intelx Subdomains, Emails Intelix.io intelligence

Social Media and People Sources

Source Data Types Description
linkedin Employee Names, Emails LinkedIn professional network
github Subdomains, Emails, Code GitHub repositories
twitter Employee Names Twitter/X social media
facebook Employee Names Facebook social media
phonebook Employee Names, Emails Phonebook lookup services

Specialized Sources

Source Data Types Description
dnsdumpster Subdomains, IPs DNSdumpster.com DNS research
threatminer Subdomains, IPs ThreatMiner threat intelligence
passivetotal Subdomains, IPs PassiveTotal DNS data
bufferover Subdomains BufferOver DNS API
bevigil Subdomains, IPs BeVigil attack surface
c99 Subdomains C99.nl DNS API
whois IPs, Emails WHOIS database records
hunter Emails Hunter.io email finder
netlas Subdomains, IPs Netlas.io network data
redhunt Subdomains, IPs RedHunt Labs API
subdomaincenter Subdomains Subdomain.center API
whois IPs, Emails WHOIS database records
zoomeye Subdomains, IPs ZoomEye search engine

Output Formats

theHarvester supports multiple output formats:

Format Extension Description
HTML .html Human-readable report with tables
JSON .json Machine-readable structured data
CSV .csv Spreadsheet-compatible format
XLS .xls Excel spreadsheet format
XLSX .xlsx Modern Excel format
SQLite .db SQLite database for querying

Practical Command Examples

Basic Usage

# Basic subdomain and email harvest
theHarvester -d example.com -b bing,google,duckduckgo -l 500

# Harvest with DNS resolution
theHarvester -d example.com -b bing,google,virustotal -l 500 -n

# Harvest with all sources
theHarvester -d example.com --full -l 1000

# Harvest with specific output format
theHarvester -d example.com -b bing,google,virustotal -l 500 -f results_example

Advanced Usage

# Comprehensive harvest with all options
theHarvester -d example.com \
  -b bing,google,duckduckgo,virustotal,crtsh,dnsdumpster,github,linkedin \
  -l 1000 \
  -f comprehensive_results \
  -n \
  -v \
  -t

# Harvest with Shodan integration
theHarvester -d example.com \
  -b shodan,virustotal,crtsh \
  -l 500 \
  --shodan YOUR_SHODAN_API_KEY \
  -f shodan_results

# Harvest emails specifically
theHarvester -d example.com \
  -b hunter,linkedin,google,bing \
  -l 500 \
  -f email_results

# Harvest with takeover detection
theHarvester -d example.com \
  -b bing,google,virustotal,crtsh \
  -l 500 \
  -t \
  -f takeover_results

DNS Resolution and Reverse Lookup

# Forward DNS resolution
theHarvester -d example.com -b bing,google -l 500 -n

# Reverse DNS lookup on found IPs
theHarvester -d example.com -b bing,google -l 500 -c

# Both forward and reverse
theHarvester -d example.com -b bing,google -l 500 -n -c

DNS Brute Force

# DNS brute force using built-in subdomain list
theHarvester -d example.com -b bing,google -l 500 -j

# Combine passive sources with brute force
theHarvester -d example.com -b bing,google,virustotal -l 500 -j -n

Takeover Detection

theHarvester can detect potential subdomain takeover vulnerabilities:

# Enable takeover detection
theHarvester -d example.com -b bing,google,virustotal,crtsh -l 500 -t

# What takeover detection checks:
# 1. CNAME records pointing to non-existent services
# 2. Subdomains with no A/AAAA records but with CNAME
# 3. CNAMEs to services that no longer host the subdomain
# 4. Common takeover-prone services (GitHub Pages, Heroku, S3, etc.)

Common Takeover-Prone Services:

Service Indicator Risk
GitHub Pages CNAME to *.github.io High
Heroku CNAME to *.herokuapp.com High
Amazon S3 CNAME to *.s3.amazonaws.com High
Azure Blob CNAME to *.blob.core.windows.net Medium
Firebase CNAME to *.firebaseapp.com Medium
Netlify CNAME to *.netlify.app Medium
GitLab Pages CNAME to *.gitlab.io High
WordPress CNAME to *.wordpress.com Medium

Result Interpretation

HTML Output Structure

┌─────────────────────────────────────────────────────────────┐
│ theHarvester Report: example.com                            │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Emails:                                                    │
│  ┌─────────────────────────────────────────────────────┐   │
│  │ admin@example.com                                   │   │
│  │ info@example.com                                    │   │
│  │ john.doe@example.com                                │   │
│  └─────────────────────────────────────────────────────┘   │
│                                                             │
│  Hosts:                                                     │
│  ┌─────────────────────────────────────────────────────┐   │
│  │ www.example.com -> 93.184.216.34                    │   │
│  │ mail.example.com -> 93.184.216.35                   │   │
│  │ api.example.com -> 93.184.216.36                    │   │
│  └─────────────────────────────────────────────────────┘   │
│                                                             │
│  IPs:                                                       │
│  ┌─────────────────────────────────────────────────────┐   │
│  │ 93.184.216.34                                       │   │
│  │ 93.184.216.35                                       │   │
│  │ 93.184.216.36                                       │   │
│  └─────────────────────────────────────────────────────┘   │
│                                                             │
│  Virtual Hosts:                                             │
│  ┌─────────────────────────────────────────────────────┐   │
│  │ dev.example.com                                     │   │
│  │ staging.example.com                                 │   │
│  └─────────────────────────────────────────────────────┘   │
│                                                             │
└─────────────────────────────────────────────────────────────┘

JSON Output Structure

{
  "emails": [
    "admin@example.com",
    "info@example.com",
    "john.doe@example.com"
  ],
  "hosts": [
    "www.example.com",
    "mail.example.com",
    "api.example.com",
    "dev.example.com",
    "staging.example.com"
  ],
  "ips": [
    "93.184.216.34",
    "93.184.216.35",
    "93.184.216.36"
  ],
  "virtual_hosts": [
    "dev.example.com",
    "staging.example.com"
  ],
  "names": [
    "John Doe",
    "Jane Smith"
  ],
  "sources": [
    "bing",
    "google",
    "virustotal"
  ]
}

Source Selection Strategy

Goal Recommended Sources Reason
Maximum subdomain coverage bing,google,virustotal,crtsh,dnsdumpster,shodan Broadest coverage across different data types
Email harvesting hunter,linkedin,google,bing Best sources for email addresses
Fast results bing,google,crtsh Quick response times
Deep historical data virustotal,securitytrails,shodan Historical DNS and certificate data
Code and infrastructure github,shodan Source code and exposed services
Employee intelligence linkedin,google,twitter People and organizational data

Maltego Deep Dive

Overview and Purpose

Maltego is a visual link analysis and open-source intelligence (OSINT) tool developed by Paterva (formerly Paterva Ltd.). It transforms data from various sources into an interactive graph of entities and relationships.

Primary Purpose: Visual mapping of relationships between entities (domains, IPs, people, organizations, etc.) to discover connections and patterns that might not be apparent from raw data.

Key Characteristics:

  • Visual interface: Graph-based representation of data
  • Entity-based: Works with predefined entity types
  • Transformations: Pre-built queries that expand the graph
  • Multi-source: Integrates with dozens of data sources
  • Commercial and Community editions: Free Community edition available

Installation

# Method 1: Download from Paterva website
# Visit: https://www.maltego.com/products/maltego/
# Download Maltego Community Edition (free)

# Method 2: Linux package
wget https://downloads.maltego.com/maltego/Maltego_4.4.1-1_amd64.deb
sudo dpkg -i Maltego_4.4.1-1_amd64.deb

# Method 3: Windows installer
# Download .exe from Paterva website

# Launch Maltego
maltego

Entity Types

Maltego works with predefined entity types — structured representations of real-world objects:

Core Entity Types

Entity Type Icon Description Common Transformations
Domain Name 🌐 A registered domain DNS Resolution, WHOIS, Subdomains
IP Address 📍 IPv4 or IPv6 address Reverse DNS, Geolocation, Hostnames
Hostname 💻 A specific host name DNS Resolution, Port Scan, Services
URL 🔗 A web address Content Analysis, Links, Technologies
Person 👤 An individual Email, Social Media, Affiliations
Email Address 📧 An email address Domain Extraction, Person Lookup
Phone Number 📞 A telephone number Geolocation, Owner Lookup
Organization 🏢 A company or entity Domains, People, Technologies
ASN 🌍 Autonomous System Number IP Ranges, Organization, Peers
Certificate 🔒 SSL/TLS certificate Domain Extraction, Issuer Info
MX Record 📬 Mail exchange record Mail Server, IP Resolution
NS Record 📋 Name server record DNS Server, IP Resolution
TXT Record 📝 DNS TXT record SPF, DKIM, DMARC Analysis
SSH Key 🔑 SSH public key Fingerprint, Host Association
SSL Certificate 🛡️ SSL certificate details Domain, Issuer, Validity
Whois Record 📄 WHOIS registration data Registrant, Dates, Nameservers

Entity Hierarchy

graph TD
    A[Domain Name] --> B[IP Address]
    A --> C[Hostname]
    A --> D[MX Record]
    A --> E[NS Record]
    A --> F[TXT Record]
    A --> G[SSL Certificate]
    A --> H[Whois Record]
    
    B --> I[ASN]
    B --> J[Geolocation]
    B --> K[Reverse DNS]
    
    C --> L[URL]
    C --> M[Services]
    
    G --> N[Issuer Organization]
    
    H --> O[Registrant Person]
    H --> P[Registrant Organization]
    
    style A fill:#f9d56e,stroke:#333,stroke-width:2px

Key Transformations

Transformations are the core mechanism in Maltego — they query data sources and return new entities linked to the selected entity.

Domain Name Transformations

Transformation Source Returns Description
DNS: Resolve to IP DNS IP Address Resolve domain to IP addresses
DNS: Reverse Lookup DNS Hostname Reverse DNS lookup on IPs
DNS: Get MX Records DNS MX Record Get mail exchange records
DNS: Get NS Records DNS NS Record Get name server records
DNS: Get TXT Records DNS TXT Record Get TXT records (SPF, DKIM)
VirusTotal: Get Subdomains VirusTotal Domain Name Get subdomains from VT
PassiveTotal: Get Subdomains PassiveTotal Domain Name Get subdomains from PT
Certificate: Get Certificates CT Logs SSL Certificate Get SSL certificates
Whois: Get Registration WHOIS Whois Record Get domain registration info
Shodan: Get Hosts Shodan Hostname Get hosts from Shodan
Google: Get Subdomains Google Domain Name Search Google for subdomains
Bing: Get Subdomains Bing Domain Name Search Bing for subdomains

IP Address Transformations

Transformation Source Returns Description
DNS: Reverse Lookup DNS Hostname Reverse DNS on IP
GeoIP: Get Location MaxMind Geolocation Get geographic location
Shodan: Get Info Shodan Host details Get Shodan host details
Censys: Get Host Censys Host details Get Censys host details
ASN: Get ASN Info RIPEDB ASN Get ASN information
VirusTotal: Get Report VirusTotal VT Report Get VirusTotal report
AbuseIPDB: Get Report AbuseIPDB Report Get abuse reports

Email Address Transformations

Transformation Source Returns Description
Extract Domain Built-in Domain Name Extract domain from email
Hunter: Get Person Hunter.io Person Get person details
Snovio: Get Person Snovio Person Get person details
Phonebook: Lookup Phonebook Person Phonebook lookup

Maltego Workflow

graph TD
    A[Start: Add Domain Entity] --> B[Right-click: Run Transformations]
    B --> C[DNS: Resolve to IP]
    B --> D[DNS: Get MX Records]
    B --> E[VirusTotal: Get Subdomains]
    B --> F[Certificate: Get Certificates]
    
    C --> G[Analyze IP Entities]
    D --> H[Analyze MX Entities]
    E --> I[Analyze Subdomain Entities]
    F --> J[Analyze Certificate Entities]
    
    G --> K[Run IP Transformations]
    H --> L[Run MX Transformations]
    I --> M[Run Subdomain Transformations]
    J --> N[Run Certificate Transformations]
    
    K --> O[Build Complete Graph]
    L --> O
    M --> O
    N --> O
    
    O --> P[Identify Patterns]
    P --> Q[Export Results]
    Q --> R[Use in Further Recon]
    
    style A fill:#f9d56e,stroke:#333,stroke-width:2px
    style O fill:#a8e6cf,stroke:#333,stroke-width:2px

Step-by-Step Workflow

Step 1: Create a New Graph

  1. Open Maltego
  2. Click File > New Graph
  3. Give the graph a name (e.g., “example.com Recon”)

Step 2: Add Initial Entities

  1. Right-click in the canvas
  2. Select Add Entity
  3. Choose Domain Name
  4. Enter the target domain (e.g., example.com)

Step 3: Run Transformations

  1. Right-click the domain entity
  2. Select Run Transformations
  3. Choose transformations to run:
    • DNS: Resolve to IP
    • DNS: Get MX Records
    • DNS: Get NS Records
    • VirusTotal: Get Subdomains
    • Certificate: Get Certificates

Step 4: Analyze Results

  1. Review the generated graph
  2. Look for patterns and connections
  3. Identify high-value targets
  4. Expand interesting branches with additional transformations

Step 5: Export Results

  1. Click File > Export
  2. Choose export format:
    • PNG/SVG: Visual graph image
    • CSV: Tabular data
    • Maltego Package: Full graph for sharing
    • Text: Plain text entity list

Graph Analysis Techniques

Visual Pattern Recognition

Pattern What It Looks Like What It Means
Star Pattern One central node with many connections Central infrastructure (main domain)
Chain Pattern Linear sequence of connected nodes DNS delegation chain
Cluster Pattern Dense group of interconnected nodes Related services or environments
Island Pattern Isolated group with few external connections Separate infrastructure segment
Hub Pattern Node with many incoming connections Shared resource (CDN, cloud provider)

Graph Navigation Tips

Action Shortcut Description
Zoom In/Out Mouse wheel Zoom the graph view
Pan Middle mouse + drag Move around the graph
Select Multiple Ctrl + click Select multiple entities
Expand All Right-click > Expand Run all available transforms
Collapse Right-click > Collapse Hide child entities
Filter Filter panel Show/hide entity types
Search Ctrl + F Search entities by name
Auto-Layout Right-click > Auto-Layout Reorganize graph layout

Export Options

Format Use Case Command
PNG Documentation, reports File > Export > PNG
SVG Scalable graphics, presentations File > Export > SVG
CSV Data analysis, spreadsheets File > Export > CSV
Maltego Package Sharing graphs with team File > Export > Package
Text Quick entity lists File > Export > Text
GEXF Import to other graph tools File > Export > GEXF

Maltego Machine Accounts

Maltego uses Machine Accounts to authenticate with data sources:

Machine Account Purpose Free Tier?
Maltego Service Built-in transforms Yes
VirusTotal VT data access Yes (limited)
PassiveTotal PT DNS data Yes (limited)
Shodan Shodan data Yes (limited)
Google Google search Yes
Bing Bing search Yes
DNS DNS resolution Yes
WHOIS WHOIS data Yes
Certificate CT log data Yes

Setting Up Machine Accounts:

  1. Click Configuration > Machine Accounts
  2. Add new machine account
  3. Enter credentials (API keys where required)
  4. Test connection
  5. Save configuration

theHarvester vs Maltego Comparison

Feature Comparison Table

Feature theHarvester Maltego
Interface Command-line Graphical (GUI)
Cost Free Free (Community) / Paid (Enterprise)
Learning Curve Low Medium-High
Speed Fast (seconds-minutes) Medium (minutes)
Visualization None (text/HTML/JSON) Interactive graph
Data Sources 30+ sources 50+ sources (with transforms)
Customization Source selection Custom transforms, machine accounts
Automation Easy (CLI, scripts) Medium (Paterva transforms)
Export Formats HTML, JSON, CSV, XLS, SQLite PNG, SVG, CSV, Package, Text
Relationship Mapping No Yes (core feature)
Team Collaboration No Yes (Enterprise)
API Access No Yes (Enterprise)
Resource Usage Low Medium-High
Installation pip install Download installer
Best For Quick subdomain/email harvest Deep relationship analysis

When to Use Each Tool

Scenario Recommended Tool Reason
Quick subdomain list theHarvester Fast, simple, CLI-friendly
Email harvesting theHarvester Dedicated email sources
Visual relationship mapping Maltego Graph-based visualization
Deep infrastructure analysis Maltego Multi-hop relationship discovery
Automated pipelines theHarvester Easy to script and automate
Team collaboration Maltego Shared graphs and reports
Time-constrained (OSCP) theHarvester Faster setup and execution
Comprehensive OSINT Both Complementary strengths

Strengths and Weaknesses

theHarvester

Strengths:

  • Fast and lightweight
  • Easy to automate
  • No GUI overhead
  • Great for quick results
  • Integrates well with scripts

Weaknesses:

  • No visual representation
  • Limited relationship mapping
  • Source availability varies
  • No built-in correlation

Maltego

Strengths:

  • Visual relationship mapping
  • Deep multi-hop analysis
  • Rich entity types
  • Professional reporting
  • Team collaboration features

Weaknesses:

  • Steeper learning curve
  • Slower execution
  • Resource-intensive
  • Free edition has limitations
  • Requires API keys for best results

Combining theHarvester and Maltego

Integrated Workflow

graph LR
    A[theHarvester] --> B[Subdomain List]
    A --> C[Email List]
    A --> D[IP List]
    
    B --> E[Import to Maltego]
    C --> E
    D --> E
    
    E --> F[Run Maltego Transforms]
    F --> G[Visual Graph]
    G --> H[Identify Patterns]
    H --> I[Export Results]
    
    style A fill:#a8e6cf,stroke:#333,stroke-width:1px
    style E fill:#f9d56e,stroke:#333,stroke-width:2px

Step-by-Step Integration

Step 1: Run theHarvester

# Comprehensive theHarvester run
theHarvester -d example.com \
  -b bing,google,virustotal,crtsh,dnsdumpster,github \
  -l 1000 \
  -f harvester_results \
  -n \
  -v

Step 2: Import Results to Maltego

Method 1: Manual Import

  1. Open Maltego
  2. Right-click in canvas
  3. Select Add Entity
  4. Add each subdomain as a Domain Name entity

Method 2: CSV Import

# Export theHarvester results as CSV
theHarvester -d example.com -b bing,google,virustotal -l 500 -f results

# The CSV file can be imported into Maltego
# File > Import > CSV

Method 3: Scripted Import

#!/usr/bin/env python3
"""
harvester_to_maltego.py - Import theHarvester results into Maltego
"""

import csv
import json

def harvester_csv_to_maltego(csv_file, output_file):
    """Convert theHarvester CSV to Maltego import format."""
    entities = []
    
    with open(csv_file, 'r') as f:
        reader = csv.DictReader(f)
        for row in reader:
            # Add domain entities
            if row.get('hosts'):
                for host in row['hosts'].split(','):
                    host = host.strip()
                    if host:
                        entities.append({
                            'Type': 'DomainName',
                            'RawData': host,
                            'DisplayName': host
                        })
            
            # Add IP entities
            if row.get('ips'):
                for ip in row['ips'].split(','):
                    ip = ip.strip()
                    if ip:
                        entities.append({
                            'Type': 'IPv4Address',
                            'RawData': ip,
                            'DisplayName': ip
                        })
    
    with open(output_file, 'w') as f:
        json.dump(entities, f, indent=2)
    
    print(f"[*] Exported {len(entities)} entities to {output_file}")

if __name__ == "__main__":
    harvester_csv_to_maltego("results_example.csv", "maltego_import.json")

Step 3: Run Maltego Transformations

  1. Import theHarvester results
  2. Select all domain entities
  3. Right-click > Run Transformations
  4. Choose:
    • DNS: Resolve to IP
    • DNS: Get MX Records
    • Certificate: Get Certificates
    • VirusTotal: Get Subdomains

Step 4: Analyze and Export

  1. Review the visual graph
  2. Identify patterns and relationships
  3. Export results in desired format

Combined Command Workflow

#!/bin/bash
# combined_recon.sh - theHarvester + Maltego workflow

DOMAIN="${1:-example.com}"

echo "[*] Phase 1: theHarvester passive gathering"
theHarvester -d "$DOMAIN" \
  -b bing,google,virustotal,crtsh,dnsdumpster \
  -l 1000 \
  -f "${DOMAIN}_harvester" \
  -n \
  -v

echo "[*] Phase 2: Extract subdomains for Maltego"
grep -oP '"hosts"\s*:\s*\[\K[^\]]+' "${DOMAIN}_harvester.json" | \
  tr ',' '\n' | \
  sed 's/[" ]//g' | \
  sort -u > "${DOMAIN}_subdomains.txt"

echo "[*] Phase 3: Prepare Maltego import"
python3 harvester_to_maltego.py "${DOMAIN}_harvester.csv" "${DOMAIN}_maltego.json"

echo "[*] Phase 4: Open Maltego and import ${DOMAIN}_maltego.json"
echo "[*] Then run transformations on imported entities"

Other Passive Gathering Tools

Recon-ng

Recon-ng is a full-featured web reconnaissance framework written by Tim Tomes.

Features:

  • Modular framework architecture
  • Built-in modules for various sources
  • Database-backed results storage
  • Custom module development
  • Professional reporting

Installation:

git clone https://github.com/lanmaster53/recon-ng.git
cd recon-ng
pip3 install -r requirements.txt
./recon-ng

Basic Usage:

# Start Recon-ng
./recon-ng

# Install modules
marketplace install

# Search for modules
search domain

# Use a module
modules load recon/domains-hosts/google_site_search
options set SOURCE example.com
run

# View results
show hosts
show emails

# Export results
creds export --csv results.csv

Key Modules:

Module Purpose Data Source
recon/domains-hosts/google_site_search Subdomain discovery Google
recon/domains-hosts/bing_domain_search Subdomain discovery Bing
recon/domains-hosts/virustotal_passive_dns Passive DNS VirusTotal
recon/domains-hosts/crt_sh CT log subdomains crt.sh
recon/contacts-emails/hunter_io Email discovery Hunter.io
recon/contacts-names/google_gmail_search Name discovery Google
recon/hosts-ports/shodan Port discovery Shodan
recon/hosts-tech/wappalyzer Technology detection Wappalyzer

SpiderFoot

SpiderFoot is an automated OSINT and threat intelligence platform.

Features:

  • Fully automated scanning
  • 200+ modules
  • Web interface
  • API access
  • Real-time results

Installation:

# Docker installation (recommended)
docker pull spiderfoot/spiderfoot
docker run -d -p 5001:5001 spiderfoot/spiderfoot

# From source
git clone https://github.com/smicallef/spiderfoot.git
cd spiderfoot
pip3 install -r requirements.txt
python3 sf.py --web

Basic Usage:

# CLI mode
python3 sf.py --target example.com --modules all --scan myscan

# Web interface
# Navigate to http://localhost:5001
# Create new scan
# Select target and modules
# Start scan
# View results

Amass (Passive Mode)

Amass by OWASP is a comprehensive subdomain enumeration tool.

Passive Mode Usage:

# Passive-only enumeration
amass enum -passive -d example.com -o amass_passive.txt

# Passive with specific sources
amass enum -passive -d example.com -src bing,google,virustotal,crtsh

# Passive with API keys
amass enum -passive -d example.com \
  -df /path/to/api.conf \
  -o amass_passive.txt

Tool Comparison

Tool Interface Sources Automation Best For
theHarvester CLI 30+ High Quick harvest
Maltego GUI 50+ Medium Visual analysis
Recon-ng CLI/Menu 40+ High Modular recon
SpiderFoot Web/GUI 200+ Very High Automated scanning
Amass (passive) CLI 25+ High Subdomain focus

Automating Passive Gathering

Automated Pipeline Script

#!/usr/bin/env python3
"""
passive_recon_pipeline.py - Automated passive reconnaissance pipeline
"""

import subprocess
import json
import os
import argparse
from datetime import datetime
from pathlib import Path

class PassiveReconPipeline:
    def __init__(self, domain, output_dir="passive_results"):
        self.domain = domain
        self.output_dir = Path(output_dir)
        self.output_dir.mkdir(parents=True, exist_ok=True)
        self.results = {
            'subdomains': set(),
            'emails': set(),
            'ips': set(),
            'names': set(),
        }
    
    def run_theharvester(self, sources=None, limit=500):
        """Run theHarvester with specified sources."""
        if not sources:
            sources = ['bing', 'google', 'virustotal', 'crtsh', 'dnsdumpster']
        
        sources_str = ','.join(sources)
        output_file = self.output_dir / f"{self.domain}_theharvester"
        
        cmd = [
            'theHarvester',
            '-d', self.domain,
            '-b', sources_str,
            '-l', str(limit),
            '-f', str(output_file),
            '-n',
            '-v'
        ]
        
        print(f"[*] Running theHarvester with sources: {sources_str}")
        try:
            result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
            print(result.stdout)
            
            # Parse JSON output if available
            json_file = output_file.with_suffix('.json')
            if json_file.exists():
                with open(json_file, 'r') as f:
                    data = json.load(f)
                    self.results['subdomains'].update(data.get('hosts', []))
                    self.results['emails'].update(data.get('emails', []))
                    self.results['ips'].update(data.get('ips', []))
                    self.results['names'].update(data.get('names', []))
            
            print(f"[+] theHarvester complete. Found {len(self.results['subdomains'])} subdomains")
        
        except subprocess.TimeoutExpired:
            print("[!] theHarvester timed out")
        except Exception as e:
            print(f"[!] theHarvester error: {e}")
   
    def run_amass_passive(self):
        """Run Amass in passive mode."""
        output_file = self.output_dir / f"{self.domain}_amass.txt"
        
        cmd = [
            'amass', 'enum',
            '-passive',
            '-d', self.domain,
            '-o', str(output_file)
        ]
        
        print(f"[*] Running Amass passive enumeration")
        try:
            result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
            
            if output_file.exists():
                with open(output_file, 'r') as f:
                    for line in f:
                        sub = line.strip()
                        if sub and sub != self.domain:
                            self.results['subdomains'].add(sub)
            
            print(f"[+] Amass complete. Found {len(self.results['subdomains'])} total subdomains")
        
        except subprocess.TimeoutExpired:
            print("[!] Amass timed out")
        except Exception as e:
            print(f"[!] Amass error: {e}")
    
    def run_subfinder(self):
        """Run Subfinder for passive subdomain discovery."""
        output_file = self.output_dir / f"{self.domain}_subfinder.txt"
        
        cmd = [
            'subfinder',
            '-d', self.domain,
            '-o', str(output_file),
            '-silent'
        ]
        
        print(f"[*] Running Subfinder")
        try:
            result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
            
            if output_file.exists():
                with open(output_file, 'r') as f:
                    for line in f:
                        sub = line.strip()
                        if sub:
                            self.results['subdomains'].add(sub)
            
            print(f"[+] Subfinder complete")
        
        except subprocess.TimeoutExpired:
            print("[!] Subfinder timed out")
        except Exception as e:
            print(f"[!] Subfinder error: {e}")
    
    def generate_report(self):
        """Generate comprehensive report."""
        report_file = self.output_dir / f"{self.domain}_report.md"
        
        with open(report_file, 'w') as f:
            f.write(f"# Passive Reconnaissance Report: {self.domain}\n\n")
            f.write(f"**Date:** {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n\n")
            
            f.write("## Summary\n\n")
            f.write(f"- **Subdomains:** {len(self.results['subdomains'])}\n")
            f.write(f"- **Emails:** {len(self.results['emails'])}\n")
            f.write(f"- **IPs:** {len(self.results['ips'])}\n")
            f.write(f"- **Names:** {len(self.results['names'])}\n\n")
            
            f.write("## Subdomains\n\n")
            f.write("```text\n")
            for sub in sorted(self.results['subdomains']):
                f.write(f"{sub}\n")
            f.write("```\n\n")
            
            f.write("## Emails\n\n")
            f.write("```text\n")
            for email in sorted(self.results['emails']):
                f.write(f"{email}\n")
            f.write("```\n\n")
            
            f.write("## IPs\n\n")
            f.write("```text\n")
            for ip in sorted(self.results['ips']):
                f.write(f"{ip}\n")
            f.write("```\n\n")
            
            f.write("## Names\n\n")
            f.write("```text\n")
            for name in sorted(self.results['names']):
                f.write(f"{name}\n")
            f.write("```\n")
        
        print(f"[+] Report saved to {report_file}")
        return report_file
    
    def save_results(self):
        """Save results to individual files."""
        # Save subdomains
        with open(self.output_dir / "subdomains.txt", 'w') as f:
            for sub in sorted(self.results['subdomains']):
                f.write(f"{sub}\n")
        
        # Save emails
        with open(self.output_dir / "emails.txt", 'w') as f:
            for email in sorted(self.results['emails']):
                f.write(f"{email}\n")
        
        # Save IPs
        with open(self.output_dir / "ips.txt", 'w') as f:
            for ip in sorted(self.results['ips']):
                f.write(f"{ip}\n")
        
        # Save combined results as JSON
        json_results = {
            'domain': self.domain,
            'date': datetime.now().isoformat(),
            'subdomains': sorted(self.results['subdomains']),
            'emails': sorted(self.results['emails']),
            'ips': sorted(self.results['ips']),
            'names': sorted(self.results['names']),
        }
        
        with open(self.output_dir / "results.json", 'w') as f:
            json.dump(json_results, f, indent=2)
        
        print(f"[+] Results saved to {self.output_dir}/")
    
    def run(self):
        """Run the complete pipeline."""
        print(f"{'='*60}")
        print(f"Passive Reconnaissance Pipeline: {self.domain}")
        print(f"{'='*60}\n")
        
        self.run_theharvester()
        self.run_amass_passive()
        self.run_subfinder()
        self.save_results()
        self.generate_report()
        
        print(f"\n{'='*60}")
        print(f"Pipeline Complete!")
        print(f"{'='*60}")
        print(f"Subdomains: {len(self.results['subdomains'])}")
        print(f"Emails: {len(self.results['emails'])}")
        print(f"IPs: {len(self.results['ips'])}")
        print(f"Names: {len(self.results['names'])}")

def main():
    parser = argparse.ArgumentParser(description="Passive Recon Pipeline")
    parser.add_argument("domain", help="Target domain")
    parser.add_argument("-o", "--output", default="passive_results",
                       help="Output directory")
    
    args = parser.parse_args()
    pipeline = PassiveReconPipeline(args.domain, args.output)
    pipeline.run()

if __name__ == "__main__":
    main()

Cron-Based Automation

# Daily passive recon for multiple domains
0 6 * * * /usr/bin/python3 /opt/tools/passive_recon_pipeline.py example.com -o /opt/results/example.com

# Weekly comprehensive recon
0 6 * * 0 /usr/bin/python3 /opt/tools/passive_recon_pipeline.py target.org -o /opt/results/target.org

Data Correlation Techniques

Cross-Source Correlation

graph TD
    A[theHarvester Results] --> D[Correlation Engine]
    B[Maltego Graph] --> D
    C[Amass Results] --> D
    E[Subfinder Results] --> D
    F[CT Log Results] --> D
    
    D --> G[Unified Dataset]
    G --> H[Priority Scoring]
    H --> I[High Priority]
    H --> J[Medium Priority]
    H --> K[Low Priority]
    
    style D fill:#f9d56e,stroke:#333,stroke-width:2px

Correlation Script

def correlate_passive_results(domain):
    """Correlate results from multiple passive sources."""
    sources = {
        'theharvester': set(),
        'amass': set(),
        'subfinder': set(),
        'crtsh': set(),
    }
    
    # Load results from each source
    source_files = {
        'theharvester': f"{domain}_theharvester.json",
        'amass': f"{domain}_amass.txt",
        'subfinder': f"{domain}_subfinder.txt",
        'crtsh': f"{domain}_crtsh.txt",
    }
    
    for source, filepath in source_files.items():
        try:
            if filepath.endswith('.json'):
                with open(filepath, 'r') as f:
                    data = json.load(f)
                    sources[source].update(data.get('hosts', []))
            else:
                with open(filepath, 'r') as f:
                    for line in f:
                        sub = line.strip()
                        if sub:
                            sources[source].add(sub)
        except FileNotFoundError:
            print(f"[!] {filepath} not found")
    
    # Find subdomains found by multiple sources
    all_subs = set()
    for subs in sources.values():
        all_subs.update(subs)
    
    correlation = {}
    for sub in all_subs:
        found_in = [src for src, subs in sources.items() if sub in subs]
        correlation[sub] = {
            'sources': found_in,
            'source_count': len(found_in),
            'priority': 'high' if len(found_in) >= 3 else 
                       'medium' if len(found_in) == 2 else 'low'
        }
    
    # Sort by priority
    high_priority = [sub for sub, data in correlation.items() if data['priority'] == 'high']
    medium_priority = [sub for sub, data in correlation.items() if data['priority'] == 'medium']
    low_priority = [sub for sub, data in correlation.items() if data['priority'] == 'low']
    
    return {
        'high_priority': sorted(high_priority),
        'medium_priority': sorted(medium_priority),
        'low_priority': sorted(low_priority),
        'total_unique': len(all_subs),
        'source_counts': {src: len(subs) for src, subs in sources.items()},
    }

Priority Scoring Matrix

Source Count Priority Action
4+ sources Critical Immediate active scanning
3 sources High Priority target for enumeration
2 sources Medium Include in standard scanning
1 source Low Include in comprehensive scan

Building Passive Recon Pipelines

Pipeline Architecture

graph LR
    A[Input: Domain] --> B[Passive Phase]
    B --> C[theHarvester]
    B --> D[Amass Passive]
    B --> E[Subfinder]
    B --> F[CT Logs]
    
    C --> G[Deduplication]
    D --> G
    E --> G
    F --> G
    
    G --> H[Correlation]
    H --> I[Priority Scoring]
    I --> J[Output: Prioritized List]
    
    J --> K[Active Phase]
    K --> L[DNS Resolution]
    K --> M[Port Scanning]
    K --> N[Service Enumeration]
    
    style B fill:#a8e6cf,stroke:#333,stroke-width:2px
    style K fill:#f9d56e,stroke:#333,stroke-width:2px

Pipeline Best Practices

  1. Start passive, go active later: Always exhaust passive sources before active scanning
  2. Deduplicate early: Remove duplicates after each source to save processing time
  3. Correlate results: Cross-reference sources to identify high-confidence targets
  4. Prioritize targets: Focus active scanning on high-priority subdomains first
  5. Document everything: Save all results with timestamps and source attribution
  6. Automate where possible: Use scripts to run multiple tools in sequence
  7. Set time limits: Don’t let passive recon run indefinitely

Managing Large Result Sets

Deduplication

# Simple deduplication
sort -u all_subdomains.txt > unique_subdomains.txt

# Deduplication with source tracking
python3 -c "
import csv

sources = {}
with open('all_results.csv', 'r') as f:
    reader = csv.DictReader(f)
    for row in reader:
        sub = row['subdomain']
        source = row['source']
        if sub not in sources:
            sources[sub] = set()
        sources[sub].add(source)

with open('deduplicated_results.csv', 'w') as f:
    writer = csv.writer(f)
    writer.writerow(['subdomain', 'sources', 'source_count'])
    for sub, subs in sorted(sources.items()):
        writer.writerow([sub, ','.join(subs), len(subs)])
"

Result Filtering

# Filter for specific TLDs
grep '\.com$' all_subdomains.txt > com_subdomains.txt

# Filter out known CDNs and cloud providers
grep -v -E '(cloudfront\.net|azurewebsites\.net|amazonaws\.com)' all_subdomains.txt > filtered_subdomains.txt

# Filter for subdomains only (exclude base domain)
grep -v '^example\.com$' all_subdomains.txt > subdomains_only.txt

# Filter for specific patterns
grep -E '(dev|staging|test|admin|api|portal)' all_subdomains.txt > interesting_subdomains.txt

Storage and Organization

passive_results/
├── example.com/
│   ├── raw/
│   │   ├── theharvester.json
│   │   ├── amass.txt
│   │   ├── subfinder.txt
│   │   └── crtsh.txt
│   ├── processed/
│   │   ├── subdomains.txt
│   │   ├── emails.txt
│   │   ├── ips.txt
│   │   └── deduplicated.txt
│   ├── correlated/
│   │   ├── high_priority.txt
│   │   ├── medium_priority.txt
│   │   └── low_priority.txt
│   └── reports/
│       ├── summary.md
│       └── results.json
└── target.org/
    └── ...

OSCP Value Assessment

Tool Value for OSCP

Tool OSCP Value Time Investment Recommendation
theHarvester Medium-High 5-10 minutes Use for quick subdomain harvest
Maltego Medium 15-30 minutes Use if time allows for visual analysis
Recon-ng Medium 10-15 minutes Good alternative to theHarvester
SpiderFoot Low-Medium 10-20 minutes Automated but may be overkill
Amass (passive) High 5-10 minutes Excellent subdomain coverage

OSCP-Specific Considerations

  1. Time is critical: OSCP exams have strict time limits. Don’t spend too much time on passive recon.
  2. Lab constraints: Some lab environments may not have internet access for external APIs.
  3. Focus on results: The goal is to find attackable targets, not to build perfect graphs.
  4. Documentation matters: Save all results for the final report.
  5. Have backups: If one tool fails, have alternatives ready.
# OSCP Passive Recon Quick Workflow (10-15 minutes max)

# Step 1: Quick theHarvester run (3-5 minutes)
theHarvester -d target.lab -b bing,google,virustotal,crtsh -l 500 -f target_passive -n

# Step 2: Quick Amass passive (3-5 minutes)
amass enum -passive -d target.lab -o target_amass.txt

# Step 3: Combine and deduplicate (1-2 minutes)
cat target_passive.txt target_amass.txt | sort -u > target_all_subs.txt

# Step 4: Quick DNS resolution (2-3 minutes)
while read sub; do
  host "$sub" 2>/dev/null | grep "has address" | awk '{print $4}'
done < target_all_subs.txt > target_resolved_ips.txt

# Step 5: Move to active scanning
nmap -sV -sC -iL target_resolved_ips.txt -oA target_nmap

OSCP Exam Tips

Key Takeaways

┌─────────────────────────────────────────────────────────────────┐
│              OSCP Passive Gathering Tips                         │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  theHarvester:                                                  │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ • Fast and reliable for subdomain discovery             │   │
│  │ • Use -b flag to select specific sources                │   │
│  │ • Add -n for DNS resolution                             │   │
│  │ • Save results in multiple formats                      │   │
│  │ • Quick command:                                        │   │
│  │   theHarvester -d TARGET -b bing,google,virustotal     │   │
│  │     -l 500 -f results -n                               │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  Maltego:                                                       │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ • Best for visual relationship mapping                  │   │
│  │ • Steeper learning curve - practice before exam         │   │
│  │ • Community edition is sufficient for OSCP              │   │
│  │ • Set up machine accounts before exam day               │   │
│  │ • Export graphs for documentation                       │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  General Tips:                                                  │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ • Limit passive recon to 15-20 minutes per target       │   │
│  │ • Always deduplicate results from multiple sources      │   │
│  │ • Cross-reference sources for high-confidence targets   │   │
│  │ • Save all raw results for the final report             │   │
│  │ • Have backup tools if primary tool fails               │   │
│  │ • Focus on finding attackable targets, not perfection   │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Common Mistakes

Mistake Consequence Fix
Spending too much time on passive recon Not enough time for exploitation Set a 15-20 minute timer
Not saving raw results Missing data for report Save all outputs immediately
Relying on a single source Missing subdomains Use at least 2-3 sources
Not deduplicating results Wasted scanning time Always deduplicate before active scanning
Ignoring email results Missing social engineering targets Save and review email results
Not checking for takeovers Missing easy wins Enable takeover detection

Quick Reference Commands

# theHarvester quick commands
theHarvester -d TARGET -b bing,google,virustotal -l 500 -f results -n
theHarvester -d TARGET --full -l 1000 -f comprehensive -n -t
theHarvester -d TARGET -b hunter,linkedin,google -l 500 -f emails

# Maltego quick workflow
# 1. Open Maltego
# 2. Add Domain entity
# 3. Run: DNS Resolve, Get MX, Get Subdomains, Get Certificates
# 4. Export graph as PNG for documentation

# Amass passive
amass enum -passive -d TARGET -o amass_results.txt

# Subfinder
subfinder -d TARGET -o subfinder_results.txt

# Combine and deduplicate
cat results*.txt | sort -u > all_unique_subs.txt

Summary

theHarvester and Maltego are complementary tools that form a powerful passive reconnaissance toolkit:

theHarvester excels at:

  • Fast, command-line subdomain and email harvesting
  • Easy automation and scripting
  • Quick results with minimal setup
  • Integration into automated pipelines

Maltego excels at:

  • Visual relationship mapping
  • Deep multi-hop analysis
  • Professional reporting and documentation
  • Team collaboration and sharing

For OSCP:

  • Use theHarvester for quick, reliable subdomain discovery
  • Use Maltego if you have time for visual analysis
  • Always combine multiple sources for comprehensive coverage
  • Deduplicate and prioritize results before active scanning
  • Document everything for the final report

Next Steps: After passive gathering, proceed to:

  • DNS resolution and validation
  • Active port scanning with Nmap
  • Service enumeration on discovered hosts
  • Web application reconnaissance
  • Vulnerability identification and exploitation