Directory and File Discovery
Attack Chain Position
graph LR
A[Target Identification] --> B[DNS Enumeration]
B --> C[Port Scanning]
C --> D[Service Identification]
D --> E[Directory Enumeration]
E --> F[Web Application Analysis]
F --> G[Vulnerability Exploitation]
E --> H[File Discovery]
H --> F
E --> I[Virtual Host Discovery]
I --> F
style E fill:#f9d56e,stroke:#333,stroke-width:2px
Directory and file discovery is one of the most critical techniques in web application penetration testing. It helps you:
- Find hidden directories — Discover admin panels, APIs, and backup directories
- Locate sensitive files — Find configuration files, backups, and logs
- Map the web application — Understand the application structure and functionality
- Identify attack surfaces — Find upload forms, login pages, and debug endpoints
- Discover development artifacts — Find .git, .svn, and other version control directories
Table of Contents
- Why Directory Enumeration Matters
- gobuster Deep Dive
- ffuf Deep Dive
- dirb Usage
- Directory Enumeration Wordlists
- Directory Enumeration Workflow
- Handling Large Wordlists
- Recursive Enumeration Strategies
- Virtual Host Discovery
- File Extension Discovery
- Common Hidden Files
- Interpreting Status Codes
- OSCP Directory Enumeration Best Practices
1. Why Directory Enumeration Matters
The Value of Directory Enumeration
Directory enumeration is often the difference between finding a foothold and staring at a blank webpage. Here’s why it matters:
| Discovery Type |
Example |
Value |
| Admin Panels |
/admin, /administrator, /wp-admin |
Direct access to management interfaces |
| API Endpoints |
/api/v1/, /api/v2/, /graphql |
Data exposure and manipulation |
| Backup Files |
/backup/, /backups/, /db_backup/ |
Database dumps and source code |
| Configuration Files |
/config/, /configuration/, /settings/ |
Credentials and settings |
| Debug Endpoints |
/debug/, /phpinfo.php, /server-status |
System information disclosure |
| Upload Forms |
/upload/, /uploads/, /file-upload/ |
File upload vulnerabilities |
| Login Pages |
/login, /signin, /auth/ |
Authentication bypass targets |
| Version Control |
/.git/, /.svn/, /.hg/ |
Source code exposure |
| Documentation |
/docs/, /documentation/, /wiki/ |
Internal documentation |
| Test/Staging |
/test/, /staging/, /dev/ |
Less secured environments |
Common Directories and Their Value
High-Value Directories
| Directory |
Typical Content |
Exploitation Potential |
/admin |
Admin panel |
Full site control |
/api |
API endpoints |
Data extraction, manipulation |
/backup |
Backup files |
Database dumps, source code |
/config |
Configuration files |
Credentials, settings |
/console |
Management console |
System access |
/dashboard |
User dashboard |
Account access |
/debug |
Debug information |
System details, stack traces |
/dev |
Development environment |
Less secured, debug enabled |
/docs |
Documentation |
Internal information |
/files |
File storage |
Sensitive documents |
/images |
Image storage |
Image upload vulnerabilities |
/includes |
Include files |
Source code exposure |
/logs |
Log files |
Credentials, error messages |
/manager |
Manager interface |
Application management |
/portal |
User portal |
Account access |
/scripts |
Script files |
Code execution |
/server-status |
Server status |
System information |
/soap |
SOAP endpoints |
API manipulation |
/src |
Source code |
Full source exposure |
/staging |
Staging environment |
Less secured |
/status |
Status page |
System information |
/test |
Test environment |
Debug features enabled |
/tmp |
Temporary files |
Sensitive data |
/upload |
Upload directory |
File upload vulnerabilities |
/uploads |
Uploaded files |
Arbitrary file access |
/web |
Web root |
Application structure |
/wp-admin |
WordPress admin |
WordPress exploitation |
/wp-content |
WordPress content |
Plugin/theme exploitation |
Hidden Files and Directories
| File/Directory |
Content |
Exploitation Potential |
/.git/ |
Git repository |
Full source code recovery |
/.env |
Environment variables |
Database credentials, API keys |
/.htaccess |
Apache configuration |
Server configuration |
/.svn/ |
SVN repository |
Source code exposure |
/.DS_Store |
macOS metadata |
Directory structure |
/.htpasswd |
Apache passwords |
Credential exposure |
/web.config |
IIS configuration |
Application settings |
/robots.txt |
Crawl directives |
Hidden directory hints |
/sitemap.xml |
Site map |
Complete site structure |
/.well-known/ |
Well-known URIs |
Security configuration |
2. gobuster Deep Dive
Installation
# Kali Linux (pre-installed)
gobuster --version
# Manual installation
# Download from: https://github.com/OJ/gobuster/releases
wget https://github.com/OJ/gobuster/releases/download/vX.X.X/gobuster-linux-amd64.X.X.X.deb
sudo dpkg -i gobuster-linux-amd64.X.X.X.deb
# Go installation
go install github.com/OJ/gobuster/v3@latest
gobuster Modes
gobuster supports multiple modes:
| Mode |
Command |
Purpose |
| dir |
gobuster dir |
Directory/file enumeration |
| dns |
gobuster dns |
Subdomain enumeration |
| vhost |
gobuster vhost |
Virtual host enumeration |
| s3 |
gobuster s3 |
AWS S3 bucket enumeration |
| gcs |
gobuster gcs |
Google Cloud Storage enumeration |
| azure |
gobuster azure |
Azure blob storage enumeration |
gobuster dir Mode
Basic Usage
# Basic directory enumeration
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt
# With thread count
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -t 50
# With extensions
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -x php,txt,html,js,css,backup,bak,old,sql,log,conf,config,xml,json,yml,yaml
# Output to file
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -o /tmp/gobuster_results.txt
# JSON output
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -o /tmp/gobuster_results.json -j
# Verbose output
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -v
# Quiet mode (only found entries)
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -q
All gobuster dir Flags
| Flag |
Short |
Description |
Example |
-u, --url |
|
Target URL |
-u https://target.com |
-w, --wordlist |
|
Wordlist file |
-w /path/to/wordlist |
-t, --threads |
|
Number of threads |
-t 50 |
-e, --expanded |
|
Expanded progress output |
-e |
-r, --no-response-storage |
|
Don’t store responses |
-r |
-k, --no-tls-validation |
|
Skip TLS validation |
-k |
-n, --no-number |
|
Don’t display number of matches |
-n |
-c, --no-color |
|
Disable color output |
-c |
-f, --discover-backup |
|
Discover backup files |
-f |
-x, --extensions |
|
File extensions |
-x php,txt,html |
-L, --allow-no-files |
|
Allow wordlist with no files |
-L |
-b, --status-codes |
|
Status codes to display |
-b 200,204,301,302,307,403 |
--delay |
|
Delay between requests |
--delay 1s |
--timeout |
|
HTTP request timeout |
--timeout 10s |
-a, --useragent |
|
HTTP User-Agent header |
-a "Mozilla/5.0" |
-H, --headers |
|
Custom headers |
-H "Cookie: session=abc" |
-z, --hide-length |
|
Hide response length |
-z |
-o, --output |
|
Output file |
-o results.txt |
-j, --json |
|
JSON output format |
-j |
-s, --status-codes |
|
Successful status codes |
-s 200,301,302,403 |
-d, --domain |
|
Domain for vhost mode |
-d target.com |
-p, --proxy |
|
HTTP proxy |
-p http://127.0.0.1:8080 |
-S, --cookies |
|
Cookies |
-S "session=abc" |
-m, --method |
|
HTTP method |
-m POST |
-i, --include |
|
Include status codes |
-i 200,301,403 |
-exclude |
|
Exclude status codes |
--exclude 404 |
-r |
|
Follow redirects |
-r |
-D |
|
Don’t follow redirects |
-D |
-c |
|
No color |
-c |
-v |
|
Verbose |
-v |
-q |
|
Quiet mode |
-q |
--random-agent |
|
Random User-Agent |
--random-agent |
--encoding |
|
URL encoding |
--encoding utf-8 |
--wildcard |
|
Handle wildcard responses |
--wildcard |
--wildcard-match |
|
Wildcard matching |
--wildcard-match |
Advanced gobuster dir Usage
# With custom headers
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-H "Cookie: session=abc123" \
-H "Authorization: Bearer token123" \
-H "X-Forwarded-For: 127.0.0.1"
# With proxy
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-p http://127.0.0.1:8080
# With delay (for rate-limited targets)
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
--delay 500ms
# With custom status codes
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-s 200,204,301,302,307,403
# Exclude certain status codes
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
--exclude 404,500
# With POST method
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-m POST
# With custom User-Agent
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-a "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
# Random User-Agent
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
--random-agent
# Handle wildcard responses
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
--wildcard
# Recursive enumeration
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-r
# With multiple extensions
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-x php,asp,aspx,jsp,html,htm,txt,js,css,xml,json,yml,yaml,sql,log,conf,config,backup,bak,old,swp,save,tmp
# Discover backup files
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-f
# No TLS validation (for self-signed certs)
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-k
gobuster dir Output Interpretation
$ gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -x php,txt,html
===============================================================
Gobuster v3.6
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url: https://target.com
[+] Method: GET
[+] Threads: 10
[+] Wordlist: /usr/share/wordlists/dirb/common.txt
[+] Negative Status codes: 404
[+] User Agent: gobuster/3.6
[+] Extensions: php,txt,html
[+] Timeout: 10s
===============================================================
Starting gobuster in directory enumeration mode
===============================================================
/.env (Status: 200) [Size: 1234]
/admin (Status: 301) [Size: 234] [--> https://target.com/admin/]
/backup (Status: 403) [Size: 123]
/config.php (Status: 200) [Size: 567]
/index.html (Status: 200) [Size: 8901]
/login.php (Status: 200) [Size: 2345]
/server-status (Status: 200) [Size: 4567]
/wp-admin (Status: 301) [Size: 234] [--> https://target.com/wp-admin/]
===============================================================
Finished
===============================================================
Output Fields Explained
| Field |
Description |
| Path |
The discovered directory or file |
| Status |
HTTP status code returned |
| Size |
Response size in bytes |
| Redirect |
Redirect target (if applicable) |
gobuster dns Mode
# Basic subdomain enumeration
gobuster dns -d target.com -w /usr/share/wordlists/subdomains/top-10000.txt
# With thread count
gobuster dns -d target.com -w /usr/share/wordlists/subdomains/top-10000.txt -t 50
# Output to file
gobuster dns -d target.com -w /usr/share/wordlists/subdomains/top-10000.txt -o /tmp/subdomains.txt
# Find all record types
gobuster dns -d target.com -w /usr/share/wordlists/subdomains/top-10000.txt -r
# Quiet mode
gobuster dns -d target.com -w /usr/share/wordlists/subdomains/top-10000.txt -q
gobuster vhost Mode
# Basic virtual host enumeration
gobuster vhost -u https://target.com -w /usr/share/wordlists/vhosts/vhost-list.txt
# With thread count
gobuster vhost -u https://target.com -w /usr/share/wordlists/vhosts/vhost-list.txt -t 50
# Output to file
gobuster vhost -u https://target.com -w /usr/share/wordlists/vhosts/vhost-list.txt -o /tmp/vhosts.txt
# With domain specification
gobuster vhost -u https://target.com -w /usr/share/wordlists/vhosts/vhost-list.txt -d target.com
gobuster s3 Mode
# AWS S3 bucket enumeration
gobuster s3 -w /usr/share/wordlists/buckets/s3bucket-names.txt
# With thread count
gobuster s3 -w /usr/share/wordlists/buckets/s3bucket-names.txt -t 50
# Show all buckets (including private)
gobuster s3 -w /usr/share/wordlists/buckets/s3bucket-names.txt -c
# Output to file
gobuster s3 -w /usr/share/wordlists/buckets/s3bucket-names.txt -o /tmp/s3_buckets.txt
3. ffuf Deep Dive
Installation
# Kali Linux (pre-installed)
ffuf -V
# Manual installation
# Download from: https://github.com/ffuf/ffuf/releases
wget https://github.com/ffuf/ffuf/releases/download/vX.X.X/ffuf_X.X.X_linux_amd64.tar.gz
tar -xzf ffuf_X.X.X_linux_amd64.tar.gz
sudo mv ffuf /usr/local/bin/
# Go installation
go install github.com/ffuf/ffuf/v2@latest
# Homebrew (macOS)
brew install ffuf
ffuf Basics
ffuf (Fuzz Faster U Fool) uses the FUZZ keyword as a placeholder for wordlist entries.
# Basic directory enumeration
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt
# With thread count
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -t 50
# With extensions
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -e .php,.txt,.html,.js,.css
# Output to file
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -o /tmp/ffuf_results.json -of json
# Verbose output
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -v
# Quiet mode
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -s
ffuf Filters and Matchers
Response Code Filters
# Filter by status code (show only 200)
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -mc 200
# Show multiple status codes
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -mc 200,301,302,403
# Filter out status codes (hide 404)
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fc 404
# Filter out multiple status codes
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fc 404,403,500
Response Size Filters
# Filter by response size (exact match)
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fs 1234
# Filter out response size
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fs 0
# Filter by response size range (using recode)
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fs 1234,5678
Word Count Filters
# Filter by word count in response
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fw 100
# Filter out by word count
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fw 0
Line Count Filters
# Filter by line count in response
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fl 50
# Filter out by line count
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fl 0
Regex Filters
# Filter by regex match in response
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -mr "admin|login|dashboard"
# Filter out by regex
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fr "not found|404"
# Filter by regex in headers
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -hr "Server: Apache"
# Match header value
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -mr "Content-Type: text/html"
# Filter header value
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fr "Content-Type: text/html"
ffuf Advanced Features
Multiple FUZZ Positions
# Multiple FUZZ keywords with different wordlists
ffuf -u https://target.com/FUZZ1/FUZZ2 \
-w FUZZ1:/usr/share/wordlists/dirb/common.txt \
-w FUZZ2:/usr/share/wordlists/extensions/by-ext/php.txt
# Example: This will try:
# https://target.com/admin/login.php
# https://target.com/admin/index.php
# https://target.com/backup/db.php
# etc.
Recursive Enumeration
# Recursive mode (max depth 3)
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-recursion \
-recursion-depth 3
# With recursive extensions
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-recursion \
-recursion-depth 2 \
-e .php,.txt,.html
Virtual Host Discovery
# Vhost discovery using Host header
ffuf -u https://target.com \
-w /usr/share/wordlists/vhosts/vhost-list.txt \
-H "Host: FUZZ.target.com" \
-mc 200,301,302,403 \
-fc 404
# With multiple domains
ffuf -u https://target.com \
-w /usr/share/wordlists/vhosts/vhost-list.txt \
-H "Host: FUZZ" \
-mc 200,301,302,403
POST Fuzzing
# POST body fuzzing
ffuf -u https://target.com/login \
-w /usr/share/wordlists/usernames.txt \
-X POST \
-d "username=FUZZ&password=password123" \
-mc 200,302
# POST with JSON body
ffuf -u https://target.com/api/login \
-w /usr/share/wordlists/usernames.txt \
-X POST \
-H "Content-Type: application/json" \
-d '{"username":"FUZZ","password":"password123"}' \
-mc 200
# POST with file upload
ffuf -u https://target.com/upload \
-w /usr/share/wordlists/files/php-reverse-shell.php \
-X POST \
-H "Content-Type: multipart/form-data" \
-d "file=@FUZZ" \
-mc 200,302
Cookie Fuzzing
# Cookie fuzzing
ffuf -u https://target.com/admin \
-w /usr/share/wordlists/sessions.txt \
-H "Cookie: session=FUZZ" \
-mc 200
# Multiple cookies
ffuf -u https://target.com/admin \
-w /usr/share/wordlists/sessions.txt \
-H "Cookie: session=FUZZ1; token=FUZZ2" \
-w FUZZ1:/usr/share/wordlists/sessions.txt \
-w FUZZ2:/usr/share/wordlists/tokens.txt \
-mc 200
Rate Limiting
# Rate limiting with delay
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-rate 10 # 10 requests per second
# With delay between requests
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-rate 5 \
-t 10
Auto-Calibration
# Auto-calibration to filter out common responses
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-ac # Auto-calibration
# Auto-calibration with ignore
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-ac \
-ic # Ignore calibrations
# JSON output
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-o /tmp/ffuf_results.json \
-of json
# CSV output
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-o /tmp/ffuf_results.csv \
-of csv
# EJSON output (extended JSON with all data)
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-o /tmp/ffuf_results.ejson \
-of ejson
# ECSV output (extended CSV)
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-o /tmp/ffuf_results.ecsv \
-of ecsv
# HTTP output (full HTTP response)
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-o /tmp/ffuf_results.http \
-of http
# MDJSON output (JSON with metadata)
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-o /tmp/ffuf_results.mdjson \
-of mdjson
# TXT output (simple text)
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-o /tmp/ffuf_results.txt \
-of txt
# All output formats
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-o /tmp/ffuf_results \
-of all
ffuf All Flags Reference
| Flag |
Description |
Example |
-u, --url |
Target URL |
-u https://target.com/FUZZ |
-w, --wordlist |
Wordlist(s) |
-w /path/to/wordlist |
-ic, --ignore-calibration |
Ignore calibrations |
-ic |
-ack, --calibration-keywords |
Calibration keywords |
-ack MYKEYWORD |
-acc, --calibration-content-lengths |
Calibration content lengths |
-acc 1234,5678 |
-rate |
Rate limit (req/sec) |
-rate 10 |
-t, --threads |
Number of threads |
-t 50 |
-c, --color |
Enable colors |
-c |
-nc, --no-color |
Disable colors |
-nc |
-s, --silent |
Silent mode |
-s |
-v, --verbose |
Verbose mode |
-v |
-ac, --auto-calibration |
Auto-calibration |
-ac |
-r, --follow-redir |
Follow redirects |
-r |
-H, --header |
Custom header |
-H "Cookie: session=abc" |
-X, --request |
HTTP method |
-X POST |
-d, --data |
Request body |
-d "key=value" |
-b, --cookie |
Cookie |
-b "session=abc" |
-recursion |
Enable recursion |
-recursion |
-recursion-depth |
Recursion depth |
-recursion-depth 3 |
-e, --extensions |
Extensions |
-e .php,.txt |
-mc, --match-codes |
Match status codes |
-mc 200,301 |
-fc, --filter-codes |
Filter status codes |
-fc 404 |
-ml, --match-lines |
Match line count |
-ml 100 |
-fl, --filter-lines |
Filter line count |
-fl 0 |
-mw, --match-words |
Match word count |
-mw 100 |
-fw, --filter-words |
Filter word count |
-fw 0 |
-ms, --match-sizes |
Match sizes |
-ms 1234 |
-fs, --filter-sizes |
Filter sizes |
-fs 0 |
-mr, --match-regex |
Match regex |
-mr "admin" |
-fr, --filter-regex |
Filter regex |
-fr "404" |
-mh, --match-header |
Match header |
-mh "Server: Apache" |
-fh, --filter-header |
Filter header |
-fh "Server: Apache" |
-maxtime |
Max run time |
-maxtime 3600 |
-maxtime-retry |
Max retry time |
-maxtime-retry 60 |
-retry |
Retry failed requests |
-retry 3 |
-replay-proxy |
Replay proxy |
-replay-proxy http://127.0.0.1:8080 |
-proxy |
Proxy URL |
-proxy http://127.0.0.1:8080 |
-ignore-body |
Ignore response body |
-ignore-body |
-noninteractive |
Disable interactive mode |
-noninteractive |
-o |
Output file |
-o results.json |
-of |
Output format |
-of json |
-sa, --skip-auto-calibration |
Skip auto-calibration |
-sa |
-sk, --skip-calibration |
Skip calibration |
-sk |
-debug |
Debug mode |
-debug |
-debug-request |
Debug request |
-debug-request |
-json |
JSON input |
-json |
ffuf vs gobuster Comparison
| Feature |
ffuf |
gobuster |
| Speed |
Very Fast |
Fast |
| Flexibility |
Very High |
Medium |
| Filters |
Extensive |
Basic |
| Recursion |
Yes |
Yes |
| Vhost Mode |
Via headers |
Built-in |
| DNS Mode |
No |
Yes |
| S3 Mode |
No |
Yes |
| POST Fuzzing |
Yes |
Limited |
| Cookie Fuzzing |
Yes |
Limited |
| Multiple FUZZ |
Yes |
No |
| Rate Limiting |
Yes |
Yes |
| Output Formats |
Many |
Few |
| Auto-Calibration |
Yes |
No |
| Learning Curve |
Medium |
Low |
4. dirb Usage
Installation
# Kali Linux (pre-installed)
dirb --version
# Manual installation
sudo apt install dirb
Basic Usage
# Basic directory enumeration
dirb https://target.com
# With custom wordlist
dirb https://target.com /usr/share/wordlists/dirb/common.txt
# With extensions
dirb https://target.com -X .php,.txt,.html,.js
# Output to file
dirb https://target.com -o /tmp/dirb_results.txt
# With custom wordlist and extensions
dirb https://target.com /usr/share/wordlists/dirb/common.txt -X .php,.txt,.html
# Verbose output
dirb https://target.com -v
# Recursive mode
dirb https://target.com -r
# With delay
dirb https://target.com -d 1
# With custom User-Agent
dirb https://target.com -a "Mozilla/5.0"
# With proxy
dirb https://target.com -p http://127.0.0.1:8080
# With authentication
dirb https://target.com -u username:password
# Skip certain status codes
dirb https://target.com -S
# Only show certain status codes
dirb https://target.com -c 200,301,403
dirb Flags Reference
| Flag |
Description |
Example |
-X |
Extensions |
-X .php,.txt |
-x |
Extensions (alternate) |
-x .php,.txt |
-r |
Recursive |
-r |
-R |
Recursive depth |
-R 3 |
-f |
Don’t append / |
-f |
-n |
No color |
-n |
-O |
OS detection |
-O |
-s |
Status codes |
-s 200,301,403 |
-S |
Skip status codes |
-S |
-t |
Delay (seconds) |
-t 1 |
-T |
Timeout (seconds) |
-T 10 |
-u |
Authentication |
-u user:pass |
-a |
User-Agent |
-a "Mozilla/5.0" |
-p |
Proxy |
-p http://127.0.0.1:8080 |
-o |
Output file |
-o results.txt |
-c |
Cookies |
-c "session=abc" |
-H |
Custom header |
-H "X-Forwarded-For: 127.0.0.1" |
-v |
Verbose |
-v |
-l |
Lowercase only |
-l |
-U |
Uppercase only |
-U |
-i |
Ignore case |
-i |
-k |
Keep alive |
-k |
-L |
Follow redirects |
-L |
dirb Output Example
$ dirb https://target.com
-----------------
DIRB v2.22
By The Dark Raver
-----------------
START_TIME: Mon Jan 01 12:00:00 2024
URL_BASE: https://target.com/
WORDLIST_FILES: /usr/share/dirb/wordlists/common.txt
-----------------
GENERATED WORDS: 4612
---- Scanning URL: https://target.com/ ----
+ https://target.com/admin (CODE:301|SIZE:234)
+ https://target.com/backup (CODE:403|SIZE:123)
+ https://target.com/config.php (CODE:200|SIZE:567)
+ https://target.com/index.html (CODE:200|SIZE:8901)
+ https://target.com/login.php (CODE:200|SIZE:2345)
+ https://target.com/server-status (CODE:200|SIZE:4567)
+ https://target.com/wp-admin (CODE:301|SIZE:234)
---- Entering directory: https://target.com/admin/ ----
+ https://target.com/admin/config.php (CODE:200|SIZE:890)
+ https://target.com/admin/login.php (CODE:200|SIZE:1234)
-----------------
END_TIME: Mon Jan 01 12:05:00 2024
DOWNLOADED: 4612 - FOUND: 9
-----------------
5. Directory Enumeration Wordlists
SecLists Wordlists
SecLists is the most comprehensive collection of wordlists for penetration testing.
# Clone SecLists
git clone --depth 1 https://github.com/danielmiessler/SecLists.git /usr/share/SecLists
# Common directory wordlists
ls /usr/share/SecLists/Discovery/Web-Content/
# Most commonly used wordlists:
# - common.txt (4,612 entries)
# - common.txt (20,000 entries)
# - directory-list-2.3-small.txt
# - directory-list-2.3-medium.txt
# - directory-list-2.3-large.txt
# - raft-medium-directories-lowercase.txt
# - raft-large-directories-lowercase.txt
SecLists Wordlist Sizes
| Wordlist |
Entries |
Use Case |
common.txt |
4,612 |
Quick scan |
common.txt (extended) |
20,000 |
Standard scan |
directory-list-2.3-small.txt |
5,000 |
Quick scan |
directory-list-2.3-medium.txt |
20,000 |
Standard scan |
directory-list-2.3-large.txt |
4,300,000 |
Exhaustive scan |
raft-medium-directories-lowercase.txt |
25,000 |
Medium scan |
raft-large-directories-lowercase.txt |
2,200,000 |
Large scan |
raft-huge-directories-lowercase.txt |
4,500,000 |
Huge scan |
Technology-Specific Wordlists
| Wordlist |
Technology |
Path |
apache.txt |
Apache |
SecLists/Discovery/Web-Content/apache.txt |
aspnet-common.txt |
ASP.NET |
SecLists/Discovery/Web-Content/aspnet-common.txt |
coldfusion.txt |
ColdFusion |
SecLists/Discovery/Web-Content/coldfusion.txt |
drupal.txt |
Drupal |
SecLists/Discovery/Web-Content/drupal.txt |
jboss.txt |
JBoss |
SecLists/Discovery/Web-Content/jboss.txt |
jsp.txt |
JSP |
SecLists/Discovery/Web-Content/jsp.txt |
laravel.txt |
Laravel |
SecLists/Discovery/Web-Content/laravel.txt |
nodejs.txt |
Node.js |
SecLists/Discovery/Web-Content/nodejs.txt |
php.txt |
PHP |
SecLists/Discovery/Web-Content/php.txt |
rails.txt |
Ruby on Rails |
SecLists/Discovery/Web-Content/rails.txt |
tomcat.txt |
Tomcat |
SecLists/Discovery/Web-Content/tomcat.txt |
wordpress.txt |
WordPress |
SecLists/Discovery/Web-Content/wordpress.txt |
django.txt |
Django |
SecLists/Discovery/Web-Content/django.txt |
flask.txt |
Flask |
SecLists/Discovery/Web-Content/flask.txt |
Raft Wordlists
# Raft wordlists location in Kali
ls /usr/share/wordlists/raft/
# Common Raft wordlists:
# - raft-medium-directories-lowercase.txt
# - raft-large-directories-lowercase.txt
# - raft-huge-directories-lowercase.txt
# - raft-directories-lowercase.txt
Creating Custom Wordlists
Method 1: From Target Website
# Extract words from the target website
curl -s https://target.com | \
grep -oE '[a-zA-Z0-9_-]+' | \
sort -u > /tmp/custom_wordlist.txt
# Extract from multiple pages
for page in index about contact team services products; do
curl -s "https://target.com/$page" | grep -oE '[a-zA-Z0-9_-]+'
done | sort -u > /tmp/custom_wordlist.txt
# Extract JavaScript variable names
curl -s https://target.com | \
grep -oE 'var [a-zA-Z0-9_]+' | \
sed 's/var //' | \
sort -u > /tmp/js_variables.txt
Method 2: From DNS Records
# Use subdomain names as directory names
subfinder -d target.com | \
cut -d'.' -f1 | \
sort -u > /tmp/dns_wordlist.txt
# Combine with common prefixes
for sub in $(subfinder -d target.com | cut -d'.' -f1); do
echo "$sub"
echo "api-$sub"
echo "old-$sub"
echo "new-$sub"
echo "dev-$sub"
echo "test-$sub"
done | sort -u > /tmp/dns_wordlist.txt
Method 3: From Employee Names
# Use employee names as directory names
cat /tmp/employees.txt | while read name; do
echo "$name"
echo "${name,,}" # lowercase
echo "${name// /-}" # hyphenated
echo "${name// /_}" # underscored
done | sort -u > /tmp/name_wordlist.txt
Method 4: Combination Wordlist
# Combine multiple wordlists
cat /usr/share/wordlists/dirb/common.txt \
/usr/share/SecLists/Discovery/Web-Content/common.txt \
/tmp/custom_wordlist.txt \
/tmp/dns_wordlist.txt \
/tmp/name_wordlist.txt | \
sort -u > /tmp/combined_wordlist.txt
# Count entries
wc -l /tmp/combined_wordlist.txt
Wordlist Optimization
# Remove duplicates
sort -u /tmp/wordlist.txt -o /tmp/wordlist_unique.txt
# Filter by length
awk 'length >= 3 && length <= 30' /tmp/wordlist.txt > /tmp/wordlist_filtered.txt
# Remove special characters
grep -E '^[a-zA-Z0-9._-]+$' /tmp/wordlist.txt > /tmp/wordlist_clean.txt
# Convert to lowercase
tr '[:upper:]' '[:lower:]' < /tmp/wordlist.txt > /tmp/wordlist_lower.txt
# Add extensions to each entry
while read word; do
echo "$word"
echo "$word.php"
echo "$word.txt"
echo "$word.html"
echo "$word.js"
echo "$word.json"
echo "$word.xml"
echo "$word.bak"
echo "$word.old"
echo "$word.backup"
done < /tmp/wordlist.txt > /tmp/wordlist_with_extensions.txt
6. Directory Enumeration Workflow
Comprehensive Workflow Diagram
graph TD
A[Start: Target URL] --> B[Check robots.txt]
B --> C[Check sitemap.xml]
C --> D[Quick Scan: common.txt]
D --> E{Found Interesting?}
E -->|Yes| F[Analyze Findings]
E -->|No| G[Medium Scan: directory-list-2.3-medium.txt]
G --> H{Found Interesting?}
H -->|Yes| F
H -->|No| I[Large Scan: raft-large.txt]
I --> J{Found Interesting?}
J -->|Yes| F
J -->|No| K[Custom Wordlist Scan]
K --> F
F --> L[Recursive Scan on Found Dirs]
L --> M[Extension Scan]
M --> N[Virtual Host Discovery]
N --> O[Hidden File Check]
O --> P[Analyze and Document]
style A fill:#e1f5fe
style P fill:#e8f5e9
style F fill:#fff3e0
Step-by-Step Workflow
Step 1: Initial Reconnaissance
# Check robots.txt
curl -s https://target.com/robots.txt
# Check sitemap.xml
curl -s https://target.com/sitemap.xml
# Check for common files
for file in robots.txt sitemap.xml .env .git/config web.config; do
status=$(curl -s -o /dev/null -w "%{http_code}" "https://target.com/$file")
if [ "$status" != "404" ]; then
echo "[+] Found: $file (Status: $status)"
fi
done
Step 2: Quick Scan
# Quick scan with common wordlist
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-t 50 \
-q \
-o /tmp/gobuster_quick.txt
# Or with ffuf
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-t 50 \
-mc 200,301,302,403 \
-fc 404 \
-o /tmp/ffuf_quick.json \
-of json
Step 3: Analyze Quick Scan Results
# Review results
cat /tmp/gobuster_quick.txt
# Look for interesting findings
grep -E "admin|login|api|backup|config|debug|console|dashboard|manager|portal|status|test|upload" /tmp/gobuster_quick.txt
# Check response sizes for anomalies
sort -t' ' -k3 -n /tmp/gobuster_quick.txt | tail -20
Step 4: Medium Scan (if needed)
# Medium scan with larger wordlist
gobuster dir -u https://target.com \
-w /usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-medium.txt \
-t 50 \
-q \
-o /tmp/gobuster_medium.txt
# With extensions
gobuster dir -u https://target.com \
-w /usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-medium.txt \
-t 50 \
-x php,txt,html,js,css,xml,json,yml,yaml,sql,log,conf,config,backup,bak,old \
-q \
-o /tmp/gobuster_medium_ext.txt
Step 5: Recursive Scan on Interesting Directories
# Recursive scan on found directories
for dir in admin api backup config; do
echo "[*] Scanning /$dir/"
gobuster dir -u "https://target.com/$dir" \
-w /usr/share/wordlists/dirb/common.txt \
-t 50 \
-q \
-o "/tmp/gobuster_${dir}.txt"
done
Step 6: Extension Scan
# Scan with multiple extensions
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-t 50 \
-x php,asp,aspx,jsp,html,htm,txt,js,css,xml,json,yml,yaml,sql,log,conf,config,backup,bak,old,swp,save,tmp \
-q \
-o /tmp/gobuster_extensions.txt
Step 7: Hidden File Check
# Check for hidden files
for file in .git .env .htaccess .htpasswd .svn .DS_Store .well-known security.txt; do
status=$(curl -s -o /dev/null -w "%{http_code}" "https://target.com/$file")
if [ "$status" != "404" ]; then
echo "[+] Found: $file (Status: $status)"
fi
done
Step 8: Virtual Host Discovery
# Vhost discovery
gobuster vhost -u https://target.com \
-w /usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt \
-t 50 \
-q \
-o /tmp/gobuster_vhost.txt
Step 9: Document Findings
# Create summary report
echo "=== Directory Enumeration Report ===" > /tmp/dir_enum_report.txt
echo "Target: https://target.com" >> /tmp/dir_enum_report.txt
echo "Date: $(date)" >> /tmp/dir_enum_report.txt
echo "" >> /tmp/dir_enum_report.txt
echo "--- Quick Scan Results ---" >> /tmp/dir_enum_report.txt
cat /tmp/gobuster_quick.txt >> /tmp/dir_enum_report.txt
echo "" >> /tmp/dir_enum_report.txt
echo "--- Medium Scan Results ---" >> /tmp/dir_enum_report.txt
cat /tmp/gobuster_medium.txt >> /tmp/dir_enum_report.txt
echo "" >> /tmp/dir_enum_report.txt
echo "--- Hidden Files ---" >> /tmp/dir_enum_report.txt
for file in .git .env .htaccess .htpasswd .svn .DS_Store; do
status=$(curl -s -o /dev/null -w "%{http_code}" "https://target.com/$file")
if [ "$status" != "404" ]; then
echo "Found: $file (Status: $status)" >> /tmp/dir_enum_report.txt
fi
done
7. Handling Large Wordlists
Challenges with Large Wordlists
| Challenge |
Impact |
Solution |
| Time |
Hours to complete |
Use progressive scanning |
| Storage |
Large output files |
Filter and compress |
| False Positives |
Many irrelevant results |
Use smart filtering |
| Rate Limiting |
IP blocked |
Use delays and proxies |
| Resource Usage |
High CPU/memory |
Limit threads |
Progressive Scanning Strategy
# Phase 1: Quick scan (small wordlist)
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-t 50 -q -o /tmp/phase1.txt
# Phase 2: Medium scan (if phase 1 found interesting results)
gobuster dir -u https://target.com \
-w /usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-medium.txt \
-t 50 -q -o /tmp/phase2.txt
# Phase 3: Large scan (only if necessary)
gobuster dir -u https://target.com \
-w /usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-large.txt \
-t 25 -q -o /tmp/phase3.txt
# Phase 4: Custom wordlist (target-specific)
gobuster dir -u https://target.com \
-w /tmp/custom_wordlist.txt \
-t 50 -q -o /tmp/phase4.txt
Wordlist Filtering
# Filter wordlist by size
wc -l /usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-large.txt
# Take top N entries
head -10000 /usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-large.txt > /tmp/top10000.txt
# Filter by relevance
grep -E "admin|api|backup|config|debug|login|upload|test|dev|staging" \
/usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-large.txt > /tmp/relevant.txt
# Remove very long entries
awk 'length <= 20' /usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-large.txt > /tmp/short.txt
Parallel Scanning
# Split wordlist and scan in parallel
split -l 10000 /usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-large.txt /tmp/wordlist_part_
# Scan each part in parallel
for part in /tmp/wordlist_part_*; do
gobuster dir -u https://target.com \
-w "$part" \
-t 25 -q \
-o "${part}.results.txt" &
done
# Wait for all to complete
wait
# Combine results
cat /tmp/wordlist_part_*.results.txt | sort -u > /tmp/combined_results.txt
Rate Limiting and Evasion
# Use delay between requests
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
--delay 500ms
# Use lower thread count
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-t 10
# Use proxy rotation
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-p http://127.0.0.1:8080
# Use random User-Agent
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
--random-agent
8. Recursive Enumeration Strategies
Why Recursive Enumeration?
Recursive enumeration discovers nested directories that a flat scan would miss:
Flat scan finds:
/admin
Recursive scan finds:
/admin
/admin/config
/admin/config/backup
/admin/users
/admin/users/export
gobuster Recursive Mode
# Basic recursive scan
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-r
# With depth limit
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-r \
--delay 500ms
# With extensions
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-r \
-x php,txt,html
ffuf Recursive Mode
# Basic recursive scan
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-recursion
# With depth limit
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-recursion \
-recursion-depth 3
# With extensions and recursion
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-recursion \
-recursion-depth 2 \
-e .php,.txt,.html
Manual Recursive Strategy
# Manual recursive approach
FOUND_DIRS=("/admin" "/api" "/backup")
for dir in "${FOUND_DIRS[@]}"; do
echo "[*] Scanning: $dir"
gobuster dir -u "https://target.com${dir}" \
-w /usr/share/wordlists/dirb/common.txt \
-t 50 -q \
-o "/tmp/recursive_${dir//\//_}.txt"
# Extract new directories from results
NEW_DIRS=$(grep -E "\(Status: (200|301|302|403)\)" "/tmp/recursive_${dir//\//_}.txt" | \
awk '{print $1}' | grep -v "\.")
for new_dir in $NEW_DIRS; do
echo "[+] New directory found: ${dir}${new_dir}"
FOUND_DIRS+=("${dir}${new_dir}")
done
done
Recursive Enumeration Best Practices
| Practice |
Reason |
| Limit depth |
Prevents excessive scanning |
| Use delays |
Avoids rate limiting |
| Filter results |
Reduces false positives |
| Document findings |
Tracks discovered paths |
| Prioritize targets |
Focus on interesting directories |
| Use smaller wordlists |
Faster recursive scans |
9. Virtual Host Discovery
What are Virtual Hosts?
Virtual hosts allow multiple websites to run on a single server/IP address. Discovering virtual hosts can reveal:
- Hidden applications — Internal tools and admin panels
- Development environments — Less secured staging sites
- Legacy applications — Older, potentially vulnerable systems
- Internal services — Services not exposed to the public
gobuster vhost Mode
# Basic vhost discovery
gobuster vhost -u https://target.com \
-w /usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt
# With thread count
gobuster vhost -u https://target.com \
-w /usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt \
-t 50
# Output to file
gobuster vhost -u https://target.com \
-w /usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt \
-o /tmp/vhosts.txt
# With domain specification
gobuster vhost -u https://target.com \
-w /usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt \
-d target.com
ffuf vhost Discovery
# Vhost discovery with ffuf
ffuf -u https://target.com \
-w /usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt \
-H "Host: FUZZ.target.com" \
-mc 200,301,302,403 \
-fc 404 \
-t 50 \
-o /tmp/vhosts_ffuf.json \
-of json
# With different domain patterns
ffuf -u https://target.com \
-w /usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt \
-H "Host: FUZZ" \
-mc 200,301,302,403 \
-fc 404
# With multiple headers
ffuf -u https://target.com \
-w /usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt \
-H "Host: FUZZ.target.com" \
-H "X-Forwarded-Host: FUZZ.target.com" \
-mc 200,301,302,403
Manual vhost Discovery
# Manual vhost testing
for vhost in admin dev test staging internal portal api; do
response=$(curl -s -o /dev/null -w "%{http_code}" \
-H "Host: ${vhost}.target.com" \
https://target.com)
if [ "$response" != "404" ] && [ "$response" != "000" ]; then
echo "[+] Possible vhost: ${vhost}.target.com (Status: $response)"
fi
done
vhost Wordlists
# Common vhost wordlists
/usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt
/usr/share/SecLists/Discovery/DNS/subdomains-top1million-110000.txt
/usr/share/wordlists/vhosts/vhost-list.txt
# Create custom vhost wordlist
cat > /tmp/custom_vhosts.txt << 'EOF'
admin
dev
test
staging
internal
portal
api
app
web
mail
vpn
intranet
extranet
dashboard
console
manager
status
monitor
EOF
10. File Extension Discovery
Why Extension Discovery Matters
Different file extensions reveal different technologies and vulnerabilities:
| Extension |
Technology |
Vulnerability Types |
.php |
PHP |
RCE, LFI, SQLi |
.asp |
ASP |
RCE, LFI |
.aspx |
ASP.NET |
RCE, LFI, deserialization |
.jsp |
Java |
RCE, LFI, deserialization |
.py |
Python |
RCE, LFI |
.rb |
Ruby |
RCE, LFI |
.js |
JavaScript |
XSS, API exposure |
.json |
JSON |
Data exposure |
.xml |
XML |
XXE, data exposure |
.yaml |
YAML |
Config exposure |
.yml |
YAML |
Config exposure |
.sql |
SQL |
Database dumps |
.log |
Log |
Credential exposure |
.conf |
Config |
Credential exposure |
.config |
Config |
Credential exposure |
.bak |
Backup |
Source code exposure |
.old |
Backup |
Source code exposure |
.backup |
Backup |
Source code exposure |
.swp |
Swap |
Source code exposure |
.save |
Save |
Source code exposure |
.tmp |
Temp |
Sensitive data |
Extension Discovery with gobuster
# Single extension
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-x php
# Multiple extensions
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-x php,asp,aspx,jsp,html,htm,txt,js,css,xml,json,yml,yaml
# All common extensions
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-x php,asp,aspx,jsp,html,htm,txt,js,css,xml,json,yml,yaml,sql,log,conf,config,backup,bak,old,swp,save,tmp,cgi,pl,py,rb,sh,bat,cmd,ps1
Extension Discovery with ffuf
# Single extension
ffuf -u https://target.com/FUZZ.php \
-w /usr/share/wordlists/dirb/common.txt
# Multiple extensions with FUZZ
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-e .php,.asp,.aspx,.jsp,.html,.htm,.txt,.js,.css,.xml,.json
# Extension-specific wordlist
ffuf -u https://target.com/FUZZ \
-w /usr/share/SecLists/Discovery/Web-Content/php.txt
Technology-Specific Extension Scans
# PHP application
gobuster dir -u https://target.com \
-w /usr/share/SecLists/Discovery/Web-Content/php.txt \
-x php,inc,phtml,php3,php4,php5,phps
# ASP.NET application
gobuster dir -u https://target.com \
-w /usr/share/SecLists/Discovery/Web-Content/aspnet-common.txt \
-x aspx,ashx,asmx,aspx.cs,aspx.vb,config
# Java application
gobuster dir -u https://target.com \
-w /usr/share/SecLists/Discovery/Web-Content/jsp.txt \
-x jsp,jspf,do,action,jsf
# Node.js application
gobuster dir -u https://target.com \
-w /usr/share/SecLists/Discovery/Web-Content/nodejs.txt \
-x js,json,html
11. Common Hidden Files
Critical Hidden Files to Check
Version Control Files
| File |
Content |
Exploitation |
/.git/ |
Git repository |
Full source code recovery |
/.git/config |
Git configuration |
Remote repository URL |
/.git/HEAD |
Current branch |
Repository structure |
/.svn/ |
SVN repository |
Source code exposure |
/.hg/ |
Mercurial repository |
Source code exposure |
/.bzr/ |
Bazaar repository |
Source code exposure |
Environment and Configuration Files
| File |
Content |
Exploitation |
/.env |
Environment variables |
Database credentials, API keys |
/.env.local |
Local environment |
Development credentials |
/.env.backup |
Backup environment |
Old credentials |
/.htaccess |
Apache configuration |
Server settings |
/.htpasswd |
Apache passwords |
User credentials |
/web.config |
IIS configuration |
Application settings |
/config.php |
PHP configuration |
Database credentials |
/configuration.php |
PHP configuration |
Database credentials |
/settings.py |
Python settings |
Application settings |
/config.yml |
YAML configuration |
Application settings |
/config.json |
JSON configuration |
Application settings |
| File |
Content |
Exploitation |
/.DS_Store |
macOS metadata |
Directory structure |
/Thumbs.db |
Windows thumbnails |
File names |
/.well-known/security.txt |
Security policy |
Contact information |
/.well-known/acme-challenge/ |
Let’s Encrypt |
Domain validation |
/robots.txt |
Crawl directives |
Hidden directory hints |
/sitemap.xml |
Site map |
Complete site structure |
Backup and Temporary Files
| File |
Content |
Exploitation |
/index.php.bak |
Backup PHP file |
Source code |
/index.php.old |
Old PHP file |
Source code |
/index.php.swp |
Swap file |
Source code |
/index.php.save |
Save file |
Source code |
/index.php~ |
Backup file |
Source code |
/database.sql |
Database dump |
Full database |
/dump.sql |
Database dump |
Full database |
/backup.sql |
Database dump |
Full database |
/db.sql |
Database dump |
Full database |
Automated Hidden File Check
#!/bin/bash
# Check for common hidden files
TARGET="https://target.com"
HIDDEN_FILES=(
".git/config"
".git/HEAD"
".env"
".env.local"
".env.backup"
".htaccess"
".htpasswd"
".svn/entries"
".DS_Store"
"web.config"
"robots.txt"
"sitemap.xml"
".well-known/security.txt"
"config.php"
"configuration.php"
"settings.py"
"config.yml"
"config.json"
"Thumbs.db"
)
echo "=== Hidden File Check ==="
echo "Target: $TARGET"
echo ""
for file in "${HIDDEN_FILES[@]}"; do
status=$(curl -s -o /dev/null -w "%{http_code}" "${TARGET}/${file}")
size=$(curl -s -o /dev/null -w "%{size_download}" "${TARGET}/${file}")
if [ "$status" != "404" ] && [ "$status" != "000" ]; then
echo "[+] FOUND: /${file} (Status: ${status}, Size: ${size})"
fi
done
.git Directory Exploitation
# Check if .git is accessible
curl -s https://target.com/.git/config
# Download entire .git directory
git clone https://target.com/.git /tmp/target_git
# Or use git-dumper
git clone https://github.com/arthaud/git-dumper.git
cd git-dumper
python3 git-dumper.py https://target.com/.git /tmp/target_dump
# Or use dg (dumpgit)
pip3 install dumpgit
dg https://target.com/.git /tmp/target_dump
# Recover source code
cd /tmp/target_dump
git log --oneline
git checkout master
ls -la
.env File Exploitation
# Check for .env file
curl -s https://target.com/.env
# Common .env contents:
# DB_HOST=localhost
# DB_DATABASE=app_database
# DB_USERNAME=root
# DB_PASSWORD=secret_password
# APP_KEY=base64:xxxxxxxx
# AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
# AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
# MAIL_PASSWORD=secret
# REDIS_PASSWORD=secret
12. Interpreting Status Codes
HTTP Status Code Reference
Success Codes (2xx)
| Code |
Meaning |
Directory Enumeration Value |
| 200 |
OK |
Directory/file exists and is accessible |
| 201 |
Created |
Resource created (POST response) |
| 204 |
No Content |
Exists but no content returned |
Redirection Codes (3xx)
| Code |
Meaning |
Directory Enumeration Value |
| 301 |
Moved Permanently |
Directory exists, redirects elsewhere |
| 302 |
Found (Temporary) |
Directory exists, temporary redirect |
| 307 |
Temporary Redirect |
Directory exists, preserve method |
| 308 |
Permanent Redirect |
Directory exists, preserve method |
Client Error Codes (4xx)
| Code |
Meaning |
Directory Enumeration Value |
| 400 |
Bad Request |
Malformed request |
| 401 |
Unauthorized |
Exists but requires authentication |
| 403 |
Forbidden |
Exists but access denied |
| 404 |
Not Found |
Directory/file does not exist |
| 405 |
Method Not Allowed |
Exists but method not allowed |
| 407 |
Proxy Auth Required |
Proxy authentication needed |
| 408 |
Request Timeout |
Server timeout |
| 429 |
Too Many Requests |
Rate limited |
Server Error Codes (5xx)
| Code |
Meaning |
Directory Enumeration Value |
| 500 |
Internal Server Error |
May exist, server error |
| 502 |
Bad Gateway |
Backend error |
| 503 |
Service Unavailable |
Server overloaded |
| 504 |
Gateway Timeout |
Backend timeout |
Status Code Interpretation Strategy
graph TD
A[HTTP Response] --> B{Status Code}
B -->|200| C[Exists - Accessible]
B -->|204| D[Exists - No Content]
B -->|301| E[Exists - Permanent Redirect]
B -->|302| F[Exists - Temporary Redirect]
B -->|401| G[Exists - Auth Required]
B -->|403| H[Exists - Access Denied]
B -->|404| I[Does Not Exist]
B -->|405| J[Exists - Method Not Allowed]
B -->|500| K[May Exist - Server Error]
B -->|Other| L[Investigate Further]
style C fill:#c8e6c9
style G fill:#fff9c4
style H fill:#fff9c4
style I fill:#ffcdd2
style K fill:#ffe0b2
Handling Wildcard Responses
Some servers return the same response for all requests (wildcard responses). This makes enumeration difficult.
# Detect wildcard responses
# 1. Check response for non-existent path
curl -s -o /dev/null -w "%{http_code} %{size_download}" https://target.com/nonexistent12345
# 2. Check response for another non-existent path
curl -s -o /dev/null -w "%{http_code} %{size_download}" https://target.com/nonexistent67890
# 3. If both return same status and size, it's likely a wildcard
# Handle wildcard with gobuster
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
--wildcard
# Handle wildcard with ffuf (auto-calibration)
ffuf -u https://target.com/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-ac
Response Size Analysis
# Analyze response sizes to find anomalies
gobuster dir -u https://target.com \
-w /usr/share/wordlists/dirb/common.txt \
-o /tmp/gobuster_sizes.txt
# Sort by size to find outliers
sort -t' ' -k3 -n /tmp/gobuster_sizes.txt | tail -20
# Look for unusually large or small responses
awk '{if ($3 > 10000 || $3 < 100) print}' /tmp/gobuster_sizes.txt
13. OSCP Directory Enumeration Best Practices
OSCP-Specific Workflow
graph TD
A[Target IP/Domain] --> B[Identify Web Services]
B --> C[Check robots.txt]
C --> D[Quick gobuster Scan]
D --> E{Found Interesting?}
E -->|Yes| F[Deep Dive on Findings]
E -->|No| G[Extended Wordlist Scan]
G --> H{Found Interesting?}
H -->|Yes| F
H -->|No| I[Check Hidden Files]
I --> J{Found Interesting?}
J -->|Yes| F
J -->|No| K[Move to Next Target]
F --> L[Exploit Vulnerabilities]
OSCP Essential Commands
# Step 1: Quick scan
gobuster dir -u http://target.local \
-w /usr/share/wordlists/dirb/common.txt \
-t 50 -q
# Step 2: With extensions
gobuster dir -u http://target.local \
-w /usr/share/wordlists/dirb/common.txt \
-x php,txt,html,js,asp,aspx,jsp \
-t 50 -q
# Step 3: Check hidden files
for file in .git .env .htaccess .htpasswd .svn .DS_Store robots.txt sitemap.xml; do
curl -s -o /dev/null -w "%{http_code}" "http://target.local/$file" | grep -v "404" && echo "Found: $file"
done
# Step 4: Recursive on interesting dirs
gobuster dir -u http://target.local/admin \
-w /usr/share/wordlists/dirb/common.txt \
-t 50 -q
# Step 5: ffuf for advanced filtering
ffuf -u http://target.local/FUZZ \
-w /usr/share/wordlists/dirb/common.txt \
-mc 200,301,403 \
-fc 404 \
-t 50
OSCP Time Management
| Activity |
Recommended Time |
Priority |
| Quick gobuster scan |
5-10 minutes |
High |
| Extension scan |
10-15 minutes |
High |
| Hidden file check |
2-5 minutes |
High |
| Recursive scan |
10-20 minutes |
Medium |
| Extended wordlist scan |
15-30 minutes |
Medium |
| Virtual host discovery |
10-15 minutes |
Low |
| Total |
50-95 minutes |
— |
OSCP Tips and Tricks
- Start with common.txt — It’s small and fast, catches most common directories
- Always check robots.txt — It often reveals hidden directories
- Check for .git directories — Source code exposure is common in labs
- Look for .env files — Credentials are often exposed
- Use -x for extensions — Don’t miss PHP, ASP, and JSP files
- Follow redirects — 301/302 responses indicate existing directories
- Check 403 responses — Forbidden directories may have weak access controls
- Recursive scan admin panels — Admin directories often have subdirectories
- Document everything — Record all findings for your report
- Don’t spend too long — If nothing found after 30 minutes, move on
Common OSCP Lab Directories
| Directory |
Frequency |
Notes |
/admin |
Very Common |
Admin panel |
/api |
Common |
API endpoints |
/backup |
Common |
Backup files |
/config |
Common |
Configuration files |
/debug |
Common |
Debug information |
/files |
Common |
File storage |
/images |
Very Common |
Image storage |
/includes |
Common |
Include files |
/js |
Very Common |
JavaScript files |
/login |
Common |
Login page |
/logs |
Common |
Log files |
/phpmyadmin |
Common |
Database admin |
/scripts |
Common |
Script files |
/server-status |
Common |
Server status |
/src |
Common |
Source code |
/uploads |
Common |
Upload directory |
/wp-admin |
Common |
WordPress admin |
/.git |
Common |
Git repository |
/.env |
Common |
Environment variables |
Quick Reference Cheat Sheet
Essential Commands
# gobuster
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -t 50 -q
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -x php,txt,html -t 50 -q
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -r -t 50 -q
gobuster dns -d target.com -w /usr/share/wordlists/subdomains/top-10000.txt -t 50 -q
gobuster vhost -u https://target.com -w /usr/share/wordlists/vhosts/vhost-list.txt -t 50 -q
# ffuf
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -t 50 -mc 200,301,403 -fc 404
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -e .php,.txt,.html -t 50
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -recursion -recursion-depth 3
ffuf -u https://target.com -w /usr/share/wordlists/vhosts/vhost-list.txt -H "Host: FUZZ.target.com" -mc 200,301,403
# dirb
dirb https://target.com /usr/share/wordlists/dirb/common.txt -X .php,.txt,.html -r
# Hidden file check
for file in .git .env .htaccess .htpasswd .svn .DS_Store robots.txt sitemap.xml; do
curl -s -o /dev/null -w "%{http_code}" "https://target.com/$file" | grep -v "404" && echo "Found: $file"
done
# .git exploitation
git clone https://target.com/.git /tmp/target_git
python3 git-dumper.py https://target.com/.git /tmp/target_dump
Useful One-Liners
# Quick directory scan with common extensions
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -x php,txt,html,js,css,xml,json -t 50 -q -o /tmp/dir_scan.txt
# Find all PHP files
ffuf -u https://target.com/FUZZ.php -w /usr/share/wordlists/dirb/common.txt -mc 200,301,403 -fc 404 -t 50
# Find backup files
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -x bak,old,backup,swp,save,tmp,sql -t 50 -q
# Check for common admin panels
for dir in admin administrator panel dashboard console manager; do
status=$(curl -s -o /dev/null -w "%{http_code}" "https://target.com/$dir")
[ "$status" != "404" ] && echo "[+] Found: /$dir (Status: $status)"
done
# Extract directories from robots.txt
curl -s https://target.com/robots.txt | grep -oE 'Disallow: [^\s]+' | cut -d' ' -f2
# Combine multiple wordlists
cat /usr/share/wordlists/dirb/common.txt /usr/share/SecLists/Discovery/Web-Content/common.txt | sort -u > /tmp/combined.txt
Summary
Directory and file discovery is one of the most important techniques in web application penetration testing. Key takeaways:
- gobuster is the go-to tool for directory enumeration with multiple modes (dir, dns, vhost, s3)
- ffuf offers more flexibility with advanced filtering, multiple FUZZ positions, and recursion
- dirb is a simpler alternative with basic functionality
- Wordlist selection is critical — start small and progress to larger lists
- Recursive enumeration discovers nested directories that flat scans miss
- Virtual host discovery reveals hidden applications and internal services
- Extension discovery helps identify the technology stack
- Hidden files like .git, .env, and .htaccess often contain sensitive information
- Status code interpretation is essential for accurate results
- In OSCP, focus on quick scans with common.txt, hidden file checks, and recursive scanning of interesting directories
Master these techniques to thoroughly map web application attack surfaces and discover hidden entry points for exploitation.