Domain 1Recon & ReportingCORE

Directory and File Discovery

Comprehensive guide to directory and file discovery techniques for OSCP, covering gobuster, ffuf, dirb, wordlist strategies, recursive enumeration, virtual host discovery, status code interpretation, and hidden file detection.

Domain: Domain 1: Reconnaissance & Reporting•Updated: 2026-09-01
#directory-enumeration#gobuster#ffuf#web-discovery

Directory and File Discovery

Attack Chain Position

graph LR
    A[Target Identification] --> B[DNS Enumeration]
    B --> C[Port Scanning]
    C --> D[Service Identification]
    D --> E[Directory Enumeration]
    E --> F[Web Application Analysis]
    F --> G[Vulnerability Exploitation]
    E --> H[File Discovery]
    H --> F
    E --> I[Virtual Host Discovery]
    I --> F
    style E fill:#f9d56e,stroke:#333,stroke-width:2px

Directory and file discovery is one of the most critical techniques in web application penetration testing. It helps you:

  • Find hidden directories — Discover admin panels, APIs, and backup directories
  • Locate sensitive files — Find configuration files, backups, and logs
  • Map the web application — Understand the application structure and functionality
  • Identify attack surfaces — Find upload forms, login pages, and debug endpoints
  • Discover development artifacts — Find .git, .svn, and other version control directories

Table of Contents

  1. Why Directory Enumeration Matters
  2. gobuster Deep Dive
  3. ffuf Deep Dive
  4. dirb Usage
  5. Directory Enumeration Wordlists
  6. Directory Enumeration Workflow
  7. Handling Large Wordlists
  8. Recursive Enumeration Strategies
  9. Virtual Host Discovery
  10. File Extension Discovery
  11. Common Hidden Files
  12. Interpreting Status Codes
  13. OSCP Directory Enumeration Best Practices

1. Why Directory Enumeration Matters

The Value of Directory Enumeration

Directory enumeration is often the difference between finding a foothold and staring at a blank webpage. Here’s why it matters:

Discovery Type Example Value
Admin Panels /admin, /administrator, /wp-admin Direct access to management interfaces
API Endpoints /api/v1/, /api/v2/, /graphql Data exposure and manipulation
Backup Files /backup/, /backups/, /db_backup/ Database dumps and source code
Configuration Files /config/, /configuration/, /settings/ Credentials and settings
Debug Endpoints /debug/, /phpinfo.php, /server-status System information disclosure
Upload Forms /upload/, /uploads/, /file-upload/ File upload vulnerabilities
Login Pages /login, /signin, /auth/ Authentication bypass targets
Version Control /.git/, /.svn/, /.hg/ Source code exposure
Documentation /docs/, /documentation/, /wiki/ Internal documentation
Test/Staging /test/, /staging/, /dev/ Less secured environments

Common Directories and Their Value

High-Value Directories

Directory Typical Content Exploitation Potential
/admin Admin panel Full site control
/api API endpoints Data extraction, manipulation
/backup Backup files Database dumps, source code
/config Configuration files Credentials, settings
/console Management console System access
/dashboard User dashboard Account access
/debug Debug information System details, stack traces
/dev Development environment Less secured, debug enabled
/docs Documentation Internal information
/files File storage Sensitive documents
/images Image storage Image upload vulnerabilities
/includes Include files Source code exposure
/logs Log files Credentials, error messages
/manager Manager interface Application management
/portal User portal Account access
/scripts Script files Code execution
/server-status Server status System information
/soap SOAP endpoints API manipulation
/src Source code Full source exposure
/staging Staging environment Less secured
/status Status page System information
/test Test environment Debug features enabled
/tmp Temporary files Sensitive data
/upload Upload directory File upload vulnerabilities
/uploads Uploaded files Arbitrary file access
/web Web root Application structure
/wp-admin WordPress admin WordPress exploitation
/wp-content WordPress content Plugin/theme exploitation

Hidden Files and Directories

File/Directory Content Exploitation Potential
/.git/ Git repository Full source code recovery
/.env Environment variables Database credentials, API keys
/.htaccess Apache configuration Server configuration
/.svn/ SVN repository Source code exposure
/.DS_Store macOS metadata Directory structure
/.htpasswd Apache passwords Credential exposure
/web.config IIS configuration Application settings
/robots.txt Crawl directives Hidden directory hints
/sitemap.xml Site map Complete site structure
/.well-known/ Well-known URIs Security configuration

2. gobuster Deep Dive

Installation

# Kali Linux (pre-installed)
gobuster --version

# Manual installation
# Download from: https://github.com/OJ/gobuster/releases
wget https://github.com/OJ/gobuster/releases/download/vX.X.X/gobuster-linux-amd64.X.X.X.deb
sudo dpkg -i gobuster-linux-amd64.X.X.X.deb

# Go installation
go install github.com/OJ/gobuster/v3@latest

gobuster Modes

gobuster supports multiple modes:

Mode Command Purpose
dir gobuster dir Directory/file enumeration
dns gobuster dns Subdomain enumeration
vhost gobuster vhost Virtual host enumeration
s3 gobuster s3 AWS S3 bucket enumeration
gcs gobuster gcs Google Cloud Storage enumeration
azure gobuster azure Azure blob storage enumeration

gobuster dir Mode

Basic Usage

# Basic directory enumeration
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt

# With thread count
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -t 50

# With extensions
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -x php,txt,html,js,css,backup,bak,old,sql,log,conf,config,xml,json,yml,yaml

# Output to file
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -o /tmp/gobuster_results.txt

# JSON output
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -o /tmp/gobuster_results.json -j

# Verbose output
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -v

# Quiet mode (only found entries)
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -q

All gobuster dir Flags

Flag Short Description Example
-u, --url Target URL -u https://target.com
-w, --wordlist Wordlist file -w /path/to/wordlist
-t, --threads Number of threads -t 50
-e, --expanded Expanded progress output -e
-r, --no-response-storage Don’t store responses -r
-k, --no-tls-validation Skip TLS validation -k
-n, --no-number Don’t display number of matches -n
-c, --no-color Disable color output -c
-f, --discover-backup Discover backup files -f
-x, --extensions File extensions -x php,txt,html
-L, --allow-no-files Allow wordlist with no files -L
-b, --status-codes Status codes to display -b 200,204,301,302,307,403
--delay Delay between requests --delay 1s
--timeout HTTP request timeout --timeout 10s
-a, --useragent HTTP User-Agent header -a "Mozilla/5.0"
-H, --headers Custom headers -H "Cookie: session=abc"
-z, --hide-length Hide response length -z
-o, --output Output file -o results.txt
-j, --json JSON output format -j
-s, --status-codes Successful status codes -s 200,301,302,403
-d, --domain Domain for vhost mode -d target.com
-p, --proxy HTTP proxy -p http://127.0.0.1:8080
-S, --cookies Cookies -S "session=abc"
-m, --method HTTP method -m POST
-i, --include Include status codes -i 200,301,403
-exclude Exclude status codes --exclude 404
-r Follow redirects -r
-D Don’t follow redirects -D
-c No color -c
-v Verbose -v
-q Quiet mode -q
--random-agent Random User-Agent --random-agent
--encoding URL encoding --encoding utf-8
--wildcard Handle wildcard responses --wildcard
--wildcard-match Wildcard matching --wildcard-match

Advanced gobuster dir Usage

# With custom headers
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -H "Cookie: session=abc123" \
  -H "Authorization: Bearer token123" \
  -H "X-Forwarded-For: 127.0.0.1"

# With proxy
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -p http://127.0.0.1:8080

# With delay (for rate-limited targets)
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  --delay 500ms

# With custom status codes
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -s 200,204,301,302,307,403

# Exclude certain status codes
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  --exclude 404,500

# With POST method
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -m POST

# With custom User-Agent
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -a "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"

# Random User-Agent
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  --random-agent

# Handle wildcard responses
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  --wildcard

# Recursive enumeration
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -r

# With multiple extensions
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -x php,asp,aspx,jsp,html,htm,txt,js,css,xml,json,yml,yaml,sql,log,conf,config,backup,bak,old,swp,save,tmp

# Discover backup files
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -f

# No TLS validation (for self-signed certs)
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -k

gobuster dir Output Interpretation

$ gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -x php,txt,html

===============================================================
Gobuster v3.6
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url:                     https://target.com
[+] Method:                  GET
[+] Threads:                   10
[+] Wordlist:                  /usr/share/wordlists/dirb/common.txt
[+] Negative Status codes:     404
[+] User Agent:                gobuster/3.6
[+] Extensions:                php,txt,html
[+] Timeout:                   10s
===============================================================
Starting gobuster in directory enumeration mode
===============================================================
/.env               (Status: 200) [Size: 1234]
/admin              (Status: 301) [Size: 234] [--> https://target.com/admin/]
/backup             (Status: 403) [Size: 123]
/config.php         (Status: 200) [Size: 567]
/index.html         (Status: 200) [Size: 8901]
/login.php          (Status: 200) [Size: 2345]
/server-status      (Status: 200) [Size: 4567]
/wp-admin           (Status: 301) [Size: 234] [--> https://target.com/wp-admin/]
===============================================================
Finished
===============================================================

Output Fields Explained

Field Description
Path The discovered directory or file
Status HTTP status code returned
Size Response size in bytes
Redirect Redirect target (if applicable)

gobuster dns Mode

# Basic subdomain enumeration
gobuster dns -d target.com -w /usr/share/wordlists/subdomains/top-10000.txt

# With thread count
gobuster dns -d target.com -w /usr/share/wordlists/subdomains/top-10000.txt -t 50

# Output to file
gobuster dns -d target.com -w /usr/share/wordlists/subdomains/top-10000.txt -o /tmp/subdomains.txt

# Find all record types
gobuster dns -d target.com -w /usr/share/wordlists/subdomains/top-10000.txt -r

# Quiet mode
gobuster dns -d target.com -w /usr/share/wordlists/subdomains/top-10000.txt -q

gobuster vhost Mode

# Basic virtual host enumeration
gobuster vhost -u https://target.com -w /usr/share/wordlists/vhosts/vhost-list.txt

# With thread count
gobuster vhost -u https://target.com -w /usr/share/wordlists/vhosts/vhost-list.txt -t 50

# Output to file
gobuster vhost -u https://target.com -w /usr/share/wordlists/vhosts/vhost-list.txt -o /tmp/vhosts.txt

# With domain specification
gobuster vhost -u https://target.com -w /usr/share/wordlists/vhosts/vhost-list.txt -d target.com

gobuster s3 Mode

# AWS S3 bucket enumeration
gobuster s3 -w /usr/share/wordlists/buckets/s3bucket-names.txt

# With thread count
gobuster s3 -w /usr/share/wordlists/buckets/s3bucket-names.txt -t 50

# Show all buckets (including private)
gobuster s3 -w /usr/share/wordlists/buckets/s3bucket-names.txt -c

# Output to file
gobuster s3 -w /usr/share/wordlists/buckets/s3bucket-names.txt -o /tmp/s3_buckets.txt

3. ffuf Deep Dive

Installation

# Kali Linux (pre-installed)
ffuf -V

# Manual installation
# Download from: https://github.com/ffuf/ffuf/releases
wget https://github.com/ffuf/ffuf/releases/download/vX.X.X/ffuf_X.X.X_linux_amd64.tar.gz
tar -xzf ffuf_X.X.X_linux_amd64.tar.gz
sudo mv ffuf /usr/local/bin/

# Go installation
go install github.com/ffuf/ffuf/v2@latest

# Homebrew (macOS)
brew install ffuf

ffuf Basics

ffuf (Fuzz Faster U Fool) uses the FUZZ keyword as a placeholder for wordlist entries.

# Basic directory enumeration
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt

# With thread count
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -t 50

# With extensions
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -e .php,.txt,.html,.js,.css

# Output to file
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -o /tmp/ffuf_results.json -of json

# Verbose output
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -v

# Quiet mode
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -s

ffuf Filters and Matchers

Response Code Filters

# Filter by status code (show only 200)
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -mc 200

# Show multiple status codes
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -mc 200,301,302,403

# Filter out status codes (hide 404)
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fc 404

# Filter out multiple status codes
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fc 404,403,500

Response Size Filters

# Filter by response size (exact match)
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fs 1234

# Filter out response size
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fs 0

# Filter by response size range (using recode)
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fs 1234,5678

Word Count Filters

# Filter by word count in response
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fw 100

# Filter out by word count
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fw 0

Line Count Filters

# Filter by line count in response
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fl 50

# Filter out by line count
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fl 0

Regex Filters

# Filter by regex match in response
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -mr "admin|login|dashboard"

# Filter out by regex
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fr "not found|404"

# Filter by regex in headers
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -hr "Server: Apache"

Header Filters

# Match header value
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -mr "Content-Type: text/html"

# Filter header value
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fr "Content-Type: text/html"

ffuf Advanced Features

Multiple FUZZ Positions

# Multiple FUZZ keywords with different wordlists
ffuf -u https://target.com/FUZZ1/FUZZ2 \
  -w FUZZ1:/usr/share/wordlists/dirb/common.txt \
  -w FUZZ2:/usr/share/wordlists/extensions/by-ext/php.txt

# Example: This will try:
# https://target.com/admin/login.php
# https://target.com/admin/index.php
# https://target.com/backup/db.php
# etc.

Recursive Enumeration

# Recursive mode (max depth 3)
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -recursion \
  -recursion-depth 3

# With recursive extensions
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -recursion \
  -recursion-depth 2 \
  -e .php,.txt,.html

Virtual Host Discovery

# Vhost discovery using Host header
ffuf -u https://target.com \
  -w /usr/share/wordlists/vhosts/vhost-list.txt \
  -H "Host: FUZZ.target.com" \
  -mc 200,301,302,403 \
  -fc 404

# With multiple domains
ffuf -u https://target.com \
  -w /usr/share/wordlists/vhosts/vhost-list.txt \
  -H "Host: FUZZ" \
  -mc 200,301,302,403

POST Fuzzing

# POST body fuzzing
ffuf -u https://target.com/login \
  -w /usr/share/wordlists/usernames.txt \
  -X POST \
  -d "username=FUZZ&password=password123" \
  -mc 200,302

# POST with JSON body
ffuf -u https://target.com/api/login \
  -w /usr/share/wordlists/usernames.txt \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"username":"FUZZ","password":"password123"}' \
  -mc 200

# POST with file upload
ffuf -u https://target.com/upload \
  -w /usr/share/wordlists/files/php-reverse-shell.php \
  -X POST \
  -H "Content-Type: multipart/form-data" \
  -d "file=@FUZZ" \
  -mc 200,302
# Cookie fuzzing
ffuf -u https://target.com/admin \
  -w /usr/share/wordlists/sessions.txt \
  -H "Cookie: session=FUZZ" \
  -mc 200

# Multiple cookies
ffuf -u https://target.com/admin \
  -w /usr/share/wordlists/sessions.txt \
  -H "Cookie: session=FUZZ1; token=FUZZ2" \
  -w FUZZ1:/usr/share/wordlists/sessions.txt \
  -w FUZZ2:/usr/share/wordlists/tokens.txt \
  -mc 200

Rate Limiting

# Rate limiting with delay
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -rate 10  # 10 requests per second

# With delay between requests
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -rate 5 \
  -t 10

Auto-Calibration

# Auto-calibration to filter out common responses
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -ac  # Auto-calibration

# Auto-calibration with ignore
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -ac \
  -ic  # Ignore calibrations

Output Formats

# JSON output
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -o /tmp/ffuf_results.json \
  -of json

# CSV output
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -o /tmp/ffuf_results.csv \
  -of csv

# EJSON output (extended JSON with all data)
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -o /tmp/ffuf_results.ejson \
  -of ejson

# ECSV output (extended CSV)
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -o /tmp/ffuf_results.ecsv \
  -of ecsv

# HTTP output (full HTTP response)
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -o /tmp/ffuf_results.http \
  -of http

# MDJSON output (JSON with metadata)
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -o /tmp/ffuf_results.mdjson \
  -of mdjson

# TXT output (simple text)
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -o /tmp/ffuf_results.txt \
  -of txt

# All output formats
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -o /tmp/ffuf_results \
  -of all

ffuf All Flags Reference

Flag Description Example
-u, --url Target URL -u https://target.com/FUZZ
-w, --wordlist Wordlist(s) -w /path/to/wordlist
-ic, --ignore-calibration Ignore calibrations -ic
-ack, --calibration-keywords Calibration keywords -ack MYKEYWORD
-acc, --calibration-content-lengths Calibration content lengths -acc 1234,5678
-rate Rate limit (req/sec) -rate 10
-t, --threads Number of threads -t 50
-c, --color Enable colors -c
-nc, --no-color Disable colors -nc
-s, --silent Silent mode -s
-v, --verbose Verbose mode -v
-ac, --auto-calibration Auto-calibration -ac
-r, --follow-redir Follow redirects -r
-H, --header Custom header -H "Cookie: session=abc"
-X, --request HTTP method -X POST
-d, --data Request body -d "key=value"
-b, --cookie Cookie -b "session=abc"
-recursion Enable recursion -recursion
-recursion-depth Recursion depth -recursion-depth 3
-e, --extensions Extensions -e .php,.txt
-mc, --match-codes Match status codes -mc 200,301
-fc, --filter-codes Filter status codes -fc 404
-ml, --match-lines Match line count -ml 100
-fl, --filter-lines Filter line count -fl 0
-mw, --match-words Match word count -mw 100
-fw, --filter-words Filter word count -fw 0
-ms, --match-sizes Match sizes -ms 1234
-fs, --filter-sizes Filter sizes -fs 0
-mr, --match-regex Match regex -mr "admin"
-fr, --filter-regex Filter regex -fr "404"
-mh, --match-header Match header -mh "Server: Apache"
-fh, --filter-header Filter header -fh "Server: Apache"
-maxtime Max run time -maxtime 3600
-maxtime-retry Max retry time -maxtime-retry 60
-retry Retry failed requests -retry 3
-replay-proxy Replay proxy -replay-proxy http://127.0.0.1:8080
-proxy Proxy URL -proxy http://127.0.0.1:8080
-ignore-body Ignore response body -ignore-body
-noninteractive Disable interactive mode -noninteractive
-o Output file -o results.json
-of Output format -of json
-sa, --skip-auto-calibration Skip auto-calibration -sa
-sk, --skip-calibration Skip calibration -sk
-debug Debug mode -debug
-debug-request Debug request -debug-request
-json JSON input -json

ffuf vs gobuster Comparison

Feature ffuf gobuster
Speed Very Fast Fast
Flexibility Very High Medium
Filters Extensive Basic
Recursion Yes Yes
Vhost Mode Via headers Built-in
DNS Mode No Yes
S3 Mode No Yes
POST Fuzzing Yes Limited
Cookie Fuzzing Yes Limited
Multiple FUZZ Yes No
Rate Limiting Yes Yes
Output Formats Many Few
Auto-Calibration Yes No
Learning Curve Medium Low

4. dirb Usage

Installation

# Kali Linux (pre-installed)
dirb --version

# Manual installation
sudo apt install dirb

Basic Usage

# Basic directory enumeration
dirb https://target.com

# With custom wordlist
dirb https://target.com /usr/share/wordlists/dirb/common.txt

# With extensions
dirb https://target.com -X .php,.txt,.html,.js

# Output to file
dirb https://target.com -o /tmp/dirb_results.txt

# With custom wordlist and extensions
dirb https://target.com /usr/share/wordlists/dirb/common.txt -X .php,.txt,.html

# Verbose output
dirb https://target.com -v

# Recursive mode
dirb https://target.com -r

# With delay
dirb https://target.com -d 1

# With custom User-Agent
dirb https://target.com -a "Mozilla/5.0"

# With proxy
dirb https://target.com -p http://127.0.0.1:8080

# With authentication
dirb https://target.com -u username:password

# Skip certain status codes
dirb https://target.com -S

# Only show certain status codes
dirb https://target.com -c 200,301,403

dirb Flags Reference

Flag Description Example
-X Extensions -X .php,.txt
-x Extensions (alternate) -x .php,.txt
-r Recursive -r
-R Recursive depth -R 3
-f Don’t append / -f
-n No color -n
-O OS detection -O
-s Status codes -s 200,301,403
-S Skip status codes -S
-t Delay (seconds) -t 1
-T Timeout (seconds) -T 10
-u Authentication -u user:pass
-a User-Agent -a "Mozilla/5.0"
-p Proxy -p http://127.0.0.1:8080
-o Output file -o results.txt
-c Cookies -c "session=abc"
-H Custom header -H "X-Forwarded-For: 127.0.0.1"
-v Verbose -v
-l Lowercase only -l
-U Uppercase only -U
-i Ignore case -i
-k Keep alive -k
-L Follow redirects -L

dirb Output Example

$ dirb https://target.com

-----------------
DIRB v2.22
By The Dark Raver
-----------------

START_TIME: Mon Jan 01 12:00:00 2024
URL_BASE: https://target.com/
WORDLIST_FILES: /usr/share/dirb/wordlists/common.txt

-----------------

GENERATED WORDS: 4612

---- Scanning URL: https://target.com/ ----

+ https://target.com/admin (CODE:301|SIZE:234)
+ https://target.com/backup (CODE:403|SIZE:123)
+ https://target.com/config.php (CODE:200|SIZE:567)
+ https://target.com/index.html (CODE:200|SIZE:8901)
+ https://target.com/login.php (CODE:200|SIZE:2345)
+ https://target.com/server-status (CODE:200|SIZE:4567)
+ https://target.com/wp-admin (CODE:301|SIZE:234)

---- Entering directory: https://target.com/admin/ ----

+ https://target.com/admin/config.php (CODE:200|SIZE:890)
+ https://target.com/admin/login.php (CODE:200|SIZE:1234)

-----------------
END_TIME: Mon Jan 01 12:05:00 2024
DOWNLOADED: 4612 - FOUND: 9
-----------------

5. Directory Enumeration Wordlists

SecLists Wordlists

SecLists is the most comprehensive collection of wordlists for penetration testing.

# Clone SecLists
git clone --depth 1 https://github.com/danielmiessler/SecLists.git /usr/share/SecLists

# Common directory wordlists
ls /usr/share/SecLists/Discovery/Web-Content/

# Most commonly used wordlists:
# - common.txt (4,612 entries)
# - common.txt (20,000 entries)
# - directory-list-2.3-small.txt
# - directory-list-2.3-medium.txt
# - directory-list-2.3-large.txt
# - raft-medium-directories-lowercase.txt
# - raft-large-directories-lowercase.txt

SecLists Wordlist Sizes

Wordlist Entries Use Case
common.txt 4,612 Quick scan
common.txt (extended) 20,000 Standard scan
directory-list-2.3-small.txt 5,000 Quick scan
directory-list-2.3-medium.txt 20,000 Standard scan
directory-list-2.3-large.txt 4,300,000 Exhaustive scan
raft-medium-directories-lowercase.txt 25,000 Medium scan
raft-large-directories-lowercase.txt 2,200,000 Large scan
raft-huge-directories-lowercase.txt 4,500,000 Huge scan

Technology-Specific Wordlists

Wordlist Technology Path
apache.txt Apache SecLists/Discovery/Web-Content/apache.txt
aspnet-common.txt ASP.NET SecLists/Discovery/Web-Content/aspnet-common.txt
coldfusion.txt ColdFusion SecLists/Discovery/Web-Content/coldfusion.txt
drupal.txt Drupal SecLists/Discovery/Web-Content/drupal.txt
jboss.txt JBoss SecLists/Discovery/Web-Content/jboss.txt
jsp.txt JSP SecLists/Discovery/Web-Content/jsp.txt
laravel.txt Laravel SecLists/Discovery/Web-Content/laravel.txt
nodejs.txt Node.js SecLists/Discovery/Web-Content/nodejs.txt
php.txt PHP SecLists/Discovery/Web-Content/php.txt
rails.txt Ruby on Rails SecLists/Discovery/Web-Content/rails.txt
tomcat.txt Tomcat SecLists/Discovery/Web-Content/tomcat.txt
wordpress.txt WordPress SecLists/Discovery/Web-Content/wordpress.txt
django.txt Django SecLists/Discovery/Web-Content/django.txt
flask.txt Flask SecLists/Discovery/Web-Content/flask.txt

Raft Wordlists

# Raft wordlists location in Kali
ls /usr/share/wordlists/raft/

# Common Raft wordlists:
# - raft-medium-directories-lowercase.txt
# - raft-large-directories-lowercase.txt
# - raft-huge-directories-lowercase.txt
# - raft-directories-lowercase.txt

Creating Custom Wordlists

Method 1: From Target Website

# Extract words from the target website
curl -s https://target.com | \
  grep -oE '[a-zA-Z0-9_-]+' | \
  sort -u > /tmp/custom_wordlist.txt

# Extract from multiple pages
for page in index about contact team services products; do
  curl -s "https://target.com/$page" | grep -oE '[a-zA-Z0-9_-]+'
done | sort -u > /tmp/custom_wordlist.txt

# Extract JavaScript variable names
curl -s https://target.com | \
  grep -oE 'var [a-zA-Z0-9_]+' | \
  sed 's/var //' | \
  sort -u > /tmp/js_variables.txt

Method 2: From DNS Records

# Use subdomain names as directory names
subfinder -d target.com | \
  cut -d'.' -f1 | \
  sort -u > /tmp/dns_wordlist.txt

# Combine with common prefixes
for sub in $(subfinder -d target.com | cut -d'.' -f1); do
  echo "$sub"
  echo "api-$sub"
  echo "old-$sub"
  echo "new-$sub"
  echo "dev-$sub"
  echo "test-$sub"
done | sort -u > /tmp/dns_wordlist.txt

Method 3: From Employee Names

# Use employee names as directory names
cat /tmp/employees.txt | while read name; do
  echo "$name"
  echo "${name,,}"  # lowercase
  echo "${name// /-}"  # hyphenated
  echo "${name// /_}"  # underscored
done | sort -u > /tmp/name_wordlist.txt

Method 4: Combination Wordlist

# Combine multiple wordlists
cat /usr/share/wordlists/dirb/common.txt \
    /usr/share/SecLists/Discovery/Web-Content/common.txt \
    /tmp/custom_wordlist.txt \
    /tmp/dns_wordlist.txt \
    /tmp/name_wordlist.txt | \
  sort -u > /tmp/combined_wordlist.txt

# Count entries
wc -l /tmp/combined_wordlist.txt

Wordlist Optimization

# Remove duplicates
sort -u /tmp/wordlist.txt -o /tmp/wordlist_unique.txt

# Filter by length
awk 'length >= 3 && length <= 30' /tmp/wordlist.txt > /tmp/wordlist_filtered.txt

# Remove special characters
grep -E '^[a-zA-Z0-9._-]+$' /tmp/wordlist.txt > /tmp/wordlist_clean.txt

# Convert to lowercase
tr '[:upper:]' '[:lower:]' < /tmp/wordlist.txt > /tmp/wordlist_lower.txt

# Add extensions to each entry
while read word; do
  echo "$word"
  echo "$word.php"
  echo "$word.txt"
  echo "$word.html"
  echo "$word.js"
  echo "$word.json"
  echo "$word.xml"
  echo "$word.bak"
  echo "$word.old"
  echo "$word.backup"
done < /tmp/wordlist.txt > /tmp/wordlist_with_extensions.txt

6. Directory Enumeration Workflow

Comprehensive Workflow Diagram

graph TD
    A[Start: Target URL] --> B[Check robots.txt]
    B --> C[Check sitemap.xml]
    C --> D[Quick Scan: common.txt]
    D --> E{Found Interesting?}
    E -->|Yes| F[Analyze Findings]
    E -->|No| G[Medium Scan: directory-list-2.3-medium.txt]
    G --> H{Found Interesting?}
    H -->|Yes| F
    H -->|No| I[Large Scan: raft-large.txt]
    I --> J{Found Interesting?}
    J -->|Yes| F
    J -->|No| K[Custom Wordlist Scan]
    K --> F
    F --> L[Recursive Scan on Found Dirs]
    L --> M[Extension Scan]
    M --> N[Virtual Host Discovery]
    N --> O[Hidden File Check]
    O --> P[Analyze and Document]
    
    style A fill:#e1f5fe
    style P fill:#e8f5e9
    style F fill:#fff3e0

Step-by-Step Workflow

Step 1: Initial Reconnaissance

# Check robots.txt
curl -s https://target.com/robots.txt

# Check sitemap.xml
curl -s https://target.com/sitemap.xml

# Check for common files
for file in robots.txt sitemap.xml .env .git/config web.config; do
  status=$(curl -s -o /dev/null -w "%{http_code}" "https://target.com/$file")
  if [ "$status" != "404" ]; then
    echo "[+] Found: $file (Status: $status)"
  fi
done

Step 2: Quick Scan

# Quick scan with common wordlist
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -t 50 \
  -q \
  -o /tmp/gobuster_quick.txt

# Or with ffuf
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -t 50 \
  -mc 200,301,302,403 \
  -fc 404 \
  -o /tmp/ffuf_quick.json \
  -of json

Step 3: Analyze Quick Scan Results

# Review results
cat /tmp/gobuster_quick.txt

# Look for interesting findings
grep -E "admin|login|api|backup|config|debug|console|dashboard|manager|portal|status|test|upload" /tmp/gobuster_quick.txt

# Check response sizes for anomalies
sort -t' ' -k3 -n /tmp/gobuster_quick.txt | tail -20

Step 4: Medium Scan (if needed)

# Medium scan with larger wordlist
gobuster dir -u https://target.com \
  -w /usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-medium.txt \
  -t 50 \
  -q \
  -o /tmp/gobuster_medium.txt

# With extensions
gobuster dir -u https://target.com \
  -w /usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-medium.txt \
  -t 50 \
  -x php,txt,html,js,css,xml,json,yml,yaml,sql,log,conf,config,backup,bak,old \
  -q \
  -o /tmp/gobuster_medium_ext.txt

Step 5: Recursive Scan on Interesting Directories

# Recursive scan on found directories
for dir in admin api backup config; do
  echo "[*] Scanning /$dir/"
  gobuster dir -u "https://target.com/$dir" \
    -w /usr/share/wordlists/dirb/common.txt \
    -t 50 \
    -q \
    -o "/tmp/gobuster_${dir}.txt"
done

Step 6: Extension Scan

# Scan with multiple extensions
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -t 50 \
  -x php,asp,aspx,jsp,html,htm,txt,js,css,xml,json,yml,yaml,sql,log,conf,config,backup,bak,old,swp,save,tmp \
  -q \
  -o /tmp/gobuster_extensions.txt

Step 7: Hidden File Check

# Check for hidden files
for file in .git .env .htaccess .htpasswd .svn .DS_Store .well-known security.txt; do
  status=$(curl -s -o /dev/null -w "%{http_code}" "https://target.com/$file")
  if [ "$status" != "404" ]; then
    echo "[+] Found: $file (Status: $status)"
  fi
done

Step 8: Virtual Host Discovery

# Vhost discovery
gobuster vhost -u https://target.com \
  -w /usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt \
  -t 50 \
  -q \
  -o /tmp/gobuster_vhost.txt

Step 9: Document Findings

# Create summary report
echo "=== Directory Enumeration Report ===" > /tmp/dir_enum_report.txt
echo "Target: https://target.com" >> /tmp/dir_enum_report.txt
echo "Date: $(date)" >> /tmp/dir_enum_report.txt
echo "" >> /tmp/dir_enum_report.txt

echo "--- Quick Scan Results ---" >> /tmp/dir_enum_report.txt
cat /tmp/gobuster_quick.txt >> /tmp/dir_enum_report.txt
echo "" >> /tmp/dir_enum_report.txt

echo "--- Medium Scan Results ---" >> /tmp/dir_enum_report.txt
cat /tmp/gobuster_medium.txt >> /tmp/dir_enum_report.txt
echo "" >> /tmp/dir_enum_report.txt

echo "--- Hidden Files ---" >> /tmp/dir_enum_report.txt
for file in .git .env .htaccess .htpasswd .svn .DS_Store; do
  status=$(curl -s -o /dev/null -w "%{http_code}" "https://target.com/$file")
  if [ "$status" != "404" ]; then
    echo "Found: $file (Status: $status)" >> /tmp/dir_enum_report.txt
  fi
done

7. Handling Large Wordlists

Challenges with Large Wordlists

Challenge Impact Solution
Time Hours to complete Use progressive scanning
Storage Large output files Filter and compress
False Positives Many irrelevant results Use smart filtering
Rate Limiting IP blocked Use delays and proxies
Resource Usage High CPU/memory Limit threads

Progressive Scanning Strategy

# Phase 1: Quick scan (small wordlist)
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -t 50 -q -o /tmp/phase1.txt

# Phase 2: Medium scan (if phase 1 found interesting results)
gobuster dir -u https://target.com \
  -w /usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-medium.txt \
  -t 50 -q -o /tmp/phase2.txt

# Phase 3: Large scan (only if necessary)
gobuster dir -u https://target.com \
  -w /usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-large.txt \
  -t 25 -q -o /tmp/phase3.txt

# Phase 4: Custom wordlist (target-specific)
gobuster dir -u https://target.com \
  -w /tmp/custom_wordlist.txt \
  -t 50 -q -o /tmp/phase4.txt

Wordlist Filtering

# Filter wordlist by size
wc -l /usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-large.txt

# Take top N entries
head -10000 /usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-large.txt > /tmp/top10000.txt

# Filter by relevance
grep -E "admin|api|backup|config|debug|login|upload|test|dev|staging" \
  /usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-large.txt > /tmp/relevant.txt

# Remove very long entries
awk 'length <= 20' /usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-large.txt > /tmp/short.txt

Parallel Scanning

# Split wordlist and scan in parallel
split -l 10000 /usr/share/SecLists/Discovery/Web-Content/directory-list-2.3-large.txt /tmp/wordlist_part_

# Scan each part in parallel
for part in /tmp/wordlist_part_*; do
  gobuster dir -u https://target.com \
    -w "$part" \
    -t 25 -q \
    -o "${part}.results.txt" &
done

# Wait for all to complete
wait

# Combine results
cat /tmp/wordlist_part_*.results.txt | sort -u > /tmp/combined_results.txt

Rate Limiting and Evasion

# Use delay between requests
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  --delay 500ms

# Use lower thread count
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -t 10

# Use proxy rotation
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -p http://127.0.0.1:8080

# Use random User-Agent
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  --random-agent

8. Recursive Enumeration Strategies

Why Recursive Enumeration?

Recursive enumeration discovers nested directories that a flat scan would miss:

Flat scan finds:
  /admin

Recursive scan finds:
  /admin
  /admin/config
  /admin/config/backup
  /admin/users
  /admin/users/export

gobuster Recursive Mode

# Basic recursive scan
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -r

# With depth limit
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -r \
  --delay 500ms

# With extensions
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -r \
  -x php,txt,html

ffuf Recursive Mode

# Basic recursive scan
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -recursion

# With depth limit
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -recursion \
  -recursion-depth 3

# With extensions and recursion
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -recursion \
  -recursion-depth 2 \
  -e .php,.txt,.html

Manual Recursive Strategy

# Manual recursive approach
FOUND_DIRS=("/admin" "/api" "/backup")

for dir in "${FOUND_DIRS[@]}"; do
  echo "[*] Scanning: $dir"
  gobuster dir -u "https://target.com${dir}" \
    -w /usr/share/wordlists/dirb/common.txt \
    -t 50 -q \
    -o "/tmp/recursive_${dir//\//_}.txt"
  
  # Extract new directories from results
  NEW_DIRS=$(grep -E "\(Status: (200|301|302|403)\)" "/tmp/recursive_${dir//\//_}.txt" | \
    awk '{print $1}' | grep -v "\.")
  
  for new_dir in $NEW_DIRS; do
    echo "[+] New directory found: ${dir}${new_dir}"
    FOUND_DIRS+=("${dir}${new_dir}")
  done
done

Recursive Enumeration Best Practices

Practice Reason
Limit depth Prevents excessive scanning
Use delays Avoids rate limiting
Filter results Reduces false positives
Document findings Tracks discovered paths
Prioritize targets Focus on interesting directories
Use smaller wordlists Faster recursive scans

9. Virtual Host Discovery

What are Virtual Hosts?

Virtual hosts allow multiple websites to run on a single server/IP address. Discovering virtual hosts can reveal:

  • Hidden applications — Internal tools and admin panels
  • Development environments — Less secured staging sites
  • Legacy applications — Older, potentially vulnerable systems
  • Internal services — Services not exposed to the public

gobuster vhost Mode

# Basic vhost discovery
gobuster vhost -u https://target.com \
  -w /usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt

# With thread count
gobuster vhost -u https://target.com \
  -w /usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt \
  -t 50

# Output to file
gobuster vhost -u https://target.com \
  -w /usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt \
  -o /tmp/vhosts.txt

# With domain specification
gobuster vhost -u https://target.com \
  -w /usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt \
  -d target.com

ffuf vhost Discovery

# Vhost discovery with ffuf
ffuf -u https://target.com \
  -w /usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt \
  -H "Host: FUZZ.target.com" \
  -mc 200,301,302,403 \
  -fc 404 \
  -t 50 \
  -o /tmp/vhosts_ffuf.json \
  -of json

# With different domain patterns
ffuf -u https://target.com \
  -w /usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt \
  -H "Host: FUZZ" \
  -mc 200,301,302,403 \
  -fc 404

# With multiple headers
ffuf -u https://target.com \
  -w /usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt \
  -H "Host: FUZZ.target.com" \
  -H "X-Forwarded-Host: FUZZ.target.com" \
  -mc 200,301,302,403

Manual vhost Discovery

# Manual vhost testing
for vhost in admin dev test staging internal portal api; do
  response=$(curl -s -o /dev/null -w "%{http_code}" \
    -H "Host: ${vhost}.target.com" \
    https://target.com)
  
  if [ "$response" != "404" ] && [ "$response" != "000" ]; then
    echo "[+] Possible vhost: ${vhost}.target.com (Status: $response)"
  fi
done

vhost Wordlists

# Common vhost wordlists
/usr/share/SecLists/Discovery/DNS/subdomains-top1million-5000.txt
/usr/share/SecLists/Discovery/DNS/subdomains-top1million-110000.txt
/usr/share/wordlists/vhosts/vhost-list.txt

# Create custom vhost wordlist
cat > /tmp/custom_vhosts.txt << 'EOF'
admin
dev
test
staging
internal
portal
api
app
web
mail
vpn
intranet
extranet
dashboard
console
manager
status
monitor
EOF

10. File Extension Discovery

Why Extension Discovery Matters

Different file extensions reveal different technologies and vulnerabilities:

Extension Technology Vulnerability Types
.php PHP RCE, LFI, SQLi
.asp ASP RCE, LFI
.aspx ASP.NET RCE, LFI, deserialization
.jsp Java RCE, LFI, deserialization
.py Python RCE, LFI
.rb Ruby RCE, LFI
.js JavaScript XSS, API exposure
.json JSON Data exposure
.xml XML XXE, data exposure
.yaml YAML Config exposure
.yml YAML Config exposure
.sql SQL Database dumps
.log Log Credential exposure
.conf Config Credential exposure
.config Config Credential exposure
.bak Backup Source code exposure
.old Backup Source code exposure
.backup Backup Source code exposure
.swp Swap Source code exposure
.save Save Source code exposure
.tmp Temp Sensitive data

Extension Discovery with gobuster

# Single extension
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -x php

# Multiple extensions
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -x php,asp,aspx,jsp,html,htm,txt,js,css,xml,json,yml,yaml

# All common extensions
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -x php,asp,aspx,jsp,html,htm,txt,js,css,xml,json,yml,yaml,sql,log,conf,config,backup,bak,old,swp,save,tmp,cgi,pl,py,rb,sh,bat,cmd,ps1

Extension Discovery with ffuf

# Single extension
ffuf -u https://target.com/FUZZ.php \
  -w /usr/share/wordlists/dirb/common.txt

# Multiple extensions with FUZZ
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -e .php,.asp,.aspx,.jsp,.html,.htm,.txt,.js,.css,.xml,.json

# Extension-specific wordlist
ffuf -u https://target.com/FUZZ \
  -w /usr/share/SecLists/Discovery/Web-Content/php.txt

Technology-Specific Extension Scans

# PHP application
gobuster dir -u https://target.com \
  -w /usr/share/SecLists/Discovery/Web-Content/php.txt \
  -x php,inc,phtml,php3,php4,php5,phps

# ASP.NET application
gobuster dir -u https://target.com \
  -w /usr/share/SecLists/Discovery/Web-Content/aspnet-common.txt \
  -x aspx,ashx,asmx,aspx.cs,aspx.vb,config

# Java application
gobuster dir -u https://target.com \
  -w /usr/share/SecLists/Discovery/Web-Content/jsp.txt \
  -x jsp,jspf,do,action,jsf

# Node.js application
gobuster dir -u https://target.com \
  -w /usr/share/SecLists/Discovery/Web-Content/nodejs.txt \
  -x js,json,html

11. Common Hidden Files

Critical Hidden Files to Check

Version Control Files

File Content Exploitation
/.git/ Git repository Full source code recovery
/.git/config Git configuration Remote repository URL
/.git/HEAD Current branch Repository structure
/.svn/ SVN repository Source code exposure
/.hg/ Mercurial repository Source code exposure
/.bzr/ Bazaar repository Source code exposure

Environment and Configuration Files

File Content Exploitation
/.env Environment variables Database credentials, API keys
/.env.local Local environment Development credentials
/.env.backup Backup environment Old credentials
/.htaccess Apache configuration Server settings
/.htpasswd Apache passwords User credentials
/web.config IIS configuration Application settings
/config.php PHP configuration Database credentials
/configuration.php PHP configuration Database credentials
/settings.py Python settings Application settings
/config.yml YAML configuration Application settings
/config.json JSON configuration Application settings

System and Metadata Files

File Content Exploitation
/.DS_Store macOS metadata Directory structure
/Thumbs.db Windows thumbnails File names
/.well-known/security.txt Security policy Contact information
/.well-known/acme-challenge/ Let’s Encrypt Domain validation
/robots.txt Crawl directives Hidden directory hints
/sitemap.xml Site map Complete site structure

Backup and Temporary Files

File Content Exploitation
/index.php.bak Backup PHP file Source code
/index.php.old Old PHP file Source code
/index.php.swp Swap file Source code
/index.php.save Save file Source code
/index.php~ Backup file Source code
/database.sql Database dump Full database
/dump.sql Database dump Full database
/backup.sql Database dump Full database
/db.sql Database dump Full database

Automated Hidden File Check

#!/bin/bash
# Check for common hidden files

TARGET="https://target.com"
HIDDEN_FILES=(
  ".git/config"
  ".git/HEAD"
  ".env"
  ".env.local"
  ".env.backup"
  ".htaccess"
  ".htpasswd"
  ".svn/entries"
  ".DS_Store"
  "web.config"
  "robots.txt"
  "sitemap.xml"
  ".well-known/security.txt"
  "config.php"
  "configuration.php"
  "settings.py"
  "config.yml"
  "config.json"
  "Thumbs.db"
)

echo "=== Hidden File Check ==="
echo "Target: $TARGET"
echo ""

for file in "${HIDDEN_FILES[@]}"; do
  status=$(curl -s -o /dev/null -w "%{http_code}" "${TARGET}/${file}")
  size=$(curl -s -o /dev/null -w "%{size_download}" "${TARGET}/${file}")
  
  if [ "$status" != "404" ] && [ "$status" != "000" ]; then
    echo "[+] FOUND: /${file} (Status: ${status}, Size: ${size})"
  fi
done

.git Directory Exploitation

# Check if .git is accessible
curl -s https://target.com/.git/config

# Download entire .git directory
git clone https://target.com/.git /tmp/target_git

# Or use git-dumper
git clone https://github.com/arthaud/git-dumper.git
cd git-dumper
python3 git-dumper.py https://target.com/.git /tmp/target_dump

# Or use dg (dumpgit)
pip3 install dumpgit
dg https://target.com/.git /tmp/target_dump

# Recover source code
cd /tmp/target_dump
git log --oneline
git checkout master
ls -la

.env File Exploitation

# Check for .env file
curl -s https://target.com/.env

# Common .env contents:
# DB_HOST=localhost
# DB_DATABASE=app_database
# DB_USERNAME=root
# DB_PASSWORD=secret_password
# APP_KEY=base64:xxxxxxxx
# AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
# AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
# MAIL_PASSWORD=secret
# REDIS_PASSWORD=secret

12. Interpreting Status Codes

HTTP Status Code Reference

Success Codes (2xx)

Code Meaning Directory Enumeration Value
200 OK Directory/file exists and is accessible
201 Created Resource created (POST response)
204 No Content Exists but no content returned

Redirection Codes (3xx)

Code Meaning Directory Enumeration Value
301 Moved Permanently Directory exists, redirects elsewhere
302 Found (Temporary) Directory exists, temporary redirect
307 Temporary Redirect Directory exists, preserve method
308 Permanent Redirect Directory exists, preserve method

Client Error Codes (4xx)

Code Meaning Directory Enumeration Value
400 Bad Request Malformed request
401 Unauthorized Exists but requires authentication
403 Forbidden Exists but access denied
404 Not Found Directory/file does not exist
405 Method Not Allowed Exists but method not allowed
407 Proxy Auth Required Proxy authentication needed
408 Request Timeout Server timeout
429 Too Many Requests Rate limited

Server Error Codes (5xx)

Code Meaning Directory Enumeration Value
500 Internal Server Error May exist, server error
502 Bad Gateway Backend error
503 Service Unavailable Server overloaded
504 Gateway Timeout Backend timeout

Status Code Interpretation Strategy

graph TD
    A[HTTP Response] --> B{Status Code}
    B -->|200| C[Exists - Accessible]
    B -->|204| D[Exists - No Content]
    B -->|301| E[Exists - Permanent Redirect]
    B -->|302| F[Exists - Temporary Redirect]
    B -->|401| G[Exists - Auth Required]
    B -->|403| H[Exists - Access Denied]
    B -->|404| I[Does Not Exist]
    B -->|405| J[Exists - Method Not Allowed]
    B -->|500| K[May Exist - Server Error]
    B -->|Other| L[Investigate Further]
    
    style C fill:#c8e6c9
    style G fill:#fff9c4
    style H fill:#fff9c4
    style I fill:#ffcdd2
    style K fill:#ffe0b2

Handling Wildcard Responses

Some servers return the same response for all requests (wildcard responses). This makes enumeration difficult.

# Detect wildcard responses
# 1. Check response for non-existent path
curl -s -o /dev/null -w "%{http_code} %{size_download}" https://target.com/nonexistent12345

# 2. Check response for another non-existent path
curl -s -o /dev/null -w "%{http_code} %{size_download}" https://target.com/nonexistent67890

# 3. If both return same status and size, it's likely a wildcard

# Handle wildcard with gobuster
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  --wildcard

# Handle wildcard with ffuf (auto-calibration)
ffuf -u https://target.com/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -ac

Response Size Analysis

# Analyze response sizes to find anomalies
gobuster dir -u https://target.com \
  -w /usr/share/wordlists/dirb/common.txt \
  -o /tmp/gobuster_sizes.txt

# Sort by size to find outliers
sort -t' ' -k3 -n /tmp/gobuster_sizes.txt | tail -20

# Look for unusually large or small responses
awk '{if ($3 > 10000 || $3 < 100) print}' /tmp/gobuster_sizes.txt

13. OSCP Directory Enumeration Best Practices

OSCP-Specific Workflow

graph TD
    A[Target IP/Domain] --> B[Identify Web Services]
    B --> C[Check robots.txt]
    C --> D[Quick gobuster Scan]
    D --> E{Found Interesting?}
    E -->|Yes| F[Deep Dive on Findings]
    E -->|No| G[Extended Wordlist Scan]
    G --> H{Found Interesting?}
    H -->|Yes| F
    H -->|No| I[Check Hidden Files]
    I --> J{Found Interesting?}
    J -->|Yes| F
    J -->|No| K[Move to Next Target]
    F --> L[Exploit Vulnerabilities]

OSCP Essential Commands

# Step 1: Quick scan
gobuster dir -u http://target.local \
  -w /usr/share/wordlists/dirb/common.txt \
  -t 50 -q

# Step 2: With extensions
gobuster dir -u http://target.local \
  -w /usr/share/wordlists/dirb/common.txt \
  -x php,txt,html,js,asp,aspx,jsp \
  -t 50 -q

# Step 3: Check hidden files
for file in .git .env .htaccess .htpasswd .svn .DS_Store robots.txt sitemap.xml; do
  curl -s -o /dev/null -w "%{http_code}" "http://target.local/$file" | grep -v "404" && echo "Found: $file"
done

# Step 4: Recursive on interesting dirs
gobuster dir -u http://target.local/admin \
  -w /usr/share/wordlists/dirb/common.txt \
  -t 50 -q

# Step 5: ffuf for advanced filtering
ffuf -u http://target.local/FUZZ \
  -w /usr/share/wordlists/dirb/common.txt \
  -mc 200,301,403 \
  -fc 404 \
  -t 50

OSCP Time Management

Activity Recommended Time Priority
Quick gobuster scan 5-10 minutes High
Extension scan 10-15 minutes High
Hidden file check 2-5 minutes High
Recursive scan 10-20 minutes Medium
Extended wordlist scan 15-30 minutes Medium
Virtual host discovery 10-15 minutes Low
Total 50-95 minutes —

OSCP Tips and Tricks

  1. Start with common.txt — It’s small and fast, catches most common directories
  2. Always check robots.txt — It often reveals hidden directories
  3. Check for .git directories — Source code exposure is common in labs
  4. Look for .env files — Credentials are often exposed
  5. Use -x for extensions — Don’t miss PHP, ASP, and JSP files
  6. Follow redirects — 301/302 responses indicate existing directories
  7. Check 403 responses — Forbidden directories may have weak access controls
  8. Recursive scan admin panels — Admin directories often have subdirectories
  9. Document everything — Record all findings for your report
  10. Don’t spend too long — If nothing found after 30 minutes, move on

Common OSCP Lab Directories

Directory Frequency Notes
/admin Very Common Admin panel
/api Common API endpoints
/backup Common Backup files
/config Common Configuration files
/debug Common Debug information
/files Common File storage
/images Very Common Image storage
/includes Common Include files
/js Very Common JavaScript files
/login Common Login page
/logs Common Log files
/phpmyadmin Common Database admin
/scripts Common Script files
/server-status Common Server status
/src Common Source code
/uploads Common Upload directory
/wp-admin Common WordPress admin
/.git Common Git repository
/.env Common Environment variables

Quick Reference Cheat Sheet

Essential Commands

# gobuster
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -t 50 -q
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -x php,txt,html -t 50 -q
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -r -t 50 -q
gobuster dns -d target.com -w /usr/share/wordlists/subdomains/top-10000.txt -t 50 -q
gobuster vhost -u https://target.com -w /usr/share/wordlists/vhosts/vhost-list.txt -t 50 -q

# ffuf
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -t 50 -mc 200,301,403 -fc 404
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -e .php,.txt,.html -t 50
ffuf -u https://target.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -recursion -recursion-depth 3
ffuf -u https://target.com -w /usr/share/wordlists/vhosts/vhost-list.txt -H "Host: FUZZ.target.com" -mc 200,301,403

# dirb
dirb https://target.com /usr/share/wordlists/dirb/common.txt -X .php,.txt,.html -r

# Hidden file check
for file in .git .env .htaccess .htpasswd .svn .DS_Store robots.txt sitemap.xml; do
  curl -s -o /dev/null -w "%{http_code}" "https://target.com/$file" | grep -v "404" && echo "Found: $file"
done

# .git exploitation
git clone https://target.com/.git /tmp/target_git
python3 git-dumper.py https://target.com/.git /tmp/target_dump

Useful One-Liners

# Quick directory scan with common extensions
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -x php,txt,html,js,css,xml,json -t 50 -q -o /tmp/dir_scan.txt

# Find all PHP files
ffuf -u https://target.com/FUZZ.php -w /usr/share/wordlists/dirb/common.txt -mc 200,301,403 -fc 404 -t 50

# Find backup files
gobuster dir -u https://target.com -w /usr/share/wordlists/dirb/common.txt -x bak,old,backup,swp,save,tmp,sql -t 50 -q

# Check for common admin panels
for dir in admin administrator panel dashboard console manager; do
  status=$(curl -s -o /dev/null -w "%{http_code}" "https://target.com/$dir")
  [ "$status" != "404" ] && echo "[+] Found: /$dir (Status: $status)"
done

# Extract directories from robots.txt
curl -s https://target.com/robots.txt | grep -oE 'Disallow: [^\s]+' | cut -d' ' -f2

# Combine multiple wordlists
cat /usr/share/wordlists/dirb/common.txt /usr/share/SecLists/Discovery/Web-Content/common.txt | sort -u > /tmp/combined.txt

Summary

Directory and file discovery is one of the most important techniques in web application penetration testing. Key takeaways:

  1. gobuster is the go-to tool for directory enumeration with multiple modes (dir, dns, vhost, s3)
  2. ffuf offers more flexibility with advanced filtering, multiple FUZZ positions, and recursion
  3. dirb is a simpler alternative with basic functionality
  4. Wordlist selection is critical — start small and progress to larger lists
  5. Recursive enumeration discovers nested directories that flat scans miss
  6. Virtual host discovery reveals hidden applications and internal services
  7. Extension discovery helps identify the technology stack
  8. Hidden files like .git, .env, and .htaccess often contain sensitive information
  9. Status code interpretation is essential for accurate results
  10. In OSCP, focus on quick scans with common.txt, hidden file checks, and recursive scanning of interesting directories

Master these techniques to thoroughly map web application attack surfaces and discover hidden entry points for exploitation.