Wordlist Creation and Management
Module 12, Section 5 — The art of building intelligent, target-aware wordlists that dramatically increase cracking success rates.
Table of Contents
- Why Custom Wordlists Matter
- CeWL — Web Spidering for Wordlist Generation
- Crunch — Pattern-Based Wordlist Generation
- Wordlist Manipulation — Shell Tool Mastery
- Custom Rule Creation — Hashcat Rules Deep Dive
1. Why Custom Wordlists Matter
1.1 The Intelligence Gap
Generic wordlists like rockyou.txt are a starting point, not an endpoint. They contain passwords from real-world breaches — but those breaches were from other organizations, other industries, and other cultures. When you attack a specific target, you need intelligence that reflects that target’s environment.
Consider what a generic wordlist will never contain:
- The company’s name, product names, or internal project codenames
- Employee names, pet names, or mascot references
- Industry-specific jargon and acronyms
- Internal slogans, mottos, or team names
- Dates significant to the organization (founding, mergers, product launches)
- Password policy-enforced patterns (e.g., “must contain company name”)
1.2 Success Rate Comparison
The difference between a generic wordlist and a properly crafted, target-specific wordlist is dramatic:
| Wordlist Type | Example | Coverage | Estimated Success Rate |
|---|---|---|---|
rockyou.txt alone |
14M common passwords | General population | 30–50% |
rockyou.txt + built-in rules |
Dictionary + mutations | Modified common passwords | 50–70% |
| Target-specific wordlist | CeWL output + Crunch | Context-aware terms | 70–85% |
| Layered approach | All of the above + mask attacks | Exhaustive pattern coverage | 85–95% |
1.3 The OSCP Reality
On OSCP machines, you will frequently encounter passwords that follow these patterns:
| Pattern | Example | Why It Matters |
|---|---|---|
| Company/product name + year | Medico2024 |
Company names appear on websites |
| Username variations | admin, administrator, admin123 |
Default accounts are common |
| Service names | svc_backup, sqlservice |
Service accounts have weak passwords |
| Password policy compliance | Welcome1!, P@ssw0rd1 |
Forced complexity creates predictable patterns |
| Keyboard patterns | qwerty, asdfgh, 1q2w3e |
Lazy users pick easy patterns |
| Seasonal/yearly rotation | Summer2024!, Q3-2024 |
Quarterly password changes |
The single most impactful thing you can do on an OSCP exam is spend 5–10 minutes building a custom wordlist before launching a crack. A 2-minute CeWL run against the target’s web application can uncover the exact terms the password was built from.
1.4 The Wordlist Attack Chain
Think of wordlist creation as a pipeline:
┌──────────────┐ ┌──────────────┐ ┌────────────────┐ ┌──────────────┐
│ Intelligence │───▶│ Generation │───▶│ Manipulation │───▶│ Validation │
│ Gathering │ │ (CeWL/Crunch)│ │ (sort/awk/ │ │ (testing, │
│ │ │ │ │ grep/sed) │ │ frequency) │
└──────────────┘ └──────────────┘ └────────────────┘ └──────────────┘
│ │ │ │
Web scraping Pattern-based Deduplication Size optimization
OSINT Brute-force ranges Length filtering Frequency ranking
Username enum Policy-derived ASCII cleanup Top-N selection
2. CeWL — Web Spidering for Wordlist Generation
2.1 What Is CeWL?
CeWL (Custom Word List generator) is a Ruby-based web spider that crawls a target website and extracts unique words from the page content. It was created by digininja and is one of the most practical tools for generating target-specific wordlists.
Why CeWL matters: Passwords are often derived from terms visible on the target’s own website — product names, team names, internal references, employee names in “About Us” pages, blog post titles, and more. CeWL automates the extraction of these terms.
2.2 Installation
# Clone the repository
git clone https://github.com/digininja/CeWL.git
cd CeWL
# Install Ruby dependencies
bundle install
# Run from source
ruby cewl.rb [options] <url>
# Or install globally
sudo gem install cewl
Pre-installed on Kali Linux — no installation needed during OSCP exams.
2.3 How CeWL Works Internally
CeWL operates in three phases:
-
Crawling — Starting from the seed URL, it follows links up to a configurable depth, building a queue of URLs to visit. It maintains a visited set to avoid cycles.
-
Extraction — For each page, it parses the HTML and extracts text from:
- Visible text content (body text, headings, paragraphs)
- HTML attributes (if
-ais enabled):alt,title,value,placeholder,name,id - Link text and URLs
-
Processing — Extracted text is:
- Tokenized into individual words
- Filtered by minimum length (
-m) - Optionally broken into sub-words (
-b) - Deduplicated
- Written to the output file
2.4 Complete Options Reference
| Option | Short | Long | Description | Default |
|---|---|---|---|---|
| Depth | -d |
--depth |
How many link levels to follow | 2 |
| Word breaker | -b |
--breaker |
Split compound words into sub-words | Off |
| Minimum length | -m |
--min_word_length |
Minimum word length to include | 3 |
| Output file | -w |
--write |
Write output to file instead of stdout | stdout |
| Cookie string | -c |
--cookie |
Cookie string for authenticated access | None |
| User agent | -x |
--user_agent |
HTTP User-Agent header | Default Ruby UA |
| Verbose | -v |
--verbose |
Show progress and URLs being crawled | Off |
| Ignore HTTPS | -i |
--ignore_errors |
Ignore SSL/HTTPS certificate errors | Off |
| Regex filter | -r |
--regex |
Only include words matching this regex | All words |
| HTML attributes | -a |
--attributes |
Also extract words from HTML attributes | Off |
| Internal links only | -internal |
Only follow internal links (same domain) | Off | |
| Delay | -delay |
Delay between requests (seconds) | 0 |
|
| Threads | -threads |
Number of concurrent threads | 1 |
|
| Email extraction | -e |
--emails |
Extract email addresses | Off |
| Extensions | -extensions |
File extensions to follow | html,htm,php,asp,aspx,jsp |
|
| Proxy | -proxy |
HTTP proxy for requests | None | |
| Auth type | -auth |
HTTP auth type (basic, digest, ntlm) | None | |
| Auth user | -auth_user |
Username for HTTP authentication | None | |
| Auth pass | -auth_pass |
Password for HTTP authentication | None | |
| Depth limit per domain | -depth_limit |
Maximum depth per individual domain | Same as -d |
|
| Spelling | -spelling |
Use Google spelling suggestions | Off | |
| Dictionary | -dictionary |
Path to dictionary for spelling | None |
2.5 Deep Dive: Key Options
Depth (-d)
Controls how many levels of links CeWL will follow from the seed URL:
Depth 0: https://target.com/ (just the homepage)
Depth 1: https://target.com/about (one click away)
Depth 2: https://target.com/about/team (two clicks away)
Depth 3: https://target.com/about/team/bio (three clicks away)
Guidance:
-d 0— Fast, only the homepage. Good for quick reconnaissance.-d 2— Default. Covers most public-facing content.-d 3— Deeper coverage. May take significantly longer.-d 5+— Usually diminishing returns. Risk of hitting rate limits.
Word Breaker (-b)
Splits compound words and CamelCase into their component parts:
Input: "CyberSecuritySolutions"
Output: "cyber", "security", "solutions", "cybersecurity", "cybersecuritysolutions"
Input: "TeamAlpha"
Output: "team", "alpha", "teamalpha"
Why this matters: If a password is Alpha2024!, the base word alpha might only appear as part of TeamAlpha on the website. Without -b, CeWL would miss it.
Minimum Word Length (-m)
Filters out short words that are unlikely to be password components:
-m 3 (default): excludes "a", "I", "to", "an", "is"
-m 5: excludes "word", "this", "that", "with"
-m 7: very selective, only longer terms
Guidance: Use -m 4 or -m 5 for password-focused wordlists. Short words add noise without value.
HTML Attributes (-a)
Extracts words from HTML element attributes. This is critical because:
alttext on images often contains product namestitleattributes contain tooltips with meaningful textplaceholdertext in forms reveals field labels and hintsnameandidattributes often contain internal naming conventions
<!-- Without -a, CeWL only sees "Login" -->
<input type="text" name="username" placeholder="Enter your Medico username" />
<!-- With -a, CeWL also extracts: "username", "enter", "your", "medico" -->
Regex Filter (-r)
Only include words matching a specific regular expression:
# Only words containing digits (potential version numbers, dates)
cewl https://target.com -r "[0-9]+" -w numbers.txt
# Only words starting with uppercase (potential proper nouns)
cewl https://target.com -r "^[A-Z]" -w propernouns.txt
# Only words that are 8+ characters (likely password-worthy)
cewl https://target.com -r "^.{8,}$" -w longwords.txt
2.6 Handling JavaScript-Rendered Content
Critical limitation: CeWL uses a standard HTTP client and HTML parser. It does not execute JavaScript. This means:
- Content loaded via AJAX/fetch calls will be missed
- Single-page applications (React, Angular, Vue) will yield minimal results
- Lazy-loaded content below the fold will be missed
- Content behind JavaScript-rendered login walls is inaccessible
Workarounds for JS-rendered content:
# Method 1: Use a headless browser to render, then extract
# Install pupeteer or playwright, render the page, save HTML
# Then feed the saved HTML to a word extractor
# Method 2: Use the browser's developer tools
# View the network tab, identify API endpoints
# Fetch the API responses directly and extract words
# Method 3: Use Google Cache or Wayback Machine
# These sometimes have rendered versions
cewl "https://web.archive.org/web/2/https://target.com" -w archived.txt
# Method 4: Use Google's cache
# Search for "site:target.com" and scrape cached results
2.7 Authentication Support
CeWL can access authenticated content using cookies or HTTP authentication:
# Cookie-based authentication (most common)
# First, log in manually and copy the session cookie
cewl https://target.com/internal -c "PHPSESSID=abc123; session_token=xyz789" -w internal.txt
# HTTP Basic Authentication
cewl https://target.com/admin -auth basic -auth_user admin -auth_pass password123 -w admin.txt
# HTTP NTLM Authentication (Windows environments)
cewl https://target.com/sharepoint -auth ntlm -auth_user CORP\user -auth_pass pass123 -w sp.txt
Practical tip for OSCP: If you find a web application with a login page, try default credentials first. If you gain access, grab the session cookie and run CeWL against the authenticated pages — these often contain internal terminology that’s gold for wordlist generation.
2.8 Combining CeWL with Rules
The real power comes from combining CeWL output with Hashcat rules:
# Step 1: Generate target-specific wordlist
cewl https://target.com -d 3 -b -m 4 -a -w target_words.txt
# Step 2: Apply Hashcat rules to generate password candidates
hashcat -m 1000 --stdout target_words.txt -r rules/best64.rule > target_passwords.txt
# Step 3: Crack with the enhanced wordlist
hashcat -m 1000 -a 0 hashes.txt target_passwords.txt
# Or do it in one step:
hashcat -m 1000 -a 0 hashes.txt target_words.txt -r rules/best64.rule
2.9 Advanced CeWL Techniques
# Multi-site wordlist (combine multiple target URLs)
cewl https://target.com -d 2 -w words1.txt
cewl https://blog.target.com -d 2 -w words2.txt
cewl https://www.target.com -d 2 -w words3.txt
cat words1.txt words2.txt words3.txt | sort -u > combined_target.txt
# Extract emails for username wordlist
cewl https://target.com -d 3 -e -w emails.txt
# Use with delay to avoid rate limiting (polite crawling)
cewl https://target.com -d 3 -delay 1 -w polite_words.txt
# Use proxy for anonymity
cewl https://target.com -proxy http://127.0.0.1:8080 -w proxied_words.txt
# Combine CeWL with Google spelling suggestions
cewl https://target.com -spelling -w spelled_words.txt
2.10 CeWL Limitations and Alternatives
| Limitation | Impact | Alternative |
|---|---|---|
| No JavaScript execution | Misses SPA content | Use headless browser + custom scraper |
| Single domain focus | Misses related sites | Manually run against multiple domains |
| No semantic understanding | Extracts all words equally | Manual curation or ML-based filtering |
| Ruby dependency | Slower than native tools | Consider gobuster for directory discovery |
| No API awareness | Misses JSON responses | Manually scrape API endpoints |
Alternatives to CeWL:
- GoCeWL — Go-based, faster implementation
- Gobuster — Directory/file discovery (complementary, not replacement)
- Wfuzz — Fuzzing tool that can extract words from responses
- Custom Python scripts — Full control over extraction logic
- Harvester — OSINT tool that gathers emails, subdomains, and words
3. Crunch — Pattern-Based Wordlist Generation
3.1 What Is Crunch?
Crunch is a wordlist generator that creates passwords based on specified patterns, character sets, and length ranges. Unlike CeWL which extracts existing words, Crunch generates new combinations algorithmically.
It is the tool of choice when you know (or can guess) the password structure but not the exact password.
3.2 Architecture and Design
Crunch operates as a deterministic combinatorial engine:
- Input: Length range, character set(s), and optional pattern
- Processing: Iterates through all possible combinations in lexicographic order
- Output: Writes each combination to stdout or a file
Key design characteristics:
- Deterministic: Same inputs always produce the same output in the same order
- Exhaustive: Generates every possible combination within the specified constraints
- Streaming: Outputs combinations as they’re generated (memory efficient)
- Splittable: Can split output into multiple files by size (
-b)
3.3 Installation
# Kali Linux (pre-installed)
which crunch
# Ubuntu/Debian
sudo apt install crunch
# From source
git clone https://github.com/thesprinkler/crunch.git
cd crunch
make
sudo make install
3.4 Complete Options Reference
| Option | Description | Example |
|---|---|---|
<min> <max> |
Minimum and maximum password length | crunch 4 8 |
<charset> |
Characters to use (positional) | crunch 4 4 abc |
-o <file> |
Output file (default: stdout) | -o wordlist.txt |
-t <pattern> |
Pattern/mask with special chars | -t ,,,@@@@ |
-f <file> |
Charset file reference | -f charset.txt alpha |
-l <string> |
Custom character set | -l abc123 |
-p |
Permutation mode | -p a b c |
-s <string> |
Start position (resume) | -s abcdef |
-d <count> |
Max repeats of same character | -d 2 |
-b <size> |
Max file size before split | -b 500MB |
-z <type> |
Compress output file | -z gzip |
-c |
Print to stdout AND file | -c -o wordlist.txt |
-q |
Quiet mode (no progress) | -q |
-x <group:charset> |
Custom charset with name | -x %:12345 |
-T <threads> |
Number of output threads | -T 4 |
-S |
Shuffle output (random order) | -S |
-h |
Help | -h |
-n |
Print only N lines then exit | -n 1000 |
3.5 Pattern Syntax Deep Dive
The -t option is Crunch’s most powerful feature. It lets you define exact password patterns using special placeholder characters:
| Placeholder | Character Set | Characters | Example |
|---|---|---|---|
@ |
Uppercase | A-Z | @ → A, B, C, … Z |
, |
Lowercase | a-z | , → a, b, c, … z |
% |
Numeric | 0-9 | % → 0, 1, 2, … 9 |
^ |
Special | !@#$%^&*()_+-=[]{} | ;’:“,./<>? |
a-z, A-Z, 0-9 |
Literal | The character itself | C → always C |
Pattern examples:
# Pattern: 3 lowercase + 4 digits + 1 special
# Generates: aaa0000!, aaa0000@, ..., zzz9999?
crunch 8 8 -t ,,,%%%%^ -o pattern1.txt
# Pattern: Company name + 4 digits
# Generates: Company0000, Company0001, ..., Company9999
crunch 9 9 -t Company%%% -o pattern2.txt
# Pattern: First initial (uppercase) + 6 lowercase + 2 digits
# Generates: Aaaaaa00, Aaaaaa01, ..., Zzzzzz99
crunch 9 9 -t @,%%%%%% -o pattern3.txt
# Pattern: Season + year + exclamation
# Generates: Spring2024!, Summer2024!, Fall2024!, Winter2024!
# (Note: this requires custom charset, see below)
Important: In Crunch patterns, literal characters are case-sensitive. C means literal “C”, , means any lowercase letter.
3.6 Built-in Charset File
Crunch includes a charset file at /usr/share/crunch/charset.txt with predefined character sets:
| Charset Name | Characters | Description |
|---|---|---|
alpha |
a-zA-Z0-9 | Alphanumeric |
alpha-numeric |
a-zA-Z0-9 | Same as alpha |
numeric |
0-9 | Digits only |
alpha-lower |
a-z | Lowercase only |
alpha-upper |
A-Z | Uppercase only |
special |
!@#$%^&*()… | Special characters |
hex-upper |
0-9A-F | Hexadecimal uppercase |
hex-lower |
0-9a-f | Hexadecimal lowercase |
all |
Full ASCII printable | All printable characters |
ascii-nonprint |
Non-printable ASCII | Control characters |
ascii-print |
Printable ASCII | Space through tilde |
# Use built-in charset
crunch 6 6 -f /usr/share/crunch/charset.txt alpha -o alphanum.txt
# Use numeric charset
crunch 4 4 -f /usr/share/crunch/charset.txt numeric -o pins.txt
3.7 Custom Character Sets
Define your own character sets using -l or -x:
# Using -l (inline custom charset)
# Generate 6-char passwords using only these characters
crunch 6 6 -l abcdef1234567890 -o custom.txt
# Using -x (named custom charset for use in patterns)
# Define % as a custom set of characters
crunch 8 8 -t ,,,%%%%,, -x %:AEIOU -o vowels.txt
# Generates: aaaAAAaaa, aaaAAAbbb, ... (where % iterates through AEIOU)
# Multiple custom charsets
crunch 10 10 -t @,,,,,%%%% -x %:2024 -x @:MS \
-o custom2.txt
# Generates: Madmin2024, Madmin2025, ..., Sadmin2029
3.8 Permutation Mode
Generate all permutations of a given set of characters:
# All permutations of "admin"
crunch 5 5 -p a d m i n -o perms.txt
# Output: admin, admin, adnmi, amdin, amnid, anmdi, anmid, ...
# Total: 5! = 120 permutations
# All permutations of "Company"
crunch 7 7 -p C o m p a n y -o company_perms.txt
# Total: 7! = 5,040 permutations
3.9 File Size Management
Crunch can generate enormous files. Manage output with:
# Split into 500MB files
crunch 8 8 -b 500MB -o wordlist.txt
# Creates: wordlist.txt, wordlist.txt001, wordlist.txt002, ...
# Compress output directly
crunch 8 8 -t ,,,,,,,, -o wordlist.txt -z gzip
# Creates: wordlist.txt.gz
# Other compression formats
crunch 8 8 -t ,,,,,,,, -o wordlist.txt -z bz2 # bzip2
crunch 8 8 -t ,,,,,,,, -o wordlist.txt -z lzma # lzma
crunch 8 8 -t ,,,,,,,, -o wordlist.txt -z xz # xz
# Limit to N lines (useful for testing)
crunch 8 8 -t ,,,,,,,, -n 10000 -o sample.txt
3.10 Resuming Generation
If Crunch is interrupted, resume from where it left off:
# Original run (interrupted)
crunch 8 8 -t ,,,,,,,, -o biglist.txt
# ^C (interrupted after generating "mxyzpqr")
# Resume from the last generated password
crunch 8 8 -t ,,,,,,,, -s mxyzpqr -o biglist.txt
# Append to existing file
crunch 8 8 -t ,,,,,,,, -s mxyzpqr >> biglist.txt
3.11 Limiting Character Repeats
Prevent passwords with excessive character repetition:
# Max 2 repeats of the same character
crunch 6 6 -d 2 -o limited.txt
# Includes: abcdef, aabccD
# Excludes: aaaaaa, aaabbb
# Max 1 repeat (no character appears more than twice)
crunch 8 8 -d 1 -o very_limited.txt
3.12 OSCP Scenarios: Known Policy → Targeted Crunch
This is the most common Crunch use case on OSCP exams. You discover a password policy and use Crunch to generate a targeted wordlist:
# Scenario 1: Policy requires 8 chars, 1 uppercase, 1 digit
# Pattern: 6 lowercase + 1 uppercase + 1 digit
crunch 8 8 -t ,,,,,,@% -o policy1.txt
# Scenario 2: Password must start with company name "Medico"
# Pattern: "Medico" + 2 any characters
crunch 8 8 -t Medico@@ -o medico_prefix.txt
# Scenario 3: Policy is "Word + Year + !"
# Generate: each CeWL word + 2024!, 2025!, 24!, 25!
# (Better done with rules, but Crunch can do it)
crunch 10 10 -t ,,,,,,,,%% -o words_with_years.txt
# Scenario 4: PIN code (4 digits)
crunch 4 4 0123456789 -o pins.txt
# Scenario 5: WiFi password (WPA2, 8-63 chars, known to be 12 lowercase)
crunch 12 12 -f /usr/share/crunch/charset.txt alpha-lower -o wifi.txt
3.13 Crunch File Size Calculator
Before generating, estimate the file size:
| Configuration | Combinations | Approx. File Size |
|---|---|---|
| 4 chars, lowercase | 26⁴ = 456,976 | ~3 MB |
| 6 chars, lowercase | 26⁶ = 308M | ~5 GB |
| 6 chars, alphanumeric | 62⁶ = 56.8B | ~900 GB |
| 8 chars, lowercase | 26⁸ = 208B | ~3 TB |
| 8 chars, alphanumeric | 62⁸ = 218T | impractical |
Pattern Company%%% |
10⁴ = 10,000 | ~100 KB |
Pattern ,,,@%%^ |
26³ × 26 × 10² × 32 = ~147M | ~2 GB |
Rule of thumb: If the estimated size exceeds 10 GB, reconsider your approach. Use patterns, rules, or a more targeted strategy.
4. Wordlist Manipulation — Shell Tool Mastery
Raw wordlists from CeWL or Crunch are rarely ready to use. They need cleaning, filtering, combining, and optimization. This section is your complete reference for wordlist manipulation using standard Linux shell tools.
4.1 The Essential Toolkit
| Tool | Primary Use | Key for Wordlists |
|---|---|---|
sort |
Sorting lines | Deduplication (-u), ordering |
uniq |
Remove duplicates | Count occurrences (-c) |
awk |
Field/text processing | Length filtering, column extraction |
grep |
Pattern matching | Include/exclude patterns |
sed |
Stream editing | Find-and-replace, line deletion |
tr |
Character translation | Case conversion, character removal |
wc |
Word/line counting | Size verification |
cut |
Column extraction | Remove prefixes/suffixes |
head/tail |
First/last lines | Sampling, top-N selection |
cat |
Concatenation | Combining wordlists |
shuf |
Random shuffle | Randomized ordering |
comm |
Compare sorted files | Find differences between lists |
4.2 Deduplication
The most common first step. Remove duplicate entries:
# Method 1: sort -u (fastest, recommended)
sort -u wordlist.txt -o unique_wordlist.txt
# Method 2: sort | uniq (two-step, preserves original order with -s)
sort wordlist.txt | uniq > unique_wordlist.txt
# Method 3: Preserve original order while removing duplicates
awk '!seen[$0]++' wordlist.txt > unique_ordered.txt
# Verify deduplication
echo "Before: $(wc -l < wordlist.txt)"
echo "After: $(wc -l < unique_wordlist.txt)"
4.3 Length Filtering
Filter words by character length — critical for matching password policies:
# Minimum length (8+ characters)
awk 'length >= 8' wordlist.txt > min8.txt
# Maximum length (12 or fewer characters)
awk 'length <= 12' wordlist.txt > max12.txt
# Range (8 to 12 characters)
awk 'length >= 8 && length <= 12' wordlist.txt > range8_12.txt
# Exact length
awk 'length == 10' wordlist.txt > exactly10.txt
# Remove single-character and two-character words
awk 'length > 2' wordlist.txt > no_short.txt
# Sort by length (shortest first)
awk '{ print length, $0 }' wordlist.txt | sort -n | cut -d' ' -f2- > by_length.txt
# Sort by length (longest first)
awk '{ print length, $0 }' wordlist.txt | sort -rn | cut -d' ' -f2- > by_length_desc.txt
4.4 ASCII Normalization
Clean up non-standard characters that could cause issues:
# Remove non-ASCII characters (keep only printable ASCII)
grep -P '[^\x00-\x7F]' -v wordlist.txt > ascii_only.txt
# Alternative: keep only printable characters
tr -cd '[:print:]\n' < wordlist.txt > printable.txt
# Remove trailing whitespace/carriage returns (Windows line endings)
sed -i 's/\r$//' wordlist.txt
tr -d '\r' < wordlist.txt > cleaned.txt
# Remove leading/trailing whitespace from each line
sed -i 's/^[[:space:]]*//;s/[[:space:]]*$//' wordlist.txt
# Convert to lowercase (normalize case)
tr '[:upper:]' '[:lower:]' < wordlist.txt > lowercase.txt
# Convert to uppercase
tr '[:lower:]' '[:upper:]' < wordlist.txt > uppercase.txt
# Remove all digits
tr -d '0-9' < wordlist.txt > no_digits.txt
# Remove all special characters (keep alphanumeric + newline)
tr -cd 'a-zA-Z0-9\n' < wordlist.txt > alphanum_only.txt
4.5 Empty Line and Blank Removal
# Remove empty lines
sed -i '/^$/d' wordlist.txt
# Remove lines that are only whitespace
sed -i '/^[[:space:]]*$/d' wordlist.txt
# Remove lines shorter than minimum (combines with empty line removal)
awk 'length >= 3' wordlist.txt > cleaned.txt
4.6 Pattern-Based Filtering with grep
# Include only words containing a specific pattern
grep -i "admin" wordlist.txt > admin_words.txt
# Exclude words containing a pattern
grep -iv "test" wordlist.txt > no_test.txt
# Include only words matching exact regex
grep -P '^[a-z]{4,8}$' wordlist.txt > short_lower.txt
# Include words that start with uppercase
grep -P '^[A-Z]' wordlist.txt > capitalized.txt
# Include words containing digits
grep -P '[0-9]' wordlist.txt > with_numbers.txt
# Include words with at least one special character
grep -P '[^a-zA-Z0-9]' wordlist.txt > with_special.txt
# Case-insensitive search
grep -i "company" wordlist.txt > company_related.txt
# Multiple patterns (words containing "admin" OR "user")
grep -E "admin|user" wordlist.txt > admin_or_user.txt
4.7 Combining Wordlists
Merge multiple wordlists into a single, optimized list:
# Simple concatenation
cat wordlist1.txt wordlist2.txt wordlist3.txt > combined_raw.txt
# Concatenate and deduplicate (recommended)
cat wordlist1.txt wordlist2.txt wordlist3.txt | sort -u > combined.txt
# Merge with rockyou.txt
cat cewl_words.txt /usr/share/wordlists/rockyou.txt | sort -u > enhanced.txt
# Merge and filter by length
cat cewl_words.txt crunch_output.txt | sort -u | awk 'length >= 6' > final.txt
4.8 Frequency Analysis
Understand the distribution of your wordlist:
# Count occurrences of each word (sorted by frequency, descending)
sort wordlist.txt | uniq -c | sort -rn > frequency.txt
# Top 20 most common words
sort wordlist.txt | uniq -c | sort -rn | head -20
# Word length distribution
awk '{ print length }' wordlist.txt | sort | uniq -c | sort -rn
# Average word length
awk '{ total += length; count++ } END { print total/count }' wordlist.txt
# Count words by starting letter
awk '{ print substr($0,1,1) }' wordlist.txt | sort | uniq -c | sort -rn
4.9 Sampling and Subsetting
# First N lines
head -n 100000 wordlist.txt > top100k.txt
# Last N lines
tail -n 5000 wordlist.txt > last5k.txt
# Random sample of N lines
shuf -n 10000 wordlist.txt > random_sample.txt
# Every Nth line (decimate)
awk 'NR % 10 == 0' wordlist.txt > every_10th.txt
# Lines N through M
sed -n '1000,5000p' wordlist.txt > lines_1000_5000.txt
4.10 sed Advanced Operations
# Replace all occurrences of a string
sed 's/password/p@ssw0rd/g' wordlist.txt > leet_version.txt
# Append a suffix to every word
sed 's/$/123/' wordlist.txt > appended_123.txt
# Prepend a prefix to every word
sed 's/^!/!/' wordlist.txt > prepended_bang.txt
# Capitalize first letter of each word
sed 's/\b./\U&/g' wordlist.txt > capitalized_words.txt
# Remove lines matching multiple patterns
sed -i '/^test/d;/^demo/d;/^sample/d' wordlist.txt
# Insert text at specific position
sed '3s/.*/&\nINSERTED_WORD/' wordlist.txt > inserted.txt
4.11 Building Layered Wordlists
The most effective wordlists combine multiple sources and transformations:
# Step 1: Generate base wordlists
cewl https://target.com -d 3 -b -m 4 -a -w cewl_base.txt
crunch 8 8 -t ,,,,,,@% -o crunch_policy.txt
# Step 2: Clean each source
sort -u cewl_base.txt -o cewl_clean.txt
sort -u crunch_policy.txt -o crunch_clean.txt
# Step 3: Combine with rockyou
cat cewl_clean.txt crunch_clean.txt /usr/share/wordlists/rockyou.txt \
| sort -u | awk 'length >= 6 && length <= 20' \
| grep -P '[^\x00-\x7F]' -v \
> layered_wordlist.txt
# Step 4: Verify
echo "Total entries: $(wc -l < layered_wordlist.txt)"
echo "File size: $(du -h layered_wordlist.txt | cut -f1)"
4.12 Complete Wordlist Manipulation Cheat Sheet
# === QUICK REFERENCE ===
# Count entries
wc -l wordlist.txt
# Remove duplicates
sort -u wordlist.txt -o unique.txt
# Filter by length (8-12 chars)
awk 'length >= 8 && length <= 12' wordlist.txt > filtered.txt
# Remove non-ASCII
grep -P '[^\x00-\x7F]' -v wordlist.txt > ascii.txt
# Lowercase everything
tr '[:upper:]' '[:lower:]' < wordlist.txt > lower.txt
# Remove empty lines
sed -i '/^$/d' wordlist.txt
# Combine and deduplicate
cat list1.txt list2.txt | sort -u > combined.txt
# Frequency analysis
sort wordlist.txt | uniq -c | sort -rn | head -20
# Remove Windows line endings
tr -d '\r' < wordlist.txt > cleaned.txt
# Shuffle randomly
shuf wordlist.txt > shuffled.txt
# Top 50,000 entries
head -n 50000 wordlist.txt > top50k.txt
5. Custom Rule Creation — Hashcat Rules Deep Dive
5.1 Why Custom Rules?
Hashcat rules transform dictionary words into password candidates. A single word like password can become thousands of candidates through rule application:
password → Password, password1, password!, P@ssw0rd, Password2024, ...
The math: A wordlist of 10,000 words × a rule file of 100 rules = up to 1,000,000 password candidates. This is the force multiplier that makes dictionary attacks practical.
5.2 Hashcat Rule Syntax — Complete Reference
Hashcat rules are single-line transformations. Each line in a .rule file is one rule. Rules are applied sequentially within a single rule line (multiple operations on one line).
Character Case Rules
| Rule | Description | Example Input | Example Output |
|---|---|---|---|
l |
Lowercase first character | PASSWORD |
pASSWORD |
u |
Uppercase first character | password |
Password |
c |
Capitalize (toggle first char case) | password → PasswordPassword → password |
password |
L |
Lowercase all characters | pAssWoRd |
password |
U |
Uppercase all characters | pAssWoRd |
PASSWORD |
C |
Capitalize all words (toggle all) | pAssWoRd |
PaSSwORD |
Character Replacement Rules
| Rule | Description | Example |
|---|---|---|
sXY |
Replace character X with Y (all occurrences) | s a @ → p@ssw0rd |
SXY |
Replace character X with Y (first occurrence only) | S a @ → @assword |
Common leet speak substitutions:
s a @ # a → @
s e 3 # e → 3
s o 0 # o → 0
s s $ # s → $
s i 1 # i → 1
s l 1 # l → 1
s t 7 # t → 7
s b 8 # b → 8
Character Insertion Rules
| Rule | Description | Example |
|---|---|---|
+N |
Append string N to the end | +123 → password123 |
^N |
Prepend string N to the beginning | ^! → !password |
oNX |
Insert character X at position N | o3! → pas!sword (insert ! at position 3) |
Position numbering: Positions are 1-indexed. Position 0 means “before the first character” (same as prepend).
Character Deletion Rules
| Rule | Description | Example |
|---|---|---|
dN |
Delete character at position N | d1 → assword (delete first char) |
D |
Delete last character | password → passwor |
Character Duplication Rules
| Rule | Description | Example |
|---|---|---|
p |
Permute (swap two adjacent characters) | password → apssword |
PXY |
Permute characters at positions X and Y | P13 swaps pos 1 and 3 |
d |
Duplicate (double the entire word) | password → passwordpassword |
r |
Reverse the word | password → drowssap |
Character Increment/Decrement Rules
| Rule | Description | Example |
|---|---|---|
i |
Increment first character | password → qassword |
I |
Decrement first character | qassword → password |
j |
Increment last character | passworD → passworE |
J |
Decrement last character | passworE → passworD |
iN |
Increment character at position N | i3 → passxord |
IN |
Decrement character at position N | I3 → passvord |
Rotation Rules
| Rule | Description | Example |
|---|---|---|
f |
Rotate left (first char to end) | password → asswordp |
b |
Rotate right (last char to front) | password → dpasswor |
Mirror and Reflection
| Rule | Description | Example |
|---|---|---|
m |
Mirror (reverse and swap case) | Password → DEVSOLLAWS (conceptual) |
Skip and Filter Rules
| Rule | Description | Example |
|---|---|---|
:N |
Skip output if word length is not N | :8 → only output 8-char results |
=N |
Skip output if word length equals N | =3 → skip 3-char results |
!N |
Skip output if word length is less than N | !6 → skip words shorter than 6 |
#N |
Skip output if word length is greater than N | #12 → skip words longer than 12 |
Branching Rules
| Rule | Description |
|---|---|
?N |
Branch: if character at position N matches, continue; otherwise skip remaining rules on this line |
@N |
Branch: if character at position N does NOT match, continue; otherwise skip |
%N |
Branch: if character at position N is a digit, continue |
Passthrough
| Rule | Description |
|---|---|
. |
Passthrough: output the current word unchanged and continue processing |
5.3 Rule Chaining on a Single Line
Multiple rules on one line are applied left to right, sequentially:
# Rule: Capitalize first letter, then append "123"
c+123
# Input: "password"
# After c: "Password"
# After +123: "Password123"
# Rule: Replace a→@, then e→3, then append "!"
s a @s e 3s o 0+!
# Input: "password"
# After s a @: "p@ssword"
# After s e 3: "p@ssw0rd" (no 'e' to replace, but 'o' is next)
# After s o 0: "p@ssw0rd" (already done)
# After +!: "p@ssw0rd!"
# Rule: Reverse, then capitalize, then append year
rc+2024
# Input: "password"
# After r: "drowssap"
# After c: "Drowssap"
# After +2024: "Drowssap2024"
5.4 Strategy: Thinking Like the Target
The most effective custom rules come from understanding how real people create passwords. Here’s the mental model:
Layer 1: Base Word Selection
People pick base words from:
- Their company name, product names, project codenames
- Their username, name, or pet’s name
- Common English words (password, welcome, admin, login)
- Seasonal terms (summer, winter, christmas, halloween)
- Sports teams they support
Layer 2: Capitalization Patterns
| Pattern | Frequency | Rule |
|---|---|---|
| First letter capitalized | Very common | c |
| All lowercase | Common | (no rule needed) |
| All uppercase | Less common | U |
| Title Case (Each Word) | Occasional | C |
Layer 3: Number Appending
| Pattern | Examples | Rule |
|---|---|---|
| Current year | Password2024 |
+2024 |
| Short year | Password24 |
+24 |
| Common numbers | Password123 |
+123 |
| Single digit | Password1 |
+1 |
| Quarter | PasswordQ1 |
+Q1 |
Layer 4: Special Character Appending
| Pattern | Examples | Rule |
|---|---|---|
| Exclamation | Password! |
+! |
| At sign | Password@ |
+@ |
| Dollar | Password$ |
+$ |
| Multiple | Password!! |
+!! |
Layer 5: Leet Speak Substitution
| Pattern | Examples | Rules |
|---|---|---|
| Partial leet | P@ssword |
s a @ |
| Full leet | P@$$w0rd |
s a @s s $s o 0 |
| Mixed | Pa55w0rd |
s a As s 5s o 0 |
5.5 Company-Specific Rule Creation
When you know the target company, create rules that reflect their naming:
# Assume target company is "Medico" with products "HealthPro" and "CarePlus"
# Create: medico_company_rules.rule
cat > medico_company_rules.rule << 'EOF'
# === Company Name Rules ===
# Append company name variations
+Medico
+medico
+MEDICO
+Medico2024
+Medico!
+@Medico
# Prepend company name
^Medico
^medico
# === Product Name Rules ===
+HealthPro
+healthpro
+CarePlus
+careplus
+HealthPro2024
+CarePlus!
# === Seasonal + Company ===
+MedicoQ1
+MedicoQ2
+MedicoQ3
+MedicoQ4
+Spring2024
+Summer2024
+Fall2024
+Winter2024
# === Leet + Company ===
s a @
s e 3
s o 0
# === Combined transformations ===
c+! # Capitalize + exclamation
c+123 # Capitalize + 123
c+2024 # Capitalize + year
c+Medico # Capitalize + company
u+! # Uppercase first + exclamation
u+123 # Uppercase first + 123
rc+! # Reverse + capitalize + exclamation
# === Common password base words + company ===
# These assume the base wordlist contains: admin, password, welcome, login, user
+AdminMedico
+PasswordMedico
+WelcomeMedico
EOF
5.6 Date and Seasonal Rules
Passwords often rotate on schedules. Cover common date patterns:
# Create: date_rules.rule
cat > date_rules.rule << 'EOF'
# === Full years ===
+2024
+2025
+2026
+2023
+2022
# === Short years ===
+24
+25
+26
+23
+22
# === Quarters ===
+Q1
+Q2
+Q3
+Q4
+Q1-2024
+Q2-2024
+Q3-2024
+Q4-2024
# === Months (numeric) ===
+01
+02
+03
+04
+05
+06
+07
+08
+09
+10
+11
+12
# === Seasons ===
+Spring
+Summer
+Fall
+Winter
+spring
+summer
+fall
+winter
# === Seasonal + year ===
+Spring2024
+Summer2024
+Fall2024
+Winter2024
+spring2024
+summer2024
# === Holidays ===
+Christmas
+Halloween
+Thanksgiving
+NewYear
+xmas
+halloween
EOF
5.7 Leet Speak Rules
Comprehensive leet speak transformations:
# Create: leet_rules.rule
cat > leet_rules.rule << 'EOF'
# === Single character leet (most common) ===
s a @
s e 3
s o 0
s s $
s i 1
s l 1
s t 7
s b 8
s g 9
s z 2
# === Double leet ===
s a @s e 3
s a @s o 0
s e 3s o 0
s a @s e 3s o 0
s a @s s $
s e 3s s $
# === Full leet chains ===
s a @s e 3s o 0s s $s i 1
s a @s e 3s o 0s s $s i 1s l 1
s a @s e 3s o 0s s $s i 1s t 7
# === Capitalize + leet ===
cs a @
cs a @s e 3
cs a @s e 3s o 0
cs a @s e 3s o 0s s $
# === Leet + append ===
s a @+!
s a @s e 3+!
s a @s e 3s o 0+!
s a @+123
s a @s e 3+123
EOF
5.8 Testing and Validating Rules
Before using rules in a real attack, test them:
# Test rules against a small sample wordlist
echo -e "password\nadmin\nwelcome\nMedico\nSummer" > test_words.txt
# See what candidates your rules generate
hashcat -a 0 --stdout test_words.txt -r my_rules.rule > test_output.txt
# Check the output
wc -l test_output.txt
head -50 test_output.txt
# Verify specific transformations are present
grep "P@ssw0rd" test_output.txt
grep "Medico2024" test_output.txt
grep "Summer!" test_output.txt
# Count unique outputs
sort -u test_output.txt | wc -l
# Performance test (how many candidates per second)
hashcat -a 0 --stdout test_words.txt -r my_rules.rule | wc -l
5.9 Recommended Rule Files
| Rule File | Rules | Best For | Speed |
|---|---|---|---|
best64.rule |
64 | Quick first pass, low overhead | Fast |
one337.rule |
100+ | Leet speak passwords | Fast |
leet337.rule |
500+ | Extended leet speak | Medium |
d3ad0ne.rule |
1000+ | Comprehensive coverage | Medium |
t00m53.rule |
500+ | Community favorite | Medium |
factory-rules/ |
Multiple sets | Hashcat bundled, well-tested | Varies |
custom_rules.rule |
Your creation | Target-specific patterns | Fast |
Location on Kali: /usr/share/hashcat/rules/
5.10 The Complete Attack Workflow
# === PHASE 1: Intelligence Gathering ===
# Run CeWL against all discovered web applications
cewl https://target.com -d 3 -b -m 4 -a -w cewl_main.txt
cewl https://blog.target.com -d 2 -b -m 4 -a -w cewl_blog.txt
# === PHASE 2: Wordlist Preparation ===
# Combine and clean
cat cewl_main.txt cewl_blog.txt /usr/share/wordlists/rockyou.txt \
| sort -u | awk 'length >= 4 && length <= 20' \
| grep -P '[^\x00-\x7F]' -v \
> master_wordlist.txt
# === PHASE 3: Custom Rule Creation ===
# Create target-specific rules (see above examples)
cat > target_rules.rule << 'EOF'
c+!
c+123
c+2024
s a @
s e 3
s o 0
+CompanyName
+companyname
EOF
# === PHASE 4: Attack Execution ===
# Start with best64 (fast, effective)
hashcat -m 1000 -a 0 hashes.txt master_wordlist.txt -r /usr/share/hashcat/rules/best64.rule
# If no crack, try custom rules
hashcat -m 1000 -a 0 hashes.txt master_wordlist.txt -r target_rules.rule
# If still no crack, try leet speak
hashcat -m 1000 -a 0 hashes.txt master_wordlist.txt -r /usr/share/hashcat/rules/one337.rule
# Last resort: comprehensive rules
hashcat -m 1000 -a 0 hashes.txt master_wordlist.txt -r /usr/share/hashcat/rules/d3ad0ne.rule
# === PHASE 5: Mask Attack (if policy known) ===
# If password policy requires: 8 chars, 1 upper, 1 digit, 1 special
hashcat -m 1000 -a 6 hashes.txt master_wordlist.txt ?u?d?s
OSCP Exam Tips
Tip 1: Always Run CeWL First
Before launching any cracking attempt, spend 2-3 minutes running CeWL against every web application you’ve discovered. The words it finds are often the exact building blocks of the target passwords.
# Quick CeWL run (takes 30-60 seconds for small sites)
cewl http://target-ip -d 2 -m 4 -w target_words.txt
Tip 2: Layer Your Approach
Don’t just use rockyou.txt and call it done. The OSCP exam rewards methodical attackers:
rockyou.txtwithbest64.rule— baseline- CeWL output with
best64.rule— target-aware - CeWL output with custom rules — target-specific
- Crunch with known policy — exhaustive for the pattern
Tip 3: Watch Your Wordlist Size
Hashcat performance degrades with extremely large wordlists + large rule files. Start small:
| Wordlist Size | Rule File | Estimated Candidates | Practical? |
|---|---|---|---|
| 10K words | best64 (64 rules) | 640K | Yes, fast |
| 100K words | best64 (64 rules) | 6.4M | Yes |
| 1M words | d3ad0ne (1000+ rules) | 1B+ | Slow, consider splitting |
| 14M (rockyou) | d3ad0ne (1000+ rules) | 14B+ | Very slow |
Tip 4: Use --stdout for Testing
Before committing to a long crack, test your wordlist + rules combination:
# See how many candidates you're generating
hashcat -a 0 --stdout wordlist.txt -r rules.rule | wc -l
# Preview the first few candidates
hashcat -a 0 --stdout wordlist.txt -r rules.rule | head -20
Tip 5: Save Your Session
Long cracks can be interrupted. Always name your sessions:
hashcat -m 1000 -a 0 --session myattack hashes.txt wordlist.txt -r rules.rule
# Resume later
hashcat --restore
Tip 6: The “Company Name + Year” Pattern
This is the #1 most common OSCP password pattern. Always include rules for:
+CompanyName
+companyname
+CompanyName2024
+companyname2024
+CompanyName!
+companyname!
Tip 7: Check the Password Policy
If you can discover the password policy (through web app error messages, documentation, or group policy), use Crunch to generate a targeted wordlist that matches exactly:
# Policy: 8-12 chars, must have uppercase, lowercase, digit
# Pattern: 5 lowercase + 1 uppercase + 1 digit + 1 special
crunch 8 12 -t ,,,,,@%^ -o policy_match.txt
Quick Reference Card
┌─────────────────────────────────────────────────────────────────────┐
│ WORDLIST CREATION QUICK REF │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ CeWL: cewl https://target -d 3 -b -m 4 -a -w words.txt │
│ Crunch: crunch 8 8 -t ,,,,,,@% -o policy.txt │
│ Combine: cat w1.txt w2.txt | sort -u > combined.txt │
│ Dedup: sort -u wordlist.txt -o unique.txt │
│ Filter: awk 'length >= 8 && length <= 12' wordlist.txt > f.txt │
│ Clean: tr -d '\r' < wordlist.txt | grep -P '[^\x00-\x7F]' -v │
│ Count: wc -l wordlist.txt │
│ │
│ Hashcat Rules: │
│ c = capitalize + = append ^ = prepend │
│ sXY = replace r = reverse d = duplicate │
│ L = lowercase U = uppercase dN = delete pos N │
│ │
│ Attack Order: │
│ 1. rockyou + best64.rule │
│ 2. CeWL words + best64.rule │
│ 3. CeWL words + custom rules │
│ 4. Crunch (if policy known) │
│ 5. rockyou + d3ad0ne.rule (comprehensive) │
│ │
└─────────────────────────────────────────────────────────────────────┘
This guide covers the complete wordlist creation and management workflow for password attacks. Master these techniques and you’ll dramatically increase your cracking success rate on both OSCP exams and real-world engagements.