Certificate Transparency Logs
Attack Chain Position
graph LR
A[Reconnaissance] --> B[CT Log Analysis]
B --> C[Subdomain Discovery]
C --> D[Port Scanning]
D --> E[Service Enumeration]
E --> F[Vulnerability Identification]
F --> G[Exploitation]
style B fill:#f9d56e,stroke:#333,stroke-width:2px
Position: Passive Reconnaissance Phase — CT logs are one of the most underutilized yet powerful sources for discovering subdomains, internal hostnames, and infrastructure details without generating any traffic to the target.
Table of Contents
- What Are Certificate Transparency Logs
- Why CT Logs Matter for Recon
- crt.sh Deep Dive
- Other CT Log Sources
- Certificate Parsing and Analysis
- Historical Certificate Analysis
- Wildcard Certificate Detection
- CT Log Monitoring Setup
- Correlating CT Data with Other Sources
- CT Logs in OSCP Context
- Building CT-Driven Subdomain Lists
- OSCP Exam Tips
What Are Certificate Transparency Logs
History and Background
Certificate Transparency (CT) was proposed by Google in 2013 as a response to several high-profile incidents where Certificate Authorities (CAs) issued certificates without proper authorization:
- 2011 DigiNotar Incident: A Dutch CA was compromised, leading to the issuance of a fraudulent Google certificate that enabled man-in-the-middle attacks against Gmail users in Iran.
- 2011 Comodo Breach: Hackers obtained a code-signing certificate from Comodo by compromising a reseller, leading to the issuance of unauthorized certificates for Google, Yahoo, and Microsoft.
- 2012 GlobalSign Incident: A misconfigured system at GlobalSign allowed anyone to obtain a valid certificate for any domain.
These incidents highlighted a fundamental problem: there was no public, auditable record of which certificates had been issued by CAs. Domain owners had no way to detect if a rogue or mistakenly-issued certificate existed for their domain.
The CT Framework
Certificate Transparency is an open-framework designed to monitor and detect misissued or malicious TLS certificates. It was standardized as RFC 6962 (published in 2013) and has since become a mandatory requirement for all publicly-trusted CAs.
Key Components of the CT Framework:
| Component | Description |
|---|---|
| Certificate Authorities (CAs) | Entities that issue TLS certificates. Must submit all publicly-trusted certificates to CT logs. |
| CT Logs | Publicly-accessible, append-only registries of TLS certificates. Operated by various organizations (Google, Facebook, DigiCert, etc.). |
| CT Monitors | Services that monitor CT logs for certificates related to specific domains. Used by domain owners to detect unauthorized issuance. |
| CT Auditors | Verify the integrity and consistency of CT log data, ensuring logs cannot be tampered with. |
How CT Works
sequenceDiagram
participant CA as Certificate Authority
participant CTL as CT Log Server
participant M as CT Monitor
participant O as Observer (Pentester)
CA->>CTL: Submit certificate for inclusion
CTL->>CTL: Assign SCT (Signed Certificate Timestamp)
CTL-->>CA: Return SCT
CA->>CA: Embed SCT in issued certificate
Note over M,O: Anyone can query CT Logs
M->>CTL: Monitor for domain certificates
O->>CTL: Query for subdomain discovery
CTL-->>O: Return all certificates for domain
The CT Log Submission Process:
- Certificate Issuance: When a CA issues a publicly-trusted TLS certificate, it must submit the certificate to multiple CT log servers.
- SCT Generation: Each CT log server generates a Signed Certificate Timestamp (SCT) — a cryptographic proof that the certificate was included in the log at a specific time.
- Certificate Delivery: The CA includes the SCT(s) in the issued certificate (via TLS extensions or OCSP stapling).
- Public Availability: The certificate becomes publicly queryable in the CT log, typically within minutes to hours of issuance.
Key Properties of CT Logs:
- Append-only: Once a certificate is logged, it cannot be removed or modified.
- Publicly accessible: Anyone can query CT logs without authentication.
- Mandatory: All publicly-trusted CAs must submit certificates to CT logs (enforced by browser vendors).
- Merkle Tree Structure: CT logs use Merkle trees to ensure data integrity and enable efficient auditing.
Types of Certificates in CT Logs
| Certificate Type | Description | CT Logged? |
|---|---|---|
| Domain Validated (DV) | Basic TLS certificates validating domain ownership | Yes |
| Organization Validated (OV) | Certificates with organization verification | Yes |
| Extended Validation (EV) | Highest level of identity verification | Yes |
| Wildcard Certificates | Cover *.example.com |
Yes |
| Self-Signed Certificates | Not issued by a public CA | No |
| Internal/Private CA Certs | Issued by private/internal CAs | No |
| Let’s Encrypt Certs | Automated, free certificates | Yes |
Why CT Logs Matter for Recon
Subdomain Discovery
CT logs are arguably the single best passive source for subdomain discovery because:
- Every TLS-enabled service leaves a trace: Any subdomain that has ever had an SSL/TLS certificate issued for it will appear in CT logs.
- Historical data: Even if a subdomain is no longer active, its certificate remains in the logs indefinitely.
- No rate limiting: Unlike DNS enumeration tools, CT log queries don’t generate traffic to the target.
- High accuracy: Certificates are issued by real CAs for real services — low false positive rate compared to wordlist-based enumeration.
What You Can Discover
| Data Type | Example | Value for Recon |
|---|---|---|
| Subdomains | dev.example.com, staging.example.com |
Direct targets for further enumeration |
| Internal Hostnames | internal.corp.example.com |
May reveal internal infrastructure patterns |
| Development Environments | dev01.example.com, test.example.com |
Often less secured, easier entry points |
| Legacy Services | old-portal.example.com |
May run outdated software with known vulnerabilities |
| API Endpoints | api.example.com, api-v2.example.com |
Potential attack surface for web application testing |
| Admin Interfaces | admin.example.com, manage.example.com |
High-value targets for authentication bypass |
| Cloud Infrastructure | example.azurewebsites.net |
Reveals cloud provider usage |
| CDN/Proxy Services | example.cloudfront.net |
Reveals infrastructure architecture |
Historical Data Value
CT logs preserve a timeline of a target’s infrastructure:
2023-01-15: cert issued for dev.example.com
2023-03-22: cert issued for staging.example.com
2023-06-10: cert issued for api-v2.example.com
2023-09-05: cert issued for internal.corp.example.com
2024-01-18: cert issued for test-newapp.example.com
2024-04-30: cert issued for legacy.example.com
This timeline can reveal:
- Infrastructure growth patterns — new services being deployed
- Decommissioned services — subdomains that existed but may no longer be actively monitored
- Development cycles — patterns in how the organization names and deploys environments
- M&A activity — sudden appearance of new domain patterns
Internal Hostnames and Dev Environments
One of the most valuable aspects of CT logs is the discovery of non-production environments:
Production: www.example.com, app.example.com
Staging: staging.example.com, stage.example.com
Development: dev.example.com, dev01.example.com, dev02.example.com
Testing: test.example.com, qa.example.com, uat.example.com
Internal: internal.example.com, intranet.example.com
Legacy: old.example.com, legacy.example.com, v1.example.com
Why these matter:
- Development and staging environments often have debug features enabled
- They may run older software versions not yet patched
- Authentication may be weaker or disabled entirely
- They may expose internal APIs not available on production
- They often share infrastructure or credentials with production
crt.sh Deep Dive
Overview
crt.sh (https://crt.sh) is the most popular and accessible CT log monitoring service. It aggregates data from multiple CT log servers and provides both a web interface and an API for querying certificate data.
Key Features:
- Free and open-source (created by Farah Al Massri)
- Aggregates data from all major CT log providers
- Web interface with search and filtering
- REST API for programmatic access
- JSON output support
- SAML and certificate details extraction
Web Interface Usage
Basic Search
- Navigate to https://crt.sh
- Enter the target domain in the search box
- Results display in a table format showing:
- Not Before / Not After (validity period)
- Certificate serial number
- Certificate issuer
- Domain Name (including SANs)
- Certificate entry (downloadable)
Filtering Options
| Filter | Description | Example |
|---|---|---|
| Include Minimized | Show certificates with minimized/shortened entries | Toggle on/off |
| Exclude Minimized | Hide minimized entries | Toggle on/off |
| Show Certificate | View full certificate details | Click “cert” link |
| Show CRL | View Certificate Revocation List | Click “crl” link |
| Show Revoked | Include revoked certificates | Toggle on/off |
API Queries
Basic API Endpoint
https://crt.sh/?q=example.com&output=json
API Parameters
| Parameter | Description | Example |
|---|---|---|
q |
Query string (domain name) | q=example.com |
output |
Output format | json, jsonl, csv |
include_subdomains |
Include subdomain matches | 1 or 0 |
ignore_wildcards |
Exclude wildcard certificates | 1 or 0 |
pretty |
Pretty-print JSON output | 1 or 0 |
delay |
Delay between queries (seconds) | 5 |
Query Examples
Basic domain query (JSON output):
# Get all certificates for example.com and subdomains
curl -s "https://crt.sh/?q=example.com&output=json&include_subdomains=1" | \
python3 -m json.tool
Query with pretty printing:
curl -s "https://crt.sh/?q=example.com&output=json&pretty=1" | \
python3 -m json.tool
CSV output for spreadsheet analysis:
curl -s "https://crt.sh/?q=example.com&output=csv" > example_com_certs.csv
JSONL output (one JSON object per line, ideal for processing):
curl -s "https://crt.sh/?q=example.com&output=jsonl" > example_com_certs.jsonl
Extracting Unique Subdomains
Method 1: Using curl and grep
# Extract all unique domain names from crt.sh results
curl -s "https://crt.sh/?q=example.com&output=json&include_subdomains=1" | \
grep -oP '"name_value"\s*:\s*"\K[^"]+' | \
sort -u > subdomains_crtsh.txt
Method 2: Using Python
import json
import requests
import re
def extract_subdomains_from_crtsh(domain):
"""Extract unique subdomains from crt.sh for a given domain."""
url = f"https://crt.sh/?q={domain}&output=json&include_subdomains=1"
try:
response = requests.get(url, timeout=30)
response.raise_for_status()
data = response.json()
subdomains = set()
for cert in data:
name_value = cert.get('name_value', '')
if name_value:
# Handle wildcard certificates
clean_name = name_value.lstrip('*.')
subdomains.add(clean_name)
return sorted(subdomains)
except requests.exceptions.RequestException as e:
print(f"Error querying crt.sh: {e}")
return []
# Usage
domain = "example.com"
subdomains = extract_subdomains_from_crtsh(domain)
for sub in subdomains:
print(sub)
Method 3: One-liner with jq
# Using jq to extract and deduplicate subdomains
curl -s "https://crt.sh/?q=example.com&output=json&include_subdomains=1" | \
jq -r '.[].name_value' | \
sed 's/\*\.//' | \
sort -u > subdomains_crtsh.txt
Method 4: Comprehensive Extraction Script
#!/bin/bash
# crtsh_extractor.sh - Comprehensive crt.sh subdomain extractor
DOMAIN="${1:-example.com}"
OUTPUT_FILE="subdomains_${DOMAIN}.txt"
echo "[*] Querying crt.sh for ${DOMAIN}..."
# Query crt.sh and extract all name_values
curl -s "https://crt.sh/?q=${DOMAIN}&output=json&include_subdomains=1" | \
python3 -c "
import sys, json
data = json.load(sys.stdin)
subdomains = set()
for cert in data:
name = cert.get('name_value', '')
if name:
# Remove wildcard prefix
name = name.lstrip('*.')
# Only keep domains matching our target
if name.endswith('${DOMAIN}'):
subdomains.add(name)
for sub in sorted(subdomains):
print(sub)
" > "$OUTPUT_FILE"
COUNT=$(wc -l < "$OUTPUT_FILE")
echo "[+] Found ${COUNT} unique subdomains"
echo "[+] Results saved to ${OUTPUT_FILE}"
Understanding crt.sh JSON Output
A typical crt.sh JSON response contains the following fields per certificate:
{
"id": 1234567890,
"serial_number": "03:AB:CD:EF:12:34:56:78",
"score": 2,
"modulus_publication_type": "",
"leaf_certificates_in_chain": 1,
"not_before": "2024-01-15T00:00:00.0000000Z",
"not_after": "2024-04-15T23:59:59.0000000Z",
"revoked": 0,
"revocation_reason": "",
"revocation_date": "",
"crl_reason": "",
"issuer_id": 98765,
"issuer": {
"C": "US",
"O": "Let's Encrypt",
"CN": "R10"
},
"entry_timestamp": "2024-01-15T12:30:45.0000000Z",
"exact_timestamp": "2024-01-15T12:30:45.1234567Z",
"parsed": {
"subject": {
"CN": "example.com"
},
"tbs": {
"extensions": {
"subject_alt_name": [
"DNS:example.com",
"DNS:www.example.com",
"DNS:api.example.com"
]
}
}
},
"name_value": "api.example.com",
"domain_name": "api.example.com",
"common_name": "example.com",
"cert_link": "https://crt.sh/?d=ABC123...",
"crl_link": "",
"ct_log": "Google Argon 2024",
"log_id": "F4:25:3B:1D:6E:8A:9C:0D"
}
Key Fields for Recon:
| Field | Description | Recon Value |
|---|---|---|
name_value |
The specific domain this entry matches | Direct subdomain discovery |
not_before |
Certificate validity start date | Timeline analysis |
not_after |
Certificate validity end date | Identify active vs expired services |
issuer |
Certificate Authority information | Identify CA patterns, Let’s Encrypt usage |
parsed.tbs.extensions.subject_alt_name |
All SANs in the certificate | Multiple subdomains per certificate |
ct_log |
Which CT log recorded this entry | Cross-reference with other sources |
revoked |
Whether the certificate was revoked | Identify decommissioned services |
Wildcard Certificate Handling
Wildcard certificates (*.example.com) can both help and hinder subdomain discovery:
Challenges:
- A single wildcard certificate can cover thousands of subdomains
- crt.sh will show
*.example.comas a single entry - You cannot determine which specific subdomains are actually in use
Strategies:
# 1. Separate wildcard from non-wildcard entries
curl -s "https://crt.sh/?q=example.com&output=json&include_subdomains=1" | \
python3 -c "
import sys, json
data = json.load(sys.stdin)
wildcards = set()
specific = set()
for cert in data:
name = cert.get('name_value', '')
if name.startswith('*.'):
wildcards.add(name)
elif name:
specific.add(name)
print('=== Wildcard Certificates ===')
for w in sorted(wildcards):
print(w)
print('\n=== Specific Subdomains ===')
for s in sorted(specific):
print(s)
print(f'\n[*] Wildcards: {len(wildcards)}, Specific: {len(specific)}')
"
# 2. For wildcard domains, combine with active subdomain enumeration
# First get the wildcard base domains
WILDCARD_BASES=$(curl -s "https://crt.sh/?q=example.com&output=json" | \
grep -oP '"name_value"\s*:\s*"\*\.\K[^"]+' | sort -u)
# Then enumerate actual subdomains using other methods
for base in $WILDCARD_BASES; do
echo "[*] Enumerating subdomains for: $base"
# Use subfinder, amass, or gobuster DNS here
subfinder -d "$base" >> all_subdomains.txt
done
Other CT Log Sources
Censys CT
Censys (https://censys.io) provides CT log data as part of its broader internet scanning platform.
Features:
- CT log data combined with active internet scanning
- Certificate details with associated IP addresses
- Historical certificate data
- API access (free tier available)
Querying Censys CT:
# Web interface: https://censys.io/certificates
# Search query: "names: *.example.com"
# API query (requires authentication)
curl -s -u "API_ID:API_SECRET" \
"https://search.censys.io/api/v2/certificates/search?q=names:example.com&per_page=100" | \
python3 -m json.tool
Censys Python API:
from censys.search import CensysCertSearch
# Initialize with API credentials
cert_search = CensysCertSearch(api_id="YOUR_ID", api_secret="YOUR_SECRET")
# Search for certificates
results = cert_search.search("names: example.com", fields=[
"names",
"parsed.subject_cn",
"parsed.validity.not_before",
"parsed.validity.not_after",
"cert_integrity"
], per_page=100)
for cert in results:
print(cert.get('names', []))
Google CT Transparency Report
Google maintains the CT Transparency Report (https://transparencyreport.google.com/ssl-certificate):
Features:
- Visual dashboard of CT log coverage
- Statistics on certificate issuance
- Browser enforcement policies
- CT log operator information
Usage for Recon:
- Verify CT log coverage for a domain
- Understand which CT logs are being used
- Check for CT compliance issues
SecurityTrails
SecurityTrails (https://securitytrails.com) aggregates CT log data with DNS history and other sources.
Features:
- Historical DNS records
- Certificate data
- WHOIS history
- API access (paid)
API Query:
curl -s -H "Accept: application/json" \
-H "APIKEY: YOUR_API_KEY" \
"https://api.securitytrails.com/v1/domain/example.com/certificates" | \
python3 -m json.tool
CertSpotter
CertSpotter (https://certspotter.com) by SSLMate provides a clean interface for CT log queries.
Features:
- Simple web interface
- Domain-based search
- Certificate details view
- API access
API Query:
curl -s "https://api.certspotter.com/v0/issuances?domain=example.com&include_subdomains=true&expand=dns_names" | \
python3 -m json.tool
CT Log Source Comparison
| Source | Free Tier | API Access | Historical Data | Ease of Use | Best For |
|---|---|---|---|---|---|
| crt.sh | Full | Yes (no auth) | Yes | Medium | Quick queries, scripting |
| Censys | Limited | Yes (auth) | Yes | Medium | Combined cert + IP data |
| SecurityTrails | Limited | Yes (paid) | Yes | High | Comprehensive DNS + cert data |
| CertSpotter | Full | Yes (no auth) | Yes | High | Simple queries |
| Google CT Report | Full | No | Yes | High | Statistics, compliance |
| Google Transparency | Full | No | Yes | High | Visual dashboard |
Certificate Parsing and Analysis
Extracting SANs (Subject Alternative Names)
SANs are the most valuable field in a certificate for subdomain discovery. A single certificate can contain multiple SANs:
# Extract SANs from a certificate file
openssl x509 -in certificate.pem -noout -text | \
grep -A 1 "Subject Alternative Name"
Example SAN output:
X509v3 Subject Alternative Name:
DNS:example.com, DNS:www.example.com, DNS:api.example.com,
DNS:mail.example.com, DNS:vpn.example.com
Extracting Issuer Information
# Get issuer details
openssl x509 -in certificate.pem -noout -issuer
# Output example:
# issuer=C = US, O = Let's Encrypt, CN = R10
Common Issuers and Their Significance:
| Issuer | Significance |
|---|---|
| Let’s Encrypt | Automated certificate management, likely web services |
| DigiCert | Enterprise-grade, likely critical infrastructure |
| GlobalSign | Enterprise-grade, common for corporate sites |
| Comodo/Sectigo | Wide range of services, common for web hosting |
| Amazon RSA | AWS infrastructure, likely cloud-hosted services |
| Self-Signed | Internal services, potentially less secure |
Validity Period Analysis
# Extract validity dates
openssl x509 -in certificate.pem -noout -dates
# Output:
# notBefore=Jan 15 00:00:00 2024 GMT
# notAfter=Apr 15 23:59:59 2024 GMT
Validity Period Insights:
| Validity Duration | Likely Scenario |
|---|---|
| 90 days | Let’s Encrypt (automated renewal) |
| 1 year | Standard commercial certificate |
| 2 years | Older commercial certificate |
| Very short (<30 days) | Development/testing environment |
| Expired | Decommissioned service or neglected renewal |
Full Certificate Parsing Script
#!/usr/bin/env python3
"""
certificate_parser.py - Parse certificate data from crt.sh for recon analysis
"""
import json
import requests
import re
from datetime import datetime
from collections import defaultdict
def parse_crtsh_certificates(domain):
"""Parse and analyze certificates from crt.sh."""
url = f"https://crt.sh/?q={domain}&output=json&include_subdomains=1"
try:
response = requests.get(url, timeout=30)
response.raise_for_status()
data = response.json()
results = {
'total_certificates': len(data),
'unique_subdomains': set(),
'issuers': defaultdict(int),
'wildcards': set(),
'expired': [],
'active': [],
'timeline': [],
'sans_per_cert': [],
}
for cert in data:
name = cert.get('name_value', '')
if not name:
continue
# Extract subdomain
clean_name = name.lstrip('*.')
results['unique_subdomains'].add(clean_name)
# Track wildcards
if name.startswith('*.'):
results['wildcards'].add(name)
# Track issuers
issuer = cert.get('issuer', {})
issuer_cn = issuer.get('CN', 'Unknown')
results['issuers'][issuer_cn] += 1
# Validity analysis
not_before = cert.get('not_before', '')
not_after = cert.get('not_after', '')
if not_after:
try:
expiry = datetime.fromisoformat(not_after.replace('Z', '+00:00'))
now = datetime.now(expiry.tzinfo)
if expiry < now:
results['expired'].append({
'domain': clean_name,
'expired': not_after,
})
else:
results['active'].append({
'domain': clean_name,
'expires': not_after,
})
results['timeline'].append({
'domain': clean_name,
'issued': not_before,
'expires': not_after,
})
except ValueError:
pass
# Extract SANs from parsed data
parsed = cert.get('parsed', {})
tbs = parsed.get('tbs', {})
extensions = tbs.get('extensions', {})
sans = extensions.get('subject_alt_name', [])
if sans:
results['sans_per_cert'].append({
'common_name': cert.get('common_name', ''),
'sans': sans,
})
# Convert sets to sorted lists for output
results['unique_subdomains'] = sorted(results['unique_subdomains'])
results['wildcards'] = sorted(results['wildcards'])
return results
except requests.exceptions.RequestException as e:
print(f"Error: {e}")
return None
def print_analysis(domain, results):
"""Print formatted analysis results."""
if not results:
return
print(f"\n{'='*60}")
print(f"Certificate Transparency Analysis: {domain}")
print(f"{'='*60}")
print(f"\n[*] Total Certificates Found: {results['total_certificates']}")
print(f"[*] Unique Subdomains: {len(results['unique_subdomains'])}")
print(f"[*] Wildcard Certificates: {len(results['wildcards'])}")
print(f"[*] Active Certificates: {len(results['active'])}")
print(f"[*] Expired Certificates: {len(results['expired'])}")
print(f"\n--- Issuers ---")
for issuer, count in sorted(results['issuers'].items(), key=lambda x: x[1], reverse=True):
print(f" {issuer}: {count}")
print(f"\n--- Wildcard Certificates ---")
for wc in results['wildcards']:
print(f" {wc}")
print(f"\n--- Unique Subdomains ---")
for sub in results['unique_subdomains']:
print(f" {sub}")
print(f"\n--- Recent Certificates (last 30 days) ---")
now = datetime.now()
for entry in results['timeline'][-10:]:
print(f" {entry['domain']} | Issued: {entry['issued'][:10]} | Expires: {entry['expires'][:10]}")
if __name__ == "__main__":
import sys
domain = sys.argv[1] if len(sys.argv) > 1 else "example.com"
results = parse_crtsh_certificates(domain)
print_analysis(domain, results)
Historical Certificate Analysis
Why Historical Analysis Matters
Historical certificate data reveals the evolution of a target’s infrastructure:
- Decommissioned Services: Subdomains that had certificates but no longer do may still be accessible or may have been forgotten.
- Infrastructure Changes: Patterns in certificate issuance can reveal migration patterns, cloud adoption, or rebranding.
- Security Posture Changes: Shifts in CA usage, validity periods, or certificate types can indicate changes in security practices.
- M&A Indicators: Sudden appearance of certificates for new domain patterns may indicate acquisitions.
Timeline Analysis
# Create a timeline of certificate issuances
curl -s "https://crt.sh/?q=example.com&output=json&include_subdomains=1" | \
python3 -c "
import sys, json
from datetime import datetime
data = json.load(sys.stdin)
timeline = []
for cert in data:
name = cert.get('name_value', '').lstrip('*.')
issued = cert.get('not_before', '')
if issued and name:
timeline.append((issued[:10], name))
# Sort by date
timeline.sort()
# Group by month
from collections import defaultdict
by_month = defaultdict(list)
for date, name in timeline:
month = date[:7] # YYYY-MM
by_month[month].append(name)
for month in sorted(by_month.keys()):
subs = sorted(set(by_month[month]))
print(f'{month}: {len(subs)} unique subdomains')
for sub in subs[:5]: # Show first 5
print(f' - {sub}')
if len(subs) > 5:
print(f' ... and {len(subs) - 5} more')
"
Identifying Decommissioned Services
def find_decommissioned_services(domain):
"""Find subdomains with expired certificates that may still be accessible."""
url = f"https://crt.sh/?q={domain}&output=json&include_subdomains=1"
response = requests.get(url, timeout=30)
data = response.json()
now = datetime.now()
decommissioned = []
for cert in data:
name = cert.get('name_value', '').lstrip('*.')
not_after = cert.get('not_after', '')
if not_after:
try:
expiry = datetime.fromisoformat(not_after.replace('Z', '+00:00'))
# Certificate expired more than 30 days ago
if (now - expiry).days > 30:
decommissioned.append({
'subdomain': name,
'expired': not_after[:10],
'days_ago': (now - expiry).days,
})
except ValueError:
pass
# Sort by most recently expired
decommissioned.sort(key=lambda x: x['days_ago'])
return decommissioned
Wildcard Certificate Detection
Identifying Wildcard Certificates
# Find all wildcard certificates for a domain
curl -s "https://crt.sh/?q=example.com&output=json" | \
python3 -c "
import sys, json
data = json.load(sys.stdin)
wildcards = []
for cert in data:
name = cert.get('name_value', '')
if name.startswith('*.'):
wildcards.append({
'domain': name,
'issuer': cert.get('issuer', {}).get('CN', 'Unknown'),
'issued': cert.get('not_before', '')[:10],
'expires': cert.get('not_after', '')[:10],
})
for wc in wildcards:
print(f\"{wc['domain']} | {wc['issuer']} | {wc['issued']} - {wc['expires']}\")
"
Wildcard Impact Assessment
| Scenario | Impact on Recon | Mitigation |
|---|---|---|
Single wildcard (*.example.com) |
All subdomains covered by one cert | Combine with active enumeration |
Multiple wildcards (*.dev.example.com, *.staging.example.com) |
Reveals environment structure | Enumerate each wildcard base |
| No wildcards | Each subdomain has individual cert | Maximum subdomain discovery from CT |
| Wildcard + specific | Mixed coverage | Filter wildcards, focus on specific entries |
Strategy for Wildcard Domains
# Step 1: Identify wildcard bases
WILDCARD_BASES=$(curl -s "https://crt.sh/?q=example.com&output=json" | \
grep -oP '"name_value"\s*:\s*"\*\.\K[^"]+' | sort -u)
# Step 2: For each wildcard base, enumerate actual subdomains
for base in $WILDCARD_BASES; do
echo "[*] Enumerating subdomains for wildcard base: $base"
# Use subfinder for passive enumeration
subfinder -d "$base" -silent >> "${base}_subdomains.txt"
# Use amass for deeper enumeration
amass enum -passive -d "$base" -o "${base}_amass.txt"
# Combine and deduplicate
cat "${base}_subdomains.txt" "${base}_amass.txt" | sort -u > "${base}_all.txt"
done
CT Log Monitoring Setup
Automated Monitoring Script
#!/usr/bin/env python3
"""
ct_monitor.py - Monitor CT logs for new certificates on target domains
"""
import json
import requests
import time
import sqlite3
import argparse
from datetime import datetime
class CTMonitor:
def __init__(self, db_path="ct_monitor.db"):
self.db_path = db_path
self.init_db()
def init_db(self):
"""Initialize SQLite database for tracking certificates."""
conn = sqlite3.connect(self.db_path)
cursor = conn.cursor()
cursor.execute('''
CREATE TABLE IF NOT EXISTS certificates (
id INTEGER PRIMARY KEY AUTOINCREMENT,
domain TEXT,
name_value TEXT,
not_before TEXT,
not_after TEXT,
issuer_cn TEXT,
first_seen TEXT,
ct_log TEXT
)
''')
conn.commit()
conn.close()
def query_crtsh(self, domain):
"""Query crt.sh for certificates."""
url = f"https://crt.sh/?q={domain}&output=json&include_subdomains=1"
response = requests.get(url, timeout=30)
return response.json()
def get_known_certs(self, domain):
"""Get known certificates from database."""
conn = sqlite3.connect(self.db_path)
cursor = conn.cursor()
cursor.execute(
"SELECT name_value FROM certificates WHERE domain = ?",
(domain,)
)
return {row[0] for row in cursor.fetchall()}
def save_cert(self, domain, cert):
"""Save certificate to database."""
conn = sqlite3.connect(self.db_path)
cursor = conn.cursor()
cursor.execute('''
INSERT INTO certificates
(domain, name_value, not_before, not_after, issuer_cn, first_seen, ct_log)
VALUES (?, ?, ?, ?, ?, ?, ?)
''', (
domain,
cert.get('name_value', ''),
cert.get('not_before', ''),
cert.get('not_after', ''),
cert.get('issuer', {}).get('CN', ''),
datetime.now().isoformat(),
cert.get('ct_log', '')
))
conn.commit()
conn.close()
def check_new_certs(self, domain):
"""Check for new certificates since last check."""
print(f"[*] Checking CT logs for {domain}...")
known = self.get_known_certs(domain)
certs = self.query_crtsh(domain)
new_certs = []
for cert in certs:
name = cert.get('name_value', '')
if name and name not in known:
new_certs.append(cert)
self.save_cert(domain, cert)
print(f"[!] NEW CERTIFICATE: {name}")
print(f" Issuer: {cert.get('issuer', {}).get('CN', 'Unknown')}")
print(f" Issued: {cert.get('not_before', '')[:10]}")
print(f" Expires: {cert.get('not_after', '')[:10]}")
if not new_certs:
print("[+] No new certificates found")
return new_certs
def main():
parser = argparse.ArgumentParser(description="CT Log Monitor")
parser.add_argument("domain", help="Domain to monitor")
parser.add_argument("--interval", type=int, default=3600,
help="Check interval in seconds (default: 3600)")
parser.add_argument("--once", action="store_true",
help="Run once and exit")
args = parser.parse_args()
monitor = CTMonitor()
if args.once:
monitor.check_new_certs(args.domain)
else:
print(f"[*] Monitoring {args.domain} every {args.interval} seconds...")
while True:
monitor.check_new_certs(args.domain)
time.sleep(args.interval)
if __name__ == "__main__":
main()
Cron-Based Monitoring
# Add to crontab for daily monitoring
0 8 * * * /usr/bin/python3 /opt/tools/ct_monitor.py example.com --once >> /var/log/ct_monitor.log 2>&1
# For multiple domains
0 8 * * * for domain in example.com target.org another.net; do
/usr/bin/python3 /opt/tools/ct_monitor.py "$domain" --once
done >> /var/log/ct_monitor.log 2>&1
Correlating CT Data with Other Sources
Cross-Reference Strategy
graph TD
A[CT Logs] --> D[Correlation Engine]
B[DNS Enumeration] --> D
C[Search Engine Dorking] --> D
E[Shodan/Censys] --> D
D --> F[Unified Subdomain List]
F --> G[Priority Scoring]
G --> H[Active Targets]
style D fill:#f9d56e,stroke:#333,stroke-width:2px
Correlation Script
def correlate_sources(domain):
"""Correlate CT data with other recon sources."""
results = {
'ct_only': set(),
'dns_only': set(),
'both': set(),
'ct_with_live_dns': set(),
}
# Get CT subdomains
ct_subs = set(extract_subdomains_from_crtsh(domain))
# Get DNS subdomains (from subfinder, amass, etc.)
dns_subs = set()
with open(f"{domain}_dns_subs.txt", 'r') as f:
dns_subs = {line.strip() for line in f if line.strip()}
# Correlate
results['ct_only'] = ct_subs - dns_subs
results['dns_only'] = dns_subs - ct_subs
results['both'] = ct_subs & dns_subs
# Check which CT-only subdomains have live DNS
import subprocess
for sub in results['ct_only']:
try:
result = subprocess.run(
['host', sub],
capture_output=True, text=True, timeout=5
)
if result.returncode == 0 and 'has address' in result.stdout:
results['ct_with_live_dns'].add(sub)
except (subprocess.TimeoutExpired, Exception):
pass
return results
Priority Scoring
| Source Match | Priority | Reasoning |
|---|---|---|
| CT + DNS + Live | Critical | Active service with certificate |
| CT + DNS | High | Registered subdomain, may be active |
| CT Only | Medium | Historical, may still be accessible |
| DNS Only | Medium | Active DNS, no cert (HTTP service?) |
| CT + Shodan | High | Active service with banner data |
CT Logs in OSCP Context
OSCP Relevance Assessment
| OSCP Scenario | CT Log Value | Notes |
|---|---|---|
| External Pentest | High | Primary subdomain discovery source |
| Internal Pentest | Low-Medium | Internal CAs not in public CT logs |
| Web App Focus | High | Discover web-facing subdomains |
| Infrastructure Focus | Medium | Supplemental to DNS enumeration |
| Time-Constrained | High | Fast, passive, no rate limiting |
OSCP-Specific Workflow
# OSCP CT Log Quick Workflow
# Step 1: Quick subdomain extraction
DOMAIN="target.oscp.com"
curl -s "https://crt.sh/?q=${DOMAIN}&output=json&include_subdomains=1" | \
grep -oP '"name_value"\s*:\s*"\K[^"]+' | \
sed 's/\*\.//' | \
sort -u > ct_subdomains.txt
# Step 2: Filter for target domain only
grep "${DOMAIN}$" ct_subdomains.txt > ct_target_subdomains.txt
# Step 3: Resolve DNS for live targets
while read sub; do
host "$sub" 2>/dev/null | grep "has address" | awk '{print $4}'
done < ct_target_subdomains.txt > ct_resolved_ips.txt
# Step 4: Quick port scan on resolved IPs
nmap -sV -p 80,443,8080,8443 -iL ct_resolved_ips.txt -oA ct_nmap_quick
Limitations in OSCP
- Internal CAs: OSCP labs often use internal/private CAs that don’t submit to public CT logs.
- Self-Signed Certs: Many lab services use self-signed certificates.
- No Internet Access: Some lab environments don’t have external internet access.
- Time Factor: CT logs are most valuable for external assessments with real-world targets.
Bottom Line: CT logs are a valuable supplemental technique for OSCP. Use them when available, but don’t rely on them as your primary subdomain source for lab exams.
Building CT-Driven Subdomain Lists
Complete Workflow Script
#!/bin/bash
# ct_recon_pipeline.sh - Complete CT-driven reconnaissance pipeline
DOMAIN="${1:-example.com}"
OUTPUT_DIR="ct_recon_${DOMAIN}"
mkdir -p "$OUTPUT_DIR"
echo "[*] Starting CT-driven reconnaissance for ${DOMAIN}"
echo "[*] Output directory: ${OUTPUT_DIR}"
# Step 1: Query crt.sh
echo "[1/6] Querying crt.sh..."
curl -s "https://crt.sh/?q=${DOMAIN}&output=json&include_subdomains=1" > \
"${OUTPUT_DIR}/raw_certs.json"
# Step 2: Extract subdomains
echo "[2/6] Extracting subdomains..."
python3 -c "
import json, sys
with open('${OUTPUT_DIR}/raw_certs.json', 'r') as f:
data = json.load(f)
subdomains = set()
for cert in data:
name = cert.get('name_value', '')
if name:
name = name.lstrip('*.')
if name.endswith('${DOMAIN}'):
subdomains.add(name)
with open('${OUTPUT_DIR}/subdomains.txt', 'w') as f:
for sub in sorted(subdomains):
f.write(sub + '\n')
print(f'[*] Found {len(subdomains)} unique subdomains')
"
# Step 3: Separate wildcards
echo "[3/6] Identifying wildcard certificates..."
python3 -c "
import json
with open('${OUTPUT_DIR}/raw_certs.json', 'r') as f:
data = json.load(f)
wildcards = set()
for cert in data:
name = cert.get('name_value', '')
if name.startswith('*.'):
wildcards.add(name)
with open('${OUTPUT_DIR}/wildcards.txt', 'w') as f:
for wc in sorted(wildcards):
f.write(wc + '\n')
print(f'[*] Found {len(wildcards)} wildcard certificates')
"
# Step 4: DNS resolution
echo "[4/6] Resolving DNS for subdomains..."
while IFS= read -r sub; do
ip=$(host "$sub" 2>/dev/null | grep "has address" | awk '{print $4}' | head -1)
if [ -n "$ip" ]; then
echo "${sub} ${ip}" >> "${OUTPUT_DIR}/resolved.txt"
fi
done < "${OUTPUT_DIR}/subdomains.txt"
# Step 5: Quick HTTP check
echo "[5/6] Checking HTTP services..."
if [ -f "${OUTPUT_DIR}/resolved.txt" ]; then
while IFS= read -r line; do
sub=$(echo "$line" | awk '{print $1}')
ip=$(echo "$line" | awk '{print $2}')
# Quick HTTP check
status=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 3 \
"http://${sub}" 2>/dev/null)
if [ "$status" != "000" ]; then
echo "${sub} ${ip} HTTP:${status}" >> "${OUTPUT_DIR}/http_services.txt"
fi
# HTTPS check
status_ssl=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 3 -k \
"https://${sub}" 2>/dev/null)
if [ "$status_ssl" != "000" ]; then
echo "${sub} ${ip} HTTPS:${status_ssl}" >> "${OUTPUT_DIR}/https_services.txt"
fi
done < "${OUTPUT_DIR}/resolved.txt"
fi
# Step 6: Summary report
echo "[6/6] Generating summary report..."
cat > "${OUTPUT_DIR}/SUMMARY.md" << EOF
# CT Reconnaissance Report: ${DOMAIN}
## Summary
- **Total Subdomains Found:** $(wc -l < "${OUTPUT_DIR}/subdomains.txt")
- **Wildcard Certificates:** $(wc -l < "${OUTPUT_DIR}/wildcards.txt" 2>/dev/null || echo 0)
- **Resolved IPs:** $(wc -l < "${OUTPUT_DIR}/resolved.txt" 2>/dev/null || echo 0)
- **HTTP Services:** $(wc -l < "${OUTPUT_DIR}/http_services.txt" 2>/dev/null || echo 0)
- **HTTPS Services:** $(wc -l < "${OUTPUT_DIR}/https_services.txt" 2>/dev/null || echo 0)
## Top-Level Subdomains
$(head -20 "${OUTPUT_DIR}/subdomains.txt")
## Wildcard Certificates
$(cat "${OUTPUT_DIR}/wildcards.txt" 2>/dev/null || echo "None found")
## Active HTTP Services
$(cat "${OUTPUT_DIR}/http_services.txt" 2>/dev/null || echo "None found")
## Active HTTPS Services
$(cat "${OUTPUT_DIR}/https_services.txt" 2>/dev/null || echo "None found")
EOF
echo "[+] Complete! Results in ${OUTPUT_DIR}/"
echo "[+] See ${OUTPUT_DIR}/SUMMARY.md for overview"
Integrating with Other Tools
# Combine CT results with subfinder and amass
cat ct_recon_*.txt subfinder_*.txt amass_*.txt | sort -u > all_subdomains_combined.txt
# Use the combined list for further enumeration
gobuster dns -d example.com -w all_subdomains_combined.txt -o gobuster_results.txt
# Or feed into nmap
nmap -sV -sC -iL all_subdomains_combined.txt -oA nmap_full_scan
OSCP Exam Tips
Key Takeaways for OSCP
┌─────────────────────────────────────────────────────────────────┐
│ OSCP CT Log Tips │
├─────────────────────────────────────────────────────────────────┤
│ │
│ 1. CT logs are PASSIVE - no traffic to target │
│ Use them early in recon to build subdomain lists │
│ │
│ 2. crt.sh is the easiest source - no API key needed │
│ Bookmark: https://crt.sh │
│ │
│ 3. Focus on the "name_value" field for subdomain extraction │
│ This contains the actual domain/SAN entries │
│ │
│ 4. Wildcard certs limit discovery value │
│ If you see *.target.com, combine with active enumeration │
│ │
│ 5. CT logs may not work in all OSCP labs │
│ Internal CAs and self-signed certs won't appear │
│ Always have backup subdomain discovery methods │
│ │
│ 6. Quick one-liner for exam day: │
│ curl -s "https://crt.sh/?q=TARGET&output=json" | │
│ grep -oP '"name_value"\s*:\s*"\K[^"]+' | sort -u │
│ │
│ 7. Cross-reference CT results with DNS enumeration │
│ Subdomains in CT + live DNS = high priority targets │
│ │
│ 8. Don't spend too much time on CT logs │
│ 5-10 minutes max per target domain │
│ Move on to active scanning after initial discovery │
│ │
└─────────────────────────────────────────────────────────────────┘
Common Mistakes to Avoid
| Mistake | Consequence | Fix |
|---|---|---|
| Ignoring wildcard certificates | Missing entire subdomain trees | Always check for *.domain entries |
| Not filtering by target domain | Too much noise in results | Filter results to match target domain |
| Relying solely on CT logs | Missing non-TLS services | Combine with DNS enumeration and search engines |
| Not checking SANs | Missing subdomains in multi-domain certs | Parse subject_alt_name field |
| Skipping historical data | Missing decommissioned services | Check expired certificates too |
Quick Reference Commands
# Quick subdomain extraction (exam-ready)
curl -s "https://crt.sh/?q=TARGET&output=json&include_subdomains=1" | \
grep -oP '"name_value"\s*:\s*"\K[^"]+' | \
sed 's/\*\.//' | sort -u
# Quick wildcard check
curl -s "https://crt.sh/?q=TARGET&output=json" | \
grep -oP '"name_value"\s*:\s*"\*\.[^"]+'
# Quick issuer analysis
curl -s "https://crt.sh/?q=TARGET&output=json" | \
grep -oP '"CN"\s*:\s*"\K[^"]+' | sort | uniq -c | sort -rn
Summary
Certificate Transparency logs represent one of the most powerful passive reconnaissance techniques available. By leveraging crt.sh and other CT sources, you can:
- Discover subdomains that have ever had TLS certificates issued for them
- Map infrastructure through historical certificate analysis
- Identify development and staging environments that may be less secure
- Build comprehensive subdomain lists without generating any traffic to the target
- Correlate with other sources to prioritize active targets
While CT logs may not always be applicable in OSCP lab environments (due to internal CAs and self-signed certificates), they are an essential technique for real-world engagements and external assessments. Master this technique as a supplement to your primary subdomain discovery methods.
Next Steps: After gathering subdomains from CT logs, proceed to:
- DNS resolution and validation
- Port scanning on discovered hosts
- Service enumeration on live services
- Web application reconnaissance on HTTP/HTTPS services