1. Introduction
In modern Site Reliability Engineering (SRE) and Linux system administration, web server log analysis is a foundational skill. System administrators regularly analyze access logs to diagnose traffic anomalies, identify scraping bots, investigate Distributed Denial of Service (DDoS) attempts, and audit infrastructure usage.
This article provides a technical breakdown of the Saskatoon challenge from SadServers. The challenge requires analyzing a standard NGINX/Apache HTTP access log to isolate the single IP address generating the highest volume of requests.
Beyond solving the challenge, this guide explores Unix text-processing paradigms, evaluates memory and performance trade-offs between sorted pipelines and single-pass hash tables, and examines how log analysis fits into real-world production incident response and observability workflows.
2. Problem Description and Environment
Challenge Overview
- Scenario Name: Saskatoon: counting IPs
- Difficulty Level: Easy
- Access Level: Non-root user
adminwithsudoprivileges - Target Log File:
/home/admin/access.log - Output Destination:
/home/admin/highestip.txt
Requirements & Constraints
- Parse
/home/admin/access.log, where each line represents a single HTTP request with the requester’s IP address located in the first column. - Determine the single IP address with the highest number of request occurrences (guaranteed unique, no ties).
- Write only the winning IP address string into
/home/admin/highestip.txt.
Verification Criteria
- Hit Count Check: The correct IP address must appear exactly 482 times in
access.log. Runninggrep -c -F -f /home/admin/highestip.txt /home/admin/access.logmust return482. - Checksum Verification: The SHA1 checksum of
/home/admin/highestip.txtmust match6ef426c40652babc0d081d438b9f353709008e93. - Automated Test Script: Submission is verified by executing
/home/admin/agent/check.sh.
3. Initial Investigation and Hypotheses
Before executing commands, a systematic investigation requires formulating hypotheses regarding file size, structural consistency, and field delimiters.
| Hypothesis | Supporting Evidence | Evidence Against | Diagnostic Step |
|---|---|---|---|
| H1: The file follows standard space-delimited NGINX/Apache HTTP Combined Log Format. | Common convention for web server access logs. | Non-standard or custom log formats (e.g., CSV, JSON). | Inspect top lines with head -n 5 access.log. |
H2: The file is small enough to process in-memory using standard POSIX text pipelines (awk, sort, uniq). |
Challenge is categorized as “Easy” on a local virtual machine. | Massive multi-gigabyte log files requiring chunking or streaming. | Count total log lines with wc -l access.log. |
| H3: Field 1 strictly contains clean IP strings without leading whitespace or malformed records. | Standard web server access log specifications. | Malformed log entries or corrupted lines. | Inspect extracted column 1 entries. |
4. Troubleshooting Process
Step 1: Inspecting Log Format and Dataset Size
Objective
Verify the structural delimiter of /home/admin/access.log, confirm that the IP address resides in column 1, and measure total file length.
Command
head -n 5 /home/admin/access.log && wc -l /home/admin/access.log
Command Explanation
head -n 5: Reads and displays the first 5 lines from the target file.&&: Sequential execution control operator; runswc -lonly ifheadsucceeds.wc -l: Counts the number of newline characters, providing total line count.
Output and Analysis
83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:43 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1" 200 171717 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:47 +0000] "GET /presentations/logstash-monitorama-2013/plugin/highlight/highlight.js HTTP/1.1" 200 26185 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:12 +0000] "GET /presentations/logstash-monitorama-2013/plugin/zoom-js/zoom.js HTTP/1.1" 200 7697 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:07 +0000] "GET /presentations/logstash-monitorama-2013/plugin/notes/notes.js HTTP/1.1" 200 2892 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
10000 access.log
Decision
The log consists of 10,000 lines formatted in standard Combined Log Format. Field 1 ($1) is space-delimited and contains the requester’s IP address (e.g., 83.149.9.216). We can safely use column-extraction tools.
Step 2: Extracting, Aggregating, and Ranking IP Frequencies
Objective
Isolate column 1, count occurrences of each distinct IP address, and rank them in descending numerical order.
Command
awk '{print $1}' /home/admin/access.log | sort | uniq -c | sort -nr | head -n 5
Command Explanation
awk '{print $1}': Extracts field 1 (the IP address) from each record.sort: Sorts the raw extracted IP addresses lexicographically. This step is mandatory becauseuniqonly aggregates contiguous identical lines.uniq -c: Replaces adjacent identical lines with a single count-prefixed line.sort -nr: Performs a reverse numerical sort on the output ofuniq -c, placing the highest occurrence counts at the top.head -n 5: Limits output to the top 5 highest-frequency entries.
Output and Analysis
482 66.249.73.135
364 46.105.14.53
357 130.237.218.86
273 75.97.9.59
113 50.16.19.13
Decision
The IP address 66.249.73.135 ranks highest with 482 occurrences. This exactly matches the challenge specification constraint (482 matches).
Step 3: Output Persistence and Verification
Objective
Write 66.249.73.135 to /home/admin/highestip.txt and run validation checks.
Commands Executed
# 1. Write IP address to destination file
echo "66.249.73.135" > /home/admin/highestip.txt
# 2. Verify hit count in source log
grep -c -F -f /home/admin/highestip.txt /home/admin/access.log
# 3. Verify SHA1 hash checksum
sha1sum /home/admin/highestip.txt
# 4. Execute official evaluation script
/home/admin/agent/check.sh
Output and Analysis
482
6ef426c40652babc0d081d438b9f353709008e93 /home/admin/highestip.txt
Status: SUCCESS - Solution verified!
All verification checks returned expected results: exact count of 482, SHA1 checksum matching 6ef426c40652babc0d081d438b9f353709008e93, and successful execution of check.sh.
5. Root Cause Analysis
In this challenge, the target IP 66.249.73.135 accounted for 4.82% of all log traffic (482 out of 10,000 requests).
In real-world web infrastructure, identifying high-frequency IP addresses reveals critical operational context:
- Search Engine Indexing: Reverse DNS lookup (
host 66.249.73.135) revealscrawl-66-249-73-135.googlebot.com. This IP belongs to Googlebot, Google’s web crawling bot. - Scraper / Bot Activity: Automated clients often generate disproportionately high request volumes without respecting
robots.txt. - Application Layer DDoS / Brute Force: Unusually high request density from a single IP can signal an HTTP flood attack or credential stuffing attempt.
6. Solution and Remediation
Primary Solution: Manual File Creation
echo "66.249.73.135" > /home/admin/highestip.txt
Automated One-Liner Pipeline Solution
To automate extraction and output creation without hardcoding the IP string:
awk '{print $1}' /home/admin/access.log | sort | uniq -c | sort -nr | head -n 1 | awk '{print $2}' > /home/admin/highestip.txt
Mechanical Breakdown of Automated Pipeline:
awk '{print $1}': Isolates IP address.sort | uniq -c: Aggregates counts.sort -nr: Ranks descending.head -n 1: Selects482 66.249.73.135.awk '{print $2}': Extracts field 2 (the raw IP string66.249.73.135).>: Truncates target file/home/admin/highestip.txtand writes the output string.
7. Verification and Validation
Verification ensures both statistical correctness and target criteria compliance:
# Verify literal count against original log
grep -c -F -f /home/admin/highestip.txt /home/admin/access.log
# Expected Output: 482
# Verify cryptographic hash of destination file
sha1sum /home/admin/highestip.txt
# Expected Output: 6ef426c40652babc0d081d438b9f353709008e93 /home/admin/highestip.txt
# Run SadServers agent evaluation script
/home/admin/agent/check.sh
Why Each Flag Matters:
grep -c: Returns line count rather than matching lines.grep -F: Treats string as fixed literal (prevents.in IP from acting as a regex wildcard).grep -f <file>: Reads search pattern directly from file.
8. Alternative Troubleshooting Approaches
While awk | sort | uniq -c | sort -nr is standard, alternative text processing tools offer performance and architectural advantages.
Approach A: Single-Pass In-Memory Aggregation (awk Associative Arrays)
awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' /home/admin/access.log | sort -nr | head -n 1
- Mechanism: Maintains an in-memory hash table (
count) insideawk’s runtime. For each line, it increments the value stored at key$1. TheENDblock iterates over the array to output final counts. - Advantages: Avoids sorting un-aggregated raw log lines. Memory usage scales with unique IPs rather than total log lines.
- Complexity: Time $O(N + U \log U)$ where $N$ is total lines and $U$ is unique IPs.
Approach B: High-Speed Stream Processing (datamash)
cut -d ' ' -f 1 /home/admin/access.log | sort | datamash -g 1 count 1 | sort -k2,2nr | head -n 1
- Mechanism: GNU
datamashperforms statistical operations on text streams. - Advantages: Native C implementation optimized for large tabular datasets.
Approach C: Native Python One-Liner
python3 -c '
from collections import Counter
import sys
with open("/home/admin/access.log") as f:
counts = Counter(line.split()[0] for line in f if line.strip())
top_ip, count = counts.most_common(1)[0]
print(f"{count} {top_ip}")
'
- Mechanism: Uses Python’s C-optimized
collections.Counterhash map. - Advantages: Memory-efficient, highly readable, easily handles custom parsing rules or JSON logs.
Performance & Memory Comparison Table
| Approach | Memory Complexity | Time Complexity | Stream Processing? | Best Use Case |
|---|---|---|---|---|
| **`awk | sort | uniq`** | $O(N)$ (Disk/RAM) | $O(N \log N)$ |
awk Hash Map |
$O(U)$ (RAM) | $O(N + U \log U)$ | Partial | Large files where unique keys $\ll$ total lines. |
**Python Counter** |
$O(U)$ (RAM) | $O(N)$ | Yes | Complex multi-field parsing and scripting. |
**GNU datamash** |
$O(U)$ (RAM) | $O(N \log N)$ | Partial | Heavy numeric analysis on structured logs. |
Note: $N$ = total log lines (10,000), $U$ = total unique IP addresses.
9. Linux Command Deep Dive
1. awk
- Purpose: Pattern scanning and processing language.
- Syntax:
awk 'pattern { action }' file - Key Concepts: Splits records into fields (
$1,$2, …$NF) based on field separator (FS, default: contiguous whitespace).
| Flag / Variable | Description | Example Usage |
|---|---|---|
{print $1} |
Prints column 1 | awk '{print $1}' access.log |
-F',' |
Sets field separator to comma | awk -F',' '{print $2}' file.csv |
NR |
Total record number processed so far | awk 'NR==100 {print $0}' access.log |
NF |
Total fields in current line | awk '{print $NF}' access.log (Prints last column) |
2. sort
- Purpose: Sorts lines of text files.
- Syntax:
sort [options] [file]
| Flag | Full Name | Description | Practical Use |
|---|---|---|---|
-n |
--numeric-sort |
Sorts by numerical value instead of string collation. | sort -n (ranks 2 before 10) |
-r |
--reverse |
Reverses sort order (descending). | sort -nr (highest count first) |
-k |
--key=POS1,POS2 |
Sorts based on specific column range. | sort -k 2,2nr (sorts by 2nd column) |
-u |
--unique |
Suppresses duplicate entries (sorts and unique-ifies). | sort -u file.txt |
-T |
--temporary-directory |
Directory for temporary spillover files during big sorts. | sort -T /var/tmp |
3. uniq
- Purpose: Report or omit repeated lines.
- Syntax:
uniq [options] [input [output]] - Crucial Rule:
uniqonly detects adjacent duplicate lines. Input MUST be sorted first!
| Flag | Full Name | Description |
|---|---|---|
-c |
--count |
Prefixes output lines with number of occurrences. |
-d |
--repeated |
Only prints duplicate lines. |
-u |
--unique |
Only prints lines that are not repeated. |
-i |
--ignore-case |
Ignores case differences when comparing. |
4. grep
- Purpose: Print lines matching a pattern.
- Syntax:
grep [options] PATTERN [FILE]
| Flag | Full Name | Description |
|---|---|---|
-c |
--count |
Suppresses normal output; prints total matching line count. |
-F |
--fixed-strings |
Treats pattern as fixed string, disabling regex parsing. |
-f |
--file=FILE |
Takes search patterns from specified file. |
-v |
--invert-match |
Selects non-matching lines. |
-i |
--ignore-case |
Case-insensitive matching. |
10. Underlying Linux Concepts and Architecture
POSIX Pipelines and Anonymous Pipes
The Unix pipeline operator (|) connects the standard output (stdout, file descriptor 1) of one process directly to the standard input (stdin, file descriptor 0) of another process via a kernel-managed FIFO buffer (typically 64 KB in modern Linux kernels).
+-------------------+ Pipe Buffer (64KB) +---------------+
| awk '{print $1}' | === stdout ===> stdin ===> | sort |
+-------------------+ +---------------+
Memory vs. Disk Sorting (sort)
When executing sort, the system buffers incoming streams in memory up to a default allocation threshold. If dataset size exceeds allocated buffer space:
sortsplits input into temporary chunks.- It sorts each chunk in memory and writes temporary spillover files to
/tmp(or$TMPDIR). - It performs a multi-way merge on the sorted temporary files.
Standard Web Log Format (Combined Log Format)
83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET /images/kibana.png HTTP/1.1" 200 203023
| | | | | | |
$1 $2 $3 $4 $6 $8 $9
(Client IP) (Timestamp) (HTTP Request line) (Status)(Bytes)
11. Real-World Sysadmin and SRE Applications
Production Scale Challenges
In high-traffic environments generating tens of gigabytes of logs per hour, running unbuffered text pipelines like cat access.log | sort causes severe disk I/O thrashing and memory exhaustion.
Production Log Analysis Architecture
[ Web Nodes (NGINX/Apache) ]
│
▼ (Log Shipping via Vector / Fluentbit)
[ Log Aggregator / Stream Processor (Kafka / Loki / OpenSearch) ]
│
▼ (Automated Analytics & Rules)
[ WAF / Rate-Limiting Engine (Cloudflare / Fail2ban / iptables) ]
Incident Response Runbook Integration
When top IP extraction reveals malicious traffic (e.g., 5,000 requests/sec from a single IP):
- Immediate Temporary Block (iptables / nftables):
sudo iptables -A INPUT -s 66.249.73.135 -j DROP
- Application Level Rate-Limiting (NGINX
limit_reqModule):
limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;
12. Further Learning and Experiments
Hands-on Experiments
Experiment 1: Measuring Pipeline Execution Time vs. In-Memory Awk
Create a large dummy access log (1 million lines) and compare performance:
# Generate dummy log
for i in {1..1000000}; do echo "192.168.1.$((RANDOM % 254 + 1)) - - [11/Nov/2026:10:00:00 +0000] \"GET / HTTP/1.1\" 200 1024" >> mega_access.log; done
# Benchmark Standard Pipeline
time awk '{print $1}' mega_access.log | sort | uniq -c | sort -nr | head -n 1
# Benchmark In-Memory Hash Table
time awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' mega_access.log | sort -nr | head -n 1
Experiment 2: Analyzing Bandwidth Usage per IP
Find which IP address consumed the most egress bandwidth (summing Column 10):
awk '$10 ~ /^[0-9]+$/ {bytes[$1]+=$10} END {for (ip in bytes) print bytes[ip], ip}' /home/admin/access.log | sort -nr | head -n 5
Authoritative Documentation & References
- Linux Manual Pages:
man 1 awk,man 1 sort,man 1 uniq,man 1 grep - GNU Awk User’s Guide: Free Software Foundation (FSF) Official Documentation
- Nginx HTTP Access Log Module:
nginx.org/en/docs/http/ngx_http_log_module.html
13. Key Takeaways
- Space-Delimited Log Parsing: Default
awksplits lines by contiguous whitespace, making$1the fastest way to extract client IPs from standard access logs. uniqRequires Sorted Input: Always runsortbeforeuniq -c. Passing unsorted streams touniqproduces incorrect counts.- Numeric vs. Lexicographical Sorting: Use
sort -nfor numbers. Without-n,100ranks before2. - Performance Scaling: For large log files, in-memory
awkarray counting (count[$1]++) is significantly faster and uses less memory than pipeline sorting. - Exact Match Verification: Use
grep -Fto treat dots in IP addresses as literal string characters rather than regex wildcards.
14. Quick Troubleshooting Reference / Runbook
# ------------------------------------------------------------------------------
# QUICK LOG ANALYSIS CHEAT SHEET
# ------------------------------------------------------------------------------
# 1. Top 10 IP Addresses by Request Count
awk '{print $1}' access.log | sort | uniq -c | sort -nr | head -n 10
# 2. Top 10 Requested URLs (Column 7 in Standard Combined Log)
awk '{print $7}' access.log | sort | uniq -c | sort -nr | head -n 10
# 3. HTTP Response Status Code Distribution (Column 9)
awk '{print $9}' access.log | sort | uniq -c | sort -nr
# 4. Filter Requests by Specific IP and Count Status Codes
awk '$1 == "66.249.73.135" {print $9}' access.log | sort | uniq -c
# 5. Extract Unique IPs Associated with 404 Not Found Errors
awk '$9 == "404" {print $1}' access.log | sort -u