Domain 10Linux & SRECORE

(Saskatoon) Web Server Access Log Analysis & High-Frequency IP Extraction

A comprehensive guide and analysis of the Saskatoon SadServers challenge, demonstrating text-processing pipelines, log parsing performance, and production SRE log-analysis practices.

Domain: Domain 10: Linux Internals, SRE & Troubleshooting•Updated: 2026-11-11
#Bash#Log-Analysis#Awk#Text-Processing#SysAdmin#SRE

1. Introduction

In modern Site Reliability Engineering (SRE) and Linux system administration, web server log analysis is a foundational skill. System administrators regularly analyze access logs to diagnose traffic anomalies, identify scraping bots, investigate Distributed Denial of Service (DDoS) attempts, and audit infrastructure usage.

This article provides a technical breakdown of the Saskatoon challenge from SadServers. The challenge requires analyzing a standard NGINX/Apache HTTP access log to isolate the single IP address generating the highest volume of requests.

Beyond solving the challenge, this guide explores Unix text-processing paradigms, evaluates memory and performance trade-offs between sorted pipelines and single-pass hash tables, and examines how log analysis fits into real-world production incident response and observability workflows.


2. Problem Description and Environment

Challenge Overview

  • Scenario Name: Saskatoon: counting IPs
  • Difficulty Level: Easy
  • Access Level: Non-root user admin with sudo privileges
  • Target Log File: /home/admin/access.log
  • Output Destination: /home/admin/highestip.txt

Requirements & Constraints

  1. Parse /home/admin/access.log, where each line represents a single HTTP request with the requester’s IP address located in the first column.
  2. Determine the single IP address with the highest number of request occurrences (guaranteed unique, no ties).
  3. Write only the winning IP address string into /home/admin/highestip.txt.

Verification Criteria

  • Hit Count Check: The correct IP address must appear exactly 482 times in access.log. Running grep -c -F -f /home/admin/highestip.txt /home/admin/access.log must return 482.
  • Checksum Verification: The SHA1 checksum of /home/admin/highestip.txt must match 6ef426c40652babc0d081d438b9f353709008e93.
  • Automated Test Script: Submission is verified by executing /home/admin/agent/check.sh.

3. Initial Investigation and Hypotheses

Before executing commands, a systematic investigation requires formulating hypotheses regarding file size, structural consistency, and field delimiters.

Hypothesis Supporting Evidence Evidence Against Diagnostic Step
H1: The file follows standard space-delimited NGINX/Apache HTTP Combined Log Format. Common convention for web server access logs. Non-standard or custom log formats (e.g., CSV, JSON). Inspect top lines with head -n 5 access.log.
H2: The file is small enough to process in-memory using standard POSIX text pipelines (awk, sort, uniq). Challenge is categorized as “Easy” on a local virtual machine. Massive multi-gigabyte log files requiring chunking or streaming. Count total log lines with wc -l access.log.
H3: Field 1 strictly contains clean IP strings without leading whitespace or malformed records. Standard web server access log specifications. Malformed log entries or corrupted lines. Inspect extracted column 1 entries.

4. Troubleshooting Process

Step 1: Inspecting Log Format and Dataset Size

Objective

Verify the structural delimiter of /home/admin/access.log, confirm that the IP address resides in column 1, and measure total file length.

Command

head -n 5 /home/admin/access.log && wc -l /home/admin/access.log

Command Explanation

  • head -n 5: Reads and displays the first 5 lines from the target file.
  • &&: Sequential execution control operator; runs wc -l only if head succeeds.
  • wc -l: Counts the number of newline characters, providing total line count.

Output and Analysis

83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:43 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1" 200 171717 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:47 +0000] "GET /presentations/logstash-monitorama-2013/plugin/highlight/highlight.js HTTP/1.1" 200 26185 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:12 +0000] "GET /presentations/logstash-monitorama-2013/plugin/zoom-js/zoom.js HTTP/1.1" 200 7697 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:07 +0000] "GET /presentations/logstash-monitorama-2013/plugin/notes/notes.js HTTP/1.1" 200 2892 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
10000 access.log

Decision

The log consists of 10,000 lines formatted in standard Combined Log Format. Field 1 ($1) is space-delimited and contains the requester’s IP address (e.g., 83.149.9.216). We can safely use column-extraction tools.


Step 2: Extracting, Aggregating, and Ranking IP Frequencies

Objective

Isolate column 1, count occurrences of each distinct IP address, and rank them in descending numerical order.

Command

awk '{print $1}' /home/admin/access.log | sort | uniq -c | sort -nr | head -n 5

Command Explanation

  1. awk '{print $1}': Extracts field 1 (the IP address) from each record.
  2. sort: Sorts the raw extracted IP addresses lexicographically. This step is mandatory because uniq only aggregates contiguous identical lines.
  3. uniq -c: Replaces adjacent identical lines with a single count-prefixed line.
  4. sort -nr: Performs a reverse numerical sort on the output of uniq -c, placing the highest occurrence counts at the top.
  5. head -n 5: Limits output to the top 5 highest-frequency entries.

Output and Analysis

    482 66.249.73.135
    364 46.105.14.53
    357 130.237.218.86
    273 75.97.9.59
    113 50.16.19.13

Decision

The IP address 66.249.73.135 ranks highest with 482 occurrences. This exactly matches the challenge specification constraint (482 matches).


Step 3: Output Persistence and Verification

Objective

Write 66.249.73.135 to /home/admin/highestip.txt and run validation checks.

Commands Executed

# 1. Write IP address to destination file
echo "66.249.73.135" > /home/admin/highestip.txt

# 2. Verify hit count in source log
grep -c -F -f /home/admin/highestip.txt /home/admin/access.log

# 3. Verify SHA1 hash checksum
sha1sum /home/admin/highestip.txt

# 4. Execute official evaluation script
/home/admin/agent/check.sh

Output and Analysis

482
6ef426c40652babc0d081d438b9f353709008e93  /home/admin/highestip.txt
Status: SUCCESS - Solution verified!

All verification checks returned expected results: exact count of 482, SHA1 checksum matching 6ef426c40652babc0d081d438b9f353709008e93, and successful execution of check.sh.


5. Root Cause Analysis

In this challenge, the target IP 66.249.73.135 accounted for 4.82% of all log traffic (482 out of 10,000 requests).

In real-world web infrastructure, identifying high-frequency IP addresses reveals critical operational context:

  1. Search Engine Indexing: Reverse DNS lookup (host 66.249.73.135) reveals crawl-66-249-73-135.googlebot.com. This IP belongs to Googlebot, Google’s web crawling bot.
  2. Scraper / Bot Activity: Automated clients often generate disproportionately high request volumes without respecting robots.txt.
  3. Application Layer DDoS / Brute Force: Unusually high request density from a single IP can signal an HTTP flood attack or credential stuffing attempt.

6. Solution and Remediation

Primary Solution: Manual File Creation

echo "66.249.73.135" > /home/admin/highestip.txt

Automated One-Liner Pipeline Solution

To automate extraction and output creation without hardcoding the IP string:

awk '{print $1}' /home/admin/access.log | sort | uniq -c | sort -nr | head -n 1 | awk '{print $2}' > /home/admin/highestip.txt

Mechanical Breakdown of Automated Pipeline:

  • awk '{print $1}': Isolates IP address.
  • sort | uniq -c: Aggregates counts.
  • sort -nr: Ranks descending.
  • head -n 1: Selects 482 66.249.73.135.
  • awk '{print $2}': Extracts field 2 (the raw IP string 66.249.73.135).
  • >: Truncates target file /home/admin/highestip.txt and writes the output string.

7. Verification and Validation

Verification ensures both statistical correctness and target criteria compliance:

# Verify literal count against original log
grep -c -F -f /home/admin/highestip.txt /home/admin/access.log
# Expected Output: 482

# Verify cryptographic hash of destination file
sha1sum /home/admin/highestip.txt
# Expected Output: 6ef426c40652babc0d081d438b9f353709008e93  /home/admin/highestip.txt

# Run SadServers agent evaluation script
/home/admin/agent/check.sh

Why Each Flag Matters:

  • grep -c: Returns line count rather than matching lines.
  • grep -F: Treats string as fixed literal (prevents . in IP from acting as a regex wildcard).
  • grep -f <file>: Reads search pattern directly from file.

8. Alternative Troubleshooting Approaches

While awk | sort | uniq -c | sort -nr is standard, alternative text processing tools offer performance and architectural advantages.

Approach A: Single-Pass In-Memory Aggregation (awk Associative Arrays)

awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' /home/admin/access.log | sort -nr | head -n 1
  • Mechanism: Maintains an in-memory hash table (count) inside awk’s runtime. For each line, it increments the value stored at key $1. The END block iterates over the array to output final counts.
  • Advantages: Avoids sorting un-aggregated raw log lines. Memory usage scales with unique IPs rather than total log lines.
  • Complexity: Time $O(N + U \log U)$ where $N$ is total lines and $U$ is unique IPs.

Approach B: High-Speed Stream Processing (datamash)

cut -d ' ' -f 1 /home/admin/access.log | sort | datamash -g 1 count 1 | sort -k2,2nr | head -n 1
  • Mechanism: GNU datamash performs statistical operations on text streams.
  • Advantages: Native C implementation optimized for large tabular datasets.

Approach C: Native Python One-Liner

python3 -c '
from collections import Counter
import sys

with open("/home/admin/access.log") as f:
    counts = Counter(line.split()[0] for line in f if line.strip())
top_ip, count = counts.most_common(1)[0]
print(f"{count} {top_ip}")
'
  • Mechanism: Uses Python’s C-optimized collections.Counter hash map.
  • Advantages: Memory-efficient, highly readable, easily handles custom parsing rules or JSON logs.

Performance & Memory Comparison Table

Approach Memory Complexity Time Complexity Stream Processing? Best Use Case
**`awk sort uniq`** $O(N)$ (Disk/RAM) $O(N \log N)$
awk Hash Map $O(U)$ (RAM) $O(N + U \log U)$ Partial Large files where unique keys $\ll$ total lines.
**Python Counter** $O(U)$ (RAM) $O(N)$ Yes Complex multi-field parsing and scripting.
**GNU datamash** $O(U)$ (RAM) $O(N \log N)$ Partial Heavy numeric analysis on structured logs.

Note: $N$ = total log lines (10,000), $U$ = total unique IP addresses.


9. Linux Command Deep Dive

1. awk

  • Purpose: Pattern scanning and processing language.
  • Syntax: awk 'pattern { action }' file
  • Key Concepts: Splits records into fields ($1, $2, … $NF) based on field separator (FS, default: contiguous whitespace).
Flag / Variable Description Example Usage
{print $1} Prints column 1 awk '{print $1}' access.log
-F',' Sets field separator to comma awk -F',' '{print $2}' file.csv
NR Total record number processed so far awk 'NR==100 {print $0}' access.log
NF Total fields in current line awk '{print $NF}' access.log (Prints last column)

2. sort

  • Purpose: Sorts lines of text files.
  • Syntax: sort [options] [file]
Flag Full Name Description Practical Use
-n --numeric-sort Sorts by numerical value instead of string collation. sort -n (ranks 2 before 10)
-r --reverse Reverses sort order (descending). sort -nr (highest count first)
-k --key=POS1,POS2 Sorts based on specific column range. sort -k 2,2nr (sorts by 2nd column)
-u --unique Suppresses duplicate entries (sorts and unique-ifies). sort -u file.txt
-T --temporary-directory Directory for temporary spillover files during big sorts. sort -T /var/tmp

3. uniq

  • Purpose: Report or omit repeated lines.
  • Syntax: uniq [options] [input [output]]
  • Crucial Rule: uniq only detects adjacent duplicate lines. Input MUST be sorted first!
Flag Full Name Description
-c --count Prefixes output lines with number of occurrences.
-d --repeated Only prints duplicate lines.
-u --unique Only prints lines that are not repeated.
-i --ignore-case Ignores case differences when comparing.

4. grep

  • Purpose: Print lines matching a pattern.
  • Syntax: grep [options] PATTERN [FILE]
Flag Full Name Description
-c --count Suppresses normal output; prints total matching line count.
-F --fixed-strings Treats pattern as fixed string, disabling regex parsing.
-f --file=FILE Takes search patterns from specified file.
-v --invert-match Selects non-matching lines.
-i --ignore-case Case-insensitive matching.

10. Underlying Linux Concepts and Architecture

POSIX Pipelines and Anonymous Pipes

The Unix pipeline operator (|) connects the standard output (stdout, file descriptor 1) of one process directly to the standard input (stdin, file descriptor 0) of another process via a kernel-managed FIFO buffer (typically 64 KB in modern Linux kernels).

+-------------------+      Pipe Buffer (64KB)      +---------------+
| awk '{print $1}'  | === stdout ===> stdin ===> |  sort         |
+-------------------+                              +---------------+

Memory vs. Disk Sorting (sort)

When executing sort, the system buffers incoming streams in memory up to a default allocation threshold. If dataset size exceeds allocated buffer space:

  1. sort splits input into temporary chunks.
  2. It sorts each chunk in memory and writes temporary spillover files to /tmp (or $TMPDIR).
  3. It performs a multi-way merge on the sorted temporary files.

Standard Web Log Format (Combined Log Format)

83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET /images/kibana.png HTTP/1.1" 200 203023
|            | |  |                           |                                 |   |
$1           $2 $3 $4                          $6                                $8  $9
(Client IP)       (Timestamp)                 (HTTP Request line)              (Status)(Bytes)

11. Real-World Sysadmin and SRE Applications

Production Scale Challenges

In high-traffic environments generating tens of gigabytes of logs per hour, running unbuffered text pipelines like cat access.log | sort causes severe disk I/O thrashing and memory exhaustion.

Production Log Analysis Architecture

[ Web Nodes (NGINX/Apache) ] 
       │
       ▼ (Log Shipping via Vector / Fluentbit)
[ Log Aggregator / Stream Processor (Kafka / Loki / OpenSearch) ]
       │
       ▼ (Automated Analytics & Rules)
[ WAF / Rate-Limiting Engine (Cloudflare / Fail2ban / iptables) ]

Incident Response Runbook Integration

When top IP extraction reveals malicious traffic (e.g., 5,000 requests/sec from a single IP):

  1. Immediate Temporary Block (iptables / nftables):
sudo iptables -A INPUT -s 66.249.73.135 -j DROP
  1. Application Level Rate-Limiting (NGINX limit_req Module):
limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;

12. Further Learning and Experiments

Hands-on Experiments

Experiment 1: Measuring Pipeline Execution Time vs. In-Memory Awk

Create a large dummy access log (1 million lines) and compare performance:

# Generate dummy log
for i in {1..1000000}; do echo "192.168.1.$((RANDOM % 254 + 1)) - - [11/Nov/2026:10:00:00 +0000] \"GET / HTTP/1.1\" 200 1024" >> mega_access.log; done

# Benchmark Standard Pipeline
time awk '{print $1}' mega_access.log | sort | uniq -c | sort -nr | head -n 1

# Benchmark In-Memory Hash Table
time awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' mega_access.log | sort -nr | head -n 1

Experiment 2: Analyzing Bandwidth Usage per IP

Find which IP address consumed the most egress bandwidth (summing Column 10):

awk '$10 ~ /^[0-9]+$/ {bytes[$1]+=$10} END {for (ip in bytes) print bytes[ip], ip}' /home/admin/access.log | sort -nr | head -n 5

Authoritative Documentation & References

  • Linux Manual Pages: man 1 awk, man 1 sort, man 1 uniq, man 1 grep
  • GNU Awk User’s Guide: Free Software Foundation (FSF) Official Documentation
  • Nginx HTTP Access Log Module: nginx.org/en/docs/http/ngx_http_log_module.html

13. Key Takeaways

  1. Space-Delimited Log Parsing: Default awk splits lines by contiguous whitespace, making $1 the fastest way to extract client IPs from standard access logs.
  2. uniq Requires Sorted Input: Always run sort before uniq -c. Passing unsorted streams to uniq produces incorrect counts.
  3. Numeric vs. Lexicographical Sorting: Use sort -n for numbers. Without -n, 100 ranks before 2.
  4. Performance Scaling: For large log files, in-memory awk array counting (count[$1]++) is significantly faster and uses less memory than pipeline sorting.
  5. Exact Match Verification: Use grep -F to treat dots in IP addresses as literal string characters rather than regex wildcards.

14. Quick Troubleshooting Reference / Runbook

# ------------------------------------------------------------------------------
# QUICK LOG ANALYSIS CHEAT SHEET
# ------------------------------------------------------------------------------

# 1. Top 10 IP Addresses by Request Count
awk '{print $1}' access.log | sort | uniq -c | sort -nr | head -n 10

# 2. Top 10 Requested URLs (Column 7 in Standard Combined Log)
awk '{print $7}' access.log | sort | uniq -c | sort -nr | head -n 10

# 3. HTTP Response Status Code Distribution (Column 9)
awk '{print $9}' access.log | sort | uniq -c | sort -nr

# 4. Filter Requests by Specific IP and Count Status Codes
awk '$1 == "66.249.73.135" {print $9}' access.log | sort | uniq -c

# 5. Extract Unique IPs Associated with 404 Not Found Errors
awk '$9 == "404" {print $1}' access.log | sort -u