Domain 8Data InfrastructureCORE

(Flip) Reconciling Corrupted FIX Execution Logs with AWK and Relational Stream Processing

A systematic SRE troubleshooting guide to diagnosing symbol-flip corruption in Financial Information eXchange (FIX) logs using multi-file AWK hash joins, cross-validation, and POSIX stream utilities.

Domain: Domain 8: Data Infrastructure•Updated: 2026-11-12
#Linux#Troubleshooting#AWK#Log Analysis#FIX Protocol#SRE#Bash

1. Introduction

Production systems rarely fail exclusively through crashed daemons or exhausted disk space. Some of the most damaging incidents in financial, telemetry, and distributed backend infrastructure involve silent data corruption—where services remain up, return 200 OK or valid protocol checksums, and continue writing logs, yet emit semantically wrong fields.

In this troubleshooting exercise (based on the SadServers “Flip”: algo symbol flips scenario), an electronic trading execution algorithm intermittently stamps the wrong stock ticker symbol (55=Symbol) onto outbound trade execution reports. Because downstream clearing, risk, and accounting pipelines rely on execution logs to book trades, even a small percentage of mislabeled fills can cause severe financial discrepancies.

For a Linux systems administrator or Site Reliability Engineer (SRE), resolving this class of incident requires fluency in user-space text and stream processing. Rather than manually inspecting thousands of structured protocol messages, you need to correlate multiple log streams, enforce domain invariants, verify data completeness in read-only mode, and generate an exact, auditable remediation dataset.

In this article, you will learn:

  • How to structure a hypothesis-driven investigation for structured application logs.
  • How to parse tag-value protocols (such as FIX) safely without relying on brittle column offsets.
  • How to perform single-pass relational joins across multiple files in awk using FNR == NR and associative arrays.
  • How to cross-validate remediation data against independent invariants before writing output files.
  • Alternative approaches using python3 and disk-backed POSIX sort/join pipelines, along with their memory and scalability trade-offs.

2. Problem Description and Environment

Challenge Overview

The Financial Information eXchange (FIX) protocol transmits trading messages as sequences of tag=value pairs delimited by an ASCII SOH (0x01) character, rendered in these log files as a pipe (|). Each message begins with 8= (BeginString), 9= (BodyLength), and 35= (MsgType), and ends with 10= (CheckSum).

A bug in the execution algorithm causes some execution fills (35=8) to be stamped with an incorrect 55=Symbol (for example, an AAPL execution labeled DELL, or GOOG labeled INTC).

Available Files and FIX Tag Dictionary

Three data files and one grading script are present on the system:

  • /home/admin/market/BANDS.csv: Reference market data containing symbol,min_px,max_px rows with non-overlapping price bands.
  • /home/admin/fix/orders.log: Outbound order logs (35=D, NewOrderSingle) containing the client order ID (11=ClOrdID) and the authoritative symbol (55=Symbol).
  • /home/admin/fix/executions.log: Execution fill logs (35=8, ExecutionReport) containing a unique fill ID (17=ExecID), parent order link (11=ClOrdID), as-stamped 55=Symbol, and fill price (31=LastPx).
  • /home/admin/agent/check.sh: Automated verification script validating the output artifact.
FIX Tag Field Name Role in Investigation
35 MsgType D = NewOrderSingle (order); 8 = ExecutionReport (fill)
11 ClOrdID Client order ID linking one or more fills back to their parent order
17 ExecID Unique identifier per fill (e.g., EX000123)
55 Symbol True ticker symbol on 35=D orders; potentially flipped on 35=8 fills
31 LastPx Execution price of the individual fill

Expected vs. Actual Behaviour

  • Expected Behaviour: Every fill (35=8) in /home/admin/fix/executions.log carries the exact same 55=Symbol as its parent order (35=D) in /home/admin/fix/orders.log (joined via 11=ClOrdID), and its execution price (31=LastPx) falls strictly within that symbol’s [min_px, max_px] range in /home/admin/market/BANDS.csv.
  • Actual Behaviour: Approximately 25 out of ~960 fills in executions.log are stamped with a mismatched 55=Symbol whose price band contradicts 31=LastPx and disagrees with the parent order in orders.log.
  • Remediation Goal: Create /home/admin/fix/corrections.csv containing one ExecID,correct_symbol pair per line for only the flipped fills, with no missing rows, false positives, or duplicates.

Environment Details

  • Host & User: admin@i-051025d49a1b03ff5 (with sudo / root privileges available, though all target files reside under /home/admin/ and are readable/writable by admin).
  • OS Distribution & Kernel: Specific Linux distribution and kernel version (uname -a, /etc/os-release) were not queried during this session because the task operated entirely in user space on application log files.

3. Initial Investigation and Hypotheses

Before writing any parsing scripts, an SRE establishes what is known, what is unknown, and what edge cases could corrupt the analysis. Jumping straight into a one-liner that assumes fixed column positions or clean 1-to-1 order-to-fill mappings often leads to subtle data bugs.

What Was Known vs. Unknown

  • Known:

  • Two independent detection signals exist: relational lookup via 11=ClOrdID (orders.log vs. executions.log) and price-range validation via 31=LastPx (executions.log vs. BANDS.csv).

  • Roughly 25 fills out of ~960 are affected.

  • Unknown:

  1. CSV Format Expectations: Does /home/admin/agent/check.sh expect a header row in corrections.csv (ExecID,correct_symbol), or only data rows?
  2. Message Purity: Do orders.log and executions.log contain only 35=D and 35=8 messages, or are session-level FIX messages (such as heartbeats 35=0, logons 35=A, or order cancels 35=F) mixed in?
  3. Cardinality: Is there a 1-to-1 relationship between orders and executions, or can a single order produce multiple partial fills?
  4. Tag Ordering: Are FIX tags always at fixed pipe-delimited column indices, or can optional tags shift field offsets across lines?
  5. Referential Completeness: Does every 11=ClOrdID in executions.log exist in orders.log, or are some parent orders missing (which would require falling back on BANDS.csv price-band inference)?

Hypotheses Considered

Hypothesis Supporting Evidence Evidence Against Diagnostic Step
1. Deterministic or semi-random symbol flip during fill serialization Problem statement notes AAPL labeled DELL and GOOG labeled INTC while order records hold the true symbol. None initially; requires comparing 55= across joined logs and price bands. Join orders.log and executions.log on 11=ClOrdID and cross-check 31=LastPx against BANDS.csv.
2. Variable FIX tag positions or log prefixes breaking column-based parsing Real-world FIX engines frequently include prefix timestamps/directions (SENT ) and variable-length tag sets between orders and fills. Unknown until raw log lines are inspected. Inspect raw lines with head and parse fields by tag= prefix rather than fixed column index.
**3. Orphaned executions missing from orders.log** Multi-file log rotations or truncated order logs can leave fills without a matching 35=D record. Unknown until referential integrity is tested across all 960 fills. Run a read-only diagnostic comparing order_sym and band_sym for every execution.
4. Mixed FIX message types (35=0, 35=F, etc.) in log streams Standard FIX session logs record all inbound/outbound traffic in a single file. Problem description specifies 35=D in orders.log and 35=8 in executions.log. Extract tag 35= across every line in both logs and aggregate counts with `sort

4. Troubleshooting Process

Step 1: Initial File Structure and Volume Inspection

Objective

Inspect the exact formatting, delimiters, header rows, and line counts of /home/admin/market/BANDS.csv, /home/admin/fix/orders.log, and /home/admin/fix/executions.log using read-only commands.

Command

head -n 5 /home/admin/market/BANDS.csv /home/admin/fix/orders.log /home/admin/fix/executions.log
wc -l /home/admin/market/BANDS.csv /home/admin/fix/orders.log /home/admin/fix/executions.log

Command Explanation

  • head -n 5 ...: Prints the first 5 lines of each specified file, automatically inserting ==> <filename> <== banners when given multiple file arguments.
  • wc -l ...: Counts newline characters (\n) in each file to establish exact record volumes before processing.

Output and Analysis

==> /home/admin/market/BANDS.csv <==
symbol,min_px,max_px
GOOG,20.00,30.00
INTC,68.00,78.00
DELL,95.00,110.00
AAPL,170.00,185.00

==> /home/admin/fix/orders.log <==
SENT 8=FIX.4.2|9=93|35=D|49=ROUTER|56=EXCH|34=1|52=20260916-09:30:00.000|11=C00001|55=DELL|54=2|38=1000|44=96.11|10=058|
SENT 8=FIX.4.2|9=93|35=D|49=ROUTER|56=EXCH|34=2|52=20260916-09:30:00.392|11=C00002|55=NVDA|54=2|38=300|44=879.55|10=101|
SENT 8=FIX.4.2|9=93|35=D|49=ROUTER|56=EXCH|34=3|52=20260916-09:30:00.799|11=C00003|55=AAPL|54=2|38=300|44=174.85|10=094|
SENT 8=FIX.4.2|9=93|35=D|49=ROUTER|56=EXCH|34=4|52=20260916-09:30:01.136|11=C00004|55=NVDA|54=1|38=500|44=879.62|10=101|
SENT 8=FIX.4.2|9=92|35=D|49=ROUTER|56=EXCH|34=5|52=20260916-09:30:01.591|11=C00005|55=INTC|54=2|38=100|44=71.00|10=038|

==> /home/admin/fix/executions.log <==
SENT 8=FIX.4.2|9=140|35=8|49=EXCH|56=ROUTER|34=1|52=20260916-09:30:00.186|17=EX000001|11=C00001|37=O00001|150=F|39=2|55=DELL|54=2|32=1000|31=96.11|14=1000|151=0|10=192|
SENT 8=FIX.4.2|9=139|35=8|49=EXCH|56=ROUTER|34=2|52=20260916-09:30:00.485|17=EX000002|11=C00002|37=O00002|150=F|39=2|55=NVDA|54=2|32=150|31=879.55|14=150|151=0|10=185|
SENT 8=FIX.4.2|9=139|35=8|49=EXCH|56=ROUTER|34=3|52=20260916-09:30:00.732|17=EX000003|11=C00002|37=O00002|150=F|39=2|55=NVDA|54=2|32=150|31=879.55|14=150|151=0|10=182|
SENT 8=FIX.4.2|9=139|35=8|49=EXCH|56=ROUTER|34=4|52=20260916-09:30:01.022|17=EX000004|11=C00003|37=O00003|150=F|39=2|55=AAPL|54=2|32=300|31=174.85|14=300|151=0|10=153|
SENT 8=FIX.4.2|9=139|35=8|49=EXCH|56=ROUTER|34=5|52=20260916-09:30:01.371|17=EX000005|11=C00004|37=O00004|150=F|39=2|55=NVDA|54=1|32=500|31=879.62|14=500|151=0|10=185|
      7 /home/admin/market/BANDS.csv
    700 /home/admin/fix/orders.log
    960 /home/admin/fix/executions.log
   1667 total

Key takeaways from this output:

  1. Log Prefix & Delimiters: Every FIX log line starts with SENT immediately followed by 8=FIX.4.2, with pipe-delimited (|) fields and a trailing |.
  2. Column Offset Differences: In orders.log, 11=ClOrdID is pipe-field $8 and 55=Symbol is $9. In executions.log, 17=ExecID occupies $8, shifting 11=ClOrdID to $9 and 55=Symbol to $13. Hardcoding column indices across files would fail.
  3. One-to-Many Order-to-Fill Cardinality: orders.log has 700 lines while executions.log has 960 lines. Looking at 11=C00002 (an order for 38=300 shares of NVDA), it splits into two separate fills (17=EX000002 and 17=EX000003, each with 32=150 shares). Therefore, 11=ClOrdID is a non-unique foreign key in executions.log.

Decision

Read the entirety of BANDS.csv, inspect /home/admin/agent/check.sh to confirm the exact output contract, and verify whether all 1,660 log lines contain strictly 35=D and 35=8 messages.


Step 2: Inspecting Reference Bands, Validation Rules, and Message Purity

Objective

Confirm all symbol price intervals, understand how check.sh grades corrections.csv, and verify that no unexpected FIX message types exist in either log file.

Command

cat /home/admin/market/BANDS.csv
cat /home/admin/agent/check.sh
awk -F'|' '{print $3}' /home/admin/fix/orders.log /home/admin/fix/executions.log | sort | uniq -c

Command Explanation

  • cat /home/admin/market/BANDS.csv: Displays all 7 lines of the symbol reference table.
  • cat /home/admin/agent/check.sh: Displays the validation script (permitted by the challenge instructions) so we can verify formatting, sorting, and header rules.
  • awk -F'|' '{print $3}' ... | sort | uniq -c: Splits every line of both FIX logs on |, extracts the 3rd field (35=<MsgType>), sorts the values, and counts occurrences of each distinct message type.

Output and Analysis

symbol,min_px,max_px
GOOG,20.00,30.00
INTC,68.00,78.00
DELL,95.00,110.00
AAPL,170.00,185.00
MSFT,400.00,420.00
NVDA,850.00,880.00
#!/bin/bash
# DO NOT MODIFY THIS FILE ("Check My Solution" will fail)

D=/home/admin/fix
F="$D/corrections.csv"
T=$(mktemp -d) || { echo -n "NO"; exit 0; }
trap 'rm -rf "$T"' EXIT
[ -f "$F" ] || { echo -n "NO"; exit 0; }
EXP_HASH="c3d1e3fa97cee2bd0acf0e779fcd13a147c23441a7e2d15c3729e5cc726d4fb9"
EXP_COUNT=25
awk 'BEGIN{FS="[,; \t]+"} {sub(/\r$/,"");sub(/^[ \t]+/,"");sub(/[ \t]+$/,"");if($0=="")next;if(NR==1&&$0~/^[Ee][Xx][Ee][Cc]/)next;e="";s="";U=toupper($0);if(match(U,/17=EX[0-9]+/)){e=substr(U,RSTART+3,RLENGTH-3);s=$2}else{e=toupper($1);s=$2}if(e=="")exit 1;if(match(e,/EX[0-9]+/)){e=substr(e,RSTART,RLENGTH)}else exit 1;if(s=="")exit 1;s=toupper(s);gsub(/[^A-Z]/,"",s);if(e!~/^EX[0-9]+$/)exit 1;if(s!~/^(GOOG|INTC|DELL|AAPL|MSFT|NVDA)$/)exit 1;print e","s}' "$F" > "$T/raw" 2>/dev/null || { echo -n "NO"; exit 0; }
[ -s "$T/raw" ] || { echo -n "NO"; exit 0; }
if [ "$(wc -l < "$T/raw")" != "$(sort -u "$T/raw" | wc -l)" ]; then echo -n "NO"; exit 0; fi
sort -u "$T/raw" > "$T/got"
[ "$(wc -l < "$T/got")" = "$EXP_COUNT" ] || { echo -n "NO"; exit 0; }
if [ "$(sha256sum "$T/got" | cut -d" " -f1)" = "$EXP_HASH" ]; then echo -n "OK"; else echo -n "NO"; fi
exit 0
    960 35=8
    700 35=D

Key findings:

  1. Pure Message Streams: Every single line in orders.log (700) is 35=D and every single line in executions.log (960) is 35=8. There are no administrative or cancel messages to filter out.
  2. Disjoint Market Bands: All 6 symbols occupy wide, non-overlapping price ranges (20–30, 68–78, 95–110, 170–185, 400–420, 850–880).
  3. Validation Contract (check.sh):
  • Expects /home/admin/fix/corrections.csv to exist and contain 25 unique EX<digits>,<SYMBOL> rows (EXP_COUNT=25).
  • Normalizes whitespace and \r, skips an optional header line starting with Exec (case-insensitive), rejects duplicates (wc -l vs. sort -u), sorts rows, and verifies the SHA-256 digest against EXP_HASH.

Decision

Run a single-pass, read-only awk diagnostic across all three files (BANDS.csv, orders.log, and executions.log) to cross-validate both detection hypotheses before creating corrections.csv.


Step 3: Cross-Validating Referential Integrity and Market Price Bands

Objective

Audit all 960 executions in read-only mode to check whether any execution’s stamped 55=Symbol disagrees with its parent order’s symbol (order_sym, via 11=ClOrdID) or its price-band symbol (band_sym, via 31=LastPx), and detect any missing parent orders or out-of-band prices.

Command

awk -F'[,|]' '
FNR == NR {
    if (FNR > 1) {
        min[$1] = $2 + 0
        max[$1] = $3 + 0
    }
    next
}
FILENAME ~ /orders\.log$/ {
    id = ""; sym = ""
    for (i = 1; i <= NF; i++) {
        if ($i ~ /^11=/) id = substr($i, 4)
        else if ($i ~ /^55=/) sym = substr($i, 4)
    }
    if (id != "") ord_sym[id] = sym
    next
}
{
    exec_id = ""; id = ""; sym = ""; px = 0
    for (i = 1; i <= NF; i++) {
        if ($i ~ /^17=/) exec_id = substr($i, 4)
        else if ($i ~ /^11=/) id = substr($i, 4)
        else if ($i ~ /^55=/) sym = substr($i, 4)
        else if ($i ~ /^31=/) px = substr($i, 4) + 0
    }
    band_sym = "UNKNOWN"
    for (s in min) {
        if (px >= min[s] && px <= max[s]) band_sym = s
    }
    true_sym = (id in ord_sym) ? ord_sym[id] : "MISSING_ORDER"
    if (sym != true_sym || sym != band_sym) {
        printf "%s | ClOrdID=%s | stamped=%s | order_sym=%s | band_sym=%s (px=%.2f)\n",
               exec_id, id, sym, true_sym, band_sym, px
    }
}' /home/admin/market/BANDS.csv /home/admin/fix/orders.log /home/admin/fix/executions.log

Command Explanation

  • -F'[,|]': Sets the field separator to a regular expression matching either , (for BANDS.csv) or | (for the FIX log files).
  • FNR == NR: True only while reading the first input file (BANDS.csv), loading min[symbol] and max[symbol] as numeric values (+ 0).
  • FILENAME ~ /orders\.log$/: Matches the second file (orders.log), scanning fields for ^11= and ^55= to populate the ord_sym[ClOrdID] associative array.
  • Final block (processes executions.log): Extracts 17=ExecID, 11=ClOrdID, 55=Symbol, and 31=LastPx, determines which price band contains px, looks up ord_sym[id], and prints any record where stamped differs from either order_sym or band_sym.

Output and Analysis

EX000108 | ClOrdID=C00076 | stamped=MSFT | order_sym=NVDA | band_sym=NVDA (px=852.20)
EX000136 | ClOrdID=C00095 | stamped=NVDA | order_sym=MSFT | band_sym=MSFT (px=408.83)
EX000150 | ClOrdID=C00105 | stamped=INTC | order_sym=GOOG | band_sym=GOOG (px=22.28)
EX000177 | ClOrdID=C00124 | stamped=INTC | order_sym=GOOG | band_sym=GOOG (px=26.31)
EX000223 | ClOrdID=C00156 | stamped=GOOG | order_sym=INTC | band_sym=INTC (px=77.33)
EX000226 | ClOrdID=C00157 | stamped=AAPL | order_sym=DELL | band_sym=DELL (px=109.63)
EX000287 | ClOrdID=C00200 | stamped=INTC | order_sym=GOOG | band_sym=GOOG (px=24.47)
EX000302 | ClOrdID=C00213 | stamped=DELL | order_sym=AAPL | band_sym=AAPL (px=174.41)
EX000312 | ClOrdID=C00220 | stamped=DELL | order_sym=AAPL | band_sym=AAPL (px=180.23)
EX000414 | ClOrdID=C00291 | stamped=GOOG | order_sym=INTC | band_sym=INTC (px=68.37)
EX000444 | ClOrdID=C00312 | stamped=AAPL | order_sym=DELL | band_sym=DELL (px=97.57)
EX000464 | ClOrdID=C00327 | stamped=NVDA | order_sym=MSFT | band_sym=MSFT (px=411.01)
EX000488 | ClOrdID=C00345 | stamped=DELL | order_sym=AAPL | band_sym=AAPL (px=175.02)
EX000502 | ClOrdID=C00356 | stamped=INTC | order_sym=GOOG | band_sym=GOOG (px=25.67)
EX000536 | ClOrdID=C00381 | stamped=MSFT | order_sym=NVDA | band_sym=NVDA (px=875.51)
EX000545 | ClOrdID=C00388 | stamped=INTC | order_sym=GOOG | band_sym=GOOG (px=23.65)
EX000596 | ClOrdID=C00425 | stamped=NVDA | order_sym=MSFT | band_sym=MSFT (px=407.93)
EX000681 | ClOrdID=C00493 | stamped=AAPL | order_sym=DELL | band_sym=DELL (px=100.05)
EX000787 | ClOrdID=C00569 | stamped=GOOG | order_sym=INTC | band_sym=INTC (px=70.95)
EX000794 | ClOrdID=C00574 | stamped=DELL | order_sym=AAPL | band_sym=AAPL (px=181.01)
EX000813 | ClOrdID=C00589 | stamped=NVDA | order_sym=MSFT | band_sym=MSFT (px=402.74)
EX000862 | ClOrdID=C00627 | stamped=MSFT | order_sym=NVDA | band_sym=NVDA (px=862.94)
EX000901 | ClOrdID=C00654 | stamped=NVDA | order_sym=MSFT | band_sym=MSFT (px=401.27)
EX000919 | ClOrdID=C00667 | stamped=AAPL | order_sym=DELL | band_sym=DELL (px=101.83)
EX000947 | ClOrdID=C00690 | stamped=INTC | order_sym=GOOG | band_sym=GOOG (px=27.76)

This output provides complete clarity:

  • Exact Count: Exactly 25 lines are returned, matching EXP_COUNT=25 in check.sh.
  • 100% Cross-Method Agreement: On every single flipped fill, order_sym and band_sym match identically. There are zero MISSING_ORDER records and zero UNKNOWN out-of-band prices.

Decision

Generate /home/admin/fix/corrections.csv using a clean two-file awk hash join between orders.log and executions.log, then verify the output file and run /home/admin/agent/check.sh.


5. Root Cause Analysis

Mechanism of the Failure

Comparing the stamped symbol against the true order_sym across all 25 corrupted fills reveals a strict deterministic pattern tied directly to the symbol order in /home/admin/market/BANDS.csv:

BANDS.csv Index True Symbol Stamped Symbol on Flip Swap Pair
0 GOOG INTC 0 ↔ 1
1 INTC GOOG 0 ↔ 1
2 DELL AAPL 2 ↔ 3
3 AAPL DELL 2 ↔ 3
4 MSFT NVDA 4 ↔ 5
5 NVDA MSFT 4 ↔ 5

Every single corrupted fill swapped adjacent symbol indices (0↔1, 2↔3, 4↔5). In systems programming, this exact signature (idx ^ 1) occurs when a bitwise XOR or off-by-one array indexing error corrupts the lookup index during serialization of the outbound ExecutionReport (35=8).

flowchart LR
    A["Order Router<br/>(35=D NewOrderSingle)"] -->|"Logs true 55=Symbol<br/>& 11=ClOrdID"| B[("/home/admin/fix/orders.log")]
    A -->|"Routes order"| C["Execution Algo Engine"]
    C -->|"Bug: idx ^ 1 swaps<br/>adjacent symbol on 35=8"| D[("/home/admin/fix/executions.log")]
    B -->|"AWK Hash Join<br/>on 11=ClOrdID"| E["/home/admin/fix/corrections.csv"]
    D -->|"Compare stamped 55=<br/>vs. order 55="| E

Distinguishing Root Cause from Secondary Symptoms

  • Root Cause: Indexing/serialization defect in the execution algorithm when populating tag 55=Symbol on 35=8 messages.
  • Primary Symptom: Out-of-band fill prices (31=LastPx) relative to the stamped symbol (e.g., EX000108 executing at $852.20 while stamped MSFT, whose valid band is $400.00–$420.00).
  • Why Recovery Was Unambiguous: Because the upstream order router persisted the uncorrupted 35=D message in orders.log before execution occurred, orders.log served as an intact source of truth.

6. Solution and Remediation

Changes Applied

No existing system files or logs were modified. A single new output file, /home/admin/fix/corrections.csv, was generated from read-only inputs. Rolling back this change requires only removing that file (rm -f /home/admin/fix/corrections.csv).

Command Executed

awk -F'|' '
FNR == NR {
    id = ""; sym = ""
    for (i = 1; i <= NF; i++) {
        if ($i ~ /^11=/) id = substr($i, 4)
        else if ($i ~ /^55=/) sym = substr($i, 4)
    }
    if (id != "") ord_sym[id] = sym
    next
}
{
    exec_id = ""; id = ""; sym = ""
    for (i = 1; i <= NF; i++) {
        if ($i ~ /^17=/) exec_id = substr($i, 4)
        else if ($i ~ /^11=/) id = substr($i, 4)
        else if ($i ~ /^55=/) sym = substr($i, 4)
    }
    if (id in ord_sym && sym != ord_sym[id]) {
        print exec_id "," ord_sym[id]
    }
}' /home/admin/fix/orders.log /home/admin/fix/executions.log > /home/admin/fix/corrections.csv

How It Works

  1. In-Memory Lookup Construction (FNR == NR): While reading /home/admin/fix/orders.log, awk scans each pipe-separated token for 11= (ClOrdID) and 55= (Symbol), storing ord_sym[id] = sym in an associative array. The next statement skips the second rule block until orders.log is exhausted.
  2. Stream Comparison (Second Rule Block): When awk transitions to /home/admin/fix/executions.log, FNR resets to 1 while NR continues incrementing from 701, so FNR == NR evaluates to false. For each fill, awk extracts 17=ExecID, 11=ClOrdID, and 55=Symbol.
  3. Defensive Key Lookup (id in ord_sym): Checking id in ord_sym before evaluating ord_sym[id] prevents awk from implicitly creating empty entries for missing keys. Whenever sym != ord_sym[id], it emits ExecID,true_symbol to /home/admin/fix/corrections.csv.

7. Verification and Validation

In Linux administration, verifying that a command exited with status 0 is never enough—you must verify the resulting filesystem state and application-level acceptance criteria.

Commands Executed

wc -l /home/admin/fix/corrections.csv
head -n 5 /home/admin/fix/corrections.csv
agent/check.sh

Verification Output

25 /home/admin/fix/corrections.csv
EX000108,NVDA
EX000136,MSFT
EX000150,GOOG
EX000177,GOOG
EX000223,INTC
OK

Why Each Check Mattered

  1. wc -l /home/admin/fix/corrections.csv (25): Confirmed exact row count matching EXP_COUNT=25 without blank lines or extra header rows.
  2. head -n 5 /home/admin/fix/corrections.csv: Confirmed exact CSV formatting (EX000108,NVDA) with no stray 17= or 55= prefixes, spaces, or trailing pipes.
  3. agent/check.sh (OK): Validated uniqueness (sort -u), regex conformance (^EX[0-9]+$ and valid ticker symbol), exact record count, and the canonical SHA-256 digest (c3d1e3fa97cee2bd0acf0e779fcd13a147c23441a7e2d15c3729e5cc726d4fb9).

8. Alternative Troubleshooting Approaches

The following approaches were not executed during the live session, but represent standard SRE alternatives depending on dataset size, tooling availability, and operational constraints.

Alternative 1: Structured Parsing with Python (python3)

  • Underlying Idea: Parse each FIX message into a native Python dict mapping tag strings to values, perform an $O(1)$ dictionary lookup on ClOrdID, and write the CSV output.
  • Example Command:
python3 -c '
def parse_fix(line):
    return dict(f.split("=", 1) for f in line.strip().split("|") if "=" in f)

with open("/home/admin/fix/orders.log") as f:
    orders = {m["11"]: m["55"] for m in map(parse_fix, f)}

with open("/home/admin/fix/executions.log") as f, open("/home/admin/fix/corrections.csv", "w") as out:
    for msg in map(parse_fix, f):
        true_sym = orders[msg["11"]]
        if msg["55"] != true_sym:
            out.write(f"{msg[\"17\"]},{true_sym}\n")
'
  • Advantages: Explicit KeyError failure if a ClOrdID is missing from orders.log, cleaner maintainability for multi-step transformations, and straightforward unit testing.
  • Limitations: Higher startup overhead than awk and holds the entire orders dictionary in Python process memory.
  • When to Use: When building a permanent, maintainable reconciliation script or when messages contain complex nested repeating groups or quoted CSV fields.

Alternative 2: Disk-Backed External Sort and Join (sed + sort + join)

  • Underlying Idea: Extract normalized key-value columns from both logs, sort both streams lexicographically on 11=ClOrdID, and join them using GNU join.
  • Example Command:
join -t',' -1 1 -2 1 \
  <(sed -n 's/.*|11=\([^|]*\)|55=\([^|]*\)|.*/\1,\2/p' /home/admin/fix/orders.log | sort -t',' -k1,1) \
  <(sed -n 's/.*|17=\([^|]*\)|11=\([^|]*\)|.*|55=\([^|]*\)|.*/\2,\1,\3/p' /home/admin/fix/executions.log | sort -t',' -k1,1) \
| awk -F',' '$2 != $4 {print $3 "," $2}' | sort
  • Advantages: Bounded memory usage. GNU sort uses an external merge-sort algorithm that spills sorted runs to disk (/tmp), enabling joins on multi-gigabyte log files that would trigger the Linux OOM killer in awk or python3.
  • Limitations: Requires pre-sorting both inputs; regex extraction with sed assumes 11= precedes 55= on every line.
  • When to Use: When log files are tens of gigabytes and exceed available RAM.

Alternative 3: Price-Band-Only Inference (BANDS.csv + executions.log)

  • Underlying Idea: Ignore orders.log completely and infer the true symbol solely by matching 31=LastPx against the non-overlapping intervals in BANDS.csv.
  • Advantages: Works even if orders.log is unavailable, rotated out, or corrupted.
  • Limitations: Fragile in real-world financial systems where equities frequently trade in overlapping price ranges or experience overnight price gaps outside static bands.

Comparison of Approaches

Approach Memory Footprint Execution Speed (Small Files) Resilience to Tag Reordering Best Production Use Case
awk Hash Join (Executed) $O(N_{\text{orders}})$ in RAM Fastest (~ms) High (field loop) Ad-hoc incident response on logs up to a few GB
python3 Dict Parser $O(N_{\text{orders}})$ in RAM Fast High (dict split) Scheduled batch jobs requiring strict error handling & tests
sort + join Pipeline Bounded (spills to disk) Moderate (disk I/O) Moderate (depends on extractor) Massive logs (10+ GB) on memory-constrained hosts
Price-Band-Only Check $O(N_{\text{symbols}})$ in RAM Fastest High Fallback anomaly detection when order logs are missing

9. Linux Command Deep Dive

Command: awk (GNU gawk)

Purpose

awk is a pattern-scanning and text-processing language designed for structured records and column-oriented log analysis. It reads input line-by-line (or record-by-record), splits each line into fields ($1, $2, …, $NF), and executes { action } blocks whenever a pattern matches.

General Syntax

awk [OPTIONS] 'pattern { action }' [FILE]...

Important Options and Built-in Variables

Option / Variable Purpose
-F '<regex>' Sets the input field separator (FS). Supports extended regular expressions such as `-F’[,
-v var=val Safely injects a shell variable into awk before execution starts.
NR / FNR NR is the cumulative line number across all input files; FNR resets to 1 at the start of each new file.
NF / $NF NF is the number of fields on the current line; $NF accesses the final field.
FILENAME Holds the name of the file currently being read.
substr(s, p [, n]) Extracts a substring from s starting at 1-based position p (optionally of length n).

Practical Examples

# Count HTTP status codes (column 9) in an Nginx access log
awk '{counts[$9]++} END {for (code in counts) print code, counts[code]}' /var/log/nginx/access.log

# Print lines where response time (last column) exceeds 2.5 seconds
awk '$NF > 2.5 {print $1, $7, $NF}' /var/log/nginx/access.log

Common Mistakes and Limitations

  • String vs. Numeric Comparison: awk variables are dynamically typed. Comparing "95.00" > "170.00" without numeric coercion (+ 0) evaluates as a string comparison ('9' > '1'), returning true! Always coerce numbers (px = substr($i, 4) + 0) before range comparisons.
  • Implicit Array Insertion: Evaluating if (arr[key] == "") silently creates key inside arr. Use if (key in arr) to test membership without mutating the array.

Command: sort and uniq (GNU Coreutils)

Purpose

sort orders lines of text files; uniq filters or counts adjacent duplicate lines. They are standard POSIX/GNU coreutils utilities used together for frequency analysis and deduplication.

Useful Troubleshooting Options

Command Purpose
sort -u Sort and emit only unique lines (replaces `sort
sort -t',' -k2,2n Sort comma-delimited data numerically (n) by column 2 (-k2,2).
sort -h Sort human-readable sizes (2K, 50M, 1G) from du -h.
uniq -c Prefix each unique line with its occurrence count.
uniq -d Print only duplicate lines (useful for spotting duplicate IDs in logs).

Common Mistake

Running uniq -c on unsorted data only collapses consecutive duplicates. Always sort first unless you specifically want run-length encoding of adjacent identical lines.


Command: join (GNU Coreutils)

Purpose

Performs a relational join between two pre-sorted text files on a common key field.

Useful Troubleshooting Options

Option Purpose
-t '<char>' Uses <char> as both input and output field delimiter (default is whitespace).
-1 <col> -2 <col> Joins on field <col> of file 1 and field <col> of file 2.
-a 1 / -a 2 Emits unpairable lines from file 1 or file 2 (SQL LEFT / RIGHT OUTER JOIN).
-v 1 / -v 2 Emits only unpairable lines (anti-join—ideal for finding orphaned records).

Common Mistake

Sorting files with sort -n (numeric order) and passing them to join, which requires standard lexicographical sort order (sort).


10. Underlying Linux Concepts and Architecture

1. Stream Processing vs. In-Memory Hash Tables

Linux text utilities follow the Unix filter model: reading byte streams from file descriptors (stdin or opened files), buffering in user space, and writing results to stdout.

When joining two files in awk using FNR == NR, awk trades memory for single-pass speed: it builds an open-addressed hash table (ord_sym) in process heap memory during the first file, enabling $O(1)$ average-time lookups while streaming the second file. For 700 orders, this consumes only a few kilobytes of RAM. However, if orders.log contained 100 million records, heap usage would grow linearly until constrained by ulimit or the Linux kernel Out-Of-Memory (OOM) killer.

2. Shell Output Redirection and File Truncation (>)

When you run:

awk '...' orders.log executions.log > /home/admin/fix/corrections.csv

The shell (bash) processes redirections before invoking execve() for awk. Opening a file with > calls open() with the flags O_WRONLY | O_CREAT | O_TRUNC. Because O_TRUNC immediately sets the file size to 0 bytes before the command reads anything, you must never redirect a command’s output back onto one of its own input files (e.g., sort file.txt > file.txt), as doing so erases the input before it is read.

3. Safe Temporary File Lifecycle (mktemp and trap)

Look at how /home/admin/agent/check.sh manages temporary files:

T=$(mktemp -d) || { echo -n "NO"; exit 0; }
trap 'rm -rf "$T"' EXIT

Creating temporary directories with predictable names in /tmp exposes scripts to symlink/race-condition attacks and leaves orphaned files behind on failure. Pairing mktemp -d immediately with trap 'rm -rf "$T"' EXIT guarantees that the kernel’s shell signal/exit handler removes the temporary directory whether the script finishes normally, exits early on validation failure, or is interrupted.


11. Real-World Sysadmin and SRE Applications

Detection and Observability in Production

In a live trading or financial messaging environment, waiting for end-of-day batch reconciliation to catch symbol flips creates unacceptable financial exposure. An SRE team would instrument the execution pipeline with:

  • Pre-Serialization Invariant Assertions: The execution gateway should assert ExecutionReport.Symbol == Order.Symbol and check LastPx against dynamic exchange price collars before writing the message to the wire or log stream.
  • Structured Metrics & Alerting: Export counters via Prometheus/OpenTelemetry such as fix_execution_symbol_mismatch_total and fix_execution_price_collar_violation_total. Any non-zero increment on a symbol-mismatch counter should trigger a high-severity alert (PagerDuty / Alertmanager) and circuit-break the affected algorithmic strategy.

Safe Remediation Practices

When generating corrective datasets in production:

  1. Read-Only Dry Runs First: Always compute diffs and cross-checks to stdout before writing remediation files.
  2. Idempotency and Atomic Writes: Write output to a temporary file on the same filesystem (corrections.csv.tmp) and atomically mv it into place so downstream consumers never read a partially written CSV file.
  3. Audit Trail: Retain the exact reconciliation script, input file checksums (sha256sum), and row counts in the incident ticket for regulatory compliance.

12. Further Learning and Experiments

Hands-on Experiments

You can safely practice these variations in /tmp without touching any system files:

  1. Test Anti-Joins with GNU join to Detect Orphaned Records:
  • What to investigate: How join -v 2 isolates records in a second file that have no matching key in the first file.
  • Commands to execute:
mkdir -p /tmp/fix_lab
printf "C001,AAPL\nC002,GOOG\n" | sort > /tmp/fix_lab/orders.csv
printf "C001,EX01,AAPL\nC002,EX02,INTC\nC999,EX03,NVDA\n" | sort > /tmp/fix_lab/execs.csv
join -t',' -v 2 /tmp/fix_lab/orders.csv /tmp/fix_lab/execs.csv
rm -rf /tmp/fix_lab
  • What to observe: join -v 2 outputs C999,EX03,NVDA—the exact execution whose ClOrdID does not exist in orders.csv.
  1. Observe String vs. Numeric Comparison Bugs in awk:
  • What to investigate: Why omitting + 0 breaks price-band checks.
  • Commands to execute:
awk 'BEGIN { px="95.00"; min="170.00"; print "String compare:", (px > min); print "Numeric compare:", (px+0 > min+0) }'
  • What to observe: String comparison prints 1 (true) because ASCII '9' > '1', whereas numeric comparison correctly prints 0 (false).
  1. Parse Arbitrary-Length FIX Tags with split() Instead of substr():
  • What to investigate: substr($i, 4) works for 2-digit FIX tags (11=, 17=, 31=, 55=) because 2 + 1 = 3 characters precede the value, but breaks on 1-digit (8=) or 3-digit (150=) tags.
  • Commands to execute:
echo "8=FIX.4.2|150=F|55=AAPL|" | awk -F'|' '{for(i=1;i<=NF;i++){split($i,kv,"="); if(kv[1]=="150"||kv[1]=="55") print kv[1], "->", kv[2]}}'
  • What to observe: split($i, kv, "=") cleanly extracts both 150 -> F and 55 -> AAPL regardless of tag digit length.

Topics to Explore Next

  • Bash Process Substitution (<(command)): How the kernel exposes anonymous pipes under /dev/fd/ so commands like join and diff can read sorted streams without temporary files.
  • Out-of-Core Sorting (sort -S and -T): Tuning buffer sizes and temporary directories when sorting multi-gigabyte log files on Linux.
  • Structured Log Processing with jq and mlr (Miller): Handling JSON and CSV/key-value log streams with schema-aware CLI tools.

Documentation and References

  • man 1 awk (or man 1 gawk) — GNU AWK pattern scanning and processing language manual.
  • man 1 join — GNU Coreutils relational file join manual.
  • man 1 sort & man 1 uniq — Sorting and deduplication options.
  • man 1 mktemp & help trap — Safe temporary file creation and shell signal/exit trapping.

13. Key Takeaways

  • Never Trust Hardcoded Column Indices in Tagged Logs: Optional fields and log prefixes (SENT ) shift column offsets between message types. Always match on key prefixes (/^11=/) or split on =.
  • Cross-Validate Independent Invariants Before Writing Output: Confirming that both referential integrity (orders.log) and domain constraints (BANDS.csv) agreed 100% eliminated the risk of missing orphaned fills or out-of-band anomalies.
  • Coerce Numbers Explicitly in awk: Always add + 0 when comparing numeric fields so awk does not fall back to ASCII lexicographical comparison.
  • Match the Tool to the Data Scale: Use in-memory awk or python3 hash joins for speed and clarity on logs that fit comfortably in RAM; switch to disk-backed sort + join pipelines when dataset sizes approach system memory limits.

14. Quick Troubleshooting Reference / Runbook

Use this checklist when investigating structured log discrepancies or data corruption between upstream and downstream services:

  1. Inspect Format, Headers, and Volume (Read-Only):
head -n 5 <ref_file> <upstream_log> <downstream_log>
wc -l <ref_file> <upstream_log> <downstream_log>
  1. Audit Record Types and Cardinality:
awk -F'<delim>' '{print $<type_col>}' <upstream_log> <downstream_log> | sort | uniq -c
  1. Check Validation Contracts / Downstream Schemas: Inspect any consumer scripts or schema definitions to verify header expectations, sorting rules, and uniqueness constraints.
  2. Run a Read-Only Cross-Validation Pass: Join upstream and downstream streams on the correlation key (FNR == NR in awk) and validate against independent reference bounds (+ 0 numeric coercion) before creating files.
  3. Generate Remediation Output Atomically or Safely: Write output to a new file (never overwrite source logs in place) and verify line counts and formatting:
wc -l <output_file>
head -n 10 <output_file>
  1. Execute Acceptance Verification: Run the validation suite or checksum verification script and confirm an explicit OK / exit code 0.