1. Introduction
Production systems rarely fail exclusively through crashed daemons or exhausted disk space. Some of the most damaging incidents in financial, telemetry, and distributed backend infrastructure involve silent data corruption—where services remain up, return 200 OK or valid protocol checksums, and continue writing logs, yet emit semantically wrong fields.
In this troubleshooting exercise (based on the SadServers “Flip”: algo symbol flips scenario), an electronic trading execution algorithm intermittently stamps the wrong stock ticker symbol (55=Symbol) onto outbound trade execution reports. Because downstream clearing, risk, and accounting pipelines rely on execution logs to book trades, even a small percentage of mislabeled fills can cause severe financial discrepancies.
For a Linux systems administrator or Site Reliability Engineer (SRE), resolving this class of incident requires fluency in user-space text and stream processing. Rather than manually inspecting thousands of structured protocol messages, you need to correlate multiple log streams, enforce domain invariants, verify data completeness in read-only mode, and generate an exact, auditable remediation dataset.
In this article, you will learn:
- How to structure a hypothesis-driven investigation for structured application logs.
- How to parse tag-value protocols (such as FIX) safely without relying on brittle column offsets.
- How to perform single-pass relational joins across multiple files in
awkusingFNR == NRand associative arrays. - How to cross-validate remediation data against independent invariants before writing output files.
- Alternative approaches using
python3and disk-backed POSIXsort/joinpipelines, along with their memory and scalability trade-offs.
2. Problem Description and Environment
Challenge Overview
The Financial Information eXchange (FIX) protocol transmits trading messages as sequences of tag=value pairs delimited by an ASCII SOH (0x01) character, rendered in these log files as a pipe (|). Each message begins with 8= (BeginString), 9= (BodyLength), and 35= (MsgType), and ends with 10= (CheckSum).
A bug in the execution algorithm causes some execution fills (35=8) to be stamped with an incorrect 55=Symbol (for example, an AAPL execution labeled DELL, or GOOG labeled INTC).
Available Files and FIX Tag Dictionary
Three data files and one grading script are present on the system:
/home/admin/market/BANDS.csv: Reference market data containingsymbol,min_px,max_pxrows with non-overlapping price bands./home/admin/fix/orders.log: Outbound order logs (35=D,NewOrderSingle) containing the client order ID (11=ClOrdID) and the authoritative symbol (55=Symbol)./home/admin/fix/executions.log: Execution fill logs (35=8,ExecutionReport) containing a unique fill ID (17=ExecID), parent order link (11=ClOrdID), as-stamped55=Symbol, and fill price (31=LastPx)./home/admin/agent/check.sh: Automated verification script validating the output artifact.
| FIX Tag | Field Name | Role in Investigation |
|---|---|---|
35 |
MsgType |
D = NewOrderSingle (order); 8 = ExecutionReport (fill) |
11 |
ClOrdID |
Client order ID linking one or more fills back to their parent order |
17 |
ExecID |
Unique identifier per fill (e.g., EX000123) |
55 |
Symbol |
True ticker symbol on 35=D orders; potentially flipped on 35=8 fills |
31 |
LastPx |
Execution price of the individual fill |
Expected vs. Actual Behaviour
- Expected Behaviour: Every fill (
35=8) in/home/admin/fix/executions.logcarries the exact same55=Symbolas its parent order (35=D) in/home/admin/fix/orders.log(joined via11=ClOrdID), and its execution price (31=LastPx) falls strictly within that symbol’s[min_px, max_px]range in/home/admin/market/BANDS.csv. - Actual Behaviour: Approximately 25 out of ~960 fills in
executions.logare stamped with a mismatched55=Symbolwhose price band contradicts31=LastPxand disagrees with the parent order inorders.log. - Remediation Goal: Create
/home/admin/fix/corrections.csvcontaining oneExecID,correct_symbolpair per line for only the flipped fills, with no missing rows, false positives, or duplicates.
Environment Details
- Host & User:
admin@i-051025d49a1b03ff5(withsudo/ root privileges available, though all target files reside under/home/admin/and are readable/writable byadmin). - OS Distribution & Kernel: Specific Linux distribution and kernel version (
uname -a,/etc/os-release) were not queried during this session because the task operated entirely in user space on application log files.
3. Initial Investigation and Hypotheses
Before writing any parsing scripts, an SRE establishes what is known, what is unknown, and what edge cases could corrupt the analysis. Jumping straight into a one-liner that assumes fixed column positions or clean 1-to-1 order-to-fill mappings often leads to subtle data bugs.
What Was Known vs. Unknown
-
Known:
-
Two independent detection signals exist: relational lookup via
11=ClOrdID(orders.logvs.executions.log) and price-range validation via31=LastPx(executions.logvs.BANDS.csv). -
Roughly 25 fills out of ~960 are affected.
-
Unknown:
- CSV Format Expectations: Does
/home/admin/agent/check.shexpect a header row incorrections.csv(ExecID,correct_symbol), or only data rows? - Message Purity: Do
orders.logandexecutions.logcontain only35=Dand35=8messages, or are session-level FIX messages (such as heartbeats35=0, logons35=A, or order cancels35=F) mixed in? - Cardinality: Is there a 1-to-1 relationship between orders and executions, or can a single order produce multiple partial fills?
- Tag Ordering: Are FIX tags always at fixed pipe-delimited column indices, or can optional tags shift field offsets across lines?
- Referential Completeness: Does every
11=ClOrdIDinexecutions.logexist inorders.log, or are some parent orders missing (which would require falling back onBANDS.csvprice-band inference)?
Hypotheses Considered
| Hypothesis | Supporting Evidence | Evidence Against | Diagnostic Step |
|---|---|---|---|
| 1. Deterministic or semi-random symbol flip during fill serialization | Problem statement notes AAPL labeled DELL and GOOG labeled INTC while order records hold the true symbol. |
None initially; requires comparing 55= across joined logs and price bands. |
Join orders.log and executions.log on 11=ClOrdID and cross-check 31=LastPx against BANDS.csv. |
| 2. Variable FIX tag positions or log prefixes breaking column-based parsing | Real-world FIX engines frequently include prefix timestamps/directions (SENT ) and variable-length tag sets between orders and fills. |
Unknown until raw log lines are inspected. | Inspect raw lines with head and parse fields by tag= prefix rather than fixed column index. |
**3. Orphaned executions missing from orders.log** |
Multi-file log rotations or truncated order logs can leave fills without a matching 35=D record. |
Unknown until referential integrity is tested across all 960 fills. | Run a read-only diagnostic comparing order_sym and band_sym for every execution. |
4. Mixed FIX message types (35=0, 35=F, etc.) in log streams |
Standard FIX session logs record all inbound/outbound traffic in a single file. | Problem description specifies 35=D in orders.log and 35=8 in executions.log. |
Extract tag 35= across every line in both logs and aggregate counts with `sort |
4. Troubleshooting Process
Step 1: Initial File Structure and Volume Inspection
Objective
Inspect the exact formatting, delimiters, header rows, and line counts of /home/admin/market/BANDS.csv, /home/admin/fix/orders.log, and /home/admin/fix/executions.log using read-only commands.
Command
head -n 5 /home/admin/market/BANDS.csv /home/admin/fix/orders.log /home/admin/fix/executions.log
wc -l /home/admin/market/BANDS.csv /home/admin/fix/orders.log /home/admin/fix/executions.log
Command Explanation
head -n 5 ...: Prints the first 5 lines of each specified file, automatically inserting==> <filename> <==banners when given multiple file arguments.wc -l ...: Counts newline characters (\n) in each file to establish exact record volumes before processing.
Output and Analysis
==> /home/admin/market/BANDS.csv <==
symbol,min_px,max_px
GOOG,20.00,30.00
INTC,68.00,78.00
DELL,95.00,110.00
AAPL,170.00,185.00
==> /home/admin/fix/orders.log <==
SENT 8=FIX.4.2|9=93|35=D|49=ROUTER|56=EXCH|34=1|52=20260916-09:30:00.000|11=C00001|55=DELL|54=2|38=1000|44=96.11|10=058|
SENT 8=FIX.4.2|9=93|35=D|49=ROUTER|56=EXCH|34=2|52=20260916-09:30:00.392|11=C00002|55=NVDA|54=2|38=300|44=879.55|10=101|
SENT 8=FIX.4.2|9=93|35=D|49=ROUTER|56=EXCH|34=3|52=20260916-09:30:00.799|11=C00003|55=AAPL|54=2|38=300|44=174.85|10=094|
SENT 8=FIX.4.2|9=93|35=D|49=ROUTER|56=EXCH|34=4|52=20260916-09:30:01.136|11=C00004|55=NVDA|54=1|38=500|44=879.62|10=101|
SENT 8=FIX.4.2|9=92|35=D|49=ROUTER|56=EXCH|34=5|52=20260916-09:30:01.591|11=C00005|55=INTC|54=2|38=100|44=71.00|10=038|
==> /home/admin/fix/executions.log <==
SENT 8=FIX.4.2|9=140|35=8|49=EXCH|56=ROUTER|34=1|52=20260916-09:30:00.186|17=EX000001|11=C00001|37=O00001|150=F|39=2|55=DELL|54=2|32=1000|31=96.11|14=1000|151=0|10=192|
SENT 8=FIX.4.2|9=139|35=8|49=EXCH|56=ROUTER|34=2|52=20260916-09:30:00.485|17=EX000002|11=C00002|37=O00002|150=F|39=2|55=NVDA|54=2|32=150|31=879.55|14=150|151=0|10=185|
SENT 8=FIX.4.2|9=139|35=8|49=EXCH|56=ROUTER|34=3|52=20260916-09:30:00.732|17=EX000003|11=C00002|37=O00002|150=F|39=2|55=NVDA|54=2|32=150|31=879.55|14=150|151=0|10=182|
SENT 8=FIX.4.2|9=139|35=8|49=EXCH|56=ROUTER|34=4|52=20260916-09:30:01.022|17=EX000004|11=C00003|37=O00003|150=F|39=2|55=AAPL|54=2|32=300|31=174.85|14=300|151=0|10=153|
SENT 8=FIX.4.2|9=139|35=8|49=EXCH|56=ROUTER|34=5|52=20260916-09:30:01.371|17=EX000005|11=C00004|37=O00004|150=F|39=2|55=NVDA|54=1|32=500|31=879.62|14=500|151=0|10=185|
7 /home/admin/market/BANDS.csv
700 /home/admin/fix/orders.log
960 /home/admin/fix/executions.log
1667 total
Key takeaways from this output:
- Log Prefix & Delimiters: Every FIX log line starts with
SENTimmediately followed by8=FIX.4.2, with pipe-delimited (|) fields and a trailing|. - Column Offset Differences: In
orders.log,11=ClOrdIDis pipe-field$8and55=Symbolis$9. Inexecutions.log,17=ExecIDoccupies$8, shifting11=ClOrdIDto$9and55=Symbolto$13. Hardcoding column indices across files would fail. - One-to-Many Order-to-Fill Cardinality:
orders.loghas 700 lines whileexecutions.loghas 960 lines. Looking at11=C00002(an order for38=300shares ofNVDA), it splits into two separate fills (17=EX000002and17=EX000003, each with32=150shares). Therefore,11=ClOrdIDis a non-unique foreign key inexecutions.log.
Decision
Read the entirety of BANDS.csv, inspect /home/admin/agent/check.sh to confirm the exact output contract, and verify whether all 1,660 log lines contain strictly 35=D and 35=8 messages.
Step 2: Inspecting Reference Bands, Validation Rules, and Message Purity
Objective
Confirm all symbol price intervals, understand how check.sh grades corrections.csv, and verify that no unexpected FIX message types exist in either log file.
Command
cat /home/admin/market/BANDS.csv
cat /home/admin/agent/check.sh
awk -F'|' '{print $3}' /home/admin/fix/orders.log /home/admin/fix/executions.log | sort | uniq -c
Command Explanation
cat /home/admin/market/BANDS.csv: Displays all 7 lines of the symbol reference table.cat /home/admin/agent/check.sh: Displays the validation script (permitted by the challenge instructions) so we can verify formatting, sorting, and header rules.awk -F'|' '{print $3}' ... | sort | uniq -c: Splits every line of both FIX logs on|, extracts the 3rd field (35=<MsgType>), sorts the values, and counts occurrences of each distinct message type.
Output and Analysis
symbol,min_px,max_px
GOOG,20.00,30.00
INTC,68.00,78.00
DELL,95.00,110.00
AAPL,170.00,185.00
MSFT,400.00,420.00
NVDA,850.00,880.00
#!/bin/bash
# DO NOT MODIFY THIS FILE ("Check My Solution" will fail)
D=/home/admin/fix
F="$D/corrections.csv"
T=$(mktemp -d) || { echo -n "NO"; exit 0; }
trap 'rm -rf "$T"' EXIT
[ -f "$F" ] || { echo -n "NO"; exit 0; }
EXP_HASH="c3d1e3fa97cee2bd0acf0e779fcd13a147c23441a7e2d15c3729e5cc726d4fb9"
EXP_COUNT=25
awk 'BEGIN{FS="[,; \t]+"} {sub(/\r$/,"");sub(/^[ \t]+/,"");sub(/[ \t]+$/,"");if($0=="")next;if(NR==1&&$0~/^[Ee][Xx][Ee][Cc]/)next;e="";s="";U=toupper($0);if(match(U,/17=EX[0-9]+/)){e=substr(U,RSTART+3,RLENGTH-3);s=$2}else{e=toupper($1);s=$2}if(e=="")exit 1;if(match(e,/EX[0-9]+/)){e=substr(e,RSTART,RLENGTH)}else exit 1;if(s=="")exit 1;s=toupper(s);gsub(/[^A-Z]/,"",s);if(e!~/^EX[0-9]+$/)exit 1;if(s!~/^(GOOG|INTC|DELL|AAPL|MSFT|NVDA)$/)exit 1;print e","s}' "$F" > "$T/raw" 2>/dev/null || { echo -n "NO"; exit 0; }
[ -s "$T/raw" ] || { echo -n "NO"; exit 0; }
if [ "$(wc -l < "$T/raw")" != "$(sort -u "$T/raw" | wc -l)" ]; then echo -n "NO"; exit 0; fi
sort -u "$T/raw" > "$T/got"
[ "$(wc -l < "$T/got")" = "$EXP_COUNT" ] || { echo -n "NO"; exit 0; }
if [ "$(sha256sum "$T/got" | cut -d" " -f1)" = "$EXP_HASH" ]; then echo -n "OK"; else echo -n "NO"; fi
exit 0
960 35=8
700 35=D
Key findings:
- Pure Message Streams: Every single line in
orders.log(700) is35=Dand every single line inexecutions.log(960) is35=8. There are no administrative or cancel messages to filter out. - Disjoint Market Bands: All 6 symbols occupy wide, non-overlapping price ranges (
20–30,68–78,95–110,170–185,400–420,850–880). - Validation Contract (
check.sh):
- Expects
/home/admin/fix/corrections.csvto exist and contain 25 uniqueEX<digits>,<SYMBOL>rows (EXP_COUNT=25). - Normalizes whitespace and
\r, skips an optional header line starting withExec(case-insensitive), rejects duplicates (wc -lvs.sort -u), sorts rows, and verifies the SHA-256 digest againstEXP_HASH.
Decision
Run a single-pass, read-only awk diagnostic across all three files (BANDS.csv, orders.log, and executions.log) to cross-validate both detection hypotheses before creating corrections.csv.
Step 3: Cross-Validating Referential Integrity and Market Price Bands
Objective
Audit all 960 executions in read-only mode to check whether any execution’s stamped 55=Symbol disagrees with its parent order’s symbol (order_sym, via 11=ClOrdID) or its price-band symbol (band_sym, via 31=LastPx), and detect any missing parent orders or out-of-band prices.
Command
awk -F'[,|]' '
FNR == NR {
if (FNR > 1) {
min[$1] = $2 + 0
max[$1] = $3 + 0
}
next
}
FILENAME ~ /orders\.log$/ {
id = ""; sym = ""
for (i = 1; i <= NF; i++) {
if ($i ~ /^11=/) id = substr($i, 4)
else if ($i ~ /^55=/) sym = substr($i, 4)
}
if (id != "") ord_sym[id] = sym
next
}
{
exec_id = ""; id = ""; sym = ""; px = 0
for (i = 1; i <= NF; i++) {
if ($i ~ /^17=/) exec_id = substr($i, 4)
else if ($i ~ /^11=/) id = substr($i, 4)
else if ($i ~ /^55=/) sym = substr($i, 4)
else if ($i ~ /^31=/) px = substr($i, 4) + 0
}
band_sym = "UNKNOWN"
for (s in min) {
if (px >= min[s] && px <= max[s]) band_sym = s
}
true_sym = (id in ord_sym) ? ord_sym[id] : "MISSING_ORDER"
if (sym != true_sym || sym != band_sym) {
printf "%s | ClOrdID=%s | stamped=%s | order_sym=%s | band_sym=%s (px=%.2f)\n",
exec_id, id, sym, true_sym, band_sym, px
}
}' /home/admin/market/BANDS.csv /home/admin/fix/orders.log /home/admin/fix/executions.log
Command Explanation
-F'[,|]': Sets the field separator to a regular expression matching either,(forBANDS.csv) or|(for the FIX log files).FNR == NR: True only while reading the first input file (BANDS.csv), loadingmin[symbol]andmax[symbol]as numeric values (+ 0).FILENAME ~ /orders\.log$/: Matches the second file (orders.log), scanning fields for^11=and^55=to populate theord_sym[ClOrdID]associative array.- Final block (processes
executions.log): Extracts17=ExecID,11=ClOrdID,55=Symbol, and31=LastPx, determines which price band containspx, looks upord_sym[id], and prints any record wherestampeddiffers from eitherorder_symorband_sym.
Output and Analysis
EX000108 | ClOrdID=C00076 | stamped=MSFT | order_sym=NVDA | band_sym=NVDA (px=852.20)
EX000136 | ClOrdID=C00095 | stamped=NVDA | order_sym=MSFT | band_sym=MSFT (px=408.83)
EX000150 | ClOrdID=C00105 | stamped=INTC | order_sym=GOOG | band_sym=GOOG (px=22.28)
EX000177 | ClOrdID=C00124 | stamped=INTC | order_sym=GOOG | band_sym=GOOG (px=26.31)
EX000223 | ClOrdID=C00156 | stamped=GOOG | order_sym=INTC | band_sym=INTC (px=77.33)
EX000226 | ClOrdID=C00157 | stamped=AAPL | order_sym=DELL | band_sym=DELL (px=109.63)
EX000287 | ClOrdID=C00200 | stamped=INTC | order_sym=GOOG | band_sym=GOOG (px=24.47)
EX000302 | ClOrdID=C00213 | stamped=DELL | order_sym=AAPL | band_sym=AAPL (px=174.41)
EX000312 | ClOrdID=C00220 | stamped=DELL | order_sym=AAPL | band_sym=AAPL (px=180.23)
EX000414 | ClOrdID=C00291 | stamped=GOOG | order_sym=INTC | band_sym=INTC (px=68.37)
EX000444 | ClOrdID=C00312 | stamped=AAPL | order_sym=DELL | band_sym=DELL (px=97.57)
EX000464 | ClOrdID=C00327 | stamped=NVDA | order_sym=MSFT | band_sym=MSFT (px=411.01)
EX000488 | ClOrdID=C00345 | stamped=DELL | order_sym=AAPL | band_sym=AAPL (px=175.02)
EX000502 | ClOrdID=C00356 | stamped=INTC | order_sym=GOOG | band_sym=GOOG (px=25.67)
EX000536 | ClOrdID=C00381 | stamped=MSFT | order_sym=NVDA | band_sym=NVDA (px=875.51)
EX000545 | ClOrdID=C00388 | stamped=INTC | order_sym=GOOG | band_sym=GOOG (px=23.65)
EX000596 | ClOrdID=C00425 | stamped=NVDA | order_sym=MSFT | band_sym=MSFT (px=407.93)
EX000681 | ClOrdID=C00493 | stamped=AAPL | order_sym=DELL | band_sym=DELL (px=100.05)
EX000787 | ClOrdID=C00569 | stamped=GOOG | order_sym=INTC | band_sym=INTC (px=70.95)
EX000794 | ClOrdID=C00574 | stamped=DELL | order_sym=AAPL | band_sym=AAPL (px=181.01)
EX000813 | ClOrdID=C00589 | stamped=NVDA | order_sym=MSFT | band_sym=MSFT (px=402.74)
EX000862 | ClOrdID=C00627 | stamped=MSFT | order_sym=NVDA | band_sym=NVDA (px=862.94)
EX000901 | ClOrdID=C00654 | stamped=NVDA | order_sym=MSFT | band_sym=MSFT (px=401.27)
EX000919 | ClOrdID=C00667 | stamped=AAPL | order_sym=DELL | band_sym=DELL (px=101.83)
EX000947 | ClOrdID=C00690 | stamped=INTC | order_sym=GOOG | band_sym=GOOG (px=27.76)
This output provides complete clarity:
- Exact Count: Exactly 25 lines are returned, matching
EXP_COUNT=25incheck.sh. - 100% Cross-Method Agreement: On every single flipped fill,
order_symandband_symmatch identically. There are zeroMISSING_ORDERrecords and zeroUNKNOWNout-of-band prices.
Decision
Generate /home/admin/fix/corrections.csv using a clean two-file awk hash join between orders.log and executions.log, then verify the output file and run /home/admin/agent/check.sh.
5. Root Cause Analysis
Mechanism of the Failure
Comparing the stamped symbol against the true order_sym across all 25 corrupted fills reveals a strict deterministic pattern tied directly to the symbol order in /home/admin/market/BANDS.csv:
BANDS.csv Index |
True Symbol | Stamped Symbol on Flip | Swap Pair |
|---|---|---|---|
0 |
GOOG |
INTC |
0 ↔ 1 |
1 |
INTC |
GOOG |
0 ↔ 1 |
2 |
DELL |
AAPL |
2 ↔ 3 |
3 |
AAPL |
DELL |
2 ↔ 3 |
4 |
MSFT |
NVDA |
4 ↔ 5 |
5 |
NVDA |
MSFT |
4 ↔ 5 |
Every single corrupted fill swapped adjacent symbol indices (0↔1, 2↔3, 4↔5). In systems programming, this exact signature (idx ^ 1) occurs when a bitwise XOR or off-by-one array indexing error corrupts the lookup index during serialization of the outbound ExecutionReport (35=8).
flowchart LR
A["Order Router<br/>(35=D NewOrderSingle)"] -->|"Logs true 55=Symbol<br/>& 11=ClOrdID"| B[("/home/admin/fix/orders.log")]
A -->|"Routes order"| C["Execution Algo Engine"]
C -->|"Bug: idx ^ 1 swaps<br/>adjacent symbol on 35=8"| D[("/home/admin/fix/executions.log")]
B -->|"AWK Hash Join<br/>on 11=ClOrdID"| E["/home/admin/fix/corrections.csv"]
D -->|"Compare stamped 55=<br/>vs. order 55="| E
Distinguishing Root Cause from Secondary Symptoms
- Root Cause: Indexing/serialization defect in the execution algorithm when populating tag
55=Symbolon35=8messages. - Primary Symptom: Out-of-band fill prices (
31=LastPx) relative to the stamped symbol (e.g.,EX000108executing at$852.20while stampedMSFT, whose valid band is$400.00–$420.00). - Why Recovery Was Unambiguous: Because the upstream order router persisted the uncorrupted
35=Dmessage inorders.logbefore execution occurred,orders.logserved as an intact source of truth.
6. Solution and Remediation
Changes Applied
No existing system files or logs were modified. A single new output file, /home/admin/fix/corrections.csv, was generated from read-only inputs. Rolling back this change requires only removing that file (rm -f /home/admin/fix/corrections.csv).
Command Executed
awk -F'|' '
FNR == NR {
id = ""; sym = ""
for (i = 1; i <= NF; i++) {
if ($i ~ /^11=/) id = substr($i, 4)
else if ($i ~ /^55=/) sym = substr($i, 4)
}
if (id != "") ord_sym[id] = sym
next
}
{
exec_id = ""; id = ""; sym = ""
for (i = 1; i <= NF; i++) {
if ($i ~ /^17=/) exec_id = substr($i, 4)
else if ($i ~ /^11=/) id = substr($i, 4)
else if ($i ~ /^55=/) sym = substr($i, 4)
}
if (id in ord_sym && sym != ord_sym[id]) {
print exec_id "," ord_sym[id]
}
}' /home/admin/fix/orders.log /home/admin/fix/executions.log > /home/admin/fix/corrections.csv
How It Works
- In-Memory Lookup Construction (
FNR == NR): While reading/home/admin/fix/orders.log,awkscans each pipe-separated token for11=(ClOrdID) and55=(Symbol), storingord_sym[id] = symin an associative array. Thenextstatement skips the second rule block untilorders.logis exhausted. - Stream Comparison (Second Rule Block): When
awktransitions to/home/admin/fix/executions.log,FNRresets to1whileNRcontinues incrementing from701, soFNR == NRevaluates to false. For each fill,awkextracts17=ExecID,11=ClOrdID, and55=Symbol. - Defensive Key Lookup (
id in ord_sym): Checkingid in ord_symbefore evaluatingord_sym[id]preventsawkfrom implicitly creating empty entries for missing keys. Wheneversym != ord_sym[id], it emitsExecID,true_symbolto/home/admin/fix/corrections.csv.
7. Verification and Validation
In Linux administration, verifying that a command exited with status 0 is never enough—you must verify the resulting filesystem state and application-level acceptance criteria.
Commands Executed
wc -l /home/admin/fix/corrections.csv
head -n 5 /home/admin/fix/corrections.csv
agent/check.sh
Verification Output
25 /home/admin/fix/corrections.csv
EX000108,NVDA
EX000136,MSFT
EX000150,GOOG
EX000177,GOOG
EX000223,INTC
OK
Why Each Check Mattered
wc -l /home/admin/fix/corrections.csv(25): Confirmed exact row count matchingEXP_COUNT=25without blank lines or extra header rows.head -n 5 /home/admin/fix/corrections.csv: Confirmed exact CSV formatting (EX000108,NVDA) with no stray17=or55=prefixes, spaces, or trailing pipes.agent/check.sh(OK): Validated uniqueness (sort -u), regex conformance (^EX[0-9]+$and valid ticker symbol), exact record count, and the canonical SHA-256 digest (c3d1e3fa97cee2bd0acf0e779fcd13a147c23441a7e2d15c3729e5cc726d4fb9).
8. Alternative Troubleshooting Approaches
The following approaches were not executed during the live session, but represent standard SRE alternatives depending on dataset size, tooling availability, and operational constraints.
Alternative 1: Structured Parsing with Python (python3)
- Underlying Idea: Parse each FIX message into a native Python
dictmapping tag strings to values, perform an $O(1)$ dictionary lookup onClOrdID, and write the CSV output. - Example Command:
python3 -c '
def parse_fix(line):
return dict(f.split("=", 1) for f in line.strip().split("|") if "=" in f)
with open("/home/admin/fix/orders.log") as f:
orders = {m["11"]: m["55"] for m in map(parse_fix, f)}
with open("/home/admin/fix/executions.log") as f, open("/home/admin/fix/corrections.csv", "w") as out:
for msg in map(parse_fix, f):
true_sym = orders[msg["11"]]
if msg["55"] != true_sym:
out.write(f"{msg[\"17\"]},{true_sym}\n")
'
- Advantages: Explicit
KeyErrorfailure if aClOrdIDis missing fromorders.log, cleaner maintainability for multi-step transformations, and straightforward unit testing. - Limitations: Higher startup overhead than
awkand holds the entireordersdictionary in Python process memory. - When to Use: When building a permanent, maintainable reconciliation script or when messages contain complex nested repeating groups or quoted CSV fields.
Alternative 2: Disk-Backed External Sort and Join (sed + sort + join)
- Underlying Idea: Extract normalized key-value columns from both logs, sort both streams lexicographically on
11=ClOrdID, and join them using GNUjoin. - Example Command:
join -t',' -1 1 -2 1 \
<(sed -n 's/.*|11=\([^|]*\)|55=\([^|]*\)|.*/\1,\2/p' /home/admin/fix/orders.log | sort -t',' -k1,1) \
<(sed -n 's/.*|17=\([^|]*\)|11=\([^|]*\)|.*|55=\([^|]*\)|.*/\2,\1,\3/p' /home/admin/fix/executions.log | sort -t',' -k1,1) \
| awk -F',' '$2 != $4 {print $3 "," $2}' | sort
- Advantages: Bounded memory usage. GNU
sortuses an external merge-sort algorithm that spills sorted runs to disk (/tmp), enabling joins on multi-gigabyte log files that would trigger the Linux OOM killer inawkorpython3. - Limitations: Requires pre-sorting both inputs; regex extraction with
sedassumes11=precedes55=on every line. - When to Use: When log files are tens of gigabytes and exceed available RAM.
Alternative 3: Price-Band-Only Inference (BANDS.csv + executions.log)
- Underlying Idea: Ignore
orders.logcompletely and infer the true symbol solely by matching31=LastPxagainst the non-overlapping intervals inBANDS.csv. - Advantages: Works even if
orders.logis unavailable, rotated out, or corrupted. - Limitations: Fragile in real-world financial systems where equities frequently trade in overlapping price ranges or experience overnight price gaps outside static bands.
Comparison of Approaches
| Approach | Memory Footprint | Execution Speed (Small Files) | Resilience to Tag Reordering | Best Production Use Case |
|---|---|---|---|---|
awk Hash Join (Executed) |
$O(N_{\text{orders}})$ in RAM | Fastest (~ms) | High (field loop) | Ad-hoc incident response on logs up to a few GB |
python3 Dict Parser |
$O(N_{\text{orders}})$ in RAM | Fast | High (dict split) |
Scheduled batch jobs requiring strict error handling & tests |
sort + join Pipeline |
Bounded (spills to disk) | Moderate (disk I/O) | Moderate (depends on extractor) | Massive logs (10+ GB) on memory-constrained hosts |
| Price-Band-Only Check | $O(N_{\text{symbols}})$ in RAM | Fastest | High | Fallback anomaly detection when order logs are missing |
9. Linux Command Deep Dive
Command: awk (GNU gawk)
Purpose
awk is a pattern-scanning and text-processing language designed for structured records and column-oriented log analysis. It reads input line-by-line (or record-by-record), splits each line into fields ($1, $2, …, $NF), and executes { action } blocks whenever a pattern matches.
General Syntax
awk [OPTIONS] 'pattern { action }' [FILE]...
Important Options and Built-in Variables
| Option / Variable | Purpose |
|---|---|
-F '<regex>' |
Sets the input field separator (FS). Supports extended regular expressions such as `-F’[, |
-v var=val |
Safely injects a shell variable into awk before execution starts. |
NR / FNR |
NR is the cumulative line number across all input files; FNR resets to 1 at the start of each new file. |
NF / $NF |
NF is the number of fields on the current line; $NF accesses the final field. |
FILENAME |
Holds the name of the file currently being read. |
substr(s, p [, n]) |
Extracts a substring from s starting at 1-based position p (optionally of length n). |
Practical Examples
# Count HTTP status codes (column 9) in an Nginx access log
awk '{counts[$9]++} END {for (code in counts) print code, counts[code]}' /var/log/nginx/access.log
# Print lines where response time (last column) exceeds 2.5 seconds
awk '$NF > 2.5 {print $1, $7, $NF}' /var/log/nginx/access.log
Common Mistakes and Limitations
- String vs. Numeric Comparison:
awkvariables are dynamically typed. Comparing"95.00" > "170.00"without numeric coercion (+ 0) evaluates as a string comparison ('9' > '1'), returning true! Always coerce numbers (px = substr($i, 4) + 0) before range comparisons. - Implicit Array Insertion: Evaluating
if (arr[key] == "")silently createskeyinsidearr. Useif (key in arr)to test membership without mutating the array.
Command: sort and uniq (GNU Coreutils)
Purpose
sort orders lines of text files; uniq filters or counts adjacent duplicate lines. They are standard POSIX/GNU coreutils utilities used together for frequency analysis and deduplication.
Useful Troubleshooting Options
| Command | Purpose |
|---|---|
sort -u |
Sort and emit only unique lines (replaces `sort |
sort -t',' -k2,2n |
Sort comma-delimited data numerically (n) by column 2 (-k2,2). |
sort -h |
Sort human-readable sizes (2K, 50M, 1G) from du -h. |
uniq -c |
Prefix each unique line with its occurrence count. |
uniq -d |
Print only duplicate lines (useful for spotting duplicate IDs in logs). |
Common Mistake
Running uniq -c on unsorted data only collapses consecutive duplicates. Always sort first unless you specifically want run-length encoding of adjacent identical lines.
Command: join (GNU Coreutils)
Purpose
Performs a relational join between two pre-sorted text files on a common key field.
Useful Troubleshooting Options
| Option | Purpose |
|---|---|
-t '<char>' |
Uses <char> as both input and output field delimiter (default is whitespace). |
-1 <col> -2 <col> |
Joins on field <col> of file 1 and field <col> of file 2. |
-a 1 / -a 2 |
Emits unpairable lines from file 1 or file 2 (SQL LEFT / RIGHT OUTER JOIN). |
-v 1 / -v 2 |
Emits only unpairable lines (anti-join—ideal for finding orphaned records). |
Common Mistake
Sorting files with sort -n (numeric order) and passing them to join, which requires standard lexicographical sort order (sort).
10. Underlying Linux Concepts and Architecture
1. Stream Processing vs. In-Memory Hash Tables
Linux text utilities follow the Unix filter model: reading byte streams from file descriptors (stdin or opened files), buffering in user space, and writing results to stdout.
When joining two files in awk using FNR == NR, awk trades memory for single-pass speed: it builds an open-addressed hash table (ord_sym) in process heap memory during the first file, enabling $O(1)$ average-time lookups while streaming the second file. For 700 orders, this consumes only a few kilobytes of RAM. However, if orders.log contained 100 million records, heap usage would grow linearly until constrained by ulimit or the Linux kernel Out-Of-Memory (OOM) killer.
2. Shell Output Redirection and File Truncation (>)
When you run:
awk '...' orders.log executions.log > /home/admin/fix/corrections.csv
The shell (bash) processes redirections before invoking execve() for awk. Opening a file with > calls open() with the flags O_WRONLY | O_CREAT | O_TRUNC. Because O_TRUNC immediately sets the file size to 0 bytes before the command reads anything, you must never redirect a command’s output back onto one of its own input files (e.g., sort file.txt > file.txt), as doing so erases the input before it is read.
3. Safe Temporary File Lifecycle (mktemp and trap)
Look at how /home/admin/agent/check.sh manages temporary files:
T=$(mktemp -d) || { echo -n "NO"; exit 0; }
trap 'rm -rf "$T"' EXIT
Creating temporary directories with predictable names in /tmp exposes scripts to symlink/race-condition attacks and leaves orphaned files behind on failure. Pairing mktemp -d immediately with trap 'rm -rf "$T"' EXIT guarantees that the kernel’s shell signal/exit handler removes the temporary directory whether the script finishes normally, exits early on validation failure, or is interrupted.
11. Real-World Sysadmin and SRE Applications
Detection and Observability in Production
In a live trading or financial messaging environment, waiting for end-of-day batch reconciliation to catch symbol flips creates unacceptable financial exposure. An SRE team would instrument the execution pipeline with:
- Pre-Serialization Invariant Assertions: The execution gateway should assert
ExecutionReport.Symbol == Order.Symboland checkLastPxagainst dynamic exchange price collars before writing the message to the wire or log stream. - Structured Metrics & Alerting: Export counters via Prometheus/OpenTelemetry such as
fix_execution_symbol_mismatch_totalandfix_execution_price_collar_violation_total. Any non-zero increment on a symbol-mismatch counter should trigger a high-severity alert (PagerDuty/Alertmanager) and circuit-break the affected algorithmic strategy.
Safe Remediation Practices
When generating corrective datasets in production:
- Read-Only Dry Runs First: Always compute diffs and cross-checks to
stdoutbefore writing remediation files. - Idempotency and Atomic Writes: Write output to a temporary file on the same filesystem (
corrections.csv.tmp) and atomicallymvit into place so downstream consumers never read a partially written CSV file. - Audit Trail: Retain the exact reconciliation script, input file checksums (
sha256sum), and row counts in the incident ticket for regulatory compliance.
12. Further Learning and Experiments
Hands-on Experiments
You can safely practice these variations in /tmp without touching any system files:
- Test Anti-Joins with GNU
jointo Detect Orphaned Records:
- What to investigate: How
join -v 2isolates records in a second file that have no matching key in the first file. - Commands to execute:
mkdir -p /tmp/fix_lab
printf "C001,AAPL\nC002,GOOG\n" | sort > /tmp/fix_lab/orders.csv
printf "C001,EX01,AAPL\nC002,EX02,INTC\nC999,EX03,NVDA\n" | sort > /tmp/fix_lab/execs.csv
join -t',' -v 2 /tmp/fix_lab/orders.csv /tmp/fix_lab/execs.csv
rm -rf /tmp/fix_lab
- What to observe:
join -v 2outputsC999,EX03,NVDA—the exact execution whoseClOrdIDdoes not exist inorders.csv.
- Observe String vs. Numeric Comparison Bugs in
awk:
- What to investigate: Why omitting
+ 0breaks price-band checks. - Commands to execute:
awk 'BEGIN { px="95.00"; min="170.00"; print "String compare:", (px > min); print "Numeric compare:", (px+0 > min+0) }'
- What to observe: String comparison prints
1(true) because ASCII'9'>'1', whereas numeric comparison correctly prints0(false).
- Parse Arbitrary-Length FIX Tags with
split()Instead ofsubstr():
- What to investigate:
substr($i, 4)works for 2-digit FIX tags (11=,17=,31=,55=) because2 + 1 = 3characters precede the value, but breaks on 1-digit (8=) or 3-digit (150=) tags. - Commands to execute:
echo "8=FIX.4.2|150=F|55=AAPL|" | awk -F'|' '{for(i=1;i<=NF;i++){split($i,kv,"="); if(kv[1]=="150"||kv[1]=="55") print kv[1], "->", kv[2]}}'
- What to observe:
split($i, kv, "=")cleanly extracts both150 -> Fand55 -> AAPLregardless of tag digit length.
Topics to Explore Next
- Bash Process Substitution (
<(command)): How the kernel exposes anonymous pipes under/dev/fd/so commands likejoinanddiffcan read sorted streams without temporary files. - Out-of-Core Sorting (
sort -Sand-T): Tuning buffer sizes and temporary directories when sorting multi-gigabyte log files on Linux. - Structured Log Processing with
jqandmlr(Miller): Handling JSON and CSV/key-value log streams with schema-aware CLI tools.
Documentation and References
man 1 awk(orman 1 gawk) — GNU AWK pattern scanning and processing language manual.man 1 join— GNU Coreutils relational file join manual.man 1 sort&man 1 uniq— Sorting and deduplication options.man 1 mktemp&help trap— Safe temporary file creation and shell signal/exit trapping.
13. Key Takeaways
- Never Trust Hardcoded Column Indices in Tagged Logs: Optional fields and log prefixes (
SENT) shift column offsets between message types. Always match on key prefixes (/^11=/) or split on=. - Cross-Validate Independent Invariants Before Writing Output: Confirming that both referential integrity (
orders.log) and domain constraints (BANDS.csv) agreed 100% eliminated the risk of missing orphaned fills or out-of-band anomalies. - Coerce Numbers Explicitly in
awk: Always add+ 0when comparing numeric fields soawkdoes not fall back to ASCII lexicographical comparison. - Match the Tool to the Data Scale: Use in-memory
awkorpython3hash joins for speed and clarity on logs that fit comfortably in RAM; switch to disk-backedsort+joinpipelines when dataset sizes approach system memory limits.
14. Quick Troubleshooting Reference / Runbook
Use this checklist when investigating structured log discrepancies or data corruption between upstream and downstream services:
- Inspect Format, Headers, and Volume (Read-Only):
head -n 5 <ref_file> <upstream_log> <downstream_log>
wc -l <ref_file> <upstream_log> <downstream_log>
- Audit Record Types and Cardinality:
awk -F'<delim>' '{print $<type_col>}' <upstream_log> <downstream_log> | sort | uniq -c
- Check Validation Contracts / Downstream Schemas: Inspect any consumer scripts or schema definitions to verify header expectations, sorting rules, and uniqueness constraints.
- Run a Read-Only Cross-Validation Pass:
Join upstream and downstream streams on the correlation key (
FNR == NRinawk) and validate against independent reference bounds (+ 0numeric coercion) before creating files. - Generate Remediation Output Atomically or Safely: Write output to a new file (never overwrite source logs in place) and verify line counts and formatting:
wc -l <output_file>
head -n 10 <output_file>
- Execute Acceptance Verification:
Run the validation suite or checksum verification script and confirm an explicit
OK/ exit code0.