Domain 8Data InfrastructureCORE

(Suzhou) Troubleshooting MongoDB Replica Set Startup Failures and Loopback Networking Misconfigurations

A comprehensive SRE post-mortem and deep-dive reference guide for diagnosing process startup assertions, keyfile permission security constraints, and loopback socket binding issues in a multi-instance MongoDB deployment.

Domain: Domain 8: Data Infrastructure•Updated: 2026-11-10
#MongoDB#ReplicaSets#Systemd#Permissions#Networking#SRE#Troubleshooting

1. Introduction

In modern microservice and database architectures, high availability is commonly achieved through database replication clusters. When deploying multi-instance database systems on Linux, system administrators and Site Reliability Engineers (SREs) frequently encounter failures stemming from subtle interactions between application-level security policies, system-level access controls, and network socket binding behavior.

This article details a step-by-step diagnostic and remediation process for resolving a broken multi-node MongoDB replica set deployment (the Suzhou scenario from SadServers). The scenario involves three local MongoDB instances (mongo1, mongo2, mongo3) managed via systemd that failed to form a functional cluster.

By reading this guide, you will learn:

  • How to systematically diagnose rapid systemd service startup failures (code=exited, status=1/FAILURE).
  • The security mechanics behind Linux Posix Discretionary Access Control (DAC) file permissions and MongoDB keyfile authentication constraints.
  • How Linux socket binding (bindIp) interacts with local loopback interfaces (127.0.0.0/8) and /etc/hosts name resolution.
  • Practical command-line diagnostic techniques utilizing systemctl, tail, ls, chmod, chown, md5sum, sed, and mongosh.
  • SRE production best practices for observability, Infrastructure as Code (IaC), and automated remediation.

2. Problem Description and Environment

Scenario Background

A new MongoDB replica set named rs0 was configured in a development environment. The initialization script /home/admin/app/rs0.js was provided to bootstrap the cluster. However, attempts to bring up the replica set failed.

Environment Specifications

  • Operating System: Linux (Debian/Ubuntu-derived environment running systemd)
  • Database Engine: MongoDB (mongod)
  • Management Services: mongo1.service, mongo2.service, mongo3.service
  • Configuration Files: /etc/mongo1.conf, /etc/mongo2.conf, /etc/mongo3.conf
  • Log Location: /var/log/mongodb/mongo1.log, mongo2.log, mongo3.log
  • Replica Bootstrap Script: /home/admin/app/rs0.js
  • Validation Script: /home/admin/agent/check.sh

Expected vs. Actual Behavior

  • Expected Behavior: All three services (mongo1, mongo2, mongo3) start cleanly, listen on their configured local ports (27017, 27018, 27019), communicate over local hostnames, and respond with health: 1 across all nodes when executing rs.status().
  • Actual Behavior:
    • mongo1.service remained running in an unclustered state.
    • mongo2.service and mongo3.service failed immediately upon invocation, repeatedly crashing and entering systemd auto-restart limits.
    • Executing mongosh --file app/rs0.js failed to form a working 3-node cluster.

3. Initial Investigation and Hypotheses

When approaching a multi-service failure on Linux, an engineer must formulate testable hypotheses before making state-changing operations.

Diagnostic Strategy

  1. Scope the failure: Determine which services are running and which are failing.
  2. Examine runtime assets: Read configuration files and startup scripts to understand expected topology.
  3. Inspect log evidence: Inspect system and application logs for explicit error messages.
  4. Audit permissions & networking: Check filesystem ownership/modes and network resolution parameters.

Hypothesis Matrix

Hypothesis Supporting Evidence Evidence Against Diagnostic Step
H1: Configuration syntax error in /etc/mongo2.conf or /etc/mongo3.conf Services exit in < 100ms with exit status 1. Scenario text noted /etc/mongo.conf is not the issue, but instance configs could differ. Inspect .conf files with cat.
H2: Filesystem permissions or ownership issue on security keyfiles mongod enforces strict 0600 mode on keyfiles when authentication is active. mongo1 is running fine; keyfiles 2 & 3 might differ. Check ls -la /etc/mongo* and log files.
H3: Network resolution or socket binding failure Replica script uses hostnames (mongo1:27017, mongo2:27018, mongo3:27019). Node startup wouldn’t necessarily crash immediately unless bind port was already in use. Inspect /etc/hosts and /etc/mongo*.conf.

4. Troubleshooting Process

Step 1: Query System Service Baseline

Objective:
Establish the current runtime status of all three systemd units (mongo1, mongo2, mongo3) and inspect the contents of the replica set initialization script.

Commands Executed:

systemctl status mongo1 --no-pager
systemctl status mongo2 --no-pager
systemctl status mongo3 --no-pager
cat /home/admin/app/rs0.js

Command Explanation:

  • systemctl status <unit>: Queries the systemd daemon for unit operational state, recent lifecycle events, process execution flags, and memory/CPU resource statistics.
  • --no-pager: Disables automatic piping into less, forcing full standard output directly to the terminal.
  • cat: Concatenates and prints file contents to stdout.

Output & Analysis:

● mongo1.service - MongoDB mongo1
     Loaded: loaded (/etc/systemd/system/mongo1.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-11-10 15:26:07 UTC; 30s ago
   Main PID: 764 (mongod)
     CGroup: /system.slice/mongo1.service
             └─764 /usr/bin/mongod --config /etc/mongo1.conf

× mongo2.service - MongoDB mongo2
     Active: failed (Result: exit-code) since Tue 2026-11-10 15:26:20 UTC; 22s ago
    Process: 1388 ExecStart=/usr/bin/mongod --config /etc/mongo2.conf (code=exited, status=1/FAILURE)

× mongo3.service - MongoDB mongo3
     Active: failed (Result: exit-code) since Tue 2026-11-10 15:26:20 UTC; 29s ago
    Process: 1389 ExecStart=/usr/bin/mongod --config /etc/mongo3.conf (code=exited, status=1/FAILURE)

Contents of /home/admin/app/rs0.js:

rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "mongo1:27017" },
    { _id: 1, host: "mongo2:27018" },
    { _id: 2, host: "mongo3:27019" }
  ]
})

Deductions:

  • mongo1 is alive (PID 764) using port 27017.
  • mongo2 and mongo3 execute /usr/bin/mongod, but exit almost instantaneously (duration ~80-90ms) with code=exited, status=1/FAILURE.
  • The replica set configuration requires resolution for three hostnames: mongo1, mongo2, and mongo3.

Decision:
The under-100ms crash duration points to an early application assertion error during initialization. The next logical step is inspecting the log files under /var/log/mongodb/.


Step 2: Log File Inspection

Objective:
Read the latest log entries for mongo1.log, mongo2.log, and mongo3.log to determine why process initialization failed for nodes 2 and 3.

Command Executed:

sudo tail -n 25 /var/log/mongodb/mongo*.log

Command Explanation:

  • tail: Prints the final portion of files.
  • -n 25: Specifies retrieving the last 25 lines per file.
  • sudo: Required because /var/log/mongodb/ log files are restricted to the mongodb user/group.

Output & Analysis:

Key excerpts from /var/log/mongodb/mongo2.log:

{"t":{"$date":"2026-11-10T15:26:19.777+00:00"},"s":"I", "c":"ACCESS",   "id":20254,   "ctx":"main","msg":"Read security file failed","attr":{"error":{"code":30,"codeName":"InvalidPath","errmsg":"permissions on /etc/mongo2-keyfile are too open"}}}
{"t":{"$date":"2026-11-10T15:26:19.777+00:00"},"s":"F", "c":"CONTROL",  "id":20575,   "ctx":"main","msg":"Error creating service context","attr":{"error":"Location5579201: Unable to acquire security key[s]"}}

Key excerpts from /var/log/mongodb/mongo3.log:

{"t":{"$date":"2026-11-10T15:26:19.773+00:00"},"s":"I", "c":"ACCESS",   "id":20254,   "ctx":"main","msg":"Read security file failed","attr":{"error":{"code":30,"codeName":"InvalidPath","errmsg":"permissions on /etc/mongo3-keyfile are too open"}}}
{"t":{"$date":"2026-11-10T15:26:19.773+00:00"},"s":"F", "c":"CONTROL",  "id":20575,   "ctx":"main","msg":"Error creating service context","attr":{"error":"Location5579201: Unable to acquire security key[s]"}}

Deductions:

  • Both mongo2 and mongo3 logged explicit fatal errors (s:"F"):
    • permissions on /etc/mongo2-keyfile are too open
    • permissions on /etc/mongo3-keyfile are too open
  • mongod refused to initialize the engine due to keyfile security violations.

Decision:
We must inspect the file modes and ownership of all keyfiles and configuration files in /etc/mongo*. Additionally, we need to inspect network binding configurations to ensure node communication will succeed once services start.


Step 3: Audit Keyfile Security and System Configurations

Objective:
Check file permissions, ownership, keyfile hash consistency, instance network binding configurations, and local system host resolution (/etc/hosts).

Commands Executed:

ls -la /etc/mongo*
cat /etc/mongo1.conf
cat /etc/mongo2.conf
cat /etc/mongo3.conf
grep -E "mongo" /etc/hosts

Output & Analysis:

File permissions listing (ls -la /etc/mongo*):

-rw------- 1 mongodb mongodb  14 Feb 18  2026 /etc/mongo-keyfile
-rw------- 1 mongodb mongodb  14 Feb 18  2026 /etc/mongo1-keyfile
-rw-r--r-- 1 root    root    234 Feb 18  2026 /etc/mongo1.conf
-rw-r--r-- 1 root    root     14 Feb 18  2026 /etc/mongo2-keyfile
-rw-r--r-- 1 root    root    234 Feb 18  2026 /etc/mongo2.conf
-rw-r--r-- 1 root    root     14 Feb 18  2026 /etc/mongo3-keyfile
-rw-r--r-- 1 root    root    234 Feb 18  2026 /etc/mongo3.conf
-rw-r--r-- 1 root    root    316 Feb 18  2026 /etc/mongod.conf

Configuration contents (/etc/mongo1.conf, /etc/mongo2.conf, /etc/mongo3.conf):

  • All three files specify:
    • replication.replSetName: rs0
    • net.bindIp: 127.0.0.1
    • Ports: 27017 (node 1), 27018 (node 2), 27019 (node 3)
    • Keyfiles: /etc/mongo1-keyfile, /etc/mongo2-keyfile, /etc/mongo3-keyfile

Host file entries (grep -E "mongo" /etc/hosts):

127.0.0.1 mongo1
127.0.0.2 mongo2
127.0.0.3 mongo3

Critical Diagnostic Findings:

  1. Permission Defect:

    • /etc/mongo1-keyfile: -rw------- (0600), owned by mongodb:mongodb. Valid.
    • /etc/mongo2-keyfile: -rw-r--r-- (0644), owned by root:root. Invalid.
    • /etc/mongo3-keyfile: -rw-r--r-- (0644), owned by root:root. Invalid.
  2. Networking & Socket Binding Defect:

    • /etc/hosts maps mongo2 to 127.0.0.2 and mongo3 to 127.0.0.3.
    • However, /etc/mongo2.conf and /etc/mongo3.conf specify net.bindIp: 127.0.0.1.
    • On Linux, standard loopback sockets bound explicitly to 127.0.0.1 will not listen on aliased loopback addresses like 127.0.0.2 or 127.0.0.3. When mongo1 attempts to contact mongo2:27018 at 127.0.0.2, the connection will be refused.

Step 4: Remediation Phase 1 — Correcting Keyfile Security

Objective:
Restrict permissions on keyfiles to 0600 (read/write by owner only), re-assign ownership to user/group mongodb:mongodb, and confirm hash equality.

Commands Executed:

sudo chmod 600 /etc/mongo*
sudo chown mongodb:mongodb /etc/mongo*
sudo md5sum /etc/mongo*

Output & Analysis:

Checksum output (md5sum):

d855496646e88b7c12e0a80135bef652  /etc/mongo-keyfile
d855496646e88b7c12e0a80135bef652  /etc/mongo1-keyfile
99f24a1bced9bc954cf44a42bceb85b5  /etc/mongo1.conf
d855496646e88b7c12e0a80135bef652  /etc/mongo2-keyfile
d55377e2caa42055377caf7ec3e84625  /etc/mongo2.conf
d855496646e88b7c12e0a80135bef652  /etc/mongo3-keyfile
bc75bb1a4f6b5f06bcf61f0c341ff67a  /etc/mongo3.conf
df6614a6fd62ae53968b5feb39fd97bb  /etc/mongod.conf

Deductions:

  • Hash comparison verifies that /etc/mongo1-keyfile, /etc/mongo2-keyfile, and /etc/mongo3-keyfile all share identical MD5 signatures (d855496646e88b7c12e0a80135bef652). This guarantees cluster members share the exact same cryptographic secret.

Step 5: Remediation Phase 2 — Service Lifecycle Verification

Objective:
Start mongo2 and mongo3, then verify all three database daemons are operational.

Commands Executed:

sudo systemctl start mongo1 mongo2 mongo3
systemctl status mongo1 mongo2 mongo3 --no-pager

Output & Analysis:

● mongo1.service - MongoDB mongo1
     Active: active (running) since Tue 2026-11-10 15:26:07 UTC; 9min ago
   Main PID: 764 (mongod)

● mongo2.service - MongoDB mongo2
     Active: active (running) since Tue 2026-11-10 15:35:10 UTC; 12s ago
   Main PID: 1501 (mongod)

● mongo3.service - MongoDB mongo3
     Active: active (running) since Tue 2026-11-10 15:35:10 UTC; 12s ago
   Main PID: 1502 (mongod)

Deductions:
Fixing file modes and ownership resolved the service crash loop. All three daemons are now active and listening on their respective ports.


Step 6: Remediation Phase 3 — Fixing Loopback Host Resolution

Objective:
Align /etc/hosts resolution with the bindIp: 127.0.0.1 directive configured in /etc/mongo*.conf.

Commands Executed:

sudo sed -i '/mongo/d' /etc/hosts
echo -e "127.0.0.1 mongo1\n127.0.0.1 mongo2\n127.0.0.1 mongo3" | sudo tee -a /etc/hosts
cat /etc/hosts

Command Explanation:

  • sed -i '/mongo/d' /etc/hosts: Performs in-place deletion (-i) of any line matching regex pattern /mongo/ from /etc/hosts.
  • echo -e ...: Emits strings containing newline escape sequences (\n).
  • sudo tee -a /etc/hosts: Appends standard input to /etc/hosts with elevated root privileges.

Output & Analysis:

127.0.0.1        localhost
::1              localhost ip6-localhost ip6-loopback
ff02::1          ip6-allnodes
ff02::2          ip6-allrouters

127.0.0.1 mongo1
127.0.0.1 mongo2
127.0.0.1 mongo3

Step 7: Cluster Initialization & Health Validation

Objective:
Initiate the replica set using mongosh and evaluate cluster topology state.

Commands Executed:

mongosh --file app/rs0.js
mongosh --eval "rs.status()"
/home/admin/agent/check.sh

Output & Analysis:

Key sections of rs.status() output:

{
  set: 'rs0',
  myState: 1,
  members: [
    {
      _id: 0,
      name: 'mongo1:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      self: true
    },
    {
      _id: 1,
      name: 'mongo2:27018',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      pingMs: Long('0')
    },
    {
      _id: 2,
      name: 'mongo3:27019',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      pingMs: Long('0')
    }
  ],
  ok: 1
}

Script check result:

OK

Conclusion:
The replica set initialized cleanly. mongo1 was elected PRIMARY, and both mongo2 and mongo3 reached state SECONDARY with health: 1. The validation test passed successfully.


5. Root Cause Analysis

The failure of the MongoDB replica set was caused by two distinct defects occurring across the filesystem and networking layers.

+-----------------------------------------------------------------------------------+
|                                  FAILURE CASCADE                                  |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|  1. FILESYSTEM LAYER DEFECT                                                       |
|     /etc/mongo2-keyfile (Mode: 0644, Owner: root:root)                            |
|     /etc/mongo3-keyfile (Mode: 0644, Owner: root:root)                            |
|                                │                                                  |
|                                ▼                                                  |
|  2. APPLICATION SECURITY ASSERTION                                                |
|     mongod checks file permissions on startup                                     |
|     Rejects keyfile because permissions are > 0600                                |
|                                │                                                  |
|                                ▼                                                  |
|  3. PROCESS CRASH                                                                 |
|     mongo2.service & mongo3.service exit immediately (status=1/FAILURE)           |
|                                │                                                  |
|                                ▼                                                  |
|  4. NETWORK LAYER DEFECT (Secondary Blocker)                                      |
|     /etc/hosts mapped mongo2 -> 127.0.0.2 and mongo3 -> 127.0.0.3                 |
|     mongo*.conf specified bindIp: 127.0.0.1                                       |
|     Result: Inter-node heartbeat traffic to 127.0.0.2:27018 refused                |
|                                                                                   |
+-----------------------------------------------------------------------------------+

Detailed Mechanism Breakdowns

1. Keyfile Security Enforcement (Linux DAC & Application Layer)

MongoDB uses symmetric secret keyfiles for internal cluster node authentication. To prevent non-privileged users or compromised processes on the local system from reading the shared secret, MongoDB enforces a strict startup rule:

$$\text{Keyfile Mode} \le 0600 \quad \text{AND} \quad \text{Owner} = \text{service user}$$

If mode bits permit group or world read access (e.g., 0644), mongod throws error code Location5579201 (Unable to acquire security key[s]) and aborts startup immediately.

2. Network Socket Binding & Loopback Routing

In Linux networking, the loopback adapter (lo) is assigned the IP block 127.0.0.0/8. Although 127.0.0.1, 127.0.0.2, and 127.0.0.3 all route to the local host, network sockets bound strictly to 127.0.0.1 (bindIp: 127.0.0.1) only listen for TCP SYN packets destined for 127.0.0.1. Packets directed to 127.0.0.2 or 127.0.0.3 are rejected with TCP RST (Connection Refused).


6. Solution and Remediation

Summary of Changes

  1. Security Fix: Updated permissions and ownership of keyfiles to adhere to DAC rules.
  2. Network Fix: Pointed all node hostnames in /etc/hosts to 127.0.0.1.
  3. Cluster Bootstrap: Re-executed systemd service startups and the rs.initiate() script.

Before vs. After Configuration State

Keyfile Permissions (ls -la /etc/mongo*)

BEFORE:
-rw------- 1 mongodb mongodb  14 /etc/mongo1-keyfile
-rw-r--r-- 1 root    root     14 /etc/mongo2-keyfile
-rw-r--r-- 1 root    root     14 /etc/mongo3-keyfile

AFTER:
-rw------- 1 mongodb mongodb  14 /etc/mongo1-keyfile
-rw------- 1 mongodb mongodb  14 /etc/mongo2-keyfile
-rw------- 1 mongodb mongodb  14 /etc/mongo3-keyfile

Host Resolution (/etc/hosts)

BEFORE:
127.0.0.1 mongo1
127.0.0.2 mongo2
127.0.0.3 mongo3

AFTER:
127.0.0.1 mongo1
127.0.0.1 mongo2
127.0.0.1 mongo3

7. Verification and Validation

Verification requires confirming both individual service health and cluster state.

Verification Matrix

Target Test Method Expected Result Actual Result
Service Status systemctl status mongo1 mongo2 mongo3 All units active (running) Active: active (running) on all nodes
Listening Ports ss -tulpn | grep mongod Ports 27017, 27018, 27019 listening Confirmed listening on 127.0.0.1
Keyfile Hash md5sum /etc/mongo*-keyfile Matching MD5 digests d855496646e8... across all 3 keyfiles
Cluster Health mongosh --eval "rs.status()" health: 1 for all nodes 1 PRIMARY, 2 SECONDARY, all health: 1
Automated Check /home/admin/agent/check.sh Execution returns OK Returned OK

8. Alternative Troubleshooting Approaches

While the investigation relied on systemctl, tail, and standard file inspection utilities, alternative diagnostic workflows could achieve the same results.

Diagnostic Method Comparison

Approach Tools Example Command Pros Cons / Limitations
Log Inspection (Used) tail, cat sudo tail -n 25 /var/log/mongodb/*.log Non-intrusive, clear application context. Requires read access to log paths.
Systemd Journal journalctl journalctl -u mongo2.service -n 50 -e Centralized, survives log file deletion. May only capture stderr/stdout, not dedicated application log files.
Interactive Foreground Execution mongod direct CLI sudo -u mongodb /usr/bin/mongod --config /etc/mongo2.conf Immediate output to stdout/stderr; bypasses systemd overhead. Changes runtime environment slightly; risk of running as wrong user.
Syscall Tracing strace sudo strace -f -e trace=openat,access /usr/bin/mongod --config /etc/mongo2.conf Reveals exact system call failures (EACCES, ENOENT). Extremely verbose output; high overhead.
Socket & Network Audit ss, lsof sudo ss -tulpn | grep mongod Instantly reveals listening ports and bound interfaces. Does not indicate why a process failed to start.

9. Linux Command Deep Dive

1. systemctl

  • Purpose: Controls the systemd system and service manager.
  • Syntax: systemctl [COMMAND] [UNIT...]
  • Key Flags & Useful Options:
Variant Purpose
systemctl status <unit> Show unit runtime state, memory/CPU, PID, and recent logs.
systemctl start/stop/restart <unit> Alter runtime service state immediately.
systemctl enable/disable <unit> Configure unit autostart at system boot.
systemctl is-active <unit> Return exit status 0 if active, non-zero if inactive (scripting friendly).
systemctl cat <unit> Print the underlying unit definition file (/etc/systemd/system/...).
  • Practical Example:
    # Check all failed services on host
    systemctl list-units --type=service --state=failed

2. tail

  • Purpose: Outputs the last portion of files.
  • Syntax: tail [OPTION]... [FILE]...
  • Key Flags & Useful Options:
Option Description
-n <lines> Number of lines to output (e.g., -n 50).
-f Follow file growth continuously.
-F Follow file growth by filename (handles log rotation).
-q Quiet mode; suppress filename headers when tailing multiple files.
  • Practical Example:
    # Monitor system logs in real time while attempting a service restart
    sudo tail -F /var/log/mongodb/mongo2.log

3. chmod & chown

  • Purpose: Modify Posix file permissions (chmod) and user/group ownership (chown).

  • Syntax:

    • chmod [MODE] FILE...
    • chown OWNER[:GROUP] FILE...
  • Permission Mode Reference Table:

Octal Permission String Description SRE Use Case
600 -rw------- Read/Write by owner only Private SSH keys, database keyfiles, TLS keys
644 -rw-r--r-- Owner Read/Write; Group/World Read Standard config files (/etc/hosts, /etc/nginx.conf)
755 -rwxr-xr-x Owner Read/Write/Exec; Group/World Read/Exec System executables, directories, scripts
700 -rwx------ Full access by owner only User .ssh directories

4. sed (Stream Editor)

  • Purpose: Performs text transformations on an input stream or file.

  • Syntax: sed [OPTION] 'script' [input-file]

  • Key Options:

    • -i: Edit files in place (modifies the target file directly).
    • /pattern/d: Delete lines matching pattern.
    • s/old/new/g: Substitute all occurrences of old with new.
  • Practical Example:

    # Replace 127.0.0.2 with 127.0.0.1 in a configuration file in place
    sudo sed -i 's/127.0.0.2/127.0.0.1/g' /etc/hosts

10. Underlying Linux Concepts and Architecture

1. Posix Discretionary Access Control (DAC)

In Linux, file permissions are controlled via owner, group, and world bits stored in the inode metadata.

  File Mode Bit Structure: - rw- r-- r-- (0644)
                           │  │   │   │
  Type ( - = Regular ) ────┘  │   │   │
  Owner ( rw- = 6 ) ──────────┘   │   │
  Group ( r-- = 4 ) ──────────────┘   │
  World ( r-- = 4 ) ──────────────────┘

Security-sensitive applications like OpenSSH (sshd), PostgreSQL, and MongoDB enforce strict application-level DAC sanity checks. If permission bits permit reading by non-owners, the application explicitly refuses to launch to defend against local privilege escalation or secret leakage.

2. Socket Binding & Loopback Aliasing

Network sockets in Linux are defined by an IP address and a Port tuple (IP, Port).

  [ Interface: lo (Loopback 127.0.0.0/8) ]
   ├── IP: 127.0.0.1  <---> Socket Listening: (127.0.0.1:27018)  [ACCEPT]
   ├── IP: 127.0.0.2  <---> Packet sent to (127.0.0.2:27018)    [REFUSED / TCP RST]
   └── IP: 127.0.0.3  <---> Packet sent to (127.0.0.3:27019)    [REFUSED / TCP RST]

When a process sets bindIp: 127.0.0.1, the kernel populates the socket binding table specifically for 127.0.0.1. If an application tries to connect to 127.0.0.2, the kernel’s network stack finds no active listening socket for 127.0.0.2:27018 and immediately drops or rejects the connection.


11. Real-World Sysadmin and SRE Applications

1. Observability & Monitoring

  • Systemd Failure Alerting: Configure systemd notification integration (OnFailure=alert-mailer@%n.service) or Prometheus node_exporter alerts (node_systemd_unit_state{state="failed"} > 0).
  • Log Aggregation: Route application logs (/var/log/mongodb/*.log) to a centralized log system (e.g., Fluentbit/Vector to Grafana Loki or Elasticsearch) with log pattern alerts matching Read security file failed or Unable to acquire security key[s].

2. Infrastructure as Code (IaC) & Configuration Management

Manual permission and host configuration edits introduce human error. Using Ansible, secret file permissions and /etc/hosts resolution should be declared idempotently:

# Sample Ansible task snippet for keyfile management
- name: Ensure MongoDB keyfile has strict permissions
  ansible.builtin.file:
    path: "/etc/mongo{{ item }}-keyfile"
    owner: mongodb
    group: mongodb
    mode: '0600'
  loop: [1, 2, 3]

12. Further Learning and Experiments

Hands-on Experiments

Experiment 1: Keyfile Security Enforcement Validation

  1. In a test environment, change /etc/mongo1-keyfile permissions to 0644:
    sudo chmod 644 /etc/mongo1-keyfile
  2. Restart mongo1:
    sudo systemctl restart mongo1
  3. Check systemctl status mongo1 and inspect /var/log/mongodb/mongo1.log to observe mongod catching the permission mismatch.
  4. Restore original mode 600:
    sudo chmod 600 /etc/mongo1-keyfile

Experiment 2: Inspecting Socket Bindings with ss and nc

  1. Test connectivity across aliased loopback addresses using nc (netcat):
    nc -zv 127.0.0.1 27017
    nc -zv 127.0.0.2 27017
  2. Observe how 127.0.0.1 succeeds while 127.0.0.2 fails when bindIp is explicitly locked to 127.0.0.1.

13. Key Takeaways

  1. Check Logs First: Systemd unit status (status=1/FAILURE) indicates that a process exited; application logs (/var/log/...) reveal why it exited.
  2. Keyfiles Demand 0600: Security credentials and shared secret files should always be owned by the service user with mode 0600 or 0400.
  3. Match Bind Addresses with Name Resolution: Ensure hostnames defined in /etc/hosts resolve to IP addresses that services are actually configured to listen on.
  4. Use Checksums to Verify Identical Secrets: Use md5sum or sha256sum to quickly confirm cluster keys across multiple nodes match.

14. Quick Troubleshooting Reference / Runbook

================================================================================
          MONGODB REPLICA SET STARTUP TROUBLESHOOTING RUNBOOK
================================================================================

1. SERVICE STATUS AUDIT
   $ systemctl status mongo1 mongo2 mongo3 --no-pager

2. LOG DIAGNOSTICS (Identify crash assertions)
   $ sudo tail -n 50 /var/log/mongodb/mongo*.log | grep -E "ERROR|FATAL|errmsg"

3. KEYFILE SECURITY AUDIT
   $ ls -la /etc/mongo*-keyfile
   -> Expectation: Mode 0600 (-rw-------), Owner mongodb:mongodb
   
   Fix Permissions:
   $ sudo chmod 600 /etc/mongo*-keyfile
   $ sudo chown mongodb:mongodb /etc/mongo*-keyfile

4. KEYFILE CONSISTENCY CHECK
   $ sudo md5sum /etc/mongo*-keyfile
   -> Expectation: Identical hash output across all cluster nodes

5. NETWORK & HOST RESOLUTION AUDIT
   $ grep -E "bindIp" /etc/mongo*.conf
   $ grep -E "mongo" /etc/hosts
   -> Expectation: /etc/hosts mappings must match configured bindIp interfaces

6. RESTART & CLUSTER HEALTH VALIDATION
   $ sudo systemctl restart mongo1 mongo2 mongo3
   $ mongosh --eval "rs.status()"