1. Overview
MongoDB is an open-source, high-performance document database designed for horizontal scalability, high availability, and operational flexibility. Unlike traditional relational database management systems (RDBMS) that enforce rigid, tabular schemas across rows and columns, MongoDB stores data as flexible, schema-less documents using a binary serialization format known as BSON (Binary JSON).
In modern infrastructure, MongoDB addresses key challenges associated with rapidly evolving application data structures, high write throughput demands, and the operational complexity of scaling relational databases across multiple physical servers.
┌─────────────────────────────────────────────────────────────────────────────┐
│ Relational vs. MongoDB │
├───────────────────────────────────┬─────────────────────────────────────────┤
│ Relational (RDBMS) │ MongoDB (Document Model) │
├───────────────────────────────────┼─────────────────────────────────────────┤
│ Databases │ Databases │
│ Tables │ Collections │
│ Rows │ Documents (BSON) │
│ Columns │ Fields │
│ Foreign Keys / Joins │ Embedded Documents / References │
└───────────────────────────────────┴─────────────────────────────────────────┘
Core Production Use Cases
- Application Data Stores: Storing user profiles, product catalogs, content management system (CMS) metadata, and mobile/IoT backend payloads where JSON structures map naturally to application-level code.
- High-Volume Event and Log Ingestion: Ingesting time-series data, telemetry, and event stream streams utilizing Time-To-Live (TTL) indexes for automated data retention.
- Analytics and Data Pipelines: Leveraging MongoDB’s native Aggregation Framework to execute multi-stage data processing, analytical transformations, and reporting in-database without external ETL tools.
Key Operational Strengths
- Dynamic Schema Flexibility: Fields can vary between documents within the same collection, allowing application developers to iterate schemas rapidly without expensive schema migrations (
ALTER TABLE). - Native High Availability: Built-in automatic failover and redundancy through Replica Sets without relying on external clustering software.
- Horizontal Scalability: Native database-level partition capability (Sharding) that distributes large datasets across cluster nodes via shard keys.
- Advanced Storage Engine: Powered by the WiredTiger storage engine, providing document-level concurrency control, checkpointing, fast write paths, and inline data compression.
Limitations and Trade-offs
- Memory Footprint: WiredTiger is designed to utilize available RAM aggressively for its internal cache, requiring precise memory allocation on multi-tenant hosts.
- Lack of Multi-Document ACID Default (Across Shards without Cost): While multi-document transactions exist in modern versions, traditional single-document operations are atomic by design; distributed transactions incur performance overhead.
- Storage Overhead: BSON documents explicitly encode field names with values in every document, which consumes more disk space than strictly typed RDBMS row allocations unless compensated for by compression.
2. Architecture and Mental Model
To effectively manage and troubleshoot MongoDB in a production environment, a System Administrator or Site Reliability Engineer (SRE) must understand the underlying processes, network workflows, storage engine internals, and high-availability structures.
System Architecture and Process Model
MongoDB runs as a single primary daemon process named mongod. In a clustered topology, additional routing processes named mongos direct traffic, but mongod remains the fundamental database process responsible for data persistence, index management, client connections, and replication.
+-------------------------------------------------+
| Application Driver |
+-------------------------------------------------+
│
mongodb:// connection
│
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ mongod Daemon │
│ │
│ ┌──────────────────────────┐ ┌─────────────────────────────────┐ │
│ │ Connection Pool / Listener│ │ Authentication & Authorization │ │
│ │ (:27017) │ │ (SCRAM-SHA-256) │ │
│ └────────────┬─────────────┘ └────────────────┬────────────────┘ │
│ │ │ │
│ └────────────────────┬───────────────────────┘ │
│ ▼ │
│ ┌──────────────────────────────┐ │
│ │ Query Engine & Aggregations │ │
│ └──────────────┬───────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────┐ │
│ │ WiredTiger Storage Engine │ │
│ │ ┌──────────────────────────┐ │ │
│ │ │ WiredTiger Cache (RAM) │ │ │
│ │ └────────────┬─────────────┘ │ │
│ │ │ (Checkpoints) │ │
│ │ ▼ │ │
│ │ ┌──────────────────────────┐ │ │
│ │ │ Data Files & Journal │ │ │
│ │ └──────────────────────────┘ │ │
│ └──────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────────────┘
WiredTiger Storage Engine Mechanics
WiredTiger is the default storage engine for MongoDB. Its architecture relies on two key constructs for performance and durability:
- WiredTiger In-Memory Cache: All read and write operations interact with the cache. Reads pull uncompressed data blocks from disk into RAM. Writes update BSON data structures in RAM and write changes sequentially to an on-disk Journal for durability.
- Checkpoints and Journaling:
- Journaling: Ensures write durability between checkpoints. Every write operation writes to an in-memory journal, which flushes to disk periodically (default every 50ms). If
mongodcrashes unexpected, it replays the journal to maintain consistency. - Checkpoints: Writes dirty pages from the WiredTiger cache to persistent disk data files at set intervals (typically every 60 seconds or after 2 GB of journal data). A checkpoint presents a consistent view of the database files on disk.
Complete Client Request & Connection Lifecycle
[Client Driver] [mongod Server]
│ │
│ 1. TCP Handshake (port 27017) │
├────────────────────────────────────────────────────────►│
│ 2. Wire Protocol Handshake (Hello/isMaster) │
├────────────────────────────────────────────────────────►│
│ 3. SCRAM-SHA-256 Auth Exchange │
├────────────────────────────────────────────────────────►│
│ 4. Read / Write Operation Request │
├────────────────────────────────────────────────────────►│
│ │──┐ Check Cache/Index
│ │ │ Update Journal/Mem
│ │◄─┘
│ 5. Response BSON Wire Frame │
│◄────────────────────────────────────────────────────────┤
- TCP Connection Establishment: The application client opens a standard TCP connection to the host on port
27017. - Wire Protocol Handshake: Driver and server perform an initial negotiation (such as running
helloorisMastercommands) to exchange API capabilities, server version, and discover replica set topology. - Authentication: The client authenticates against the target or
admindatabase using SCRAM-SHA-256 or certificate authentication. - Operation Processing:
- Read Request:
mongodchecks its index cache. If index keys point to data pages in memory, BSON documents are returned immediately. If missing, pages are loaded from disk into the cache. - Write Request: The operation updates the cache, writes to the replication Oplog (if in a replica set), and logs to the WiredTiger journal.
- Response Delivery: Data is packed into standard wire format frames and transmitted back to the client pool.
3. Core Concepts and Terminology
Understanding MongoDB requires familiarity with specific document and high-availability constructs.
1. Document
A single record in a MongoDB collection stored in BSON format.
- Why it matters: Represents application data models directly without requiring joins across multiple tables.
- Example:
{
"_id": ObjectId("6543217b8f1a2b3c4d5e6f7a"),
"username": "sysadmin_jane",
"email": "jane@example.com",
"roles": ["admin", "sre"],
"active": true,
"createdAt": ISODate("2026-01-15T08:30:00Z")
}
2. Collection
A grouping of MongoDB documents, analogous to a table in a relational database. Collections do not enforce strict structural uniformity across documents unless explicit JSON Schema Validation rules are defined.
3. Replica Set
A cluster of mongod instances maintaining identical datasets for high availability.
- PRIMARY Node: The single cluster node accepting all write operations.
- SECONDARY Nodes: Nodes replicating the Primary’s operations log to maintain data parity; can optionally serve read queries.
- ARBITER Node: An optional node that participates in leader elections but stores no database data.
4. Oplog (local.oplog.rs)
A special, capped collection located in the local database that records an ordered log of all operations that modify data on the PRIMARY. Secondary members continually poll and apply this log asynchronously.
5. Read Preference and Write Concern
- Read Preference: Configuration setting defining how client applications route read requests across the replica set topology (
primary,primaryPreferred,secondary,secondaryPreferred,nearest). - Write Concern: Configuration describing the level of acknowledgment requested from MongoDB for write operations.
{ w: 1 }requests acknowledgment from the primary only, whereas{ w: "majority" }guarantees that the write has been committed to a majority of voting replica set nodes before returning success.
4. Installation and Version Management
MongoDB packages are available via official MongoDB APT and YUM/DNF repositories for enterprise Linux distributions.
Installing MongoDB on Ubuntu / Debian (APT)
Step 1: Import the Public GPG Key
sudo apt-get install -y gnupg curl
curl -fsSL https://www.mongodb.org/static/pgp/server-7.0.asc | \
sudo gpg --dearmor -o /usr/share/keyrings/mongodb-server-7.0.gpg
Step 2: Add the Official Repository File
# Ubuntu 22.04 LTS (Jammy)
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" | \
sudo tee /etc/apt/sources.list.d/mongodb-org-7.0.list
Step 3: Install MongoDB Packages
sudo apt-get update
sudo apt-get install -y mongodb-org
Installing MongoDB on RHEL / Rocky Linux / AlmaLinux (DNF)
Step 1: Create the Repository Configuration
cat << 'EOF' | sudo tee /etc/yum.repos.d/mongodb-org-7.0.repo
[mongodb-org-7.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/7.0/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-7.0.asc
EOF
Step 2: Install Packages via DNF
sudo dnf install -y mongodb-org
Verifying the Installation and Version Info
# Check mongod binary version
mongod --version
# Verify shell utility (mongosh) version
mongosh --version
Default Installation Footprint
| Component / File Path | Purpose / Description |
|---|---|
/usr/bin/mongod |
Primary database daemon executable |
/usr/bin/mongosh |
Next-generation interactive MongoDB shell CLI |
/etc/mongod.conf |
Primary YAML configuration file |
/var/lib/mongodb |
Default data storage directory (storage.dbPath) |
/var/log/mongodb/mongod.log |
Default system log file (systemLog.path) |
/lib/systemd/system/mongod.service |
systemd service unit definition |
5. Configuration Files and Directory Structure
MongoDB uses the YAML syntax format for configuration files. Indentation matters; always use spaces rather than tabs.
Essential Configuration File Matrix
| File / Directory | Owner / Permissions | Purpose / Operational Details |
|---|---|---|
/etc/mongod.conf |
mongodb:mongodb (0644) |
Main configuration file. Defines paths, networking, security, and cluster options. |
/var/lib/mongodb/ |
mongodb:mongodb (0755) |
Default data directory (storage.dbPath) storing database files, indexes, and journals. |
/var/lib/mongodb/WiredTiger |
mongodb:mongodb (0600) |
Metadata file for the WiredTiger storage engine. Must remain uncorrupted. |
/var/log/mongodb/ |
mongodb:mongodb (0755) |
Default log directory. Contains operational runtime logs (mongod.log). |
/tmp/mongodb-27017.sock |
mongodb:mongodb (0777) |
UNIX domain socket for local CLI connections on port 27017. |
Production /etc/mongod.conf Reference
# /etc/mongod.conf
# Comprehensive Production MongoDB 7.0 Configuration
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
verbosity: 0
storage:
dbPath: /var/lib/mongodb
journal:
enabled: true
wiredTiger:
engineConfig:
cacheSizeGB: 4
processManagement:
timeZoneInfo: /usr/share/zoneinfo
fork: true
pidFilePath: /var/run/mongodb/mongod.pid
net:
port: 27017
bindIp: 127.0.0.1,10.0.1.15 # DO NOT set to 0.0.0.0 without auth/firewalls!
security:
authorization: enabled # Enables Role-Based Access Control (RBAC)
keyFile: /etc/mongodb/keyfile # Cluster authentication key for replica sets
replication:
replSetName: "rs0" # Mandatory name for replica set membership
Parameter Validation and Reloading Behavior
- Validation: Before restarting a production instance, syntax-check the configuration file using
mongod --config /etc/mongod.conf --validate(or dry-running startup). - Reload vs. Restart: Dynamic updates to log levels can be made without restarting via
db.setLogLevel(). However, structural configuration changes (such as changingnet.bindIp,storage.dbPath, orsecurity.authorization) require a full service restart.
6. Service Management and Common Operations
Management of the mongod process on modern Linux distributions relies on systemctl.
Basic Service Operations
# Start MongoDB service
sudo systemctl start mongod
# Stop MongoDB service gracefully (flushes cache and closes journals cleanly)
sudo systemctl stop mongod
# Restart service (stop + start sequence)
sudo systemctl restart mongod
# Enable service auto-start on boot
sudo systemctl enable mongod
# Disable service auto-start on boot
sudo systemctl disable mongod
# Check execution status and recent service log events
sudo systemctl status mongod
Process and Listener Diagnostics
# Verify the process is active and inspect process flags
ps aux | grep mongod
# Confirm listening sockets and bound IP addresses on port 27017
sudo ss -tlnp | grep 27017
# Example output expected:
# LISTEN 0 4096 10.0.1.15:27017 0.0.0.0:* users:(("mongod",pid=14202,fd=11))
7. Common Administrative Tasks and Workflows
Workflow 1: Initializing a Secured Replica Set Cluster
Objective
Configure a 3-node MongoDB Replica Set (rs0) with role-based authentication enabled.
Step 1: Configure /etc/mongod.conf across all nodes
On all three nodes (10.0.1.11, 10.0.1.12, 10.0.1.13), configure binding and replica set parameters, keeping authorization: disabled during initial setup.
net:
port: 27017
bindIp: 0.0.0.0
replication:
replSetName: "rs0"
security:
authorization: disabled
Restart mongod on all nodes: sudo systemctl restart mongod.
Step 2: Initialize the Replica Set Topology
Connect to the primary node via mongosh and execute initialization:
// Connect locally
// mongosh "mongodb://10.0.1.11:27017"
rs.initiate({
_id: "rs0",
members: [
{ _id: 0, host: "10.0.1.11:27017" },
{ _id: 1, host: "10.0.1.12:27017" },
{ _id: 2, host: "10.0.1.13:27017" }
]
})
Verify status using rs.status() until one node shows PRIMARY and others show SECONDARY.
Step 3: Create the Administrative User
While connected to the PRIMARY node, switch to the admin database and create the root user:
use admin
db.createUser({
user: "siteAdmin",
pwd: "SuperSecurePassword123!",
roles: [
{ role: "userAdminAnyDatabase", db: "admin" },
{ role: "readWriteAnyDatabase", db: "admin" },
{ role: "clusterAdmin", db: "admin" }
]
})
Step 4: Generate and Deploy Shared Internal Security Keyfile
Generate a keyfile on node 1 for internal cluster communication:
openssl rand -base64 756 | sudo tee /etc/mongodb/keyfile
sudo chmod 400 /etc/mongodb/keyfile
sudo chown mongodb:mongodb /etc/mongodb/keyfile
Securely copy (scp) this keyfile to the exact location on nodes 2 and 3 with identical permissions.
Step 5: Enable Authorization and Keyfile across the Cluster
Update /etc/mongod.conf on all nodes:
security:
authorization: enabled
keyFile: /etc/mongodb/keyfile
Restart all instances. Now, connection to the replica set requires authentication.
Workflow 2: Performing a Safe Rolling Cluster Upgrade
Objective
Upgrade a 3-node Replica Set from version 6.0 to 7.0 with zero database downtime.
Procedure:
- Upgrade Secondaries First:
- Connect to Secondary Node 2 (
10.0.1.12). - Stop the daemon:
sudo systemctl stop mongod. - Update APT/YUM repos to point to MongoDB 7.0 binaries.
- Update software:
sudo apt-get install -y mongodb-org. - Restart daemon:
sudo systemctl start mongod. - Connect with
mongoshand verify node catches up:rs.status(). EnsurestateStrreturnsSECONDARY. - Repeat step 1 for Secondary Node 3 (
10.0.1.13).
- Step Down the Primary Node:
- Connect to Primary Node 1 (
10.0.1.11). - Execute step down command to force election of an upgraded secondary node:
rs.stepDown(60) // Primary becomes secondary for 60 seconds
- Verify via
rs.status()that Node 2 or Node 3 has transitioned toPRIMARY.
- Upgrade the Old Primary Node:
- Stop
mongodon Node 1. - Upgrade APT/YUM binaries to version 7.0.
- Start
mongodon Node 1. Node 1 rejoins the set as aSECONDARY.
8. CLI Command Reference and Cheatsheet
1. Version & Environment Diagnostics
| Command | Purpose | Example / Context |
|---|---|---|
mongod --version |
Displays mongod server binary build details and target architecture. |
mongod --version |
mongosh --version |
Displays mongosh shell client version. |
mongosh --version |
mongosh --eval "db.adminCommand({ ping: 1 })" |
Runs quick non-interactive health checkping via CLI. | mongosh --eval "db.ping()" |
2. Interactive mongosh Connection Utility
# Connect to local instance without authentication
mongosh "mongodb://127.0.0.1:27017"
# Connect to specific database with authentication credentials
mongosh "mongodb://siteAdmin:SuperSecurePassword123!@127.0.0.1:27017/admin"
# Production Replica Set Connection String (Handles Failover automatically)
mongosh "mongodb://appUser:Pass123@10.0.1.11:27017,10.0.1.12:27017,10.0.1.13:27017/appDB?replicaSet=rs0&authSource=admin"
3. Service & Storage Diagnostics (Executed inside mongosh)
| Command | Purpose | Important Flags / Parameters | Example / Output Explanation |
|---|---|---|---|
db.serverStatus() |
Comprehensive state dump (RAM, connections, storage engine stats). | db.serverStatus().connections |
Shows current, available, and totalCreated client connections. |
db.stats() |
Database level sizing and storage allocation metrics. | db.stats() |
Displays document counts, data size, scale factor, and index total size. |
db.currentOp() |
Lists active operations currently executing on database. | db.currentOp({ "secs_running": { $gt: 5 } }) |
Filters ops running longer than 5 seconds. |
db.killOp(opid) |
Terminates a long-running runaway operation by operation ID. | db.killOp(14290) |
Warning: Use with care; avoid killing internal system ops. |
4. Indexing & Query Analysis Commands
| Command | Purpose | Code Example |
|---|---|---|
| List Indexes | Displays all configured indexes on a collection. | db.users.getIndexes() |
| Create Index | Creates index on fields (1 for Ascending, -1 for Descending). | db.users.createIndex({ "email": 1 }, { unique: true }) |
| Query Plan Analysis | Generates detailed execution metrics and plan for query profiling. | db.users.explain("executionStats").find({ "email": "test@domain.com" }) |
5. Replica Set Management Commands
| Command | Purpose | Operational Details |
|---|---|---|
rs.status() |
Detailed state, health status, and optimes of all cluster nodes. | Check stateStr (PRIMARY, SECONDARY, RECOVERING). |
rs.conf() |
Returns current replica set topology definition JSON document. | Used to inspect voting weights, member hostnames, and priorities. |
rs.stepDown(seconds) |
Forces current PRIMARY node to step down to SECONDARY role. | Used to trigger intentional failover for maintenance. |
rs.printReplicationInfo() |
Prints primary oplog status, total size, and time window covered. | Vital for calculating replication lag buffers. |
rs.printSecondaryReplicationInfo() |
Prints secondary nodes replication status and lag behind primary. | Identifies lagging or recovering secondary members. |
9. Logs, Monitoring and Diagnostics
Log File Analysis
MongoDB records structured log messages in JSON format (modern releases) or text strings. Logs are stored by default at /var/log/mongodb/mongod.log.
Inspecting Logs via CLI
# Live tail of MongoDB logs
tail -f /var/log/mongodb/mongod.log
# Filtering systemd journal logs for mongod unit
journalctl -u mongod -n 100 --no-pager
# Filtering for Severe Errors or Warnings (JSON logs)
grep -E '"s":"E"|"s":"W"' /var/log/mongodb/mongod.log | jq .
Interpreting Common Log Components
{
"t": { "$date": "2026-11-03T10:15:30.123+00:00" },
"s": "I",
"c": "COMMAND",
"id": 51803,
"ctx": "conn14",
"msg": "Slow query",
"attr": {
"ns": "appDB.users",
"durationMillis": 450,
"planSummary": "COLLSCAN",
"reslen": 12450
}
}
s: Severity Level:I(Info),W(Warning),E(Error),F(Fatal).c: Component:COMMAND,STORAGE,REPL(Replication),ACCESS(Auth).planSummary: Query Engine Decision:COLLSCANindicates a missing index (Full Collection Scan).IXSCANindicates an Index Scan.
Profiling and Dynamic Diagnostics
Database Profiling writes fine-grained operational performance metrics to the system collection system.profile.
// Check current profiling level (0 = Off, 1 = Log Slow Ops, 2 = Log All Ops)
db.getProfilingStatus()
// Enable Profiling for operations taking longer than 100 milliseconds
db.setProfilingLevel(1, { slowms: 100 })
// Query top 5 slowest operations recorded by profiler
db.system.profile.find().sort({ ts: -1 }).limit(5).pretty()
10. Troubleshooting Guide and Common Failure Scenarios
Scenario 1: Client Connection Refused (Connection Refused)
Symptoms
Applications report Connection Refused or SocketTimeoutException when attempting to reach mongodb://dbhost:27017.
Possible Causes
- The
mongodservice process is crashed or stopped. - Network interface configuration in
/etc/mongod.confbinds only to127.0.0.1(localhost), blocking external connections. - Firewall (e.g.,
ufw,iptables, Cloud Security Groups) is blocking TCP port27017.
Investigation and Diagnosis
- Check Service Status:
systemctl status mongod
- Interpretation: If
active (dead)orfailed, inspect service startup logs viajournalctl -u mongod -e.
- Verify Socket Bindings:
ss -tlnp | grep 27017
- Interpretation: If output displays
127.0.0.1:27017,mongodis listening locally only.
- Validate Active Configuration File:
Inspect
/etc/mongod.conffor thenet.bindIpparameter.
Resolution
Edit /etc/mongod.conf to bind to host internal interface or 0.0.0.0:
net:
port: 27017
bindIp: 127.0.0.1,10.0.1.11
Restart service: sudo systemctl restart mongod. Validate firewall status: sudo ufw allow 27017/tcp.
Verification
Execute connectivity check from remote client server:
mongosh "mongodb://10.0.1.11:27017" --eval "db.ping()"
Scenario 2: Node stuck in NotWritablePrimary / Write Failures
Symptoms
Application write operations fail with errors stating not primary or NotWritablePrimary.
Possible Causes
- Application connection string points to a single hostname that was demoted to
SECONDARYduring failover. - The replica set lost quorum and currently has no elected
PRIMARY.
Investigation and Diagnosis
- Run
db.hello()orrs.status()from shell:
db.hello().isWritablePrimary
- Interpretation: If returns
false, current node cannot accept write requests.
- Inspect entire cluster status:
rs.status()
- Look for member state strings. Determine if a
PRIMARYexists anywhere in the cluster.
Resolution
Update application database connection string URI to include all replica set node addresses and specify the replicaSet parameter:
mongodb://10.0.1.11:27017,10.0.1.12:27017,10.0.1.13:27017/appDB?replicaSet=rs0
- Explanation: Including the complete seed list allows MongoDB client drivers to automatically discover topology changes and redirect write calls to the newly elected
PRIMARYnode during failover.
Scenario 3: High Replication Lag & Growing Oplog Delay
Symptoms
Secondary instances fall behind the Primary. rs.printSecondaryReplicationInfo() displays increasing sync delays (minutes or hours). Secondary risk entering RECOVERING state.
Replication Lag Diagnostics
PRIMARY (Node 1) SECONDARY (Node 2)
┌─────────────────┐ ┌─────────────────┐
│ Oplog Write Rate│ │ Slow Disk / CPU │
│ (10,000 ops/sec)│ │ (2,000 ops/sec) │
└────────┬────────┘ └────────▲────────┘
│ │
└─────────── Replication Lag ────────┘
(Oplog Window Depleting!)
Possible Causes
- Secondary node disk I/O performance is slower than Primary node disk performance.
- Excessive read queries routed to Secondary nodes competing for I/O resources with replication processes.
- Unindexed or long-running write operations (e.g., bulk updates) overwhelming replication throughput.
- Primary
oplogsize (local.oplog.rs) is too small to buffer delays, threatening full resync requirements.
Investigation and Diagnosis
- Inspect Secondary Replication status on
PRIMARY:
rs.printSecondaryReplicationInfo()
- Identify which specific node is lagging and check duration in seconds.
- Inspect Oplog Window metrics on
PRIMARY:
rs.printReplicationInfo()
- Output reveals log size and time duration retained in memory/disk before rolling over.
- Monitor Secondary Disk Performance via Linux CLI:
iostat -xz 1 10
- Look for high
%util(>90%) or elevated%awaitlatency on secondary storage devices.
Resolution
- Temporarily redirect secondary read preferences off the lagging secondary node.
- Resize the MongoDB Oplog dynamically without downtime:
// Increase oplog size to 50 GB dynamically
db.adminCommand({ replSetResizeOplog: 1, size: 51200 })
- Upgrade disk IOPS capacity (e.g., move from standard cloud disk volumes to provisioned IOPS SSDs) on secondary nodes.
Scenario 4: Storage Space Exhaustion / Disk Full Engine Lockup
Symptoms
mongod service terminates unexpectedly or refuses write operations. System logs contain messages like WiredTiger error (-31804): memory/disk allocation failure or disk write failures.
Possible Causes
- Host root file system or dedicated
/var/lib/mongodbblock device ran out of space (0 bytes available).
Investigation and Diagnosis
- Check Linux storage allocations:
df -h /var/lib/mongodb
- Identify largest data collections from
mongosh:
db.stats()
db.collection.stats()
Resolution
- Do NOT run crude file deletions inside
/var/lib/mongodbdirectory, as this corrupts database integrity! - Extend underlying storage volume at the cloud/LVM infrastructure layer, or attach temporary emergency disk space.
- Once
mongodis online, free up collection data or invoke TTL index strategies. - To reclaim unused disk space allocated to WiredTiger data files, execute collection compaction:
db.runCommand({ compact: "largeCollectionName" })
11. Security and Best Practices
Production Security Checklist
┌────────────────────────────────────────────────────────────────────────┐
│ MongoDB Security Architecture │
├────────────────────────────────────────────────────────────────────────┤
│ [Client] ──(TLS/SSL Port 27017)──► [Authentication: SCRAM-SHA-256] │
│ │ │
│ ▼ │
│ [Database Engine] ◄──(Internal Keyfile)──► [Role-Based Access Control] │
└────────────────────────────────────────────────────────────────────────┘
- Role-Based Access Control (RBAC): Always enable
security.authorization: enabled. Never grantrootprivileges to standard application connect strings. - Bind Address Restrictions: Never bind
net.bindIpto0.0.0.0unless strict firewall rules (Security Groups,iptables) restrict access solely to trusted application servers. - Internal Cluster Keyfiles: Authenticate inter-node communication within replica sets using TLS certificates or shared keyfiles (
security.keyFile). - TLS/SSL Network Encryption: Encrypt all network traffic between drivers, applications, and MongoDB nodes by configuring TLS settings:
net:
tls:
mode: requireTLS
certificateKeyFile: /etc/ssl/mongodb.pem
CAFile: /etc/ssl/ca.pem
- Least Privilege System Users: Ensure
mongodprocess runs under dedicated, unprivileged system usermongodbowning/var/lib/mongodband/var/log/mongodbdirectories with strict permissions (0700/0600).
12. Performance and Resource Optimisation
1. Index Strategy and Index Optimization
Unindexed queries trigger full collection scans (COLLSCAN), causing massive CPU utilization and memory page faulting.
- Creating Compound Indexes: Order matters! Follow the ESR Rule (Equality, Sort, Range):
- Equality fields come first.
- Sort fields come second.
- Range query fields come last.
// Optimized index for query: db.users.find({ status: "active", age: { $gt: 21 } }).sort({ lastName: 1 })
db.users.createIndex({ status: 1, lastName: 1, age: 1 })
2. Tuning WiredTiger Cache Allocation
By default, WiredTiger allocates 50% of (RAM - 1 GB) or 256 MB, whichever is higher, for its cache.
- If running on a dedicated host, this default is generally safe.
- If co-locating services on a shared server, cap cache allocation in
/etc/mongod.confto prevent Out-Of-Memory (OOM) killer terminations:
storage:
wiredTiger:
engineConfig:
cacheSizeGB: 8 # Fixed cap allocation
3. Linux Kernel & OS Level Tuning Parameters
Add system configuration overrides in /etc/sysctl.d/99-mongodb.conf:
# Disable Memory Overcommit risks
vm.overcommit_memory = 1
# Reduce aggressive swapping tendency
vm.swappiness = 1
Disable Transparent Huge Pages (THP), which degrades database workloads by creating memory latency spikes:
# Verify status (if [always] is selected, THP is active)
cat /sys/kernel/mm/transparent_hugepage/enabled
# Disable dynamically
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag
13. Backup, Recovery and Maintenance
A secondary member of a Replica Set is NOT a backup solution! An accidental db.dropDatabase() command executed on the Primary replicates instantaneously to all Secondary nodes.
┌─────────────────────────────────────────────────────────────────────────┐
│ MongoDB Backup Method Matrix │
├───────────────────┬───────────────────────────┬─────────────────────────┤
│ Backup Strategy │ Pros │ Cons │
├───────────────────┼───────────────────────────┼─────────────────────────┤
│ Logical │ Flexible, portable, │ Slow for datasets >1TB, │
│ (mongodump) │ selective collections │ CPU intensive │
├───────────────────┼───────────────────────────┼─────────────────────────┤
│ Filesystem │ Fast restore, minimal CPU │ Must lock writes or use │
│ Snapshot (LVM/EBS)│ footprint │ storage-level snapshots │
└───────────────────┴───────────────────────────┴─────────────────────────┘
1. Logical Backups via mongodump & mongorestore
mongodump exports database data into BSON snapshot files.
# Execute point-in-time point backup incorporating Oplog entries
mongodump --uri="mongodb://siteAdmin:Pass123@10.0.1.11:27017/appDB?authSource=admin" \
--oplog \
--out=/var/backups/mongodb/$(date +%Y%m%m_%H%M%S)
# Restore Logical Backup with Oplog Application
mongorestore --uri="mongodb://siteAdmin:Pass123@10.0.1.11:27017" \
--oplogReplay \
/var/backups/mongodb/20261103_101530/
2. Storage Level Snapshots (Physical Backup)
To ensure physical file consistency without stopping mongod:
- Connect to a SECONDARY node shell.
- Pause writes to disk and flush memory pages:
db.fsyncLock()
- Create LVM snapshot or AWS EBS cloud volume snapshot.
- Release the write lock in
mongosh:
db.fsyncUnlock()
14. Automation and Integration
Managing MongoDB via Ansible
---
- name: Deploy MongoDB Server Configuration
hosts: db_servers
becom: true
tasks:
- name: Install MongoDB Package
ansible.builtin.apt:
name: mongodb-org
state: present
update_cache: yes
- name: Deploy Custom Production mongod.conf
ansible.builtin.template:
src: templates/mongod.conf.j2
dest: /etc/mongod.conf
owner: root
group: root
mode: '0644'
notify: Restart MongoDB
handlers:
- name: Restart MongoDB
ansible.builtin.systemd:
name: mongod
state: restarted
Docker Deployment Example
docker run -d \
--name mongodb-prod \
-p 27017:27017 \
-v /var/data/mongo:/data/db \
-e MONGO_INITDB_ROOT_USERNAME=admin \
-e MONGO_INITDB_ROOT_PASSWORD=SuperSecretPassword \
mongo:7.0
15. Practical Labs and Exercises
Exercise 1: Identifying Missing Indexes and Profiling Slow Queries
- Objective: Enable query profiler, generate an unindexed slow query, inspect the performance bottleneck, and construct a qualifying compound index.
- Setup Environment: Connect to local test instance via
mongosh.
Instructions:
- Populate collection with dummy data:
use lab_db
for (let i = 0; i < 50000; i++) {
db.orders.insertOne({ orderId: i, status: i % 2 === 0 ? "completed" : "pending", amount: i * 2.5 })
}
- Set profiler to record queries over 20ms:
db.setProfilingLevel(1, { slowms: 20 })
- Execute an unindexed query:
db.orders.find({ status: "completed", amount: { $gt: 100000 } }).sort({ orderId: -1 })
- Run
explain()on the query and observe execution stats:
db.orders.explain("executionStats").find({ status: "completed", amount: { $gt: 100000 } }).sort({ orderId: -1 })
- Validation: Look for
"planSummary": "COLLSCAN"and hightotalDocsExaminedcounts. - Remediation: Create index following ESR rule:
db.orders.createIndex({ status: 1, orderId: -1, amount: 1 })
- Re-run
explain()and verifyplanSummaryupdates to"IXSCAN".
Exercise 2: Simulated Troubleshooting — Primary Demotion and Node Outage
- Objective: Simulate a Primary instance failure, observe automated election dynamics, and rejoin the failed node cleanly.
- Instructions:
- Connect to active 3-node Replica set. Identify current Primary:
db.hello().me. - Simulate host failure by stopping
mongodon the Primary node:sudo systemctl stop mongod. - Connect
mongoshto one of the remaining secondary nodes. - Execute
rs.status()repeatedly to observe failover election process. - Expected Outcome: One secondary node transitions state to
PRIMARYwithin seconds. - Restart the stopped node:
sudo systemctl start mongod. - Verify via
rs.status()that the restarted node seamlessly rejoins the cluster in theSECONDARYstate.
16. Quick Reference and Cheat Sheet
Important Configuration Files
| Path / File | Description |
|---|---|
/etc/mongod.conf |
Primary YAML system configuration file |
/var/lib/mongodb |
Database file storage directory (dbPath) |
/var/log/mongodb/mongod.log |
Core service operational log file |
Important Log Locations & Flags
| Path / Log Target | Usage |
|---|---|
/var/log/mongodb/mongod.log |
General diagnostic and operational events |
journalctl -u mongod -e |
systemd service startup/shutdown failure logs |
Common Administrative Commands
| Task | Command |
|---|---|
| Start Service | sudo systemctl start mongod |
| Connect Local | mongosh "mongodb://localhost:27017" |
| Health Ping | mongosh --eval "db.adminCommand({ ping: 1 })" |
| Check Replica Set Status | rs.status() |
| Check Replication Lag | rs.printSecondaryReplicationInfo() |
| Query Execution Plan | db.collection.explain("executionStats").find(...) |
Troubleshooting Quick Reference
| Symptom | Initial Investigation Step |
|---|---|
| Connection Refused | Check systemctl status mongod and `ss -tlnp |
| Not Primary Error | Verify driver connection string contains replicaSet name and all seed hosts |
| Excessive Replication Lag | Run rs.printSecondaryReplicationInfo() and verify secondary IOPS |
| Slow Query Times | Run explain("executionStats") and verify indexing (COLLSCAN vs IXSCAN) |
Important Ports and Protocols
| Port | Protocol | Purpose / Scope |
|---|---|---|
27017 |
TCP | Standard default listener for mongod and mongos client/cluster instances |
27018 |
TCP | Default listener port when launching in shard controller mode |
17. Further Reading and References
- Official MongoDB Documentation: https://www.mongodb.com/docs
- MongoDB Replication Architecture Guide: https://www.mongodb.com/docs/manual/replication/
- MongoDB Production Security Best Practices: https://www.mongodb.com/docs/manual/security/
- MongoDB Backup and Recovery Strategies: https://www.mongodb.com/docs/manual/core/backups/
- MongoDB Indexing & Performance Optimization Manual: https://www.mongodb.com/docs/manual/applications/indexes/