Domain 10Linux & SREINTERMEDIATE

Enterprise Logging with the ELK Stack: Filebeat, Logstash, and Elasticsearch Operational Handbook

A comprehensive SysAdmin and SRE operational handbook covering architecture, configuration, administration, diagnostics, and troubleshooting for Elasticsearch, Logstash, and Filebeat.

Domain: Domain 10: Linux Internals, SRE & Troubleshooting•Updated: 2026-11-03
#linux#sysadmin#elk-stack#elasticsearch#logstash#filebeat

1. Overview

In modern production environments, distributed applications write log events to files or standard output across numerous application servers. Centralized logging is essential for site reliability engineers (SREs) and system administrators to collect these distributed log events, normalize timestamp formats and structured fields using Grok patterns, and store them in a search engine for real-time querying during incident response.

The classic Elastic logging pipeline consists of three core components:

  • Filebeat: A lightweight, low-resource agent installed directly on edge application hosts that tails log files and ships raw events.
  • Logstash: A central processing pipeline engine that ingests log streams, applies filters to parse, enrich, and transform log data, and outputs structured JSON documents.
  • Elasticsearch: A distributed RESTful search and analytics engine that indexes, stores, and enables rapid full-text searching over log events.

Pipeline Architecture Flow

The standard ingestion flow operates as follows:

$$\text{Application Logs} \longrightarrow \text{Filebeat} \longrightarrow \text{Logstash} \longrightarrow \text{Elasticsearch}$$

+------------------+      +-------------------+      +-------------------+      +-------------------+
|  App Server      |      |  Log Forwarder    |      | Central Pipeline  |      |  Search Engine    |
| /var/log/*.log   | ---> | Filebeat Agent    | ---> | Logstash (:5044)  | ---> | Elasticsearch     |
| (Raw Log Lines)  |      | (State Registry)  |      | (Grok Parsing)    |      | (JSON Indices)    |
+------------------+      +-------------------+      +-------------------+      +-------------------+

Architectural Decisions: direct vs. Tiered Ingestion

While Filebeat can be configured to forward events directly to Elasticsearch for simpler setups, introducing Logstash provides essential capabilities for heavy parsing, field transformation, and metadata enrichment. Conversely, configuring applications to send logs directly to Logstash over the network introduces operational risks: if Logstash crashes, undergoes maintenance, or experiences an internal memory buffer overflow, application threads can block or drop log entries entirely.

Filebeat mitigates this by running locally on the edge host, reading directly from disk, and using a local registry file to maintain backpressure without interrupting application delivery.

Functional Alternatives

Component Alternative Key Differences
Filebeat Fluentbit / Vector Vector and Fluentbit offer lower memory footprints in C/Rust; Filebeat integrates natively with Elastic templates and modules.
Logstash Fluentd / Vector Logstash relies on Java JVM and plugin ecosystem; alternatives offer lighter resource usage at the cost of Grok ecosystem flexibility.
Elasticsearch OpenSearch / Loki OpenSearch is a fork of Elasticsearch 7.x; Grafana Loki indexes metadata/labels instead of full-text document fields.

2. Architecture and Mental Model

To effectively diagnose system failures, an administrator must understand the internal operational mechanics and event lifecycle across all three layers.

graph TD
    subgraph Edge Host ["Edge Application Host"]
        App["Application / Web Server"] -->|Appends logs| LogFile["/var/log/nginx/access.log"]
        FB["Filebeat Daemon"] -->|Harvester tails| LogFile
        FB -->|Persists byte offset| Registry["/var/lib/filebeat/registry"]
    end

    subgraph Logstash Node ["Logstash Server"]
        BeatsInput["Input Plugin: Beats (TCP :5044)"]
        FB -->|Forward raw events| BeatsInput
        BeatsInput --> FilterStage["Filter Stage: Grok & Mutate"]
        FilterStage --> ESOutput["Output Plugin: Elasticsearch"]
    end

    subgraph ES Cluster ["Elasticsearch Cluster"]
        ESOutput -->|HTTP REST API (Port 9200)| ES_Node["Elasticsearch Node"]
        ES_Node --> PrimaryShard["Primary Shard (logs-YYYY.MM.dd)"]
        PrimaryShard --> ReplicaShard["Replica Shard"]
    end

Component Breakdown

  1. Filebeat Harvesters & Registry: Filebeat spawns a harvester thread for each matched file specifier under filebeat.inputs. Harvesters open file descriptors, tail new content, and push events to the memory spooler. The file pointer offset is written atomically to /var/lib/filebeat/registry, ensuring at-least-once delivery across agent restarts.
  2. Logstash Pipeline Execution Model: Incoming TCP streams on port 5044 enter an in-memory queue (or persistent disk queue). Worker threads process batch blocks through three discrete stages:
  • Input: Ingests incoming Beats events.
  • Filter: Executes pattern matching (Grok), type mutation, and timestamp normalization.
  • Output: Formats JSON payloads and issues HTTP POST bulk index requests to Elasticsearch.
  1. Elasticsearch Indexing Engine: Elasticsearch receives JSON payloads over HTTP (port 9200). Incoming documents are routed to a primary shard based on a hash of the document ID. Primary shards append data to Lucene segment files and asynchronously replicate data to designated replica shards across the cluster.

3. Core Concepts and Terminology

Understanding the core concepts used across the stack is vital for maintaining production logging infrastructure:

  • Document: A self-contained JSON object stored in Elasticsearch containing indexed key-value fields.

  • Index: A logical partition of documents in Elasticsearch analogous to a database table, typically organized by date (e.g., logs-2026.11.03).

  • Shard: A single Apache Lucene instance that acts as a worker unit for an index.

  • Primary Shard: Serves indexing operations.

  • Replica Shard: Provides search throughput and fault tolerance.

  • Cluster Health: Overall status of Elasticsearch indices:

  • Green: All primary and replica shards are successfully allocated.

  • Yellow: All primary shards are allocated, but one or more replica shards remain unassigned (standard for single-node setups).

  • Red: One or more primary shards are missing or unassigned, indicating active data loss or unavailability.

  • Grok Pattern: A regular expression macro used in Logstash to parse un-structured text log strings into structured named variables.

  • Registry File: A persistent file stored at /var/lib/filebeat/registry used by Filebeat to track inode IDs and byte offsets for tailed files.

  • Index Lifecycle Management (ILM): Automated Elasticsearch rules for rolling over, shrinking, archiving, or deleting daily log indices based on age or size constraints.


4. Installation and Version Management

Prerequisites & Kernel Configuration

Elasticsearch relies heavily on memory mapping and open file handles. Adjust Linux kernel sysctl bounds prior to deployment:

# Increase virtual memory limits for Lucene mmap
sudo sysctl -w vm.max_map_count=262144
echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf

# Verify limits
sysctl vm.max_map_count

Installation via APT Repository (Debian/Ubuntu)

# 1. Import Elastic GPG Key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

# 2. Add repository source
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

# 3. Update repositories and install components
sudo apt-get update
sudo apt-get install -y elasticsearch logstash filebeat

Version Inspection Commands

Verify installed software releases across hosts:

# Check Filebeat version
filebeat version

# Check Logstash version
/usr/share/logstash/bin/logstash --version

# Query Elasticsearch version endpoint
curl -s localhost:9200 | grep "number"

5. Configuration Files and Directory Structure

File/Directory Purpose Important Details
/etc/elasticsearch/elasticsearch.yml Core cluster & node settings Defines cluster.name, network.host, ports, storage paths, and xpack security toggles.
/etc/elasticsearch/jvm.options JVM Heap size allocation Sets memory allocation flags (-Xms2g -Xmx2g).
/var/lib/elasticsearch/ Persistent data directory Holds Lucene index segments, shard blocks, and cluster metadata.
/var/log/elasticsearch/ Elasticsearch log directory Contains cluster event logs and garbage collection logs.
/etc/logstash/conf.d/ Pipeline stage configurations Stores user pipeline configuration files (e.g., 01-inputs.conf, 02-filters.conf, 03-outputs.conf).
/etc/logstash/jvm.options Logstash JVM Heap setting Configures execution heap size to avoid JVM OutOfMemory errors.
/var/log/logstash/logstash-plain.log Logstash system execution log Inspect for pipeline syntax crashes or plugin exceptions.
/etc/filebeat/filebeat.yml Primary Filebeat setup Defines log inputs, harvesters, module imports, and target output end points.
/etc/filebeat/modules.d/ Pre-packaged module configs Directory containing ready-made service parsing configurations (Nginx, Apache, System logs).
/var/lib/filebeat/registry/ Harvester read offset store Maintains state data tracking current file positions.

Configuration Validation

Always test configurations prior to restarting live services:

# Validate Filebeat YML syntax
filebeat test config

# Validate Filebeat network reachability to output targets
filebeat test output

# Validate Logstash pipeline config syntax
/usr/share/logstash/bin/logstash -t

6. Service Management and Common Operations

Systemctl Management Commands

Manage service states using standard systemd primitives:

# Enable services at boot
sudo systemctl enable elasticsearch logstash filebeat

# Start services
sudo systemctl start elasticsearch logstash filebeat

# Check runtime statuses
systemctl status elasticsearch logstash filebeat

Port and Socket Verification

Verify network socket bindings using ss:

# Check if Elasticsearch (9200) and Logstash Beats (5044) sockets are listening
ss -tlnp | grep -E '9200|5044'

Service Operation Details

  • Reload vs. Restart: Reloading (systemctl reload logstash or setting --config.reload.automatic in Logstash) updates active configuration definitions dynamically without dropping established client TCP connections. Restarting (systemctl restart) flushes memory queues, terminates the process, and reinstantiates the runtime environment, causing a temporary disconnection for edge shippers.

7. Common Administrative Tasks and Workflows

Task 1: Enabling and Using Filebeat Modules

Filebeat includes pre-built modules with optimal parsing rules for common server applications:

  1. Enable the Module:
filebeat modules enable nginx
filebeat modules list
  1. Load Index Templates: If writing directly to Elasticsearch, load template definitions:
filebeat setup

Task 2: Building an End-to-End Log Pipeline

Step 1: Configure Filebeat Shipper (/etc/filebeat/filebeat.yml)

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/nginx/access.log
      - /var/log/nginx/error.log

output.logstash:
  hosts: ["10.0.0.50:5044"]

Step 2: Configure Logstash Pipeline (/etc/logstash/conf.d/nginx-pipeline.conf)

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "nginx-access-%{+YYYY.MM.dd}"
  }
  # Enable stdout during debugging
  # stdout { codec => rubydebug }
}

Step 3: Validate and Restart Services

filebeat test config
/usr/share/logstash/bin/logstash -t
sudo systemctl restart logstash filebeat

Step 4: Verify Elasticsearch Document Ingestion

curl -s "localhost:9200/nginx-access-*/_search?pretty&size=1"

8. CLI Command Reference and Cheatsheet

Core Command Operations Matrix

Category Command Purpose Important Flags / Arguments Example
Service systemctl Manage host services status, start, restart systemctl status elasticsearch
Network ss Inspect listening ports -t (TCP), -l (listen), -n (numeric), -p (process) `ss -tlnp
Filebeat filebeat Agent operational inspection test config, test output, modules filebeat test output
Logstash logstash Pipeline execution and verification -t (test syntax), -e (inline pipeline string) /usr/share/logstash/bin/logstash -t
ES Health curl Interrogate cluster status API _cluster/health, pretty curl -s localhost:9200/_cluster/health?pretty
ES Indices curl Inspect shard & index allocation _cat/indices?v, _cat/shards?v curl -s localhost:9200/_cat/indices?v
ES Search curl Search documents via REST _search, size, q curl -s 'localhost:9200/logs-*/_search?size=5'
ES Admin curl Remove obsolete index data -X DELETE curl -X DELETE "localhost:9200/logs-2024.05.*"

Diagnostic & Inspection Command Examples

# Check why shards are unassigned or stuck
curl -s localhost:9200/_cluster/allocation/explain?pretty

# Quick inline Logstash pipeline test using STDIN and STDOUT
/usr/share/logstash/bin/logstash -e 'input { stdin {} } output { stdout { codec => rubydebug } }'

# Count indexed events across date-stamped indices
curl -s 'localhost:9200/logs-*/_count?pretty'

# Perform explicit Lucene segment force-merge (off-peak maintenance)
curl -X POST "localhost:9200/nginx-access-2026.11.03/_forcemerge?max_num_segments=1"

9. Logs, Monitoring and Diagnostics

Log File Locations

  • Filebeat System Journal:
journalctl -u filebeat -n 50 --no-pager
  • Logstash System Execution Log:
tail -f /var/log/logstash/logstash-plain.log
journalctl -u logstash -e
  • Elasticsearch Engine Log:
tail -f /var/log/elasticsearch/elasticsearch.log

Live Debugging Techniques

When pipelines ingest data without populating Elasticsearch indices:

  1. Temporarily edit /etc/logstash/conf.d/output.conf to comment out the elasticsearch output block.
  2. Enable direct stdout logging using the rubydebug codec:
output {
  stdout { codec => rubydebug }
}
  1. Restart Logstash and monitor output via journalctl -u logstash -f. Inspect raw parsed JSON keys directly.

10. Troubleshooting Guide and Common Failure Scenarios

Problem 1: Filebeat Not Shipping Logs to Logstash

Symptoms

  • Elasticsearch indices are not updating.
  • No network activity seen on Logstash port 5044.

Possible Causes

  • Missing file read permissions for the filebeat service user.
  • Filebeat pointing to invalid log paths in filebeat.yml.
  • Host firewall blocking TCP port 5044.
  • Registry offset corruption.

Investigation and Diagnosis

  1. Validate Filebeat configuration syntax:
filebeat test config
  1. Test network layer connection to Logstash:
filebeat test output
nc -zv logstash-host 5044
  1. Verify Logstash service is bound to port 5044:
ss -tlnp | grep 5044

Root Cause Identification

If nc fails, host firewalls (e.g., ufw, iptables) are dropping packets. If filebeat test output reports Permission denied opening /var/log/nginx/access.log, Filebeat lacks permission to access target files.

Resolution

  • Fix File Permissions:
chmod 644 /var/log/nginx/access.log
  • Open Firewall Port:
sudo ufw allow 5044/tcp
  • Reset Corrupted Registry State:
systemctl stop filebeat
rm -rf /var/lib/filebeat/registry
systemctl start filebeat

Verification

Execute filebeat test output and confirm standard return Connection to backends OK.

Prevention

Ensure configuration management (Ansible/Puppet) enforces log permissions and deploys firewall rules concurrently with service installation.


Problem 2: Logstash Pipeline Fails to Start or Out-Of-Memory (OOM)

Symptoms

  • Logstash service crashes immediately after launch.
  • journalctl -u logstash shows java.lang.OutOfMemoryError: Java heap space.

Possible Causes

  • Syntax error inside /etc/logstash/conf.d/ pipeline files.
  • System heap limit in /etc/logstash/jvm.options exceeds host memory or is set too small for active event volume.

Investigation and Diagnosis

  1. Execute explicit syntax checks:
/usr/share/logstash/bin/logstash -t
  1. Review system logs for OOM invocation:
journalctl -u logstash -e | grep -i "OutOfMemory"

Root Cause Identification

Syntax failures highlight exact lines in .conf files. OOM errors indicate that JVM options must be adjusted.

Resolution

Adjust JVM heap boundaries inside /etc/logstash/jvm.options:

-Xms1g
-Xmx1g

Verification

Restart Logstash and ensure service remains active:

sudo systemctl restart logstash && systemctl status logstash

Prevention

Always run config validation before starting production deployments, and allocate adequate memory to JVM options based on workload expectations.


Problem 3: Elasticsearch Cluster Status is Red or Yellow

Symptoms

  • Cluster state returns yellow or red via REST queries.
  • Red status indicates missing primary shards; Yellow indicates unassigned replica shards.

Possible Causes

  • Yellow: Single-node cluster running default configuration expecting at least 2 nodes to place replica shards.
  • Red: Disk space full, corrupt index segment files, or offline node holding primary shards.

Investigation and Diagnosis

  1. Check overall cluster health:
curl -s localhost:9200/_cluster/health?pretty
  1. Identify unassigned shards:
curl -s localhost:9200/_cat/shards?v | grep UNASSIGNED
  1. Request detailed allocation diagnostic report:
curl -s localhost:9200/_cluster/allocation/explain?pretty

Resolution

  • For Single-Node Local Labs (Fixing Yellow State): Set replica counts to 0 for matching indices:
curl -X PUT "localhost:9200/nginx-access-*/_settings" -H 'Content-Type: application/json' -d'
{
  "index": {
    "number_of_replicas": 0
  }
}'
  • For Missing Primary Shards (Fixing Red State): Restore missing primary shards from cluster backups or clear damaged indices if data is recoverable elsewhere.

Verification

Ensure cluster health query returns status green.

Prevention

Configure single-node dev environments to default to number_of_replicas: 0 in index templates.


Problem 4: Elasticsearch Read-Only Lockout (Disk Flood Watermark)

Symptoms

  • Ingestion fails with HTTP error 403 Forbidden or log entries mentioning cluster_block_exception.
  • Index write operations are blocked.

Possible Causes

  • Host disk usage exceeded Elasticsearch’s flood-stage watermark threshold (default 95%).

Investigation and Diagnosis

  1. Inspect physical disk usage:
df -h /var/lib/elasticsearch
  1. Inspect Elasticsearch cluster allocation metrics:
curl -s localhost:9200/_cat/allocation?v

Root Cause Identification

Disk usage above 95% triggers an automatic, cluster-wide index read-only block.

Resolution

  1. Delete obsolete indices to free disk space:
curl -X DELETE "localhost:9200/logs-2024.05.*"
  1. Remove the global read-only index block:
curl -X PUT "localhost:9200/_settings" -H 'Content-Type: application/json' -d'
{
  "index.blocks.read_only_allow_delete": null
}'

Verification

Verify write operations resume by outputting a sample test document.

Prevention

Implement Index Lifecycle Management (ILM) policies to delete or archive old indices before disk usage reaches capacity limits.


Problem 5: Log Events Tagged with _grokparsefailure

Symptoms

  • Incoming raw log lines are ingested into Elasticsearch, but custom extracted fields (e.g., clientip, response_code) are missing.
  • Events contain a tags: ["_grokparsefailure"] array.

Possible Causes

  • Application log format changed (e.g., Nginx access log format updated), causing Grok regular expression patterns to fail.

Investigation and Diagnosis

  1. Query Elasticsearch for failed parse records:
curl -s "localhost:9200/nginx-access-*/_search?q=tags:_grokparsefailure&pretty"
  1. Isolate raw message content strings for manual evaluation.

Resolution

Update the Logstash Grok pattern definition in /etc/logstash/conf.d/filters.conf to match the new incoming message structure, or switch to structured JSON logging.

filter {
  # If application emits native JSON:
  json {
    source => "message"
  }
}

Verification

Ingest a sample log line and confirm the _grokparsefailure tag is no longer applied.

Prevention

Prefer structured log emitters (like native JSON formatters) over free-form string text loggers wherever possible.


11. Security and Best Practices

Transport Layer Security (TLS) & Authentication Basics

In production environments, always secure network traffic across all pipeline nodes:

  1. Elasticsearch Configuration (/etc/elasticsearch/elasticsearch.yml):
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
  1. Filebeat Authenticated Output (/etc/filebeat/filebeat.yml):
output.elasticsearch:
  hosts: ["https://es-node01:9200"]
  username: "filebeat_writer"
  password: "${FILEBEAT_PASSWORD}"
  ssl.certificate_authorities: ["/etc/filebeat/certs/ca.crt"]
  1. Filebeat File Permissions: Filebeat will refuse to run if its primary configuration file is writable by non-root users:
sudo chown root:root /etc/filebeat/filebeat.yml
sudo chmod 0600 /etc/filebeat/filebeat.yml

12. Performance and Resource Optimisation

  • JVM Heap Sizing: Allocate 50% of available system RAM to the Elasticsearch JVM heap, up to a maximum threshold of 32 GB (to preserve compressed object pointers).
  • Filebeat Bulk Max Size: Adjust batching parameters in filebeat.yml to reduce network round-trip overhead:
output.logstash:
  hosts: ["10.0.0.50:5044"]
  bulk_max_size: 2048
  • Logstash Persistent Queues: Safeguard against data loss during ingestion spikes by enabling disk-backed persistent queues in /etc/logstash/logstash.yml:
queue.type: persisted

13. Backup, Recovery and Maintenance

Manual Index Purging

If automated Index Lifecycle Management (ILM) is not configured, implement routine maintenance using curl invocations:

# Preview indices targeted for deletion
curl -s "localhost:9200/_cat/indices/logs-2024.05.*?v"

# Remove matching indices
curl -X DELETE "localhost:9200/logs-2024.05.*"

Segment Force-Merging

To reclaim disk space and speed up queries on historical read-only indices, perform segment force-merging during off-peak windows:

curl -X POST "localhost:9200/logs-2026.10.01/_forcemerge?max_num_segments=1"

14. Automation and Integration

Automated Index Cleanup via Systemd Timers

Create a bash maintenance script to purge log indices older than 30 days:

/usr/local/bin/es-index-purge.sh:

#!/usr/bin/env bash
set -euo pipefail

RETENTION_DATE=$(date -d "30 days ago" "+%Y.%m.%d")
curl -s -X DELETE "http://localhost:9200/nginx-access-${RETENTION_DATE}"

Configure execution via systemd timer unit /etc/systemd/system/es-purge.timer:

[Unit]
Description=Daily Elasticsearch Index Cleanup Timer

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

15. Practical Labs and Exercises

Lab 1 (Beginner): Filebeat Installation and Output Verification

  • Objective: Install Filebeat, configure log tailing for system authorization logs (/var/log/auth.log), and test output functionality.
  • Steps:
  1. Install Filebeat via package manager.
  2. Edit /etc/filebeat/filebeat.yml to enable input for /var/log/auth.log.
  3. Validate setup: filebeat test config.
  • Validation: Run filebeat test output to confirm target endpoint reachability.
  • Objective: Build a Logstash pipeline that parses Nginx web logs using Grok patterns and verify document creation in Elasticsearch.
  • Steps:
  1. Create /etc/logstash/conf.d/nginx.conf with beats input, grok filter, and elasticsearch output.
  2. Validate pipeline syntax: /usr/share/logstash/bin/logstash -t.
  3. Generate mock log entries using logger or curl against Nginx.
  • Validation: Run curl -s "localhost:9200/nginx-access-*/_count" to confirm event ingestion.

Lab 3 (Troubleshooting): Recovering from Read-Only Disk Watermark Lockout

  • Objective: Simulate a disk watermark lock out, identify root causes, and restore read-write cluster operations.
  • Steps:
  1. Inject index read-only lock:
curl -X PUT "localhost:9200/_settings" -H 'Content-Type: application/json' -d '{"index.blocks.read_only_allow_delete": true}'
  1. Confirm write failures when attempting to index new documents.
  2. Diagnose cause via curl -s localhost:9200/_cat/allocation?v.
  3. Clear the global block setting.
  • Validation: Verify indices accept write operations without raising HTTP 403 exceptions.

16. Quick Reference and Cheat Sheet

Important Configuration Files

Path Description
/etc/elasticsearch/elasticsearch.yml Core Elasticsearch settings
/etc/logstash/conf.d/*.conf Logstash pipeline processing rules
/etc/filebeat/filebeat.yml Filebeat shipper settings

Important Log Locations

Path Component
/var/log/elasticsearch/ Elasticsearch node & query logs
/var/log/logstash/logstash-plain.log Logstash system execution log
journalctl -u filebeat Filebeat daemon log stream

Common Commands

Task Command
Test Filebeat Config filebeat test config
Test Filebeat Connectivity filebeat test output
Test Logstash Syntax /usr/share/logstash/bin/logstash -t
Cluster Health Check curl -s localhost:9200/_cluster/health?pretty
List Indices curl -s localhost:9200/_cat/indices?v

Troubleshooting Quick Reference

Symptom Initial Diagnostic Command
Missing documents in ES journalctl -u logstash -n 50 --no-pager
Beats output failing filebeat test output
Shards unassigned curl -s localhost:9200/_cluster/allocation/explain?pretty
Parsing errors Search ES for tags: "_grokparsefailure"

Important Ports and Protocols

Port Protocol Purpose
5044 TCP Logstash Beats Input listener
9200 HTTP / REST Elasticsearch REST API & client ingestion
9300 TCP Elasticsearch inter-node transport network

17. Further Reading and References

  1. Elastic. Elasticsearch Guide & Reference. Available at: [https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html)
  2. Elastic. Logstash Reference Guide. Available at: [https://www.elastic.co/guide/en/logstash/current/index.html](https://www.elastic.co/guide/en/logstash/current/index.html)
  3. Elastic. Filebeat Overview & Documentation. Available at: [https://www.elastic.co/guide/en/beats/filebeat/current/index.html](https://www.elastic.co/guide/en/beats/filebeat/current/index.html)
  4. Elastic. Grok Filter Plugin & Regular Expressions. Available at: [https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html)