1. Overview
In modern production environments, distributed applications write log events to files or standard output across numerous application servers. Centralized logging is essential for site reliability engineers (SREs) and system administrators to collect these distributed log events, normalize timestamp formats and structured fields using Grok patterns, and store them in a search engine for real-time querying during incident response.
The classic Elastic logging pipeline consists of three core components:
- Filebeat: A lightweight, low-resource agent installed directly on edge application hosts that tails log files and ships raw events.
- Logstash: A central processing pipeline engine that ingests log streams, applies filters to parse, enrich, and transform log data, and outputs structured JSON documents.
- Elasticsearch: A distributed RESTful search and analytics engine that indexes, stores, and enables rapid full-text searching over log events.
Pipeline Architecture Flow
The standard ingestion flow operates as follows:
$$\text{Application Logs} \longrightarrow \text{Filebeat} \longrightarrow \text{Logstash} \longrightarrow \text{Elasticsearch}$$
+------------------+ +-------------------+ +-------------------+ +-------------------+
| App Server | | Log Forwarder | | Central Pipeline | | Search Engine |
| /var/log/*.log | ---> | Filebeat Agent | ---> | Logstash (:5044) | ---> | Elasticsearch |
| (Raw Log Lines) | | (State Registry) | | (Grok Parsing) | | (JSON Indices) |
+------------------+ +-------------------+ +-------------------+ +-------------------+
Architectural Decisions: direct vs. Tiered Ingestion
While Filebeat can be configured to forward events directly to Elasticsearch for simpler setups, introducing Logstash provides essential capabilities for heavy parsing, field transformation, and metadata enrichment. Conversely, configuring applications to send logs directly to Logstash over the network introduces operational risks: if Logstash crashes, undergoes maintenance, or experiences an internal memory buffer overflow, application threads can block or drop log entries entirely.
Filebeat mitigates this by running locally on the edge host, reading directly from disk, and using a local registry file to maintain backpressure without interrupting application delivery.
Functional Alternatives
| Component | Alternative | Key Differences |
|---|---|---|
| Filebeat | Fluentbit / Vector | Vector and Fluentbit offer lower memory footprints in C/Rust; Filebeat integrates natively with Elastic templates and modules. |
| Logstash | Fluentd / Vector | Logstash relies on Java JVM and plugin ecosystem; alternatives offer lighter resource usage at the cost of Grok ecosystem flexibility. |
| Elasticsearch | OpenSearch / Loki | OpenSearch is a fork of Elasticsearch 7.x; Grafana Loki indexes metadata/labels instead of full-text document fields. |
2. Architecture and Mental Model
To effectively diagnose system failures, an administrator must understand the internal operational mechanics and event lifecycle across all three layers.
graph TD
subgraph Edge Host ["Edge Application Host"]
App["Application / Web Server"] -->|Appends logs| LogFile["/var/log/nginx/access.log"]
FB["Filebeat Daemon"] -->|Harvester tails| LogFile
FB -->|Persists byte offset| Registry["/var/lib/filebeat/registry"]
end
subgraph Logstash Node ["Logstash Server"]
BeatsInput["Input Plugin: Beats (TCP :5044)"]
FB -->|Forward raw events| BeatsInput
BeatsInput --> FilterStage["Filter Stage: Grok & Mutate"]
FilterStage --> ESOutput["Output Plugin: Elasticsearch"]
end
subgraph ES Cluster ["Elasticsearch Cluster"]
ESOutput -->|HTTP REST API (Port 9200)| ES_Node["Elasticsearch Node"]
ES_Node --> PrimaryShard["Primary Shard (logs-YYYY.MM.dd)"]
PrimaryShard --> ReplicaShard["Replica Shard"]
end
Component Breakdown
- Filebeat Harvesters & Registry: Filebeat spawns a harvester thread for each matched file specifier under
filebeat.inputs. Harvesters open file descriptors, tail new content, and push events to the memory spooler. The file pointer offset is written atomically to/var/lib/filebeat/registry, ensuring at-least-once delivery across agent restarts. - Logstash Pipeline Execution Model: Incoming TCP streams on port 5044 enter an in-memory queue (or persistent disk queue). Worker threads process batch blocks through three discrete stages:
- Input: Ingests incoming Beats events.
- Filter: Executes pattern matching (Grok), type mutation, and timestamp normalization.
- Output: Formats JSON payloads and issues HTTP POST bulk index requests to Elasticsearch.
- Elasticsearch Indexing Engine: Elasticsearch receives JSON payloads over HTTP (port 9200). Incoming documents are routed to a primary shard based on a hash of the document ID. Primary shards append data to Lucene segment files and asynchronously replicate data to designated replica shards across the cluster.
3. Core Concepts and Terminology
Understanding the core concepts used across the stack is vital for maintaining production logging infrastructure:
-
Document: A self-contained JSON object stored in Elasticsearch containing indexed key-value fields.
-
Index: A logical partition of documents in Elasticsearch analogous to a database table, typically organized by date (e.g.,
logs-2026.11.03). -
Shard: A single Apache Lucene instance that acts as a worker unit for an index.
-
Primary Shard: Serves indexing operations.
-
Replica Shard: Provides search throughput and fault tolerance.
-
Cluster Health: Overall status of Elasticsearch indices:
-
Green: All primary and replica shards are successfully allocated.
-
Yellow: All primary shards are allocated, but one or more replica shards remain unassigned (standard for single-node setups).
-
Red: One or more primary shards are missing or unassigned, indicating active data loss or unavailability.
-
Grok Pattern: A regular expression macro used in Logstash to parse un-structured text log strings into structured named variables.
-
Registry File: A persistent file stored at
/var/lib/filebeat/registryused by Filebeat to track inode IDs and byte offsets for tailed files. -
Index Lifecycle Management (ILM): Automated Elasticsearch rules for rolling over, shrinking, archiving, or deleting daily log indices based on age or size constraints.
4. Installation and Version Management
Prerequisites & Kernel Configuration
Elasticsearch relies heavily on memory mapping and open file handles. Adjust Linux kernel sysctl bounds prior to deployment:
# Increase virtual memory limits for Lucene mmap
sudo sysctl -w vm.max_map_count=262144
echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
# Verify limits
sysctl vm.max_map_count
Installation via APT Repository (Debian/Ubuntu)
# 1. Import Elastic GPG Key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
# 2. Add repository source
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
# 3. Update repositories and install components
sudo apt-get update
sudo apt-get install -y elasticsearch logstash filebeat
Version Inspection Commands
Verify installed software releases across hosts:
# Check Filebeat version
filebeat version
# Check Logstash version
/usr/share/logstash/bin/logstash --version
# Query Elasticsearch version endpoint
curl -s localhost:9200 | grep "number"
5. Configuration Files and Directory Structure
| File/Directory | Purpose | Important Details |
|---|---|---|
/etc/elasticsearch/elasticsearch.yml |
Core cluster & node settings | Defines cluster.name, network.host, ports, storage paths, and xpack security toggles. |
/etc/elasticsearch/jvm.options |
JVM Heap size allocation | Sets memory allocation flags (-Xms2g -Xmx2g). |
/var/lib/elasticsearch/ |
Persistent data directory | Holds Lucene index segments, shard blocks, and cluster metadata. |
/var/log/elasticsearch/ |
Elasticsearch log directory | Contains cluster event logs and garbage collection logs. |
/etc/logstash/conf.d/ |
Pipeline stage configurations | Stores user pipeline configuration files (e.g., 01-inputs.conf, 02-filters.conf, 03-outputs.conf). |
/etc/logstash/jvm.options |
Logstash JVM Heap setting | Configures execution heap size to avoid JVM OutOfMemory errors. |
/var/log/logstash/logstash-plain.log |
Logstash system execution log | Inspect for pipeline syntax crashes or plugin exceptions. |
/etc/filebeat/filebeat.yml |
Primary Filebeat setup | Defines log inputs, harvesters, module imports, and target output end points. |
/etc/filebeat/modules.d/ |
Pre-packaged module configs | Directory containing ready-made service parsing configurations (Nginx, Apache, System logs). |
/var/lib/filebeat/registry/ |
Harvester read offset store | Maintains state data tracking current file positions. |
Configuration Validation
Always test configurations prior to restarting live services:
# Validate Filebeat YML syntax
filebeat test config
# Validate Filebeat network reachability to output targets
filebeat test output
# Validate Logstash pipeline config syntax
/usr/share/logstash/bin/logstash -t
6. Service Management and Common Operations
Systemctl Management Commands
Manage service states using standard systemd primitives:
# Enable services at boot
sudo systemctl enable elasticsearch logstash filebeat
# Start services
sudo systemctl start elasticsearch logstash filebeat
# Check runtime statuses
systemctl status elasticsearch logstash filebeat
Port and Socket Verification
Verify network socket bindings using ss:
# Check if Elasticsearch (9200) and Logstash Beats (5044) sockets are listening
ss -tlnp | grep -E '9200|5044'
Service Operation Details
- Reload vs. Restart: Reloading (
systemctl reload logstashor setting--config.reload.automaticin Logstash) updates active configuration definitions dynamically without dropping established client TCP connections. Restarting (systemctl restart) flushes memory queues, terminates the process, and reinstantiates the runtime environment, causing a temporary disconnection for edge shippers.
7. Common Administrative Tasks and Workflows
Task 1: Enabling and Using Filebeat Modules
Filebeat includes pre-built modules with optimal parsing rules for common server applications:
- Enable the Module:
filebeat modules enable nginx
filebeat modules list
- Load Index Templates: If writing directly to Elasticsearch, load template definitions:
filebeat setup
Task 2: Building an End-to-End Log Pipeline
Step 1: Configure Filebeat Shipper (/etc/filebeat/filebeat.yml)
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
- /var/log/nginx/error.log
output.logstash:
hosts: ["10.0.0.50:5044"]
Step 2: Configure Logstash Pipeline (/etc/logstash/conf.d/nginx-pipeline.conf)
input {
beats {
port => 5044
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "nginx-access-%{+YYYY.MM.dd}"
}
# Enable stdout during debugging
# stdout { codec => rubydebug }
}
Step 3: Validate and Restart Services
filebeat test config
/usr/share/logstash/bin/logstash -t
sudo systemctl restart logstash filebeat
Step 4: Verify Elasticsearch Document Ingestion
curl -s "localhost:9200/nginx-access-*/_search?pretty&size=1"
8. CLI Command Reference and Cheatsheet
Core Command Operations Matrix
| Category | Command | Purpose | Important Flags / Arguments | Example |
|---|---|---|---|---|
| Service | systemctl |
Manage host services | status, start, restart |
systemctl status elasticsearch |
| Network | ss |
Inspect listening ports | -t (TCP), -l (listen), -n (numeric), -p (process) |
`ss -tlnp |
| Filebeat | filebeat |
Agent operational inspection | test config, test output, modules |
filebeat test output |
| Logstash | logstash |
Pipeline execution and verification | -t (test syntax), -e (inline pipeline string) |
/usr/share/logstash/bin/logstash -t |
| ES Health | curl |
Interrogate cluster status API | _cluster/health, pretty |
curl -s localhost:9200/_cluster/health?pretty |
| ES Indices | curl |
Inspect shard & index allocation | _cat/indices?v, _cat/shards?v |
curl -s localhost:9200/_cat/indices?v |
| ES Search | curl |
Search documents via REST | _search, size, q |
curl -s 'localhost:9200/logs-*/_search?size=5' |
| ES Admin | curl |
Remove obsolete index data | -X DELETE |
curl -X DELETE "localhost:9200/logs-2024.05.*" |
Diagnostic & Inspection Command Examples
# Check why shards are unassigned or stuck
curl -s localhost:9200/_cluster/allocation/explain?pretty
# Quick inline Logstash pipeline test using STDIN and STDOUT
/usr/share/logstash/bin/logstash -e 'input { stdin {} } output { stdout { codec => rubydebug } }'
# Count indexed events across date-stamped indices
curl -s 'localhost:9200/logs-*/_count?pretty'
# Perform explicit Lucene segment force-merge (off-peak maintenance)
curl -X POST "localhost:9200/nginx-access-2026.11.03/_forcemerge?max_num_segments=1"
9. Logs, Monitoring and Diagnostics
Log File Locations
- Filebeat System Journal:
journalctl -u filebeat -n 50 --no-pager
- Logstash System Execution Log:
tail -f /var/log/logstash/logstash-plain.log
journalctl -u logstash -e
- Elasticsearch Engine Log:
tail -f /var/log/elasticsearch/elasticsearch.log
Live Debugging Techniques
When pipelines ingest data without populating Elasticsearch indices:
- Temporarily edit
/etc/logstash/conf.d/output.confto comment out theelasticsearchoutput block. - Enable direct stdout logging using the
rubydebugcodec:
output {
stdout { codec => rubydebug }
}
- Restart Logstash and monitor output via
journalctl -u logstash -f. Inspect raw parsed JSON keys directly.
10. Troubleshooting Guide and Common Failure Scenarios
Problem 1: Filebeat Not Shipping Logs to Logstash
Symptoms
- Elasticsearch indices are not updating.
- No network activity seen on Logstash port 5044.
Possible Causes
- Missing file read permissions for the
filebeatservice user. - Filebeat pointing to invalid log paths in
filebeat.yml. - Host firewall blocking TCP port 5044.
- Registry offset corruption.
Investigation and Diagnosis
- Validate Filebeat configuration syntax:
filebeat test config
- Test network layer connection to Logstash:
filebeat test output
nc -zv logstash-host 5044
- Verify Logstash service is bound to port 5044:
ss -tlnp | grep 5044
Root Cause Identification
If nc fails, host firewalls (e.g., ufw, iptables) are dropping packets. If filebeat test output reports Permission denied opening /var/log/nginx/access.log, Filebeat lacks permission to access target files.
Resolution
- Fix File Permissions:
chmod 644 /var/log/nginx/access.log
- Open Firewall Port:
sudo ufw allow 5044/tcp
- Reset Corrupted Registry State:
systemctl stop filebeat
rm -rf /var/lib/filebeat/registry
systemctl start filebeat
Verification
Execute filebeat test output and confirm standard return Connection to backends OK.
Prevention
Ensure configuration management (Ansible/Puppet) enforces log permissions and deploys firewall rules concurrently with service installation.
Problem 2: Logstash Pipeline Fails to Start or Out-Of-Memory (OOM)
Symptoms
- Logstash service crashes immediately after launch.
journalctl -u logstashshowsjava.lang.OutOfMemoryError: Java heap space.
Possible Causes
- Syntax error inside
/etc/logstash/conf.d/pipeline files. - System heap limit in
/etc/logstash/jvm.optionsexceeds host memory or is set too small for active event volume.
Investigation and Diagnosis
- Execute explicit syntax checks:
/usr/share/logstash/bin/logstash -t
- Review system logs for OOM invocation:
journalctl -u logstash -e | grep -i "OutOfMemory"
Root Cause Identification
Syntax failures highlight exact lines in .conf files. OOM errors indicate that JVM options must be adjusted.
Resolution
Adjust JVM heap boundaries inside /etc/logstash/jvm.options:
-Xms1g
-Xmx1g
Verification
Restart Logstash and ensure service remains active:
sudo systemctl restart logstash && systemctl status logstash
Prevention
Always run config validation before starting production deployments, and allocate adequate memory to JVM options based on workload expectations.
Problem 3: Elasticsearch Cluster Status is Red or Yellow
Symptoms
- Cluster state returns
yelloworredvia REST queries. - Red status indicates missing primary shards; Yellow indicates unassigned replica shards.
Possible Causes
- Yellow: Single-node cluster running default configuration expecting at least 2 nodes to place replica shards.
- Red: Disk space full, corrupt index segment files, or offline node holding primary shards.
Investigation and Diagnosis
- Check overall cluster health:
curl -s localhost:9200/_cluster/health?pretty
- Identify unassigned shards:
curl -s localhost:9200/_cat/shards?v | grep UNASSIGNED
- Request detailed allocation diagnostic report:
curl -s localhost:9200/_cluster/allocation/explain?pretty
Resolution
- For Single-Node Local Labs (Fixing Yellow State): Set replica counts to 0 for matching indices:
curl -X PUT "localhost:9200/nginx-access-*/_settings" -H 'Content-Type: application/json' -d'
{
"index": {
"number_of_replicas": 0
}
}'
- For Missing Primary Shards (Fixing Red State): Restore missing primary shards from cluster backups or clear damaged indices if data is recoverable elsewhere.
Verification
Ensure cluster health query returns status green.
Prevention
Configure single-node dev environments to default to number_of_replicas: 0 in index templates.
Problem 4: Elasticsearch Read-Only Lockout (Disk Flood Watermark)
Symptoms
- Ingestion fails with HTTP error
403 Forbiddenor log entries mentioningcluster_block_exception. - Index write operations are blocked.
Possible Causes
- Host disk usage exceeded Elasticsearch’s flood-stage watermark threshold (default 95%).
Investigation and Diagnosis
- Inspect physical disk usage:
df -h /var/lib/elasticsearch
- Inspect Elasticsearch cluster allocation metrics:
curl -s localhost:9200/_cat/allocation?v
Root Cause Identification
Disk usage above 95% triggers an automatic, cluster-wide index read-only block.
Resolution
- Delete obsolete indices to free disk space:
curl -X DELETE "localhost:9200/logs-2024.05.*"
- Remove the global read-only index block:
curl -X PUT "localhost:9200/_settings" -H 'Content-Type: application/json' -d'
{
"index.blocks.read_only_allow_delete": null
}'
Verification
Verify write operations resume by outputting a sample test document.
Prevention
Implement Index Lifecycle Management (ILM) policies to delete or archive old indices before disk usage reaches capacity limits.
Problem 5: Log Events Tagged with _grokparsefailure
Symptoms
- Incoming raw log lines are ingested into Elasticsearch, but custom extracted fields (e.g.,
clientip,response_code) are missing. - Events contain a
tags: ["_grokparsefailure"]array.
Possible Causes
- Application log format changed (e.g., Nginx access log format updated), causing Grok regular expression patterns to fail.
Investigation and Diagnosis
- Query Elasticsearch for failed parse records:
curl -s "localhost:9200/nginx-access-*/_search?q=tags:_grokparsefailure&pretty"
- Isolate raw
messagecontent strings for manual evaluation.
Resolution
Update the Logstash Grok pattern definition in /etc/logstash/conf.d/filters.conf to match the new incoming message structure, or switch to structured JSON logging.
filter {
# If application emits native JSON:
json {
source => "message"
}
}
Verification
Ingest a sample log line and confirm the _grokparsefailure tag is no longer applied.
Prevention
Prefer structured log emitters (like native JSON formatters) over free-form string text loggers wherever possible.
11. Security and Best Practices
Transport Layer Security (TLS) & Authentication Basics
In production environments, always secure network traffic across all pipeline nodes:
- Elasticsearch Configuration (
/etc/elasticsearch/elasticsearch.yml):
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
- Filebeat Authenticated Output (
/etc/filebeat/filebeat.yml):
output.elasticsearch:
hosts: ["https://es-node01:9200"]
username: "filebeat_writer"
password: "${FILEBEAT_PASSWORD}"
ssl.certificate_authorities: ["/etc/filebeat/certs/ca.crt"]
- Filebeat File Permissions: Filebeat will refuse to run if its primary configuration file is writable by non-root users:
sudo chown root:root /etc/filebeat/filebeat.yml
sudo chmod 0600 /etc/filebeat/filebeat.yml
12. Performance and Resource Optimisation
- JVM Heap Sizing: Allocate 50% of available system RAM to the Elasticsearch JVM heap, up to a maximum threshold of 32 GB (to preserve compressed object pointers).
- Filebeat Bulk Max Size: Adjust batching parameters in
filebeat.ymlto reduce network round-trip overhead:
output.logstash:
hosts: ["10.0.0.50:5044"]
bulk_max_size: 2048
- Logstash Persistent Queues: Safeguard against data loss during ingestion spikes by enabling disk-backed persistent queues in
/etc/logstash/logstash.yml:
queue.type: persisted
13. Backup, Recovery and Maintenance
Manual Index Purging
If automated Index Lifecycle Management (ILM) is not configured, implement routine maintenance using curl invocations:
# Preview indices targeted for deletion
curl -s "localhost:9200/_cat/indices/logs-2024.05.*?v"
# Remove matching indices
curl -X DELETE "localhost:9200/logs-2024.05.*"
Segment Force-Merging
To reclaim disk space and speed up queries on historical read-only indices, perform segment force-merging during off-peak windows:
curl -X POST "localhost:9200/logs-2026.10.01/_forcemerge?max_num_segments=1"
14. Automation and Integration
Automated Index Cleanup via Systemd Timers
Create a bash maintenance script to purge log indices older than 30 days:
/usr/local/bin/es-index-purge.sh:
#!/usr/bin/env bash
set -euo pipefail
RETENTION_DATE=$(date -d "30 days ago" "+%Y.%m.%d")
curl -s -X DELETE "http://localhost:9200/nginx-access-${RETENTION_DATE}"
Configure execution via systemd timer unit /etc/systemd/system/es-purge.timer:
[Unit]
Description=Daily Elasticsearch Index Cleanup Timer
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
15. Practical Labs and Exercises
Lab 1 (Beginner): Filebeat Installation and Output Verification
- Objective: Install Filebeat, configure log tailing for system authorization logs (
/var/log/auth.log), and test output functionality. - Steps:
- Install Filebeat via package manager.
- Edit
/etc/filebeat/filebeat.ymlto enable input for/var/log/auth.log. - Validate setup:
filebeat test config.
- Validation: Run
filebeat test outputto confirm target endpoint reachability.
Lab 2 (Intermediate): Grok Pipeline Development & Elasticsearch Search
- Objective: Build a Logstash pipeline that parses Nginx web logs using Grok patterns and verify document creation in Elasticsearch.
- Steps:
- Create
/etc/logstash/conf.d/nginx.confwithbeatsinput,grokfilter, andelasticsearchoutput. - Validate pipeline syntax:
/usr/share/logstash/bin/logstash -t. - Generate mock log entries using
loggerorcurlagainst Nginx.
- Validation: Run
curl -s "localhost:9200/nginx-access-*/_count"to confirm event ingestion.
Lab 3 (Troubleshooting): Recovering from Read-Only Disk Watermark Lockout
- Objective: Simulate a disk watermark lock out, identify root causes, and restore read-write cluster operations.
- Steps:
- Inject index read-only lock:
curl -X PUT "localhost:9200/_settings" -H 'Content-Type: application/json' -d '{"index.blocks.read_only_allow_delete": true}'
- Confirm write failures when attempting to index new documents.
- Diagnose cause via
curl -s localhost:9200/_cat/allocation?v. - Clear the global block setting.
- Validation: Verify indices accept write operations without raising HTTP 403 exceptions.
16. Quick Reference and Cheat Sheet
Important Configuration Files
| Path | Description |
|---|---|
/etc/elasticsearch/elasticsearch.yml |
Core Elasticsearch settings |
/etc/logstash/conf.d/*.conf |
Logstash pipeline processing rules |
/etc/filebeat/filebeat.yml |
Filebeat shipper settings |
Important Log Locations
| Path | Component |
|---|---|
/var/log/elasticsearch/ |
Elasticsearch node & query logs |
/var/log/logstash/logstash-plain.log |
Logstash system execution log |
journalctl -u filebeat |
Filebeat daemon log stream |
Common Commands
| Task | Command |
|---|---|
| Test Filebeat Config | filebeat test config |
| Test Filebeat Connectivity | filebeat test output |
| Test Logstash Syntax | /usr/share/logstash/bin/logstash -t |
| Cluster Health Check | curl -s localhost:9200/_cluster/health?pretty |
| List Indices | curl -s localhost:9200/_cat/indices?v |
Troubleshooting Quick Reference
| Symptom | Initial Diagnostic Command |
|---|---|
| Missing documents in ES | journalctl -u logstash -n 50 --no-pager |
| Beats output failing | filebeat test output |
| Shards unassigned | curl -s localhost:9200/_cluster/allocation/explain?pretty |
| Parsing errors | Search ES for tags: "_grokparsefailure" |
Important Ports and Protocols
| Port | Protocol | Purpose |
|---|---|---|
5044 |
TCP | Logstash Beats Input listener |
9200 |
HTTP / REST | Elasticsearch REST API & client ingestion |
9300 |
TCP | Elasticsearch inter-node transport network |
17. Further Reading and References
- Elastic. Elasticsearch Guide & Reference. Available at:
[https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) - Elastic. Logstash Reference Guide. Available at:
[https://www.elastic.co/guide/en/logstash/current/index.html](https://www.elastic.co/guide/en/logstash/current/index.html) - Elastic. Filebeat Overview & Documentation. Available at:
[https://www.elastic.co/guide/en/beats/filebeat/current/index.html](https://www.elastic.co/guide/en/beats/filebeat/current/index.html) - Elastic. Grok Filter Plugin & Regular Expressions. Available at:
[https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html)