1. Overview
The Domain Name System (DNS) maps human-readable names (such as api.example.com) to IP addresses (A for IPv4, AAAA for IPv6), mail exchangers (MX), aliases (CNAME), and metadata (TXT). Applications running on Linux rarely construct raw DNS network packets directly. Instead, they delegate name resolution to the standard C library (glibc or musl) by invoking POSIX socket functions such as getaddrinfo() (or legacy gethostbyname()). The operating system resolver then intercepts these calls and resolves the query through local static files, caching daemons, or upstream nameservers.
+------------------+ getaddrinfo() +------------------------+
| Application | --------------------> | OS Name Service Switch |
| (curl, web app) | | (/etc/nsswitch.conf)|
+------------------+ +------------------------+
|
+------------------+------------------+
| |
(files source) (dns source)
v v
+------------------+ +------------------+
| /etc/hosts | | /etc/resolv.conf |
+------------------+ +------------------+
|
v
+------------------+
| Upstream DNS / |
| systemd-resolved |
+------------------+
Without a functioning DNS resolution pipeline, microservices cannot communicate, database connections fail with Host not found errors, web interfaces fail to load, and secure services cannot validate TLS certificates. Understanding the distinction between direct network DNS queries (dig, host) and OS-level application resolution (getaddrinfo, getent) is one of the most critical fundamentals for a Linux System Administrator or Site Reliability Engineer (SRE).
2. Architecture and Mental Model
The Linux Name Resolution Flow
To reason accurately about resolution failures, an engineer must trace the complete path execution of a hostname lookup request. When an application calls getaddrinfo("api.example.com", ...):
- Application Call: The process invokes
getaddrinfo(), which executes inside the process address space using standard C library implementation (glibc). - Name Service Switch (NSS) Database Inspection:
glibcreads/etc/nsswitch.confto determine the lookup order defined under thehosts:configuration line. - Local File Lookup (
files): Iffilesis listed first,glibccallslibnss_files.soto parse/etc/hosts. If a matching entry exists, the IP address is immediately returned to the application, terminating the search. - DNS Lookup (
dnsorresolve): If no local match exists,glibcinvokeslibnss_dns.so(orlibnss_resolve.so).
- If using
libnss_dns.so,glibcreads/etc/resolv.confto find configurednameserveraddresses and option directives (e.g.,ndots,search). It opens a UDP socket (typically port 53) directly to the upstream nameserver. - If using
libnss_resolve.so(associated withsystemd-resolved), the query is forwarded over D-Bus or sent to the local stub listener on127.0.0.53:53.
- Upstream Recurser Execution: The targeted nameserver queries the hierarchical DNS structure (Root -> TLD -> Authoritative Nameservers) until an answer or authoritative failure (
NXDOMAIN) is returned. - Caching Layer: The answer is returned to the client and may be cached by
systemd-resolved,nscd, or an application-level cache prior to control returning to the application.
flowchart TD
App[Application Call: getaddrinfo] --> NSS[Read /etc/nsswitch.conf: hosts entry]
NSS --> OrderCheck{Lookup Order}
OrderCheck -- 1. files --> CheckHosts[Check /etc/hosts]
CheckHosts -- Match Found --> ReturnIP[Return IP to Application]
CheckHosts -- No Match --> NextSource
OrderCheck -- 2. resolve / dns --> NextSource{Resolver Engine}
NextSource -- systemd-resolved --> StubListener[Query 127.0.0.53:53 or D-Bus]
NextSource -- glibc libnss_dns --> ResolvConf[Read /etc/resolv.conf]
ResolvConf --> UpstreamDNS[Query Upstream Nameservers UDP/TCP 53]
StubListener --> UpstreamDNS
UpstreamDNS --> Recursion[Recursive Query: Root -> TLD -> Authoritative]
Recursion --> DNSResponse{DNS Response}
DNSResponse -- NOERROR A/AAAA --> Cache[Cache Result in Resolver]
Cache --> ReturnIP
DNSResponse -- NXDOMAIN / SERVFAIL --> ReturnError[Return Error to Application]
systemd-resolved Operational Modes
Modern Linux distributions (Ubuntu, Debian, Fedora, Arch) deploy systemd-resolved as a local stub resolver and network name resolution manager. It provides four main modes for managing /etc/resolv.conf:
- Stub Resolver Mode (Recommended):
/etc/resolv.confis a symbolic link pointing to/run/systemd/resolve/stub-resolv.conf./run/systemd/resolve/stub-resolv.confsetsnameserver 127.0.0.53as the sole resolver.- Enables centralized caching, DNSSEC validation, DNS-over-TLS, and per-interface routing domains.
- Static Resolver Mode:
/etc/resolv.confpoints to/usr/lib/systemd/resolv.conf.- Contains only
127.0.0.53without dynamic search domains.
- Uplink Resolver Mode:
/etc/resolv.confpoints to/run/systemd/resolve/resolv.conf.- Bypasses the local stub (
127.0.0.53) and directly lists upstream network DNS servers. Applications reading/etc/resolv.confdirectly bypasssystemd-resolvedcaching.
- Foreign / Consumer Mode:
/etc/resolv.confis a static file managed by third-party services (e.g., NetworkManager,resolvconf, manual editing).systemd-resolvedreads this file as a consumer.
3. Core Concepts and Terminology
Essential Record Types
| Record Type | Full Name | Purpose | Example Value |
|---|---|---|---|
| A | Address (IPv4) | Maps a hostname to a 32-bit IPv4 address. | 93.184.216.34 |
| AAAA | Address (IPv6) | Maps a hostname to a 128-bit IPv6 address. | 2606:2800:220:1:248:1893:25c8:1946 |
| CNAME | Canonical Name | Maps an alias to another canonical hostname. Not allowed at zone apex (example.com). |
api.example.com -> prod-lb.aws.com |
| MX | Mail Exchanger | Specifies mail servers responsible for receiving email, with priority values. | 10 mail.example.com |
| TXT | Text | Stores text metadata used for domain verification, SPF, DKIM, and DMARC. | "v=spf1 include:_spf.google.com ~all" |
| PTR | Pointer | Maps an IP address to a hostname for reverse DNS lookups. | 34.216.184.93.in-addr.arpa -> example.com |
| NS | Name Server | Delegates a DNS zone to authoritative nameservers. | ns1.example.com |
| SOA | Start of Authority | Contains core metadata for the zone (primary NS, admin email, serial number, refresh/retry timers). | ns1.example.com. admin.example.com. 2026110301 ... |
Search Domains and FQDNs
A Fully Qualified Domain Name (FQDN) ends with an implicit trailing dot (e.g., api.example.com.), indicating an absolute DNS root lookup. Unqualified or relative short hostnames (e.g., db) rely on resolver search lists.
When /etc/resolv.conf contains:
search internal.example.com example.com
options ndots:2
- If a queried name contains fewer dots than the
ndotsthreshold (default is 1), the resolver sequentially appends each domain from thesearchdirective prior to querying the raw name. - For
db(ndots= 0 < 2): The resolver queriesdb.internal.example.com, thendb.example.com, and finallydb..
Response Status Codes
NXDOMAIN(Non-Existent Domain): The authoritative nameserver confirmed that the domain name does not exist in the DNS zone. Indicates a missing record, typo, or deleted entry.SERVFAIL(Server Failure): The resolver encountered an error trying to process the query. Indicates DNSSEC validation failure, upstream timeout, network packet loss, or misconfigured authoritative servers.
4. Installation and Version Management
Core resolver libraries (glibc NSS) are built directly into Linux operating systems. Diagnostic tools (dig, host, nslookup, getent) and modern stub resolvers (systemd-resolved) are maintained in standard distribution package repositories.
Package Installation Commands
On Debian / Ubuntu:
sudo apt-get update
sudo apt-get install -y dnsutils systemd-resolved glibc-utils
On RHEL / Rocky Linux / AlmaLinux:
sudo dnf install -y bind-utils systemd-resolved glibc-common
On Arch Linux:
sudo pacman -S --needed bind systemd
Identifying Installed Versions
To inspect the installed version of systemd-resolved and client utilities:
# Check systemd version (includes systemd-resolved)
resolvectl --version
# Check dig utility version
dig -v
# Verify binaries location
which dig getent resolvectl
5. Configuration Files and Directory Structure
| File/Directory | Purpose | Important Details |
|---|---|---|
/etc/nsswitch.conf |
Defines lookup order for OS databases (hosts, passwd, group). | Controlled by line starting with hosts:. |
/etc/hosts |
Static table of hostnames mapped to IP addresses. | Evaluated before DNS when files precedes dns in nsswitch.conf. |
/etc/resolv.conf |
System resolver configuration file. | Defines nameserver, search, and options directives. Often a symlink. |
/etc/systemd/resolved.conf |
Main configuration file for systemd-resolved service. |
Controls global DNS=, FallbackDNS=, Domains=, DNSSEC=, DNSOverTLS=. |
/etc/systemd/resolved.conf.d/*.conf |
Drop-in configuration directory for systemd-resolved. |
Preferred location for modular persistent administrative edits. |
/run/systemd/resolve/stub-resolv.conf |
Generated stub configuration for systemd-resolved. |
Points to 127.0.0.53 stub listener with search domains. Target for /etc/resolv.conf link. |
/run/systemd/resolve/resolv.conf |
Generated uplink configuration listing actual upstream DNS servers. | Bypasses local stub resolver when symlinked to /etc/resolv.conf. |
Detailed File Specifications
1. /etc/nsswitch.conf Syntax
hosts: files mdns4_minimal [NOTFOUND=return] resolve dns
files: Inspects/etc/hosts.resolve: Routes queries throughsystemd-resolvedvialibnss_resolve.so.dns: Direct IPv4/IPv6 DNS socket resolution vialibnss_dns.soreading/etc/resolv.conf.
2. Standard /etc/resolv.conf Options
nameserver 10.0.0.2
nameserver 1.1.1.1
search internal.example.com example.com
options ndots:1 timeout:2 attempts:2
nameserver: IP address of DNS server to query (up to 3 maximum supported byglibc).search: List of search domains for short name expansion.options ndots:N: Minimum dot count required in name before an initial absolute query is executed.options timeout:N: Seconds resolver waits for response from a nameserver before trying another.options attempts:N: Number of times resolver queries nameservers before failing.
3. /etc/systemd/resolved.conf Example
[Resolve]
DNS=10.0.0.2 1.1.1.1
FallbackDNS=8.8.8.8 9.9.9.9
Domains=internal.example.com
DNSSEC=allow-downgrade
DNSOverTLS=opportunistic
Cache=yes
6. Service Management and Common Operations
Service Control Commands
# Check status of systemd-resolved service
systemctl status systemd-resolved
# Start, stop, or restart systemd-resolved
sudo systemctl start systemd-resolved
sudo systemctl stop systemd-resolved
sudo systemctl restart systemd-resolved
# Enable service at system boot
sudo systemctl enable systemd-resolved
Reload vs. Restart
systemctl reload systemd-resolved: Reloads configuration files without terminating active connections or flushing the main memory structure.systemctl restart systemd-resolved: Restarts the process completely, clearing in-memory runtime states and flushing cached DNS records.
Flushing DNS Caches
# Flush cache using systemd-resolved utility
sudo resolvectl flush-caches
# Verify cache statistics
sudo resolvectl statistics
# If nscd (Name Service Cache Daemon) is used on older systems:
sudo systemctl restart nscd
Port and Socket Verification
To verify that systemd-resolved is listening on the local loopback interface:
# Inspect UDP and TCP sockets on port 53
ss -tulpn | grep :53
Expected output:
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=842,fd=13))
tcp LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=842,fd=13))
7. Common Administrative Tasks and Workflows
Task 1: Setting Persistent DNS Nameservers via systemd-resolved
Objective: Configure system-wide upstream DNS servers (1.1.1.1 and 8.8.8.8) persistently across reboots.
Procedure:
- Create a drop-in configuration directory:
sudo mkdir -p /etc/systemd/resolved.conf.d
- Create a custom configuration file:
sudo tee /etc/systemd/resolved.conf.d/dns_servers.conf <<'EOF'
[Resolve]
DNS=1.1.1.1 8.8.8.8
Domains=~.
EOF
- Restart
systemd-resolved:
sudo systemctl restart systemd-resolved
- Validation:
resolvectl status
Task 2: Correcting /etc/resolv.conf Symlink to Use Local Stub Resolver
Objective: Restore standard systemd-resolved stub resolver configuration when /etc/resolv.conf has been corrupted or overwritten.
Procedure:
- Identify the current file type of
/etc/resolv.conf:
ls -la /etc/resolv.conf
- Remove the existing file or incorrect link:
sudo rm -f /etc/resolv.conf
- Re-create the symbolic link to the stub resolver:
sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
- Validation:
ls -la /etc/resolv.conf
cat /etc/resolv.conf
Expected result: /etc/resolv.conf points directly to ../run/systemd/resolve/stub-resolv.conf, containing nameserver 127.0.0.53.
Task 3: Configuring Interface-Specific Split-Horizon DNS
Objective: Route queries for *.internal.net through a private corporate DNS server (10.10.0.2) on interface eth0, while sending internet traffic to public resolvers.
Procedure:
- Apply per-interface DNS rules using
resolvectl:
sudo resolvectl dns eth0 10.10.0.2
sudo resolvectl domain eth0 ~internal.net
- Validation:
resolvectl status eth0
resolvectl query db.internal.net
8. CLI Command Reference and Cheatsheet
1. External DNS Query Tools (dig, host, nslookup)
| Command | Purpose | Important Flags / Arguments | Example | Explanation |
|---|---|---|---|---|
dig |
Perform comprehensive DNS database lookups. | @server: Query target server |
+short: Print answer section only
+trace: Follow delegation hierarchy | dig @8.8.8.8 api.example.com +short | Queries Google DNS (8.8.8.8) directly for api.example.com and outputs only the resolved IP address. |
| dig | Trace full DNS hierarchy from root. | +trace: Trace root to authoritative | dig +trace example.com | Shows step-by-step resolution path from Root DNS (.) to TLD (.com) to Authoritative NS. |
| dig | Execute Reverse DNS lookup. | -x: Reverse IP lookup | dig -x 93.184.216.34 | Queries PTR record for the given IPv4 address. |
| host | Perform simple DNS name lookups. | -t type: Query specific type | host -t MX example.com | Returns mail server hostnames and priorities for domain. |
| nslookup | Interactive/legacy lookup tool. | Non-interactive command format | nslookup example.com 10.0.0.2 | Queries 10.0.0.2 for example.com using nslookup syntax. |
2. System Resolver Diagnostics (getent, resolvectl)
| Command | Purpose | Important Flags / Arguments | Example | Explanation |
|---|---|---|---|---|
getent |
Query OS NSS databases. | hosts: Specify hosts database |
getent hosts api.example.com |
Simulates exact resolution path used by applications calling getaddrinfo() (includes /etc/hosts & NSS order). |
resolvectl |
Manage and inspect systemd-resolved. |
status: Show per-link settings |
query: Resolve via systemd-resolved | resolvectl status eth0 | Displays DNS servers, domain search paths, and DNSSEC status for eth0. |
| resolvectl | Test resolution via systemd-resolved stub. | query: Query daemon | resolvectl query example.com | Resolves host via D-Bus interface of systemd-resolved. |
| resolvectl | Flush daemon caches. | flush-caches: Clear cache | sudo resolvectl flush-caches | Immediately purges all cached DNS entries held by systemd-resolved. |
9. Logs, Monitoring and Diagnostics
Inspecting systemd-resolved Logs
systemd-resolved logs operational events to the systemd journal.
# View recent logs for systemd-resolved
sudo journalctl -u systemd-resolved -n 50 --no-pager
# Follow logs in real time
sudo journalctl -u systemd-resolved -f
Enabling Debug Logging
To observe detailed transaction logs and query processing:
# Enable debug logging dynamically
sudo resolvectl log-level debug
# View dynamic debug output
sudo journalctl -u systemd-resolved -f -o cat
# Revert back to standard info logging level
sudo resolvectl log-level info
Capturing DNS Network Traffic with tcpdump
When determining whether queries are leaving the host system over UDP/TCP port 53:
# Capture outbound DNS queries on all network interfaces
sudo tcpdump -i any -n port 53
Illustrative tcpdump log output:
18:14:02.102938 IP 192.168.1.50.54821 > 10.0.0.2.53: 41203+ A? api.example.com. (33)
18:14:02.124810 IP 10.0.0.2.53 > 192.168.1.50.54821: 41203 1/0/0 A 93.184.216.34 (49)
A?: Query requesting IPv4Arecord.41203+: Transaction ID (plus sign indicates recursion desired).1/0/0: Response containing 1 Answer, 0 Authority, 0 Additional records.
10. Troubleshooting Guide and Common Failure Scenarios
Scenario 1: dig resolves hostname successfully, but applications fail
Symptoms
- Command line utilities like
curl [https://api.example.com](https://api.example.com)or custom applications fail withCould not resolve hostorName or service not known. - Executing
dig api.example.comreturns a valid IPv4 address instantly.
Possible Causes
- An incorrect or stale static entry exists in
/etc/hosts. /etc/nsswitch.confhasfileslisted beforedns, forcinglibcto return the stale/etc/hostsIP.- The application uses standard libc
getaddrinfo(), which parses NSS databases, whereasdigbypasses NSS and queries DNS nameservers directly over the network.
Investigation and Diagnosis
- Query the host using the NSS application resolution path:
getent hosts api.example.com
- Query the host directly using raw DNS:
dig +short api.example.com
- Inspect
/etc/hostsfor overrides matching the target hostname:
grep "api.example.com" /etc/hosts
- Verify resolution order in
/etc/nsswitch.conf:
grep ^hosts /etc/nsswitch.conf
Root Cause Identification
If getent hosts api.example.com returns an old IP address (e.g. 127.0.0.1 or 10.0.1.50) matching an entry in /etc/hosts, while dig returns 93.184.216.34, the root cause is an uncoordinated local /etc/hosts override.
Resolution
Remove or update the stale record in /etc/hosts:
sudo sed -i '/api.example.com/d' /etc/hosts
Verification
Execute getent hosts api.example.com and verify that the output matches the dig IP response. Confirm application connectivity.
Prevention
Avoid adding production hostnames to /etc/hosts in automated deployment scripts; use internal DNS zones or split-horizon DNS instead.
Scenario 2: Hostname resolves on host A, but fails on host B
Symptoms
curl [https://internal-service.local](https://internal-service.local)succeeds on Host A, but returnsName or service not knownon Host B.
Possible Causes
- Host B has different nameservers defined in
/etc/resolv.conf. - Host B is missing search domains required to resolve short hostnames.
- Host A has a static entry in
/etc/hoststhat Host B lacks.
Investigation and Diagnosis
- Compare resolver configurations on Host A and Host B:
cat /etc/resolv.conf
- Compare
/etc/nsswitch.confon both hosts:
grep ^hosts /etc/nsswitch.conf
- Direct a query from Host B specifically to Host A’s nameserver:
dig @<NAMESERVER_FROM_HOST_A> internal-service.local
Root Cause Identification
If dig @<NAMESERVER_FROM_HOST_A> succeeds from Host B, Host B’s network interface is configured with an incorrect or unreachable nameserver.
Resolution
Update Host B’s configuration to include the correct nameserver or network search domain via resolvectl or netplan/NetworkManager.
Verification
Run getent hosts internal-service.local on Host B to confirm resolution.
Prevention
Use centralized configuration management (Ansible, Terraform) to maintain consistent resolver settings across all fleet servers.
Scenario 3: Distinguishing NXDOMAIN from SERVFAIL Errors
Symptoms
- DNS lookups fail, returning header status
NXDOMAINorSERVFAILindigoutput.
# Example 1:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 41203
# Example 2:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 10492
Possible Causes
- NXDOMAIN: The domain or record does not exist on the authoritative nameserver (typo, deleted record).
- SERVFAIL: Upstream resolver error, DNSSEC validation failure, authoritative server timeout, or network packet loss.
Investigation and Diagnosis
- Execute query with explicit header inspection:
dig example.com
- If status is
SERVFAIL, bypass DNSSEC validation to test if signature validation is failing:
dig example.com +cdflag
- Trace delegation from root to identify where failure occurs:
dig +trace example.com
Root Cause Identification
- If
dig +cdflagsucceeds while standarddigreturnsSERVFAIL, the upstream zone has an invalid or expired DNSSEC RRSIG record. - If
dig +tracefails at the TLD or authoritative delegation step, the authoritative nameservers are unreachable or misconfigured.
Resolution
- For
NXDOMAIN: Create the missing DNS record in the authoritative DNS console. - For DNSSEC
SERVFAIL: Re-sign the DNS zone or update DS records at the domain registrar.
Verification
Run dig example.com and ensure status returns NOERROR with valid record data.
Prevention
Implement automated DNSSEC monitoring and domain record expiration alerts.
Scenario 4: Intermittent or Slow Hostname Resolution
Symptoms
- Command line tools delay for 5 to 10 seconds before resolving hostnames.
- Web requests occasionally time out during initial connection establishment.
Possible Causes
- The primary
nameserverlisted in/etc/resolv.confis unreachable or dropping UDP packets;glibcwaits for timeout before trying the second nameserver. - Dual-stack lookups (
AandAAAA) timing out on IPv6 when the network only supports IPv4.
Investigation and Diagnosis
- Test each configured nameserver individually with tight timeouts:
# Extract nameservers and test each
grep nameserver /etc/resolv.conf | awk '{print $2}' | xargs -I {} dig @{} example.com +time=1 +tries=1
- Measure response time differences between IPv4 (
A) and IPv6 (AAAA) queries:
time dig example.com A
time dig example.com AAAA
Root Cause Identification
If the first nameserver times out after 5 seconds while the second responds in 10ms, glibc is incurring a 5-second penalty on every new socket setup due to an unresponsive primary resolver.
Resolution
- Remove unreachable nameservers from
/etc/resolv.conforsystemd-resolvedconfig. - Add conservative timeout options to
/etc/resolv.conf:
options timeout:1 attempts:2
Verification
Measure query latency with time dig example.com and confirm responses return in < 50ms.
Prevention
Ensure all configured nameservers are redundant, active-active, and monitored for latency metrics.
Scenario 5: Direct Edits to /etc/resolv.conf Overwritten After Reboot
Symptoms
- Administrative changes added directly to
/etc/resolv.confvanish whenever the server reboots or network interfaces bounce.
Possible Causes
/etc/resolv.confis a symlink dynamically generated bysystemd-resolved,NetworkManager, ornetplan.
Investigation and Diagnosis
- Inspect file path and symlink target:
ls -la /etc/resolv.conf
- Check if
systemd-resolvedis active:
systemctl is-active systemd-resolved
Root Cause Identification
/etc/resolv.conf points to /run/systemd/resolve/stub-resolv.conf, which is transient and regenerated dynamically by system daemons.
Resolution
Do not edit /etc/resolv.conf directly. Apply changes in /etc/systemd/resolved.conf.d/ drop-in files or via resolvectl:
sudo resolvectl dns eth0 1.1.1.1 8.8.8.8
Verification
Reboot the test instance or restart systemd-resolved and confirm settings persist via resolvectl status.
Prevention
Document host network management tools used across images (e.g., systemd-networkd, NetworkManager, cloud-init).
11. Security and Best Practices
File Permissions and Ownership
To prevent unprivileged users or rogue processes from altering system DNS resolution paths, enforce strict security permissions:
# Set ownership and permissions for /etc/hosts
sudo chown root:root /etc/hosts
sudo chmod 644 /etc/hosts
# Enforce secure configuration file permissions
sudo chown -R root:root /etc/systemd/resolved.conf.d
sudo chmod 755 /etc/systemd/resolved.conf.d
sudo chmod 644 /etc/systemd/resolved.conf.d/*.conf
DNS-over-TLS (DoT) Implementation
Encrypt DNS queries between the system stub resolver and upstream providers to prevent eavesdropping and MITM tampering.
In /etc/systemd/resolved.conf.d/dot.conf:
[Resolve]
DNS=1.1.1.1#cloudflare-dns.com 9.9.9.9#dns.quad9.net
DNSOverTLS=opportunistic
DNSSEC Validation
Enable DNSSEC validation to ensure DNS responses signed by authoritative servers are authentic and unaltered.
In /etc/systemd/resolved.conf.d/dnssec.conf:
[Resolve]
DNSSEC=allow-downgrade
12. Performance and Resource Optimisation
Optimizing ndots Overhead in Kubernetes and Enterprise Linux
In high-throughput microservice environments, excessive ndots settings (e.g. ndots:5 common in default Kubernetes pods) cause massive DNS lookup amplification. A query for api.example.com triggers multiple failed search domain queries before executing the actual query.
Optimization Action:
- Use Fully Qualified Domain Names ending with a trailing dot (
api.example.com.) in application configurations to bypass search domain iteration completely. - Reduce
ndotsin/etc/resolv.conftondots:1for standard host systems.
Tuning Caching Parameters
Ensure systemd-resolved caching is enabled to eliminate redundant network roundtrips:
[Resolve]
Cache=yes
CacheFromLocalhost=no
Check cache efficiency using:
resolvectl statistics
13. Backup, Recovery and Maintenance
Backup Procedures
Regularly back up key name resolution control files prior to performing system maintenance or network stack upgrades:
# Create configuration backup archive
sudo tar -czvf dns_config_backup_$(date +%F).tar.gz \
/etc/nsswitch.conf \
/etc/hosts \
/etc/resolv.conf \
/etc/systemd/resolved.conf \
/etc/systemd/resolved.conf.d/
Recovery Procedures
If networking or resolution fails completely after an upgrade:
# Restore nsswitch.conf default
sudo cp /etc/nsswitch.conf.bak /etc/nsswitch.conf
# Re-link default systemd stub resolver
sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
# Restart networking services
sudo systemctl restart systemd-resolved
14. Automation and Integration
Ansible Playbook for Persistent DNS Configuration
---
- name: Configure Production Linux Resolver Settings
hosts: all
become: yes
tasks:
- name: Ensure systemd-resolved is installed and enabled
package:
name: systemd-resolved
state: present
- name: Create systemd-resolved drop-in directory
file:
path: /etc/systemd/resolved.conf.d
state: directory
owner: root
group: root
mode: '0755'
- name: Deploy custom DNS configuration
copy:
dest: /etc/systemd/resolved.conf.d/custom_dns.conf
content: |
[Resolve]
DNS=1.1.1.1 8.8.8.8
FallbackDNS=9.9.9.9
Domains=~.
DNSOverTLS=opportunistic
Cache=yes
owner: root
group: root
mode: '0644'
notify: Restart systemd-resolved
- name: Ensure correct resolv.conf symlink
file:
src: /run/systemd/resolve/stub-resolv.conf
dest: /etc/resolv.conf
state: link
force: yes
handlers:
- name: Restart systemd-resolved
service:
name: systemd-resolved
state: restarted
15. Practical Labs and Exercises
Lab 1: Beginner — Tracing NSS vs. Direct DNS Lookups
- Objective: Observe the functional difference between application-level lookup (
getent) and direct network DNS query (dig). - Instructions:
- Add a dummy entry to
/etc/hosts:
sudo sh -c 'echo "192.0.2.123 lab.test.local" >> /etc/hosts'
- Query
lab.test.localusinggetent hosts:
getent hosts lab.test.local
- Query
lab.test.localusingdig:
dig lab.test.local
- Cleanup: Remove entry from
/etc/hosts.
- Expected Outcome:
getentreturns192.0.2.123via/etc/hosts.digfails or returnsNXDOMAINbecause the fake host is not published on upstream network DNS nameservers.
Lab 2: Intermediate — Debugging an Unresponsive Primary Nameserver
- Objective: Simulate a DNS timeout issue and diagnose latency using options and
digtiming. - Instructions:
- Temporarily configure an unreachable primary nameserver in
/etc/resolv.conf:
nameserver 192.0.2.1
nameserver 1.1.1.1
- Execute
time getent hosts example.com. - Add
options timeout:1 attempts:1to/etc/resolv.conf. - Re-run
time getent hosts example.comand measure latency reduction.
- Expected Outcome: Initial request delays for 5 seconds waiting for
192.0.2.1to time out. Adding timeout options reduces total failover delay to 1 second.
16. Quick Reference and Cheat Sheet
Important Configuration Files
| File | Primary Role |
|---|---|
/etc/nsswitch.conf |
Controls name lookup database order (hosts: files resolve dns). |
/etc/hosts |
Static IP-to-hostname mappings. |
/etc/resolv.conf |
Dynamic or static resolver configuration. |
/etc/systemd/resolved.conf |
Global configuration for systemd-resolved service. |
Important Log Locations
| Service / Tool | Command / Log Path |
|---|---|
systemd-resolved Journal |
journalctl -u systemd-resolved -f |
| Network Packet Trace | sudo tcpdump -i any port 53 -n |
Common Diagnostic Commands
| Task | Command |
|---|---|
| Application-level path lookup | getent hosts <hostname> |
| Short DNS lookup answer | dig +short <hostname> |
| Query specific nameserver | dig @<nameserver_ip> <hostname> |
| Trace root delegation path | dig +trace <hostname> |
| Inspect systemd-resolved status | resolvectl status |
| Flush DNS cache | sudo resolvectl flush-caches |
Troubleshooting Quick Reference
| Symptom | Initial Diagnostic Command |
|---|---|
dig works, app fails |
getent hosts <hostname> vs grep <hostname> /etc/hosts |
| Resolution slow/delays | grep nameserver /etc/resolv.conf + test each with dig @IP |
| Changes lost on reboot | ls -la /etc/resolv.conf (Check symlink target) |
Unexplained SERVFAIL |
dig <hostname> +cdflag (Check DNSSEC failure) |
Important Ports and Protocols
| Port | Protocol | Purpose |
|---|---|---|
53 |
UDP/TCP | Standard Domain Name System (DNS) queries. |
853 |
TCP | DNS-over-TLS (DoT) encrypted query transport. |
5353 |
UDP | Multicast DNS (mDNS / Avahi / .local). |
17. Further Reading and References
hosts(5)Linux Manual Page: Format and rules for static host tables (man7.org/linux/man-pages/man5/hosts.5.html).resolv.conf(5)Linux Manual Page: Resolver configuration file specification (man7.org/linux/man-pages/man5/resolv.conf.5.html).nsswitch.conf(5)Linux Manual Page: Name Service Switch configuration (man7.org/linux/man-pages/man5/nsswitch.conf.5.html).systemd-resolved(8)System Manager Documentation: Systemd stub resolver manual (man7.org/linux/man-pages/man8/systemd-resolved.8.html).- Cloudflare Learning DNS: Fundamentals of DNS record types, recursion, and root servers (
[cloudflare.com/learning/dns](https://cloudflare.com/learning/dns)).