Domain 2Networking & RoutingINTERMEDIATE

Demystifying Linux DNS Resolution: Architecture, Configuration, and Production Troubleshooting

A comprehensive guide to Linux DNS resolution architecture, libc NSS getaddrinfo flows, systemd-resolved stub modes, configuration files, CLI tools, diagnostic workflows, and production troubleshooting.

Domain: Domain 2: Networking & Routing•Updated: 2026-11-03
#linux#sysadmin#dns#networking#systemd-resolved

1. Overview

The Domain Name System (DNS) maps human-readable names (such as api.example.com) to IP addresses (A for IPv4, AAAA for IPv6), mail exchangers (MX), aliases (CNAME), and metadata (TXT). Applications running on Linux rarely construct raw DNS network packets directly. Instead, they delegate name resolution to the standard C library (glibc or musl) by invoking POSIX socket functions such as getaddrinfo() (or legacy gethostbyname()). The operating system resolver then intercepts these calls and resolves the query through local static files, caching daemons, or upstream nameservers.

+------------------+     getaddrinfo()     +------------------------+
|   Application    | --------------------> | OS Name Service Switch |
| (curl, web app)  |                       |    (/etc/nsswitch.conf)|
+------------------+                       +------------------------+
                                                       |
                                    +------------------+------------------+
                                    |                                     |
                             (files source)                         (dns source)
                                    v                                     v
                           +------------------+                  +------------------+
                           |    /etc/hosts    |                  | /etc/resolv.conf |
                           +------------------+                  +------------------+
                                                                          |
                                                                          v
                                                                 +------------------+
                                                                 | Upstream DNS /   |
                                                                 | systemd-resolved |
                                                                 +------------------+

Without a functioning DNS resolution pipeline, microservices cannot communicate, database connections fail with Host not found errors, web interfaces fail to load, and secure services cannot validate TLS certificates. Understanding the distinction between direct network DNS queries (dig, host) and OS-level application resolution (getaddrinfo, getent) is one of the most critical fundamentals for a Linux System Administrator or Site Reliability Engineer (SRE).


2. Architecture and Mental Model

The Linux Name Resolution Flow

To reason accurately about resolution failures, an engineer must trace the complete path execution of a hostname lookup request. When an application calls getaddrinfo("api.example.com", ...):

  1. Application Call: The process invokes getaddrinfo(), which executes inside the process address space using standard C library implementation (glibc).
  2. Name Service Switch (NSS) Database Inspection: glibc reads /etc/nsswitch.conf to determine the lookup order defined under the hosts: configuration line.
  3. Local File Lookup (files): If files is listed first, glibc calls libnss_files.so to parse /etc/hosts. If a matching entry exists, the IP address is immediately returned to the application, terminating the search.
  4. DNS Lookup (dns or resolve): If no local match exists, glibc invokes libnss_dns.so (or libnss_resolve.so).
  • If using libnss_dns.so, glibc reads /etc/resolv.conf to find configured nameserver addresses and option directives (e.g., ndots, search). It opens a UDP socket (typically port 53) directly to the upstream nameserver.
  • If using libnss_resolve.so (associated with systemd-resolved), the query is forwarded over D-Bus or sent to the local stub listener on 127.0.0.53:53.
  1. Upstream Recurser Execution: The targeted nameserver queries the hierarchical DNS structure (Root -> TLD -> Authoritative Nameservers) until an answer or authoritative failure (NXDOMAIN) is returned.
  2. Caching Layer: The answer is returned to the client and may be cached by systemd-resolved, nscd, or an application-level cache prior to control returning to the application.
flowchart TD
    App[Application Call: getaddrinfo] --> NSS[Read /etc/nsswitch.conf: hosts entry]
    NSS --> OrderCheck{Lookup Order}
    
    OrderCheck -- 1. files --> CheckHosts[Check /etc/hosts]
    CheckHosts -- Match Found --> ReturnIP[Return IP to Application]
    CheckHosts -- No Match --> NextSource
    
    OrderCheck -- 2. resolve / dns --> NextSource{Resolver Engine}
    
    NextSource -- systemd-resolved --> StubListener[Query 127.0.0.53:53 or D-Bus]
    NextSource -- glibc libnss_dns --> ResolvConf[Read /etc/resolv.conf]
    
    ResolvConf --> UpstreamDNS[Query Upstream Nameservers UDP/TCP 53]
    StubListener --> UpstreamDNS
    
    UpstreamDNS --> Recursion[Recursive Query: Root -> TLD -> Authoritative]
    Recursion --> DNSResponse{DNS Response}
    
    DNSResponse -- NOERROR A/AAAA --> Cache[Cache Result in Resolver]
    Cache --> ReturnIP
    DNSResponse -- NXDOMAIN / SERVFAIL --> ReturnError[Return Error to Application]

systemd-resolved Operational Modes

Modern Linux distributions (Ubuntu, Debian, Fedora, Arch) deploy systemd-resolved as a local stub resolver and network name resolution manager. It provides four main modes for managing /etc/resolv.conf:

  1. Stub Resolver Mode (Recommended):
  • /etc/resolv.conf is a symbolic link pointing to /run/systemd/resolve/stub-resolv.conf.
  • /run/systemd/resolve/stub-resolv.conf sets nameserver 127.0.0.53 as the sole resolver.
  • Enables centralized caching, DNSSEC validation, DNS-over-TLS, and per-interface routing domains.
  1. Static Resolver Mode:
  • /etc/resolv.conf points to /usr/lib/systemd/resolv.conf.
  • Contains only 127.0.0.53 without dynamic search domains.
  1. Uplink Resolver Mode:
  • /etc/resolv.conf points to /run/systemd/resolve/resolv.conf.
  • Bypasses the local stub (127.0.0.53) and directly lists upstream network DNS servers. Applications reading /etc/resolv.conf directly bypass systemd-resolved caching.
  1. Foreign / Consumer Mode:
  • /etc/resolv.conf is a static file managed by third-party services (e.g., NetworkManager, resolvconf, manual editing). systemd-resolved reads this file as a consumer.

3. Core Concepts and Terminology

Essential Record Types

Record Type Full Name Purpose Example Value
A Address (IPv4) Maps a hostname to a 32-bit IPv4 address. 93.184.216.34
AAAA Address (IPv6) Maps a hostname to a 128-bit IPv6 address. 2606:2800:220:1:248:1893:25c8:1946
CNAME Canonical Name Maps an alias to another canonical hostname. Not allowed at zone apex (example.com). api.example.com -> prod-lb.aws.com
MX Mail Exchanger Specifies mail servers responsible for receiving email, with priority values. 10 mail.example.com
TXT Text Stores text metadata used for domain verification, SPF, DKIM, and DMARC. "v=spf1 include:_spf.google.com ~all"
PTR Pointer Maps an IP address to a hostname for reverse DNS lookups. 34.216.184.93.in-addr.arpa -> example.com
NS Name Server Delegates a DNS zone to authoritative nameservers. ns1.example.com
SOA Start of Authority Contains core metadata for the zone (primary NS, admin email, serial number, refresh/retry timers). ns1.example.com. admin.example.com. 2026110301 ...

Search Domains and FQDNs

A Fully Qualified Domain Name (FQDN) ends with an implicit trailing dot (e.g., api.example.com.), indicating an absolute DNS root lookup. Unqualified or relative short hostnames (e.g., db) rely on resolver search lists.

When /etc/resolv.conf contains:

search internal.example.com example.com
options ndots:2
  • If a queried name contains fewer dots than the ndots threshold (default is 1), the resolver sequentially appends each domain from the search directive prior to querying the raw name.
  • For db (ndots = 0 < 2): The resolver queries db.internal.example.com, then db.example.com, and finally db..

Response Status Codes

  • NXDOMAIN (Non-Existent Domain): The authoritative nameserver confirmed that the domain name does not exist in the DNS zone. Indicates a missing record, typo, or deleted entry.
  • SERVFAIL (Server Failure): The resolver encountered an error trying to process the query. Indicates DNSSEC validation failure, upstream timeout, network packet loss, or misconfigured authoritative servers.

4. Installation and Version Management

Core resolver libraries (glibc NSS) are built directly into Linux operating systems. Diagnostic tools (dig, host, nslookup, getent) and modern stub resolvers (systemd-resolved) are maintained in standard distribution package repositories.

Package Installation Commands

On Debian / Ubuntu:

sudo apt-get update
sudo apt-get install -y dnsutils systemd-resolved glibc-utils

On RHEL / Rocky Linux / AlmaLinux:

sudo dnf install -y bind-utils systemd-resolved glibc-common

On Arch Linux:

sudo pacman -S --needed bind systemd

Identifying Installed Versions

To inspect the installed version of systemd-resolved and client utilities:

# Check systemd version (includes systemd-resolved)
resolvectl --version

# Check dig utility version
dig -v

# Verify binaries location
which dig getent resolvectl

5. Configuration Files and Directory Structure

File/Directory Purpose Important Details
/etc/nsswitch.conf Defines lookup order for OS databases (hosts, passwd, group). Controlled by line starting with hosts:.
/etc/hosts Static table of hostnames mapped to IP addresses. Evaluated before DNS when files precedes dns in nsswitch.conf.
/etc/resolv.conf System resolver configuration file. Defines nameserver, search, and options directives. Often a symlink.
/etc/systemd/resolved.conf Main configuration file for systemd-resolved service. Controls global DNS=, FallbackDNS=, Domains=, DNSSEC=, DNSOverTLS=.
/etc/systemd/resolved.conf.d/*.conf Drop-in configuration directory for systemd-resolved. Preferred location for modular persistent administrative edits.
/run/systemd/resolve/stub-resolv.conf Generated stub configuration for systemd-resolved. Points to 127.0.0.53 stub listener with search domains. Target for /etc/resolv.conf link.
/run/systemd/resolve/resolv.conf Generated uplink configuration listing actual upstream DNS servers. Bypasses local stub resolver when symlinked to /etc/resolv.conf.

Detailed File Specifications

1. /etc/nsswitch.conf Syntax

hosts: files mdns4_minimal [NOTFOUND=return] resolve dns
  • files: Inspects /etc/hosts.
  • resolve: Routes queries through systemd-resolved via libnss_resolve.so.
  • dns: Direct IPv4/IPv6 DNS socket resolution via libnss_dns.so reading /etc/resolv.conf.

2. Standard /etc/resolv.conf Options

nameserver 10.0.0.2
nameserver 1.1.1.1
search internal.example.com example.com
options ndots:1 timeout:2 attempts:2
  • nameserver: IP address of DNS server to query (up to 3 maximum supported by glibc).
  • search: List of search domains for short name expansion.
  • options ndots:N: Minimum dot count required in name before an initial absolute query is executed.
  • options timeout:N: Seconds resolver waits for response from a nameserver before trying another.
  • options attempts:N: Number of times resolver queries nameservers before failing.

3. /etc/systemd/resolved.conf Example

[Resolve]
DNS=10.0.0.2 1.1.1.1
FallbackDNS=8.8.8.8 9.9.9.9
Domains=internal.example.com
DNSSEC=allow-downgrade
DNSOverTLS=opportunistic
Cache=yes

6. Service Management and Common Operations

Service Control Commands

# Check status of systemd-resolved service
systemctl status systemd-resolved

# Start, stop, or restart systemd-resolved
sudo systemctl start systemd-resolved
sudo systemctl stop systemd-resolved
sudo systemctl restart systemd-resolved

# Enable service at system boot
sudo systemctl enable systemd-resolved

Reload vs. Restart

  • systemctl reload systemd-resolved: Reloads configuration files without terminating active connections or flushing the main memory structure.
  • systemctl restart systemd-resolved: Restarts the process completely, clearing in-memory runtime states and flushing cached DNS records.

Flushing DNS Caches

# Flush cache using systemd-resolved utility
sudo resolvectl flush-caches

# Verify cache statistics
sudo resolvectl statistics

# If nscd (Name Service Cache Daemon) is used on older systems:
sudo systemctl restart nscd

Port and Socket Verification

To verify that systemd-resolved is listening on the local loopback interface:

# Inspect UDP and TCP sockets on port 53
ss -tulpn | grep :53

Expected output:

udp   UNCONN 0      0      127.0.0.53%lo:53   0.0.0.0:*   users:(("systemd-resolve",pid=842,fd=13))
tcp   LISTEN 0      4096   127.0.0.53%lo:53   0.0.0.0:*   users:(("systemd-resolve",pid=842,fd=13))

7. Common Administrative Tasks and Workflows

Task 1: Setting Persistent DNS Nameservers via systemd-resolved

Objective: Configure system-wide upstream DNS servers (1.1.1.1 and 8.8.8.8) persistently across reboots.

Procedure:

  1. Create a drop-in configuration directory:
sudo mkdir -p /etc/systemd/resolved.conf.d
  1. Create a custom configuration file:
sudo tee /etc/systemd/resolved.conf.d/dns_servers.conf <<'EOF'
[Resolve]
DNS=1.1.1.1 8.8.8.8
Domains=~.
EOF
  1. Restart systemd-resolved:
sudo systemctl restart systemd-resolved
  1. Validation:
resolvectl status

Objective: Restore standard systemd-resolved stub resolver configuration when /etc/resolv.conf has been corrupted or overwritten.

Procedure:

  1. Identify the current file type of /etc/resolv.conf:
ls -la /etc/resolv.conf
  1. Remove the existing file or incorrect link:
sudo rm -f /etc/resolv.conf
  1. Re-create the symbolic link to the stub resolver:
sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
  1. Validation:
ls -la /etc/resolv.conf
cat /etc/resolv.conf

Expected result: /etc/resolv.conf points directly to ../run/systemd/resolve/stub-resolv.conf, containing nameserver 127.0.0.53.

Task 3: Configuring Interface-Specific Split-Horizon DNS

Objective: Route queries for *.internal.net through a private corporate DNS server (10.10.0.2) on interface eth0, while sending internet traffic to public resolvers.

Procedure:

  1. Apply per-interface DNS rules using resolvectl:
sudo resolvectl dns eth0 10.10.0.2
sudo resolvectl domain eth0 ~internal.net
  1. Validation:
resolvectl status eth0
resolvectl query db.internal.net

8. CLI Command Reference and Cheatsheet

1. External DNS Query Tools (dig, host, nslookup)

Command Purpose Important Flags / Arguments Example Explanation
dig Perform comprehensive DNS database lookups. @server: Query target server


+short: Print answer section only


+trace: Follow delegation hierarchy | dig @8.8.8.8 api.example.com +short | Queries Google DNS (8.8.8.8) directly for api.example.com and outputs only the resolved IP address. | | dig | Trace full DNS hierarchy from root. | +trace: Trace root to authoritative | dig +trace example.com | Shows step-by-step resolution path from Root DNS (.) to TLD (.com) to Authoritative NS. | | dig | Execute Reverse DNS lookup. | -x: Reverse IP lookup | dig -x 93.184.216.34 | Queries PTR record for the given IPv4 address. | | host | Perform simple DNS name lookups. | -t type: Query specific type | host -t MX example.com | Returns mail server hostnames and priorities for domain. | | nslookup | Interactive/legacy lookup tool. | Non-interactive command format | nslookup example.com 10.0.0.2 | Queries 10.0.0.2 for example.com using nslookup syntax. |

2. System Resolver Diagnostics (getent, resolvectl)

Command Purpose Important Flags / Arguments Example Explanation
getent Query OS NSS databases. hosts: Specify hosts database getent hosts api.example.com Simulates exact resolution path used by applications calling getaddrinfo() (includes /etc/hosts & NSS order).
resolvectl Manage and inspect systemd-resolved. status: Show per-link settings


query: Resolve via systemd-resolved | resolvectl status eth0 | Displays DNS servers, domain search paths, and DNSSEC status for eth0. | | resolvectl | Test resolution via systemd-resolved stub. | query: Query daemon | resolvectl query example.com | Resolves host via D-Bus interface of systemd-resolved. | | resolvectl | Flush daemon caches. | flush-caches: Clear cache | sudo resolvectl flush-caches | Immediately purges all cached DNS entries held by systemd-resolved. |


9. Logs, Monitoring and Diagnostics

Inspecting systemd-resolved Logs

systemd-resolved logs operational events to the systemd journal.

# View recent logs for systemd-resolved
sudo journalctl -u systemd-resolved -n 50 --no-pager

# Follow logs in real time
sudo journalctl -u systemd-resolved -f

Enabling Debug Logging

To observe detailed transaction logs and query processing:

# Enable debug logging dynamically
sudo resolvectl log-level debug

# View dynamic debug output
sudo journalctl -u systemd-resolved -f -o cat

# Revert back to standard info logging level
sudo resolvectl log-level info

Capturing DNS Network Traffic with tcpdump

When determining whether queries are leaving the host system over UDP/TCP port 53:

# Capture outbound DNS queries on all network interfaces
sudo tcpdump -i any -n port 53

Illustrative tcpdump log output:

18:14:02.102938 IP 192.168.1.50.54821 > 10.0.0.2.53: 41203+ A? api.example.com. (33)
18:14:02.124810 IP 10.0.0.2.53 > 192.168.1.50.54821: 41203 1/0/0 A 93.184.216.34 (49)
  • A?: Query requesting IPv4 A record.
  • 41203+: Transaction ID (plus sign indicates recursion desired).
  • 1/0/0: Response containing 1 Answer, 0 Authority, 0 Additional records.

10. Troubleshooting Guide and Common Failure Scenarios

Scenario 1: dig resolves hostname successfully, but applications fail

Symptoms

  • Command line utilities like curl [https://api.example.com](https://api.example.com) or custom applications fail with Could not resolve host or Name or service not known.
  • Executing dig api.example.com returns a valid IPv4 address instantly.

Possible Causes

  1. An incorrect or stale static entry exists in /etc/hosts.
  2. /etc/nsswitch.conf has files listed before dns, forcing libc to return the stale /etc/hosts IP.
  3. The application uses standard libc getaddrinfo(), which parses NSS databases, whereas dig bypasses NSS and queries DNS nameservers directly over the network.

Investigation and Diagnosis

  1. Query the host using the NSS application resolution path:
getent hosts api.example.com
  1. Query the host directly using raw DNS:
dig +short api.example.com
  1. Inspect /etc/hosts for overrides matching the target hostname:
grep "api.example.com" /etc/hosts
  1. Verify resolution order in /etc/nsswitch.conf:
grep ^hosts /etc/nsswitch.conf

Root Cause Identification

If getent hosts api.example.com returns an old IP address (e.g. 127.0.0.1 or 10.0.1.50) matching an entry in /etc/hosts, while dig returns 93.184.216.34, the root cause is an uncoordinated local /etc/hosts override.

Resolution

Remove or update the stale record in /etc/hosts:

sudo sed -i '/api.example.com/d' /etc/hosts

Verification

Execute getent hosts api.example.com and verify that the output matches the dig IP response. Confirm application connectivity.

Prevention

Avoid adding production hostnames to /etc/hosts in automated deployment scripts; use internal DNS zones or split-horizon DNS instead.


Scenario 2: Hostname resolves on host A, but fails on host B

Symptoms

  • curl [https://internal-service.local](https://internal-service.local) succeeds on Host A, but returns Name or service not known on Host B.

Possible Causes

  1. Host B has different nameservers defined in /etc/resolv.conf.
  2. Host B is missing search domains required to resolve short hostnames.
  3. Host A has a static entry in /etc/hosts that Host B lacks.

Investigation and Diagnosis

  1. Compare resolver configurations on Host A and Host B:
cat /etc/resolv.conf
  1. Compare /etc/nsswitch.conf on both hosts:
grep ^hosts /etc/nsswitch.conf
  1. Direct a query from Host B specifically to Host A’s nameserver:
dig @<NAMESERVER_FROM_HOST_A> internal-service.local

Root Cause Identification

If dig @<NAMESERVER_FROM_HOST_A> succeeds from Host B, Host B’s network interface is configured with an incorrect or unreachable nameserver.

Resolution

Update Host B’s configuration to include the correct nameserver or network search domain via resolvectl or netplan/NetworkManager.

Verification

Run getent hosts internal-service.local on Host B to confirm resolution.

Prevention

Use centralized configuration management (Ansible, Terraform) to maintain consistent resolver settings across all fleet servers.


Scenario 3: Distinguishing NXDOMAIN from SERVFAIL Errors

Symptoms

  • DNS lookups fail, returning header status NXDOMAIN or SERVFAIL in dig output.
# Example 1:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 41203

# Example 2:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 10492

Possible Causes

  • NXDOMAIN: The domain or record does not exist on the authoritative nameserver (typo, deleted record).
  • SERVFAIL: Upstream resolver error, DNSSEC validation failure, authoritative server timeout, or network packet loss.

Investigation and Diagnosis

  1. Execute query with explicit header inspection:
dig example.com
  1. If status is SERVFAIL, bypass DNSSEC validation to test if signature validation is failing:
dig example.com +cdflag
  1. Trace delegation from root to identify where failure occurs:
dig +trace example.com

Root Cause Identification

  • If dig +cdflag succeeds while standard dig returns SERVFAIL, the upstream zone has an invalid or expired DNSSEC RRSIG record.
  • If dig +trace fails at the TLD or authoritative delegation step, the authoritative nameservers are unreachable or misconfigured.

Resolution

  • For NXDOMAIN: Create the missing DNS record in the authoritative DNS console.
  • For DNSSEC SERVFAIL: Re-sign the DNS zone or update DS records at the domain registrar.

Verification

Run dig example.com and ensure status returns NOERROR with valid record data.

Prevention

Implement automated DNSSEC monitoring and domain record expiration alerts.


Scenario 4: Intermittent or Slow Hostname Resolution

Symptoms

  • Command line tools delay for 5 to 10 seconds before resolving hostnames.
  • Web requests occasionally time out during initial connection establishment.

Possible Causes

  1. The primary nameserver listed in /etc/resolv.conf is unreachable or dropping UDP packets; glibc waits for timeout before trying the second nameserver.
  2. Dual-stack lookups (A and AAAA) timing out on IPv6 when the network only supports IPv4.

Investigation and Diagnosis

  1. Test each configured nameserver individually with tight timeouts:
# Extract nameservers and test each
grep nameserver /etc/resolv.conf | awk '{print $2}' | xargs -I {} dig @{} example.com +time=1 +tries=1
  1. Measure response time differences between IPv4 (A) and IPv6 (AAAA) queries:
time dig example.com A
time dig example.com AAAA

Root Cause Identification

If the first nameserver times out after 5 seconds while the second responds in 10ms, glibc is incurring a 5-second penalty on every new socket setup due to an unresponsive primary resolver.

Resolution

  1. Remove unreachable nameservers from /etc/resolv.conf or systemd-resolved config.
  2. Add conservative timeout options to /etc/resolv.conf:
options timeout:1 attempts:2

Verification

Measure query latency with time dig example.com and confirm responses return in < 50ms.

Prevention

Ensure all configured nameservers are redundant, active-active, and monitored for latency metrics.


Scenario 5: Direct Edits to /etc/resolv.conf Overwritten After Reboot

Symptoms

  • Administrative changes added directly to /etc/resolv.conf vanish whenever the server reboots or network interfaces bounce.

Possible Causes

  • /etc/resolv.conf is a symlink dynamically generated by systemd-resolved, NetworkManager, or netplan.

Investigation and Diagnosis

  1. Inspect file path and symlink target:
ls -la /etc/resolv.conf
  1. Check if systemd-resolved is active:
systemctl is-active systemd-resolved

Root Cause Identification

/etc/resolv.conf points to /run/systemd/resolve/stub-resolv.conf, which is transient and regenerated dynamically by system daemons.

Resolution

Do not edit /etc/resolv.conf directly. Apply changes in /etc/systemd/resolved.conf.d/ drop-in files or via resolvectl:

sudo resolvectl dns eth0 1.1.1.1 8.8.8.8

Verification

Reboot the test instance or restart systemd-resolved and confirm settings persist via resolvectl status.

Prevention

Document host network management tools used across images (e.g., systemd-networkd, NetworkManager, cloud-init).


11. Security and Best Practices

File Permissions and Ownership

To prevent unprivileged users or rogue processes from altering system DNS resolution paths, enforce strict security permissions:

# Set ownership and permissions for /etc/hosts
sudo chown root:root /etc/hosts
sudo chmod 644 /etc/hosts

# Enforce secure configuration file permissions
sudo chown -R root:root /etc/systemd/resolved.conf.d
sudo chmod 755 /etc/systemd/resolved.conf.d
sudo chmod 644 /etc/systemd/resolved.conf.d/*.conf

DNS-over-TLS (DoT) Implementation

Encrypt DNS queries between the system stub resolver and upstream providers to prevent eavesdropping and MITM tampering.

In /etc/systemd/resolved.conf.d/dot.conf:

[Resolve]
DNS=1.1.1.1#cloudflare-dns.com 9.9.9.9#dns.quad9.net
DNSOverTLS=opportunistic

DNSSEC Validation

Enable DNSSEC validation to ensure DNS responses signed by authoritative servers are authentic and unaltered.

In /etc/systemd/resolved.conf.d/dnssec.conf:

[Resolve]
DNSSEC=allow-downgrade

12. Performance and Resource Optimisation

Optimizing ndots Overhead in Kubernetes and Enterprise Linux

In high-throughput microservice environments, excessive ndots settings (e.g. ndots:5 common in default Kubernetes pods) cause massive DNS lookup amplification. A query for api.example.com triggers multiple failed search domain queries before executing the actual query.

Optimization Action:

  • Use Fully Qualified Domain Names ending with a trailing dot (api.example.com.) in application configurations to bypass search domain iteration completely.
  • Reduce ndots in /etc/resolv.conf to ndots:1 for standard host systems.

Tuning Caching Parameters

Ensure systemd-resolved caching is enabled to eliminate redundant network roundtrips:

[Resolve]
Cache=yes
CacheFromLocalhost=no

Check cache efficiency using:

resolvectl statistics

13. Backup, Recovery and Maintenance

Backup Procedures

Regularly back up key name resolution control files prior to performing system maintenance or network stack upgrades:

# Create configuration backup archive
sudo tar -czvf dns_config_backup_$(date +%F).tar.gz \
  /etc/nsswitch.conf \
  /etc/hosts \
  /etc/resolv.conf \
  /etc/systemd/resolved.conf \
  /etc/systemd/resolved.conf.d/

Recovery Procedures

If networking or resolution fails completely after an upgrade:

# Restore nsswitch.conf default
sudo cp /etc/nsswitch.conf.bak /etc/nsswitch.conf

# Re-link default systemd stub resolver
sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

# Restart networking services
sudo systemctl restart systemd-resolved

14. Automation and Integration

Ansible Playbook for Persistent DNS Configuration

---
- name: Configure Production Linux Resolver Settings
  hosts: all
  become: yes
  tasks:
    - name: Ensure systemd-resolved is installed and enabled
      package:
        name: systemd-resolved
        state: present

    - name: Create systemd-resolved drop-in directory
      file:
        path: /etc/systemd/resolved.conf.d
        state: directory
        owner: root
        group: root
        mode: '0755'

    - name: Deploy custom DNS configuration
      copy:
        dest: /etc/systemd/resolved.conf.d/custom_dns.conf
        content: |
          [Resolve]
          DNS=1.1.1.1 8.8.8.8
          FallbackDNS=9.9.9.9
          Domains=~.
          DNSOverTLS=opportunistic
          Cache=yes
        owner: root
        group: root
        mode: '0644'
      notify: Restart systemd-resolved

    - name: Ensure correct resolv.conf symlink
      file:
        src: /run/systemd/resolve/stub-resolv.conf
        dest: /etc/resolv.conf
        state: link
        force: yes

  handlers:
    - name: Restart systemd-resolved
      service:
        name: systemd-resolved
        state: restarted

15. Practical Labs and Exercises

Lab 1: Beginner — Tracing NSS vs. Direct DNS Lookups

  • Objective: Observe the functional difference between application-level lookup (getent) and direct network DNS query (dig).
  • Instructions:
  1. Add a dummy entry to /etc/hosts:
sudo sh -c 'echo "192.0.2.123 lab.test.local" >> /etc/hosts'
  1. Query lab.test.local using getent hosts:
getent hosts lab.test.local
  1. Query lab.test.local using dig:
dig lab.test.local
  1. Cleanup: Remove entry from /etc/hosts.
  • Expected Outcome: getent returns 192.0.2.123 via /etc/hosts. dig fails or returns NXDOMAIN because the fake host is not published on upstream network DNS nameservers.

Lab 2: Intermediate — Debugging an Unresponsive Primary Nameserver

  • Objective: Simulate a DNS timeout issue and diagnose latency using options and dig timing.
  • Instructions:
  1. Temporarily configure an unreachable primary nameserver in /etc/resolv.conf:
nameserver 192.0.2.1
nameserver 1.1.1.1
  1. Execute time getent hosts example.com.
  2. Add options timeout:1 attempts:1 to /etc/resolv.conf.
  3. Re-run time getent hosts example.com and measure latency reduction.
  • Expected Outcome: Initial request delays for 5 seconds waiting for 192.0.2.1 to time out. Adding timeout options reduces total failover delay to 1 second.

16. Quick Reference and Cheat Sheet

Important Configuration Files

File Primary Role
/etc/nsswitch.conf Controls name lookup database order (hosts: files resolve dns).
/etc/hosts Static IP-to-hostname mappings.
/etc/resolv.conf Dynamic or static resolver configuration.
/etc/systemd/resolved.conf Global configuration for systemd-resolved service.

Important Log Locations

Service / Tool Command / Log Path
systemd-resolved Journal journalctl -u systemd-resolved -f
Network Packet Trace sudo tcpdump -i any port 53 -n

Common Diagnostic Commands

Task Command
Application-level path lookup getent hosts <hostname>
Short DNS lookup answer dig +short <hostname>
Query specific nameserver dig @<nameserver_ip> <hostname>
Trace root delegation path dig +trace <hostname>
Inspect systemd-resolved status resolvectl status
Flush DNS cache sudo resolvectl flush-caches

Troubleshooting Quick Reference

Symptom Initial Diagnostic Command
dig works, app fails getent hosts <hostname> vs grep <hostname> /etc/hosts
Resolution slow/delays grep nameserver /etc/resolv.conf + test each with dig @IP
Changes lost on reboot ls -la /etc/resolv.conf (Check symlink target)
Unexplained SERVFAIL dig <hostname> +cdflag (Check DNSSEC failure)

Important Ports and Protocols

Port Protocol Purpose
53 UDP/TCP Standard Domain Name System (DNS) queries.
853 TCP DNS-over-TLS (DoT) encrypted query transport.
5353 UDP Multicast DNS (mDNS / Avahi / .local).

17. Further Reading and References

  1. hosts(5) Linux Manual Page: Format and rules for static host tables (man7.org/linux/man-pages/man5/hosts.5.html).
  2. resolv.conf(5) Linux Manual Page: Resolver configuration file specification (man7.org/linux/man-pages/man5/resolv.conf.5.html).
  3. nsswitch.conf(5) Linux Manual Page: Name Service Switch configuration (man7.org/linux/man-pages/man5/nsswitch.conf.5.html).
  4. systemd-resolved(8) System Manager Documentation: Systemd stub resolver manual (man7.org/linux/man-pages/man8/systemd-resolved.8.html).
  5. Cloudflare Learning DNS: Fundamentals of DNS record types, recursion, and root servers ([cloudflare.com/learning/dns](https://cloudflare.com/learning/dns)).