Domain 2Networking & RoutingCORE

(Jakarta) Debugging Glibc Hostname Resolution and NSS (It's Always DNS)

In-depth investigation and resolution of Linux hostname resolution failures caused by Name Service Switch (/etc/nsswitch.conf) misconfigurations.

Domain: Domain 2: Networking & Routing•Updated: 2026-11-11
#DNS#SadServers#nsswitch#glibc#systemd-resolved#Networking#Troubleshooting

1. Introduction

When network troubleshooting on Linux, “it’s always DNS” is a common industry phrase. However, DNS resolution failures do not always stem from broken nameservers or network routing issues. In many cases, the breakdown occurs deep within the GNU C Library (glibc) user-space resolution stack before a single packet ever leaves the network interface.

This article details the step-by-step investigation of the SadServers challenge titled “Jakarta”: it’s always DNS. In this scenario, standard diagnostic utilities like ping fail to resolve external domain names despite healthy underlying network resolvers.

By reading this guide, you will learn:

  • How Linux applications resolve hostnames through glibc and the Name Service Switch (/etc/nsswitch.conf).
  • The fundamental difference between tools that query DNS directly (like host or dig) and tools that rely on system resolution libraries (like ping or curl).
  • How to use getent, resolvectl, and strace to pinpoint user-space resolution failures.
  • How to remediate NSS configuration drift and prevent similar issues in production systems.

2. Problem Description and Environment

Scenario Details

  • Challenge: SadServers — “Jakarta”: it’s always DNS
  • Difficulty: Hard
  • Type: Fix
  • Tags: dns, unusual-tricky

Problem Statement

The local machine is unable to resolve public hostnames. Executing ping google.com fails immediately with an OS-level resolution error:

admin@i-0f031a34668bbf083:~$ ping google.com
ping: google.com: Name or service not known

Environmental Constraints

  • Operating System: Linux (Debian/Ubuntu family).
  • Network Restrictions: As noted in the SadServers challenge environment, VM instances on standard tiers cannot transmit ICMP packets outside the network. Therefore, an IP ping (ping 8.8.8.8) will show 100% packet loss.
  • Expected Outcome: ping google.com must successfully perform hostname lookup and output the resolved IP address (e.g., PING google.com (142.251.210.206) 56(84) bytes of data).

3. Initial Investigation and Hypotheses

Systematic troubleshooting requires separating network layer connectivity from host-level configuration files. When ping google.com returns Name or service not known, the failure occurs during the getaddrinfo() system library call.

Potential Causes Matrix

Hypothesis Supporting Evidence Evidence Against Diagnostic Step
1. Upstream DNS Server Down ping google.com fails. None yet. Query local/upstream nameserver directly using host or dig.
**2. Misconfigured /etc/resolv.conf** System cannot resolve names. Stub resolver (127.0.0.53) is present in /etc/resolv.conf. Inspect /etc/resolv.conf contents and symlink target.
3. systemd-resolved Inactive nameserver 127.0.0.53 unreachable. None yet. Check resolvectl status or service state.
4. NSS (/etc/nsswitch.conf) Misconfiguration Error appears instantaneously without network wait time. None yet. Inspect the hosts: line inside /etc/nsswitch.conf.

4. Troubleshooting Process

Step 1: Baseline Network & Hostname Checks

Objective: Verify whether the failure is limited to hostname resolution or affects raw IP routing.

Commands Executed:

ping 8.8.8.8
ping google.com

Output and Analysis:

admin@i-0f031a34668bbf083:~$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3055ms

admin@i-0f031a34668bbf083:~$ ping google.com
ping: google.com: Name or service not known

Decision: The packet loss on 8.8.8.8 aligns with the laboratory firewall constraints. However, ping google.com failed instantly with Name or service not known, proving that the system aborted before attempting to generate any network traffic.


Step 2: Inspecting Resolver & NSS Configuration Files

Objective: Inspect the OS-level configuration files governing database lookups and resolver targets.

Commands Executed:

cat /etc/nsswitch.conf
cat /etc/resolv.conf

Command Explanation:

  • /etc/nsswitch.conf controls the order of sources consulted by standard glibc database lookups (e.g., passwd, hosts, services).
  • /etc/resolv.conf provides nameserver IP addresses to DNS resolvers.

Output and Analysis:

admin@i-0f031a34668bbf083:~$ cat /etc/nsswitch.conf
# /etc/nsswitch.conf
passwd:         files
group:          files
shadow:         files
gshadow:        files

hosts:          files
networks:       files
...

admin@i-0f031a34668bbf083:~$ cat /etc/resolv.conf
# Managed by systemd-resolved
nameserver 127.0.0.53
options edns0 trust-ad
search .

Critical Finding: In /etc/nsswitch.conf, the hosts: entry was set strictly to files. This configured glibc to look only inside /etc/hosts. Because dns (or resolve) was missing, glibc applications completely ignored /etc/resolv.conf and never issued network queries.


Step 3: Direct Socket DNS Query (Bypassing Glibc)

Objective: Test whether the local DNS resolver service (systemd-resolved) and network DNS path are healthy by bypassing nsswitch.conf.

Commands Executed:

host google.com 127.0.0.53
resolvectl status

Command Explanation:

  • host builds raw DNS network packets directly to the specified target IP (127.0.0.53), completely ignoring /etc/nsswitch.conf.
  • resolvectl status inspects the state of upstream DNS servers associated with each network interface.

Output and Analysis:

admin@i-0f031a34668bbf083:~$ host google.com 127.0.0.53
Using domain server:
Name: 127.0.0.53
Address: 127.0.0.53#53

google.com has address 142.251.210.206
google.com has IPv6 address 2607:f8b0:4009:80d::200e
...

admin@i-0f031a34668bbf083:~$ resolvectl status
Global
         Protocols: +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 2 (ens5)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
Current DNS Server: 10.1.0.2
       DNS Servers: 10.1.0.2

Conclusion: systemd-resolved was running normally on 127.0.0.53, and the upstream DNS server 10.1.0.2 was fully reachable. The issue was isolated entirely to user-space host resolution policy in /etc/nsswitch.conf.


Step 4: Testing NSS Host Lookup via Glibc

Objective: Confirm that glibc is dropping host queries prior to configuration changes.

Command Executed:

getent hosts google.com

Output and Analysis:

admin@i-0f031a34668bbf083:~$ getent hosts google.com
admin@i-0f031a34668bbf083:~$ 

Conclusion: getent hosts returned an empty response with exit code 1. Since getent relies directly on glibc’s gethostbyname/getaddrinfo routines, this confirmed that the NSS layer was suppressing all non-/etc/hosts name resolutions.


5. Root Cause Analysis

The root cause was an incomplete hosts: rule inside /etc/nsswitch.conf.

Mechanism of Failure

                  ┌───────────────────────────────┐
                  │    ping / curl / sudo / git   │
                  └──────────────┬────────────────┘
                                 │
                                 ▼
                  ┌───────────────────────────────┐
                  │     glibc: getaddrinfo()      │
                  └──────────────┬────────────────┘
                                 │
                                 ▼
                  ┌───────────────────────────────┐
                  │      /etc/nsswitch.conf       │
                  │        "hosts: files"         │
                  └──────────────┬────────────────┘
                                 │
            ┌────────────────────┴────────────────────┐
            ▼                                         ▼
   1. Search /etc/hosts                     2. Query DNS / Resolvers
   [Result: NOT FOUND]                      [DISABLED BY NSS CONFIG]
            │
            ▼
   Return Error:
   "Name or service not known"
  1. Standard C programs call getaddrinfo() to translate domain names into IP addresses.
  2. glibc parses /etc/nsswitch.conf to determine search ordering for the hosts database.
  3. Because the configuration was truncated to hosts: files, glibc searched /etc/hosts and immediately returned EAI_NONAME (“Name or service not known”) without proceeding to network resolution modules.
  4. Direct DNS tools (host, dig, nslookup) bypass glibc NSS calls, explaining why host google.com 127.0.0.53 succeeded while ping google.com failed.

6. Solution and Remediation

To restore DNS lookup capabilities, /etc/nsswitch.conf must instruct glibc to query DNS if a hostname is missing from /etc/hosts.

Step 1: Backup Configuration

sudo cp /etc/nsswitch.conf /etc/nsswitch.conf.bak

(Note: During execution, sudo reported sudo: unable to resolve host i-0f031a34668bbf083. This occurred because sudo checks NSS host lookup to verify the local hostname, further validating our diagnosis.)

Step 2: Update /etc/nsswitch.conf

Option A: Standard Linux Configuration Edit /etc/nsswitch.conf using sudo nano /etc/nsswitch.conf or inline sed:

sudo sed -i 's/^hosts:.*/hosts:          files dns/' /etc/nsswitch.conf

Option B: Systemd-Resolved Native Configuration (Ubuntu Standard)

hosts:          files resolve [NOTFOUND=return] dns

Before & After Comparison

# /etc/nsswitch.conf

- hosts:          files
+ hosts:          files dns

7. Verification and Validation

1. Verification via getent

Execute getent to test system resolution via glibc:

admin@i-0f031a34668bbf083:~$ getent hosts google.com
2607:f8b0:4009:80d::200e google.com

getent successfully retrieved the IP address through the updated NSS configuration.

2. Verification via ping

Verify end-to-end lookup in network applications:

admin@i-0f031a34668bbf083:~$ ping google.com
PING google.com (142.251.210.206) 56(84) bytes of data.
^C
--- google.com ping statistics ---
12 packets transmitted, 0 received, 100% packet loss, time 11270ms

Interpretation: While ICMP packets were dropped by lab firewall rules, ping successfully resolved google.com to 142.251.210.206, confirming complete resolution of the issue.


8. Alternative Troubleshooting Approaches

When troubleshooting hostname lookup failures, several diagnostic techniques can quickly reveal system-level behavior:

1. System Call Tracing with strace

strace intercepts and records system calls made by a process.

strace -e trace=openat ping -c 1 google.com
  • What it reveals: You can observe which files the binary attempts to open. If ping opens /etc/nsswitch.conf and /etc/hosts but never opens /etc/resolv.conf or opens a socket, the issue is confirmed to be inside NSS.

2. Inspection with lsof or ss

Check if local stub resolvers are actively listening on port 53:

sudo ss -tulpn | grep :53
  • What it reveals: Confirms whether systemd-resolved (127.0.0.53:53) or dnsmasq is running and ready to accept queries on loopback interfaces.

Diagnostic Method Comparison

Diagnostic Tool Subsystem Tested Bypasses nsswitch.conf? Primary Use Case
getent hosts <name> glibc NSS layer No Testing system-wide OS name resolution.
host / dig Direct DNS Protocol Yes Verifying remote DNS server health & record types.
resolvectl query systemd-resolved D-Bus Yes Testing local resolver daemon routing.
strace Kernel System Calls N/A Pinpointing missing configuration files or permissions.

9. Linux Command Deep Dive

Command: getent

Purpose:

getent (Get Entries) fetches entries from GNU Name Service Switch (NSS) administrative databases (e.g., passwd, group, hosts, services, networks).

General Syntax:

getent [options] database [key ...]

Common Troubleshooting Options:

Syntax Description
getent hosts google.com Queries host database via glibc NSS ordering.
getent passwd root Queries user account database.
getent services ssh Resolves service name to port number.
getent -s dns hosts google.com Forces getent to use a specific NSS service module (dns).

Command: host

Purpose:

A lightweight utility for performing DNS lookups directly against nameservers.

General Syntax:

host [options] domain [nameserver]

Useful Troubleshooting Flags:

Syntax / Flag Purpose
host google.com 127.0.0.53 Directs query to local loopback resolver.
host -t MX google.com Queries specific DNS record types (MX, TXT, AAAA, NS).
host -v google.com Enables verbose output, showing raw DNS query/answer headers.
host -W 2 google.com Sets wait timeout to 2 seconds.

Command: resolvectl

Purpose:

Inspect and manage the systemd-resolved network name resolution service.

Common Subcommands:

Command Purpose
resolvectl status Shows global and per-interface DNS servers and settings.
resolvectl query google.com Sends query through systemd-resolved D-Bus API.
resolvectl flush-caches Flushes local DNS caches maintained by systemd-resolved.
resolvectl statistics Displays cache hit/miss statistics and DNSSEC validation status.

10. Underlying Linux Concepts and Architecture

The Name Service Switch (NSS) Architecture

The Name Service Switch is a modular facility in glibc that allows C library functions to obtain system database entries from diverse sources (local files, DNS, LDAP, NIS, D-Bus services).

When an application calls getaddrinfo("google.com", ...):

  1. glibc reads /etc/nsswitch.conf.
  2. It identifies the modules listed under hosts:.
  3. It loads the corresponding dynamic shared objects located in /lib/x86_64-linux-gnu/libnss_<module>.so.2.
  • files -> libnss_files.so.2 (reads /etc/hosts)
  • dns -> libnss_dns.so.2 (queries IP servers in /etc/resolv.conf)
  • resolve -> libnss_resolve.so.2 (talks to systemd-resolved via D-Bus)

Reaction Controls: [NOTFOUND=return]

NSS entries support conditional control actions in brackets:

hosts: files resolve [NOTFOUND=return] dns
  • [NOTFOUND=return]: If the resolve module returns “Host Not Found”, NSS immediately aborts and stops processing subsequent modules (dns).
  • If resolve is unavailable or times out, execution falls back to dns.

11. Real-World Sysadmin and SRE Applications

1. Detecting and Preventing Configuration Drift

In production environments, /etc/nsswitch.conf is rarely edited manually. However, misconfigured cloud-init scripts, container image base builds, or broken configuration management templates can overwrite this file.

Ansible Enforcement Example:

- name: Ensure nsswitch.conf has valid host resolution rules
  ansible.builtin.lineinfile:
    path: /etc/nsswitch.conf
    regexp: '^hosts:'
    line: 'hosts:          files resolve [NOTFOUND=return] dns'
    backup: yes

2. Operational Impact of sudo Hostname Warnings

During the fix, executing sudo threw the warning sudo: unable to resolve host i-0f031a34668bbf083.

  • Why it happens: When sudo executes, it attempts to resolve the system’s own hostname to check local aliases.
  • Production Risk: If NSS is broken, sudo delays every command execution by waiting for lookup timeouts, causing automated deployment scripts and health checks to hang or fail.

12. Further Learning and Experiments

Hands-on Lab Experiments

Experiment 1: Simulating NSS Failures

  1. In an isolated test VM, edit /etc/nsswitch.conf and set hosts: files.
  2. Compare the output of getent hosts google.com vs host google.com.
  3. Restore /etc/nsswitch.conf to hosts: files dns.

Experiment 2: Tracing System Calls

  1. Run strace -f -e trace=openat,connect ping -c 1 google.com.
  2. Locate the system call opening /etc/nsswitch.conf and follow how glibc opens dependent shared library files (/lib/x86_64-linux-gnu/libnss_dns.so.2).

Documentation and References

  • man 5 nsswitch.conf — Official manual page for Name Service Switch configuration.
  • man 3 getaddrinfo — POSIX network address translation API documentation.
  • man 8 systemd-resolved — Systemd network name resolution manager manual.

13. Key Takeaways

  1. Direct DNS Utilities vs. System Utilities: Tools like host, dig, and nslookup bypass /etc/nsswitch.conf. Never rely on dig alone to verify system-wide application connectivity.
  2. getent is the Definitive OS Test: Use getent hosts <domain> to test whether applications using standard C libraries (curl, ping, python, ssh) can resolve hostnames.
  3. Check NSS First on Sudden Resolution Errors: If domain resolution fails instantaneously without network delays, inspect /etc/nsswitch.conf for missing dns or resolve entries.

14. Quick Troubleshooting Reference / Runbook

                  Linux Hostname Resolution Failures
                                   │
       1. Run: getent hosts google.com
          │
          ├──> [Success] ──> Network DNS path is working. Issue is application-specific.
          │
          └──> [Fails / Empty]
                   │
       2. Run: host google.com 127.0.0.53
          │
          ├──> [Success] ──> Resolver is healthy! Issue is USER-SPACE (NSS).
          │                  Check /etc/nsswitch.conf for "hosts: files dns".
          │
          └──> [Fails] ───> Issue is RESOLVER / NETWORK.
                             a. Check /etc/resolv.conf settings.
                             b. Check systemctl status systemd-resolved.
                             c. Check outbound firewall port 53 (UDP/TCP).