1. Introduction
When network troubleshooting on Linux, “it’s always DNS” is a common industry phrase. However, DNS resolution failures do not always stem from broken nameservers or network routing issues. In many cases, the breakdown occurs deep within the GNU C Library (glibc) user-space resolution stack before a single packet ever leaves the network interface.
This article details the step-by-step investigation of the SadServers challenge titled “Jakarta”: it’s always DNS. In this scenario, standard diagnostic utilities like ping fail to resolve external domain names despite healthy underlying network resolvers.
By reading this guide, you will learn:
- How Linux applications resolve hostnames through
glibcand the Name Service Switch (/etc/nsswitch.conf). - The fundamental difference between tools that query DNS directly (like
hostordig) and tools that rely on system resolution libraries (likepingorcurl). - How to use
getent,resolvectl, andstraceto pinpoint user-space resolution failures. - How to remediate NSS configuration drift and prevent similar issues in production systems.
2. Problem Description and Environment
Scenario Details
- Challenge: SadServers — “Jakarta”: it’s always DNS
- Difficulty: Hard
- Type: Fix
- Tags:
dns,unusual-tricky
Problem Statement
The local machine is unable to resolve public hostnames. Executing ping google.com fails immediately with an OS-level resolution error:
admin@i-0f031a34668bbf083:~$ ping google.com
ping: google.com: Name or service not known
Environmental Constraints
- Operating System: Linux (Debian/Ubuntu family).
- Network Restrictions: As noted in the SadServers challenge environment, VM instances on standard tiers cannot transmit ICMP packets outside the network. Therefore, an IP ping (
ping 8.8.8.8) will show 100% packet loss. - Expected Outcome:
ping google.commust successfully perform hostname lookup and output the resolved IP address (e.g.,PING google.com (142.251.210.206) 56(84) bytes of data).
3. Initial Investigation and Hypotheses
Systematic troubleshooting requires separating network layer connectivity from host-level configuration files. When ping google.com returns Name or service not known, the failure occurs during the getaddrinfo() system library call.
Potential Causes Matrix
| Hypothesis | Supporting Evidence | Evidence Against | Diagnostic Step |
|---|---|---|---|
| 1. Upstream DNS Server Down | ping google.com fails. |
None yet. | Query local/upstream nameserver directly using host or dig. |
**2. Misconfigured /etc/resolv.conf** |
System cannot resolve names. | Stub resolver (127.0.0.53) is present in /etc/resolv.conf. |
Inspect /etc/resolv.conf contents and symlink target. |
3. systemd-resolved Inactive |
nameserver 127.0.0.53 unreachable. |
None yet. | Check resolvectl status or service state. |
4. NSS (/etc/nsswitch.conf) Misconfiguration |
Error appears instantaneously without network wait time. | None yet. | Inspect the hosts: line inside /etc/nsswitch.conf. |
4. Troubleshooting Process
Step 1: Baseline Network & Hostname Checks
Objective: Verify whether the failure is limited to hostname resolution or affects raw IP routing.
Commands Executed:
ping 8.8.8.8
ping google.com
Output and Analysis:
admin@i-0f031a34668bbf083:~$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3055ms
admin@i-0f031a34668bbf083:~$ ping google.com
ping: google.com: Name or service not known
Decision: The packet loss on 8.8.8.8 aligns with the laboratory firewall constraints. However, ping google.com failed instantly with Name or service not known, proving that the system aborted before attempting to generate any network traffic.
Step 2: Inspecting Resolver & NSS Configuration Files
Objective: Inspect the OS-level configuration files governing database lookups and resolver targets.
Commands Executed:
cat /etc/nsswitch.conf
cat /etc/resolv.conf
Command Explanation:
/etc/nsswitch.confcontrols the order of sources consulted by standardglibcdatabase lookups (e.g.,passwd,hosts,services)./etc/resolv.confprovides nameserver IP addresses to DNS resolvers.
Output and Analysis:
admin@i-0f031a34668bbf083:~$ cat /etc/nsswitch.conf
# /etc/nsswitch.conf
passwd: files
group: files
shadow: files
gshadow: files
hosts: files
networks: files
...
admin@i-0f031a34668bbf083:~$ cat /etc/resolv.conf
# Managed by systemd-resolved
nameserver 127.0.0.53
options edns0 trust-ad
search .
Critical Finding: In /etc/nsswitch.conf, the hosts: entry was set strictly to files. This configured glibc to look only inside /etc/hosts. Because dns (or resolve) was missing, glibc applications completely ignored /etc/resolv.conf and never issued network queries.
Step 3: Direct Socket DNS Query (Bypassing Glibc)
Objective: Test whether the local DNS resolver service (systemd-resolved) and network DNS path are healthy by bypassing nsswitch.conf.
Commands Executed:
host google.com 127.0.0.53
resolvectl status
Command Explanation:
hostbuilds raw DNS network packets directly to the specified target IP (127.0.0.53), completely ignoring/etc/nsswitch.conf.resolvectl statusinspects the state of upstream DNS servers associated with each network interface.
Output and Analysis:
admin@i-0f031a34668bbf083:~$ host google.com 127.0.0.53
Using domain server:
Name: 127.0.0.53
Address: 127.0.0.53#53
google.com has address 142.251.210.206
google.com has IPv6 address 2607:f8b0:4009:80d::200e
...
admin@i-0f031a34668bbf083:~$ resolvectl status
Global
Protocols: +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 2 (ens5)
Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
Current DNS Server: 10.1.0.2
DNS Servers: 10.1.0.2
Conclusion: systemd-resolved was running normally on 127.0.0.53, and the upstream DNS server 10.1.0.2 was fully reachable. The issue was isolated entirely to user-space host resolution policy in /etc/nsswitch.conf.
Step 4: Testing NSS Host Lookup via Glibc
Objective: Confirm that glibc is dropping host queries prior to configuration changes.
Command Executed:
getent hosts google.com
Output and Analysis:
admin@i-0f031a34668bbf083:~$ getent hosts google.com
admin@i-0f031a34668bbf083:~$
Conclusion: getent hosts returned an empty response with exit code 1. Since getent relies directly on glibc’s gethostbyname/getaddrinfo routines, this confirmed that the NSS layer was suppressing all non-/etc/hosts name resolutions.
5. Root Cause Analysis
The root cause was an incomplete hosts: rule inside /etc/nsswitch.conf.
Mechanism of Failure
┌───────────────────────────────┐
│ ping / curl / sudo / git │
└──────────────┬────────────────┘
│
▼
┌───────────────────────────────┐
│ glibc: getaddrinfo() │
└──────────────┬────────────────┘
│
▼
┌───────────────────────────────┐
│ /etc/nsswitch.conf │
│ "hosts: files" │
└──────────────┬────────────────┘
│
┌────────────────────┴────────────────────┐
▼ ▼
1. Search /etc/hosts 2. Query DNS / Resolvers
[Result: NOT FOUND] [DISABLED BY NSS CONFIG]
│
▼
Return Error:
"Name or service not known"
- Standard C programs call
getaddrinfo()to translate domain names into IP addresses. glibcparses/etc/nsswitch.confto determine search ordering for thehostsdatabase.- Because the configuration was truncated to
hosts: files,glibcsearched/etc/hostsand immediately returnedEAI_NONAME(“Name or service not known”) without proceeding to network resolution modules. - Direct DNS tools (
host,dig,nslookup) bypassglibcNSS calls, explaining whyhost google.com 127.0.0.53succeeded whileping google.comfailed.
6. Solution and Remediation
To restore DNS lookup capabilities, /etc/nsswitch.conf must instruct glibc to query DNS if a hostname is missing from /etc/hosts.
Step 1: Backup Configuration
sudo cp /etc/nsswitch.conf /etc/nsswitch.conf.bak
(Note: During execution, sudo reported sudo: unable to resolve host i-0f031a34668bbf083. This occurred because sudo checks NSS host lookup to verify the local hostname, further validating our diagnosis.)
Step 2: Update /etc/nsswitch.conf
Option A: Standard Linux Configuration
Edit /etc/nsswitch.conf using sudo nano /etc/nsswitch.conf or inline sed:
sudo sed -i 's/^hosts:.*/hosts: files dns/' /etc/nsswitch.conf
Option B: Systemd-Resolved Native Configuration (Ubuntu Standard)
hosts: files resolve [NOTFOUND=return] dns
Before & After Comparison
# /etc/nsswitch.conf
- hosts: files
+ hosts: files dns
7. Verification and Validation
1. Verification via getent
Execute getent to test system resolution via glibc:
admin@i-0f031a34668bbf083:~$ getent hosts google.com
2607:f8b0:4009:80d::200e google.com
getent successfully retrieved the IP address through the updated NSS configuration.
2. Verification via ping
Verify end-to-end lookup in network applications:
admin@i-0f031a34668bbf083:~$ ping google.com
PING google.com (142.251.210.206) 56(84) bytes of data.
^C
--- google.com ping statistics ---
12 packets transmitted, 0 received, 100% packet loss, time 11270ms
Interpretation: While ICMP packets were dropped by lab firewall rules, ping successfully resolved google.com to 142.251.210.206, confirming complete resolution of the issue.
8. Alternative Troubleshooting Approaches
When troubleshooting hostname lookup failures, several diagnostic techniques can quickly reveal system-level behavior:
1. System Call Tracing with strace
strace intercepts and records system calls made by a process.
strace -e trace=openat ping -c 1 google.com
- What it reveals: You can observe which files the binary attempts to open. If
pingopens/etc/nsswitch.confand/etc/hostsbut never opens/etc/resolv.confor opens a socket, the issue is confirmed to be inside NSS.
2. Inspection with lsof or ss
Check if local stub resolvers are actively listening on port 53:
sudo ss -tulpn | grep :53
- What it reveals: Confirms whether
systemd-resolved(127.0.0.53:53) ordnsmasqis running and ready to accept queries on loopback interfaces.
Diagnostic Method Comparison
| Diagnostic Tool | Subsystem Tested | Bypasses nsswitch.conf? |
Primary Use Case |
|---|---|---|---|
getent hosts <name> |
glibc NSS layer |
No | Testing system-wide OS name resolution. |
host / dig |
Direct DNS Protocol | Yes | Verifying remote DNS server health & record types. |
resolvectl query |
systemd-resolved D-Bus |
Yes | Testing local resolver daemon routing. |
strace |
Kernel System Calls | N/A | Pinpointing missing configuration files or permissions. |
9. Linux Command Deep Dive
Command: getent
Purpose:
getent (Get Entries) fetches entries from GNU Name Service Switch (NSS) administrative databases (e.g., passwd, group, hosts, services, networks).
General Syntax:
getent [options] database [key ...]
Common Troubleshooting Options:
| Syntax | Description |
|---|---|
getent hosts google.com |
Queries host database via glibc NSS ordering. |
getent passwd root |
Queries user account database. |
getent services ssh |
Resolves service name to port number. |
getent -s dns hosts google.com |
Forces getent to use a specific NSS service module (dns). |
Command: host
Purpose:
A lightweight utility for performing DNS lookups directly against nameservers.
General Syntax:
host [options] domain [nameserver]
Useful Troubleshooting Flags:
| Syntax / Flag | Purpose |
|---|---|
host google.com 127.0.0.53 |
Directs query to local loopback resolver. |
host -t MX google.com |
Queries specific DNS record types (MX, TXT, AAAA, NS). |
host -v google.com |
Enables verbose output, showing raw DNS query/answer headers. |
host -W 2 google.com |
Sets wait timeout to 2 seconds. |
Command: resolvectl
Purpose:
Inspect and manage the systemd-resolved network name resolution service.
Common Subcommands:
| Command | Purpose |
|---|---|
resolvectl status |
Shows global and per-interface DNS servers and settings. |
resolvectl query google.com |
Sends query through systemd-resolved D-Bus API. |
resolvectl flush-caches |
Flushes local DNS caches maintained by systemd-resolved. |
resolvectl statistics |
Displays cache hit/miss statistics and DNSSEC validation status. |
10. Underlying Linux Concepts and Architecture
The Name Service Switch (NSS) Architecture
The Name Service Switch is a modular facility in glibc that allows C library functions to obtain system database entries from diverse sources (local files, DNS, LDAP, NIS, D-Bus services).
When an application calls getaddrinfo("google.com", ...):
glibcreads/etc/nsswitch.conf.- It identifies the modules listed under
hosts:. - It loads the corresponding dynamic shared objects located in
/lib/x86_64-linux-gnu/libnss_<module>.so.2.
files->libnss_files.so.2(reads/etc/hosts)dns->libnss_dns.so.2(queries IP servers in/etc/resolv.conf)resolve->libnss_resolve.so.2(talks tosystemd-resolvedvia D-Bus)
Reaction Controls: [NOTFOUND=return]
NSS entries support conditional control actions in brackets:
hosts: files resolve [NOTFOUND=return] dns
[NOTFOUND=return]: If theresolvemodule returns “Host Not Found”, NSS immediately aborts and stops processing subsequent modules (dns).- If
resolveis unavailable or times out, execution falls back todns.
11. Real-World Sysadmin and SRE Applications
1. Detecting and Preventing Configuration Drift
In production environments, /etc/nsswitch.conf is rarely edited manually. However, misconfigured cloud-init scripts, container image base builds, or broken configuration management templates can overwrite this file.
Ansible Enforcement Example:
- name: Ensure nsswitch.conf has valid host resolution rules
ansible.builtin.lineinfile:
path: /etc/nsswitch.conf
regexp: '^hosts:'
line: 'hosts: files resolve [NOTFOUND=return] dns'
backup: yes
2. Operational Impact of sudo Hostname Warnings
During the fix, executing sudo threw the warning sudo: unable to resolve host i-0f031a34668bbf083.
- Why it happens: When
sudoexecutes, it attempts to resolve the system’s own hostname to check local aliases. - Production Risk: If NSS is broken,
sudodelays every command execution by waiting for lookup timeouts, causing automated deployment scripts and health checks to hang or fail.
12. Further Learning and Experiments
Hands-on Lab Experiments
Experiment 1: Simulating NSS Failures
- In an isolated test VM, edit
/etc/nsswitch.confand sethosts: files. - Compare the output of
getent hosts google.comvshost google.com. - Restore
/etc/nsswitch.conftohosts: files dns.
Experiment 2: Tracing System Calls
- Run
strace -f -e trace=openat,connect ping -c 1 google.com. - Locate the system call opening
/etc/nsswitch.confand follow howglibcopens dependent shared library files (/lib/x86_64-linux-gnu/libnss_dns.so.2).
Documentation and References
man 5 nsswitch.conf— Official manual page for Name Service Switch configuration.man 3 getaddrinfo— POSIX network address translation API documentation.man 8 systemd-resolved— Systemd network name resolution manager manual.
13. Key Takeaways
- Direct DNS Utilities vs. System Utilities: Tools like
host,dig, andnslookupbypass/etc/nsswitch.conf. Never rely ondigalone to verify system-wide application connectivity. getentis the Definitive OS Test: Usegetent hosts <domain>to test whether applications using standard C libraries (curl,ping,python,ssh) can resolve hostnames.- Check NSS First on Sudden Resolution Errors: If domain resolution fails instantaneously without network delays, inspect
/etc/nsswitch.conffor missingdnsorresolveentries.
14. Quick Troubleshooting Reference / Runbook
Linux Hostname Resolution Failures
│
1. Run: getent hosts google.com
│
├──> [Success] ──> Network DNS path is working. Issue is application-specific.
│
└──> [Fails / Empty]
│
2. Run: host google.com 127.0.0.53
│
├──> [Success] ──> Resolver is healthy! Issue is USER-SPACE (NSS).
│ Check /etc/nsswitch.conf for "hosts: files dns".
│
└──> [Fails] ───> Issue is RESOLVER / NETWORK.
a. Check /etc/resolv.conf settings.
b. Check systemctl status systemd-resolved.
c. Check outbound firewall port 53 (UDP/TCP).