GNU/Linux host name resolution

This post describes different software components involved in host name resolutions and DNS configuration on GNU/Linux systems. It consists of a diagram and some accompanying explanations. The goal is to give some pointers and references to understand how to troubleshoot host name/DNS resolution problems and configuration problems on GNU/Linux systems.

Table of Content

Diagram

Here is a diagram of how host name resolution works on GNU/Linux systems:

Diagram

😅

Host name resolution by the C library

Most programs call functions from the C library (libc) for host name resolution:

  • getaddrinfo() is used to resolve host names to IP addresses.
  • getnameinfo() is used to resolve IP addresses to host names.

Other deprecated functions such as gethostbyname*() and gethostbyaddr*() can be used as well.

Host Aliases

The libc starts the host name resolution by trying to resolve the requested host name as a host alias. This step is optional: if the HOSTALIASES environment variable is set, it is considered to be the path to a file containing host aliases definitions. If the libc finds the requested name in the host alias file, the aliased name is used instead of the alias for the following steps.

A host alias file might look like:

example www.example.com

Note: the alias cannot contain any dot.

Example of usage of host aliases

$ echo 'example www.example.com' > ./host_aliases
$ HOSTALIASES="./host_aliases" getent hosts example
93.184.216.34   www.example.com

NSS

The GNU libc uses a system of plugins, Name Service Switch (NSS), for host name resolution10. The GNU libc finds a list of NSS plug-in modules to use in the /etc/nsswitch.conf file.

The traditional/minimal plug-in module list is:

hosts: files dns

In this example,

  1. the libc first attempts to use the files module, which uses static name/address mappings defined in the /etc/hosts file;
  2. if this fails, the libc attempts to use the dns module, which queries the resolvers listed in /etc/resolv.conf using the DNS protocol.

A more featureful plug-in module list might look like:

hosts: files mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] dns myhostname

Each plug-in module is implemented as a shared library. This shared library is dynamically loaded (dlopen()) by the libc. For example, the foo module would be implemented by a libnss_foo.so.2 shared object8.

List of some NSS plug-in modules for host name resolution

Package Plug-in module Description
glibc files Static address/name mappings defined in /etc/hosts
glibc dns DNS resolution (based on /etc/resolv.conf)
Avahi mdns_minimal mDNS resolution for foo.local using Avahi
Avahi mdns4_minimal mDNS resolution for foo.local using Avahi (IPv4 only)
Avahi mdns6_minimal mDNS resolution for foo.local using Avahi (IPv6 only)
Avahi mdns
Avahi mdns4
Avahi mdns6
systemd resolve delegate to the systemd-resolved daemon over D-Bus
systemd myhostname resolve current host name to host address, etc.
systemd mymachines resolve machines of local Virtual Machines (VMs) registered with systemd-machined
lwresd lwres delegate to a local lwresd (caching resolver)

Static associations (files)

The files plug-in module implements static name/address mappings. These static mappings are defined in the /etc/hosts file.

A typical value for this file is:

127.0.0.1   localhost
127.0.1.1   my-host-name

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouter

mDNS with Avahi (mdns[4,6]_minimal)

The mdns_minimal plug-in module implements Multicast DNS (mDNS) name resolution. It only resolves names in the local domain (eg. foo.local): these names are expected to be announced by machines on the local network using mDNS.

The mdns4_minimal and mdns6_minimal modules are variants which only work for IPv4 and IPv6 respectively.

We can test mDNS resolution through Avahi with:

avahi-resolve-host-name -4 marvin.local
avahi-resolve-host-name -6 marvin.local

Protocol used for communicating with Avahi

These modules delegate to the Avahi3 system daemon. They communicate over the /run/avahi-daemon/socket Unix socket using a custom line-oriented protocol:

$ echo 'RESOLVE-HOSTNAME foo.local' | socat STDIO UNIX:/run/avahi-daemon/socket
+ 5 0 foo.local 192.0.2.42

systemd-resolved (resolve)

The resolve plug-in module delegates host name resolution to a local systemd-resolved daemon. The features of systemd-resolved are explained afterwards but include: mDNS, LLMNR, caching, etc.

The plugin and the systemd-resolved daemon communicate over D-Bus. They use the ResolveHostname() method of the org.freedesktop.resolve1.Manager interface:

node /org/freedesktop/resolve1 {
  interface org.freedesktop.resolve1.Manager {
    methods:
      ResolveHostname(in  i ifindex,
                      in  s name,
                      in  i family,
                      in  t flags,
                      out a(iiay) addresses,
                      out s canonical,
                      out t flags);
    // ...
  }
}

We can test host name resolution through systemd-resolved with:

resolvectl -4 query www.example.com
resolvectl -6 query www.example.com
systemd-resolve -4 www.example.com
systemd-resolve -6 www.example.com

Note: If for some reason, a program cannot communicate to systemd-resolved, through D-Bus (for example because it does not use NSS), it can still make queries to systemd-resolved using the DNS protocol. systemd-resolved listens on 127.0.0.53 (TCP and UDP ports 53). When systemd-resolved is enabled, this IP address is usually configured in /etc/resolv.conf; therefore, the dns module or any other software relying on /etc/resolv.conf will end up communicating to systemd-resolved using the standard DNS protocol.

lwresd cache (lwres)

The lwres plug-in module implements system-wide caching for DNS resolutions. It communicates with an associated system daemon (lwresd). This daemon is a caching daemon (based on BIND) which forwards the queries to the resolvers defined in the /etc/resolv.conf file (not through NSS) and caches the answers.

DNS based host name resolution (dns)

The dns plug-in module implements host name resolution using the standard DNS protol. This plug-in module uses the /etc/resolv.conf file to find a list of resolvers to communicate to. These resolvers might be local resolvers such as dnsmasq, unbound or systemd-resolved or remote resolvers.

myhostname

The myhostname plug-in module resolves the name of the machine and localhost. These are traditionnaly handled by the files module.

In addition, this module handles:

  • any hostname in the localhost domain (eg. foobar.localhost) as mandated by RFC6761;
  • _gateway as the IP addresses of the default gateways.

All these are also resolved by systemd-revolved. The myhostname module is supposed to be useful as a fallback if systemd-resolved is unavailable.

NSCD

The GNU C library includes builtin support for an optional daemon, the Name Service Cache Daemon (NSCD), for caching all NSS requests (not only host names resolutions). When enabled, it listens on the /run/nscd/socket Unix socket.

This features is not implemented as a NSS plug-in module. The GNU C library always tries to contact this service for NSS requests before going through the NSS plug-in modules.

The NSCD daemon itself uses the NSS plug-in modules declared in /etc/nsswitch.conf.

This daemon is usually not installed and is apparently not very stable.

Managing resolv.conf files

The /etc/resolv.conf file defines which resolvers are used by the dns NSS module (with the nameserver directive).

It can contain:

  • IP addresses of resolverss to use (nameserver directive);
  • domain names to use for resolving dotless host names (search directive);
  • misc. options (options directive) such as,
    • enabling EDNS0 (options edns0),
    • forcing DNS-over-TDP instead of DNS-over-UDP (options vc)9;
  • ranges of IP addresses to use in priority when contacting resolved hosts (sortlist directive).

Example of revolv.conf file:

nameserver 10.0.0.42
nameserver 10.0.1.42

The programs which manage the network configuration (DHCP clients, VPN programs) usually want to reconfigure this file somehow.

The resolvconf interface

In order to avoid inconsistent state of the /etc/resolv.conf file resulting from different programs trying to configure it, the resolvconf program can be used. The idea is that different programs can register domain name configurations associated to different network interfaces and resolvconf merges them in a a resolv.conf file.

When resolvconf is installed, /etc/resolv.conf is typically a symlink to /run/resolvconf/resolv.conf:

  • resolvconf manages /run/resolvconf/resolv.conf directly;
  • /etc/resolv.conf is either a symlink to /run/resolvconf/resolv.conf or to another file (if some local resolver suh as unbound is running).

A component of the system can register a DNS configuration for an interface by sending some resolv.conf directives to the standard input of a resolvconf -a IFACE command:

resolvconf -a eth0.dhclient << EOF
nameserver 10.0.0.42
nameserver 10.0.1.42
EOF

The configuration for an interface is unregistered with:

resolvconf -d eth0.dhclient

Many programs can use this interface (usually through hooks):

  • DHCP clients (eg. dhclient) can configure the resolvers received by DHCP.
  • PPP daemons (eg. pppd) can configures resolvers advertised by the peer.
  • ifup/ifdown can configure the resolvers which are declared statically in interfaces files.
  • OpenVPN when used as a VPN client can configure resolvers advertised by the VPN server.
  • NetworkManager can configure statically or dynamically configured resolvers.

See below for some details about this.

There are several (at least three) implementations of this program:

  • traditional resolvconf;
  • openresolv;
  • systemd-resolve.

Traditional resolvconf

The /etc/resolvconf/interface-order file defines a priority of interfaces. Interfaces listed first in this file will have higher priority. Their DNS configuration (eg. DNS servers) will have precedence over the other ones. This ensures (for example) that resolvers configured from VPN interfaces (tun*, tap*) override resolvers configured from Ethernet or Wifi interfaces.

Default interface order (Debian)

  # interface-order(5)
  lo.inet6
  lo.inet
  lo.@(dnsmasq|pdnsd)
  lo.!(pdns|pdns-recursor)
  lo
  tun*
  tap*
  hso*
  em+([0-9])?(_+([0-9]))*
  p+([0-9])p+([0-9])?(_+([0-9]))*
  @(br|eth)*([^.]).inet6
  @(br|eth)*([^.]).ip6.@(dhclient|dhcpcd|pump|udhcpc)
  @(br|eth)*([^.]).inet
  @(br|eth)*([^.]).@(dhclient|dhcpcd|pump|udhcpc)
  @(br|eth)*
  @(ath|wifi|wlan)*([^.]).inet6
  @(ath|wifi|wlan)*([^.]).ip6.@(dhclient|dhcpcd|pump|udhcpc)
  @(ath|wifi|wlan)*([^.]).inet
  @(ath|wifi|wlan)*([^.]).@(dhclient|dhcpcd|pump|udhcpc)
  @(ath|wifi|wlan)*
  ppp*
  *
  

Some packages include a script (in /etc/resolvconf/update-libc.d/) to be notified of DNS configuration updates through resolvconf.

List of resolvconf hooks included in Debian

  • /etc/resolvconf/update-libc.d/avahi-daemon
  • /etc/resolvconf/update-libc.d/fetchmail
  • /etc/resolvconf/update-libc.d/postfix
  • /etc/resolvconf/update-libc.d/sendmail
  • /etc/resolvconf/update-libc.d/squid
  • /etc/resolvconf/update-libc.d/squid-deb-prox

openresolv

Openresolv is an alternative implementation of resolvconf with additional features.

In addition to configuring resolv.conf, it has support for configuring local resolvers instead: dnsmasq, named, PowerDNS Recursor, pdnsd and unbound are supported.

It has additional command-line options.

Interesting Openresolv command-line options

The metric option (-m) can be used to tweak the precedence of interfaces.

When the exclusive (-x) flag is used, the configuration of the interface override all other configurations instead of being stacked (merged) on top of them.

When the private flag (-p) is used, the resolvers configured for this interface are only used for querying host names below the domain associated with this interface. This does not work when configuring revolv.conf but only when configuring a local resolver which supports it.

Openresolv configuration file

It has a dedicated configuration file, /etc/resolvconf.conf. Of special interest are these configuration options:

  • interface_order and dynamic_order for tweaking priority;
  • inclusive_interfaces for tweaking -x;
  • private_interfaces and public_interfaces for tweaking -p;
  • search_domains, search_domains_append and domain_blacklist for tweaking search domains;
  • name_servers_append, name_server_blacklist and prepend_nameservers for tweaking name servers.

resolvectl working as resolvconf

resolvectl implements the resolvconf interface as well (when called as resolvconf). Instead of directly configuring /run/resolvconf/resolv.conf, it configures systemd-resolved (see below).

Note: on my Debian systems there is no symlink from resolvconf to resolvectl.

Command-line flags

This implementation has (partial) support for the exclusive flag (-x).

It could probably support -p as well because systemd-resolved has support for this concept.

systemd-resolved

The systemd-resolved daemon is a client for:

It additionnaly implements:

It can be queried:

  • over D-Bus;
  • over DNS (it binds to 127.0.0.53:53 on UDP and TCP).

When systemd-resolved is used, /etc/resolv.conf is usually4 a symlink to /run/systemd/resolve/stub-resolv.conf which will look like:

nameserver 127.0.0.53
options edns0

This ensures that:

  • any client relying on /etc/resolv.conf will use systemd-resolved as a resolver;
  • any client which cannot used the resolve plug-in module will still communicate to systemd-resolved5.

Network managers

ifupdown

Both openresolv and resolvconf provide hooks for configuring resolv.conf based on static DNS configuration from /etc/network/interfaces and /etc/network/interfaces.d/*.

The hooks shipping with resolvconf handle the following directives (which set the corrresponding resolv.conf directives):

  • dns-domain
  • dns-search
  • dns-sortlist
  • dns-nameservers

Example of interface definition

iface vlan10 inet static
  address 10.0.0.0/24
  network 10.10.0.0
  broadcast 10.0.0.255
  dns-nameservers 10.0.0.42

References

PPP

pppd can configure DNS resolution through resolvconf using hooks.

References

DHCP clients

dhclient can configure DNS resolution through resolvconf using hooks.

References

OpenVPN

An OpenVPN server can send DNS configuration to its clients (eg. with --dhcp-option DNS 10.0.0.42).

On Windows, the OpenVPN client can use this DNS configuration to change the DNS configuration on the system. On other systems, there is not builtin support for changing the system DNS configuration. Instead, the OpenVPN client must delegate DNS configuration to a hook7.

The hook must be explicitly enabled:

  • either as command-line argument when calling OpenVPN;
  • or in the OpenVPN configuration file.

Two such scripts are available on Debian:

Configuring OpenVPN hooks as command-line argument

openvpn $other_arguments \
  --script-security 2 \
  --up /etc/openvpn/script/update-resolv-conf \
  --down /etc/openvpn/script/update-resolv-conf \
  --down-pre

Configuring OpenVPN hooks in the configuration file

# [...]
script-security 2
up /etc/openvpn/script/update-resolv-conf
down /etc/openvpn/script/update-resolv-conf
down-pre

Directives handled by update-resolv-conf

  • DNS
  • DOMAIN-SEARCH/DOMAIN

Directives handled by update-resolv-resolved

See README:

  • DNS/DNS6, resolver
  • DOMAIN-ROUTE, use the resolvers configured for this interface to resolve the names under these domains
  • DOMAIN-SEARCH, use the resolvers configured for this interface to resolve the names under these domains and add them to the search list
  • DOMAIN/ADAPTER_DOMAIN_SUFFIX, similar to DOMAIN_SEARCH
  • DNSSEC, control whether DNSSEC should be enabled for these resolvers

NetworkManager

NetworkManager can configure DNS resolution using:

  • either static configuration (for a given network connection);
  • or dynamic configuration (DHCP, etc.).

There are several mode of operations based on its configuration (depending on the dns and rc-manager directives):

  • configure resolv.conf directly;
  • use resolvconf;
  • use netconfig;
  • configure dnsmasq;
  • configure systemd-resolved;
  • configure unbound;

When using the resolvconf method, NetworkManager registers its DNS configuration to resolvconf using a dummy network interface called NetworkManager.

Programs which do not use NSS for name resolution

This section contain a (non-exhaustive) list of programs which do not (always) rely on the libc functions (and hence NSS plug-in modules) for host name resolution.

Firefox can be configured to do DNS over HTTPS (DoH). When enabled (configured with network.trr.mode), DNS resolution does not go through the libc, /etc/resolv.conf, etc. In this mode, Firefox uses a DNS-over-HTTPS resolver coming from its configuration (configured with network.trr.custom_uri).

Chrome/Chromium has experimental support for DNS-over-HTTPS (see chrome://flags/#dns-over-https). Currently (2020-04-15), it is listed as available on Mac, Windows, Chrome OS and Android only.

DNS clients such as dig do not use the libc for name resolution but make direct DNS requests (based on /etc/resolv.conf).

Programs linked statically against the GNU libc still load NSS the plug-in modules dynamically. It is apparently possible to disable this feature: in this case, only the files and nss modules will be included (statically) in the binary.

Typical problems

Missing NSS modules on multi-arch

On a multi-arch system, if the NSS plug-in modules are not installed for all architectures, the behavior will not be consistent depending on the architecture of the program.

On Debian-based systems, we can check which NSS modules are installed with:

aptitude search '^libnss-!dbgsym~i'

Resolver not updated when using OpenVPN

You might need to enable a hook for applying the DNS configuration pulled from the OpenVPN server.

Resolver not answering

This might happen if you are contacting the resolver from the wrong network interface (using the wrong route, from the wrong source IP address).

Most recursive resolvers are not opened to the internet (they are not open resolvers) but are only available from a restricted set of IP addresses: you usally can only use the recursive resolver of the FooBar ISP from the FooBar network.

In you are using a VPN, two variants of this problem might happen:

  • you are trying to use resolver of your ISP from an IP address of your VPN;
  • you trying to use the resolver of your VPN resolver from an IP address of your ISP.

This can be a problem:

  • either of the configuration of the resolvers;
  • or the configuration of the routing table.

You should check:

  • which resolvers are used (see resolv.conf, resolvectl, etc.);
  • which route is used to reach them (ip route get $nameserver).

Internal (intranet) services not available over a VPN

When using a VPN to connect to some intranet, if the internal (intranet) host names are not resolved (but other external host names are correctly resolved), this is probably because you need to use the intranet resolvers for resolving these host names (split DNS). You should check which resolvers are configured.

If the internal resolvers are correctly configured, the problem might be that your are reaching the resolver with the wrong network interface (from the wrong IP address). You should check which route is used to reach the resolvers:

ip route get $nameserver

Useful troubleshooting commands

NSS plug-in modules

Check which NSS plug-in modules are enabled:

cat /etc/nsswitch.conf  | grep ^hosts:

Check which NSS plug-in modules are installed:

ls -l /lib/libnss_* /lib/*/libnss_*

Output

ls: cannot access '/lib/libnss_*': No such file or directory
-rw-r--r-- 1 root root  38696 10 oct.  21:54 /lib/i386-linux-gnu/libnss_compat-2.31.so
lrwxrwxrwx 1 root root     38 10 oct.  21:54 /lib/i386-linux-gnu/libnss_compat.so -> /lib/i386-linux-gnu/libnss_compat.so.2
lrwxrwxrwx 1 root root     21 10 oct.  21:54 /lib/i386-linux-gnu/libnss_compat.so.2 -> libnss_compat-2.31.so
-rw-r--r-- 1 root root  22060 10 oct.  21:54 /lib/i386-linux-gnu/libnss_dns-2.31.so
lrwxrwxrwx 1 root root     35 10 oct.  21:54 /lib/i386-linux-gnu/libnss_dns.so -> /lib/i386-linux-gnu/libnss_dns.so.2
lrwxrwxrwx 1 root root     18 10 oct.  21:54 /lib/i386-linux-gnu/libnss_dns.so.2 -> libnss_dns-2.31.so
-rw-r--r-- 1 root root  50812 10 oct.  21:54 /lib/i386-linux-gnu/libnss_files-2.31.so
lrwxrwxrwx 1 root root     37 10 oct.  21:54 /lib/i386-linux-gnu/libnss_files.so -> /lib/i386-linux-gnu/libnss_files.so.2
lrwxrwxrwx 1 root root     20 10 oct.  21:54 /lib/i386-linux-gnu/libnss_files.so.2 -> libnss_files-2.31.so
-rw-r--r-- 1 root root  22080 10 oct.  21:54 /lib/i386-linux-gnu/libnss_hesiod-2.31.so
lrwxrwxrwx 1 root root     38 10 oct.  21:54 /lib/i386-linux-gnu/libnss_hesiod.so -> /lib/i386-linux-gnu/libnss_hesiod.so.2
lrwxrwxrwx 1 root root     21 10 oct.  21:54 /lib/i386-linux-gnu/libnss_hesiod.so.2 -> libnss_hesiod-2.31.so
-rw-r--r-- 1 root root  17852 29 sept. 11:31 /lib/i386-linux-gnu/libnss_mdns4_minimal.so.2
-rw-r--r-- 1 root root  17852 29 sept. 11:31 /lib/i386-linux-gnu/libnss_mdns4.so.2
-rw-r--r-- 1 root root  17852 29 sept. 11:31 /lib/i386-linux-gnu/libnss_mdns6_minimal.so.2
-rw-r--r-- 1 root root  17852 29 sept. 11:31 /lib/i386-linux-gnu/libnss_mdns6.so.2
-rw-r--r-- 1 root root  17852 29 sept. 11:31 /lib/i386-linux-gnu/libnss_mdns_minimal.so.2
-rw-r--r-- 1 root root  17852 29 sept. 11:31 /lib/i386-linux-gnu/libnss_mdns.so.2
lrwxrwxrwx 1 root root     23 18 oct.  10:56 /lib/i386-linux-gnu/libnss_nisplus.so.2 -> libnss_nisplus.so.2.0.0
-rw-r--r-- 1 root root  58912 18 oct.  10:56 /lib/i386-linux-gnu/libnss_nisplus.so.2.0.0
lrwxrwxrwx 1 root root     19 18 oct.  10:48 /lib/i386-linux-gnu/libnss_nis.so.2 -> libnss_nis.so.2.0.0
-rw-r--r-- 1 root root  58924 18 oct.  10:48 /lib/i386-linux-gnu/libnss_nis.so.2.0.0
-rw-r--r-- 1 root root  39736 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_compat-2.31.so
lrwxrwxrwx 1 root root     40 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_compat.so -> /lib/x86_64-linux-gnu/libnss_compat.so.2
lrwxrwxrwx 1 root root     21 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_compat.so.2 -> libnss_compat-2.31.so
-rw-r--r-- 1 root root  26952 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_dns-2.31.so
lrwxrwxrwx 1 root root     37 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_dns.so -> /lib/x86_64-linux-gnu/libnss_dns.so.2
lrwxrwxrwx 1 root root     18 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_dns.so.2 -> libnss_dns-2.31.so
-rw-r--r-- 1 root root  51696 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_files-2.31.so
lrwxrwxrwx 1 root root     39 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_files.so -> /lib/x86_64-linux-gnu/libnss_files.so.2
lrwxrwxrwx 1 root root     20 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_files.so.2 -> libnss_files-2.31.so
-rw-r--r-- 1 root root  27000 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_hesiod-2.31.so
lrwxrwxrwx 1 root root     40 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_hesiod.so -> /lib/x86_64-linux-gnu/libnss_hesiod.so.2
lrwxrwxrwx 1 root root     21 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_hesiod.so.2 -> libnss_hesiod-2.31.so
-rw-r--r-- 1 root root  18504 29 sept. 11:31 /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
-rw-r--r-- 1 root root  18504 29 sept. 11:31 /lib/x86_64-linux-gnu/libnss_mdns4.so.2
-rw-r--r-- 1 root root  18504 29 sept. 11:31 /lib/x86_64-linux-gnu/libnss_mdns6_minimal.so.2
-rw-r--r-- 1 root root  18504 29 sept. 11:31 /lib/x86_64-linux-gnu/libnss_mdns6.so.2
-rw-r--r-- 1 root root  18504 29 sept. 11:31 /lib/x86_64-linux-gnu/libnss_mdns_minimal.so.2
-rw-r--r-- 1 root root  18504 29 sept. 11:31 /lib/x86_64-linux-gnu/libnss_mdns.so.2
lrwxrwxrwx 1 root root     23 18 oct.  10:56 /lib/x86_64-linux-gnu/libnss_nisplus.so.2 -> libnss_nisplus.so.2.0.0
-rw-r--r-- 1 root root  51568 18 oct.  10:56 /lib/x86_64-linux-gnu/libnss_nisplus.so.2.0.0
lrwxrwxrwx 1 root root     19 18 oct.  10:48 /lib/x86_64-linux-gnu/libnss_nis.so.2 -> libnss_nis.so.2.0.0
-rw-r--r-- 1 root root  55664 18 oct.  10:48 /lib/x86_64-linux-gnu/libnss_nis.so.2.0.0
-rw-r--r-- 1 root root 291544 15 oct.  23:48 /lib/x86_64-linux-gnu/libnss_resolve.so.2

Check which NSS plug-in modules are installed using the package manager (for Debian-based systems):

aptitude search '^libnss-!dbgsym~i'

Output

In this example, we find that libnss-resolve is only available in 64-bit. Any 32-bit program will not be able to this module: this might make it behave differently from a 64-bit program.

As systemd-resolved can be queried using the DNS protocol (through the dns module), it should be fine however. For other modules, this would be more problematic.

i   libnss-mdns           - module NSS pour la résolution de nom Multicast DNS
i   libnss-mdns:i386      - module NSS pour la résolution de nom Multicast DNS
i A libnss-nis            - NSS module for using NIS as a naming service
i A libnss-nis:i386       - NSS module for using NIS as a naming service
i A libnss-nisplus        - NSS module for using NIS+ as a naming service
i A libnss-nisplus:i386   - NSS module for using NIS+ as a naming service
i   libnss-resolve        - module NSS pour la résolution de nom avec systemd-resolved

Tip: you might want to check that they are all installed for all the available architectures.

Check whether any host aliases are defined:

echo $HOSTALIASES
if test -n "$HOSTALIASES"; then cat "$HOSTALIASES"; fi

Check which NSS plug-in modules are called for a given lookup:

ltrace -e "*gethostbyname*@libnss*" getent hosts www.example.com

Output

In this example, we see that the files, mdns4_minimal and resolve modules were used in that order:

libnss_files.so.2->_nss_files_gethostbyname3_r(0x7ffeece904fa, 10, 0x7fbbb7a6e040, 0x5622738d2a00)                                = 0
libnss_mdns4_minimal.so.2->_nss_mdns4_minimal_gethostbyname3_r(0x7ffeece904fa, 10, 0x7fbbb7a6e040, 0x5622738d2a00)                = 0xffffffff
libnss_resolve.so.2->_nss_resolve_gethostbyname3_r(0x7ffeece904fa, 10, 0x7fbbb7a6e040, 0x5622738d2a00)                            = 1
2606:2800:220:1:248:1893:25c8:1946 www.example.com
+++ exited (status 0) +++

Check socket communications for a given lookup:

strace -e "connect,sendmsg" getent hosts www.example.com

Output

In this example, we see that nscd was not found and we see the process communicating with systemd-resolved through D-Bus:

connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (Aucun fichier ou dossier de ce type)
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (Aucun fichier ou dossier de ce type)
connect(3, {sa_family=AF_UNIX, sun_path="/run/dbus/system_bus_socket"}, 30) = 0
sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\0AUTH EXTERNAL\r\nDATA\r\n", iov_len=22}, {iov_base="NEGOTIATE_UNIX_FD\r\n", iov_len=19}, {iov_base="BEGIN\r\n", iov_len=7}], msg_iovlen=3, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 48
sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\1\0\1\0\0\0\0\1\0\0\0m\0\0\0\1\1o\0\25\0\0\0/org/freedesktop/DBus\0\0\0\3\1s\0\5\0\0\0Hello\0\0\0\2\1s\0\24\0\0\0org.freedesktop.DBus\0\0\0\0\6\1s\0\24\0\0\0org.freedesktop.DBus\0\0\0\0", iov_len=128}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 128
sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\1\2\1(\0\0\0\2\0\0\0\242\0\0\0\1\1o\0\31\0\0\0/org/freedesktop/resolve1\0\0\0\0\0\0\0\3\1s\0\17\0\0\0ResolveHostname\0\2\1s\0 \0\0\0org.freedesktop.resolve1.Manager\0\0\0\0\0\0\0\0\6\1s\0\30\0\0\0org.freedesktop.resolve1\0\0\0\0\0\0\0\0\10\1g\0\4isit\0\0\0\0\0\0\0", iov_len=184}, {iov_base="\0\0\0\0\17\0\0\0www.example.com\0\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=40}], msg_iovlen=2, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 224
2606:2800:220:1:248:1893:25c8:1946 www.example.com
+++ exited with 0 ++

Check resolv.conf files

Check the content of /etc/resolv.conf:

cat /etc/resolv.conf

Check if resolvconf is a symlink and where it is pointing to (this will tell you which software manages it):

ls -l /etc/resolv.conf

Check other resolv.conf files:

cat /run/resolv.conf/resolv.conf
cat /run/systemd/resolve/stub-resolv.conf
cat /run/systemd/resolve/resolv.conf
cat /var/run/NetworkManager/resolv.conf
cat /var/run/NetworkManager/no-stub-resolv.conf

Check systemd-resolved

Check for systemd-resolved configuration;

systemd-resolve --status
# same as:
resolvectl

Test name resolution

Test host name resolution:

getent hosts www.example.com
dig A www.example.com
dig AAAA www.example.com
systemd-resolve -4 www.example.com
systemd-resolve -6 www.example.com

Test mDNS name resolution:

avahi-resolve-host-name -4 foo.local
avahi-resolve-host-name -6 foo.local
systemd-resolve -p mdns -4 foo.local
systemd-resolve -p mdns -6 foo.local

Test whether the resolvers configured in resolv.conf are answering:

for nameserver in $(cat /etc/resolv.conf | sed 's/#.*//' | grep ^nameserver | sed "s/^ *nameserver//"); do
  echo "Testing $nameserver"
  dig "@$nameserver" www.example.com
  echo
done

Check routes for resolvers configured in resolv.conf:

for nameserver in $(cat /etc/resolv.conf | sed 's/#.*//' | grep ^nameserver | sed "s/^ *nameserver//"); do
  ip route get "$nameserver"
done

Test whether the resolvers configured in systemd-resolved are answering:

for nameserver in $(resolvectl dns | sed "s/^[^:]*://"); do
  echo "Testing $nameserver"
  dig "@$nameserver" www.example.com
  echo
done

Check routes for resolvers configured in systemd-resolved:

for nameserver in $(resolvectl dns | sed "s/^[^:]*://"); do
  ip route get "$nameserver"
done

Monitor network traffic

Monitor DNS traffic6:

sudo tcpdump -i any "port 53"

Monitor mDNS traffic:

sudo tcpdump -i any "udp port 5353"

Monitor LLMNR traffic:

sudo tcpdump -i any "udp port 5355"

Monitor DNS, mDNS and LLMNR traffic:

sudo tcpdump -i any "port 53 or udp port 5353 or udp port 5355"

You might want to run these commands while you are trying some name resolutions (eg. getent hosts www.example.com, dig www.esample.com, etc.).

References

Similar articles

Update 2021-06-04: found these posts from HackerNews which discuss the same things.


  1. Multicast DNS (mDNS), defined in RFC6762, is a protocol based on DNS used to resolve host names (in foo.local) to IP addresses on the local network. 

  2. LLMNR is a protocol very similar to mDNS typically used on Windows system. 

  3. Avahi is the classical mDNS/DNS-DS implementation on Linux systems. systemd-resolved has support for mDNS as well. 

  4. systemd-resolved has four different possible modes of operations for handling resolv.conf

  5. This can happen because the plug-in module is only installed for one architecture (eg. 64-bit) on a multi-arch system, because the program is not linked against the GNU libc (for example because it uses another libc or because it's a go program). 

  6. This does not include things like DNS-over-TLS and DNS-over-HTTPS which use other ports and are encrypted anyway. 

  7. These hooks receive the informations passed by the OpenVPN server as environment variables:

    foreign_option_1='dhcp-option DNS 10.0.0.42'
    foreign_option_2='dhcp-option DNS 10.0.1.42'
    foreign_option_3='dhcp-option DOMAIN example.com'
    
     
  8. You can check which NSS features are implemented by a given NSS plug-in module with:

    $ readelf -Ws /lib/x86_64-linux-gnu/libnss_resolve.so.2 | grep FUNC | grep -v UND
    166: 0000000000036c30  1738 FUNC    GLOBAL DEFAULT   12 _nss_resolve_gethostbyname4_r
    167: 0000000000036310  2150 FUNC    GLOBAL DEFAULT   12 _nss_resolve_gethostbyname3_r
    168: 0000000000036c10    22 FUNC    GLOBAL DEFAULT   12 _nss_resolve_gethostbyname2_r
    169: 0000000000035a10  2265 FUNC    GLOBAL DEFAULT   12 _nss_resolve_gethostbyaddr2_r
    170: 00000000000362f0    24 FUNC    GLOBAL DEFAULT   12 _nss_resolve_gethostbyaddr_r
    171: 0000000000036b80   138 FUNC    GLOBAL DEFAULT   12 _nss_resolve_gethostbyname_r
     
  9. The term Virtual Circuit (VC) is used to refer to reliable bidirectional byte stream. In practice, this is usually a TCP stream. 

  10. NSS is used for other resolutions as well such as UNIX user/UID, group name/UID, etc.