GNU/Linux host name resolution

This post describes different software components involved in host name resolutions and DNS configuration on GNU/Linux systems. It consists of a diagram and some accompanying explanations. The goal is to give some pointers and references to understand how to troubleshoot host name/DNS resolution problems and configuration problems on GNU/Linux systems.

Table of content

Diagram

Here is a diagram of how host name resolution works on GNU/Linux systems:

😅

Host name resolution by the C library

Most programs call functions from the C library (libc) for host name resolution:

• getaddrinfo() is used to resolve host names to IP addresses.
• getnameinfo() is used to resolve IP addresses to host names.

Other deprecated functions such as gethostbyname*() and gethostbyaddr*() can be used as well.

Host Aliases

The libc starts the host name resolution by trying to resolve the requested host name as a host alias. This step is optional: if the HOSTALIASES environment variable is set, it is considered to be the path to a file containing host aliases definitions. If the libc finds the requested name in the host alias file, the aliased name is used instead of the alias for the following steps.

A host alias file might look like:

example www.example.com

Note: the alias cannot contain any dot.

$ echo 'example www.example.com' > ./host_aliases
$ HOSTALIASES="./host_aliases" getent hosts example
93.184.216.34   www.example.com

NSS

The GNU libc uses a system of plugins, Name Service Switch (NSS), for host name resolution^[1]. The GNU libc finds a list of NSS plug-in modules to use in the /etc/nsswitch.conf file.

The traditional/minimal plug-in module list is:

hosts: files dns

In this example,

1. the libc first attempts to use the files module, which uses static name/address mappings defined in the /etc/hosts file;
2. if this fails, the libc attempts to use the dns module, which queries the resolvers listed in /etc/resolv.conf using the DNS protocol.

A more featureful plug-in module list might look like:

hosts: files mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] dns myhostname

Each plug-in module is implemented as a shared library. This shared library is dynamically loaded (dlopen()) by the libc. For example, the foo module would be implemented by a libnss_foo.so.2 shared object^[2].

Static associations (files)

The files plug-in module implements static name/address mappings. These static mappings are defined in the /etc/hosts file.

A typical value for this file is:

127.0.0.1	localhost
127.0.1.1	my-host-name

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouter

mDNS with Avahi (mdns[4,6]_minimal)

The mdns_minimal plug-in module implements Multicast DNS (mDNS) name resolution. It only resolves names in the local domain (eg. foo.local): these names are expected to be announced by machines on the local network using mDNS.

The mdns4_minimal and mdns6_minimal modules are variants which only work for IPv4 and IPv6 respectively.

We can test mDNS resolution through Avahi with:

avahi-resolve-host-name -4 marvin.local
avahi-resolve-host-name -6 marvin.local

$ echo 'RESOLVE-HOSTNAME foo.local' | socat STDIO UNIX:/run/avahi-daemon/socket
+ 5 0 foo.local 192.0.2.42

systemd-resolved (resolve)

The resolve plug-in module delegates host name resolution to a local systemd-resolved daemon. The features of systemd-resolved are explained afterwards but include: mDNS, LLMNR, caching, etc.

The plugin and the systemd-resolved daemon communicate over D-Bus. They use the ResolveHostname() method of the org.freedesktop.resolve1.Manager interface:

node /org/freedesktop/resolve1 {
  interface org.freedesktop.resolve1.Manager {
    methods:
      ResolveHostname(in  i ifindex,
                      in  s name,
                      in  i family,
                      in  t flags,
                      out a(iiay) addresses,
                      out s canonical,
                      out t flags);
    // ...
  }
}

We can test host name resolution through systemd-resolved with:

resolvectl -4 query www.example.com
resolvectl -6 query www.example.com
systemd-resolve -4 www.example.com
systemd-resolve -6 www.example.com

Note: If for some reason, a program cannot communicate to systemd-resolved, through D-Bus (for example because it does not use NSS), it can still make queries to systemd-resolved using the DNS protocol. systemd-resolved listens on 127.0.0.53 (TCP and UDP ports 53). When systemd-resolved is enabled, this IP address is usually configured in /etc/resolv.conf; therefore, the dns module or any other software relying on /etc/resolv.conf will end up communicating to systemd-resolved using the standard DNS protocol.

lwresd cache (lwres)

The lwres plug-in module implements system-wide caching for DNS resolutions. It communicates with an associated system daemon (lwresd). This daemon is a caching daemon (based on BIND) which forwards the queries to the resolvers defined in the /etc/resolv.conf file (not through NSS) and caches the answers.

DNS based host name resolution (dns)

The dns plug-in module implements host name resolution using the standard DNS protol. This plug-in module uses the /etc/resolv.conf file to find a list of resolvers to communicate to. These resolvers might be local resolvers such as dnsmasq, unbound or systemd-resolved or remote resolvers.

myhostname

The myhostname plug-in module resolves the name of the machine and localhost. These are traditionnaly handled by the files module.

In addition, this module handles:

• any hostname in the localhost domain (eg. foobar.localhost) as mandated by RFC 6761;
• _gateway as the IP addresses of the default gateways.

All these are also resolved by systemd-revolved. The myhostname module is supposed to be useful as a fallback if systemd-resolved is unavailable.

NSCD

The GNU C library includes builtin support for an optional daemon, the Name Service Cache Daemon (NSCD), for caching all NSS requests (not only host names resolutions). When enabled, it listens on the /run/nscd/socket Unix socket.

This features is not implemented as a NSS plug-in module. The GNU C library always tries to contact this service for NSS requests before going through the NSS plug-in modules.

The NSCD daemon itself uses the NSS plug-in modules declared in /etc/nsswitch.conf.

This daemon is usually not installed and is apparently not very stable.

Managing resolv.conf files

The /etc/resolv.conf file defines which resolvers are used by the dns NSS module (with the nameserver directive).

It can contain:

• IP addresses of resolverss to use (nameserver directive);
• domain names to use for resolving dotless host names (search directive);
• misc. options (options directive) such as,
  • enabling EDNS0 (options edns0),
  • forcing DNS-over-TDP instead of DNS-over-UDP (options vc)^[4];
• ranges of IP addresses to use in priority when contacting resolved hosts (sortlist directive).

Example of revolv.conf file:

nameserver 10.0.0.42
nameserver 10.0.1.42

The programs which manage the network configuration (DHCP clients, VPN programs) usually want to reconfigure this file somehow.

The resolvconf interface

In order to avoid inconsistent state of the /etc/resolv.conf file resulting from different programs trying to configure it, the resolvconf program can be used. The idea is that different programs can register domain name configurations associated to different network interfaces and resolvconf merges them in a a resolv.conf file.

When resolvconf is installed, /etc/resolv.conf is typically a symlink to /run/resolvconf/resolv.conf:

• resolvconf manages /run/resolvconf/resolv.conf directly;
• /etc/resolv.conf is either a symlink to /run/resolvconf/resolv.conf or to another file (if some local resolver suh as unbound is running).

A component of the system can register a DNS configuration for an interface by sending some resolv.conf directives to the standard input of a resolvconf -a IFACE command:

resolvconf -a eth0.dhclient << EOF
nameserver 10.0.0.42
nameserver 10.0.1.42
EOF

The configuration for an interface is unregistered with:

resolvconf -d eth0.dhclient

Many programs can use this interface (usually through hooks):

• DHCP clients (eg. dhclient) can configure the resolvers received by DHCP.
• PPP daemons (eg. pppd) can configures resolvers advertised by the peer.
• ifup/ifdown can configure the resolvers which are declared statically in interfaces files.
• OpenVPN when used as a VPN client can configure resolvers advertised by the VPN server.
• NetworkManager can configure statically or dynamically configured resolvers.

See below for some details about this.

There are several (at least three) implementations of this program:

• traditional resolvconf;
• openresolv;
• systemd-resolve.

Traditional resolvconf

The /etc/resolvconf/interface-order file defines a priority of interfaces. Interfaces listed first in this file will have higher priority. Their DNS configuration (eg. DNS servers) will have precedence over the other ones. This ensures (for example) that resolvers configured from VPN interfaces (tun*, tap*) override resolvers configured from Ethernet or Wifi interfaces.

  # interface-order(5)
  lo.inet6
  lo.inet
  lo.@(dnsmasq|pdnsd)
  lo.!(pdns|pdns-recursor)
  lo
  tun*
  tap*
  hso*
  em+([0-9])?(_+([0-9]))*
  p+([0-9])p+([0-9])?(_+([0-9]))*
  @(br|eth)*([^.]).inet6
  @(br|eth)*([^.]).ip6.@(dhclient|dhcpcd|pump|udhcpc)
  @(br|eth)*([^.]).inet
  @(br|eth)*([^.]).@(dhclient|dhcpcd|pump|udhcpc)
  @(br|eth)*
  @(ath|wifi|wlan)*([^.]).inet6
  @(ath|wifi|wlan)*([^.]).ip6.@(dhclient|dhcpcd|pump|udhcpc)
  @(ath|wifi|wlan)*([^.]).inet
  @(ath|wifi|wlan)*([^.]).@(dhclient|dhcpcd|pump|udhcpc)
  @(ath|wifi|wlan)*
  ppp*
  *
  

Some packages include a script (in /etc/resolvconf/update-libc.d/) to be notified of DNS configuration updates through resolvconf.

openresolv

Openresolv is an alternative implementation of resolvconf with additional features.

In addition to configuring resolv.conf, it has support for configuring local resolvers instead: dnsmasq, named, PowerDNS Recursor, pdnsd and unbound are supported.

It has additional command-line options.

resolvectl working as resolvconf

resolvectl implements the resolvconf interface as well (when called as resolvconf). Instead of directly configuring /run/resolvconf/resolv.conf, it configures systemd-resolved (see below).

Note: on my Debian systems there is no symlink from resolvconf to resolvectl.

systemd-resolved

The systemd-resolved daemon is a client for:

• DNS and DNS over TLS (DoT);
• mDNS^[5] for foo.local host names;
• Link-Local Multicast Name Resolution^[6] (LLMNR) for dotless host names only.

It additionnaly implements:

• caching;
• conditional name resolution (using different resolvers for different domains);
• DNSSEC validation.

It can be queried:

• over D-Bus;
• over DNS (it binds to 127.0.0.53:53 on UDP and TCP).

When systemd-resolved is used, /etc/resolv.conf is usually^[7] a symlink to /run/systemd/resolve/stub-resolv.conf which will look like:

nameserver 127.0.0.53
options edns0

This ensures that:

• any client relying on /etc/resolv.conf will use systemd-resolved as a resolver;
• any client which cannot used the resolve plug-in module will still communicate to systemd-resolved^[8].

Network managers

ifupdown

Both openresolv and resolvconf provide hooks for configuring resolv.conf based on static DNS configuration from /etc/network/interfaces and /etc/network/interfaces.d/*.

The hooks shipping with resolvconf handle the following directives (which set the corrresponding resolv.conf directives):

• dns-domain
• dns-search
• dns-sortlist
• dns-nameservers

iface vlan10 inet static
  address 10.0.0.0/24
  network 10.10.0.0
  broadcast 10.0.0.255
  dns-nameservers 10.0.0.42

PPP

pppd can configure DNS resolution through resolvconf using hooks.

DHCP clients

dhclient can configure DNS resolution through resolvconf using hooks.

OpenVPN

An OpenVPN server can send DNS configuration to its clients (eg. with --dhcp-option DNS 10.0.0.42).

On Windows, the OpenVPN client can use this DNS configuration to change the DNS configuration on the system. On other systems, there is not builtin support for changing the system DNS configuration. Instead, the OpenVPN client must delegate DNS configuration to a hook^[9].

The hook must be explicitly enabled:

• either as command-line argument when calling OpenVPN;
• or in the OpenVPN configuration file.

Two such scripts are available on Debian:

• update-resolv-conf for resolvconf;
• update-systemd-resolved for systemd-resolved.

openvpn $other_arguments \
  --script-security 2 \
  --up /etc/openvpn/script/update-resolv-conf \
  --down /etc/openvpn/script/update-resolv-conf \
  --down-pre

# [...]
script-security 2
up /etc/openvpn/script/update-resolv-conf
down /etc/openvpn/script/update-resolv-conf
down-pre

NetworkManager

NetworkManager can configure DNS resolution using:

• either static configuration (for a given network connection);
• or dynamic configuration (DHCP, etc.).

There are several mode of operations based on its configuration (depending on the dns and rc-manager directives):

• configure resolv.conf directly;
• use resolvconf;
• use netconfig;
• configure dnsmasq;
• configure systemd-resolved;
• configure unbound;

When using the resolvconf method, NetworkManager registers its DNS configuration to resolvconf using a fake network interface called NetworkManager.

Programs which do not use NSS for name resolution

This section contain a (non-exhaustive) list of programs which do not (always) rely on the libc functions (and hence NSS plug-in modules) for host name resolution.

Firefox can be configured to do DNS over HTTPS (DoH). When enabled (configured with network.trr.mode), DNS resolution does not go through the libc, /etc/resolv.conf, etc. In this mode, Firefox uses a DNS-over-HTTPS resolver coming from its configuration (configured with network.trr.custom_uri).

Chrome/Chromium has experimental support for DNS-over-HTTPS (see chrome://flags/#dns-over-https). Currently (2020-04-15), it is listed as available on Mac, Windows, Chrome OS and Android only.

DNS clients such as dig do not use the libc for name resolution but make direct DNS requests (based on /etc/resolv.conf).

Programs linked statically against the GNU libc still load NSS the plug-in modules dynamically. It is apparently possible to disable this feature: in this case, only the files and nss modules will be included (statically) in the binary.

Typical problems

Missing NSS modules on multi-arch

On a multi-arch system, if the NSS plug-in modules are not installed for all architectures, the behavior will not be consistent depending on the architecture of the program.

On Debian-based systems, we can check which NSS modules are installed with:

aptitude search '^libnss-!dbgsym~i'

Resolver not updated when using OpenVPN

You might need to enable a hook for applying the DNS configuration pulled from the OpenVPN server.

Resolver not answering

This might happen if you are contacting the resolver from the wrong network interface (using the wrong route, from the wrong source IP address).

Most recursive resolvers are not opened to the internet (they are not open resolvers) but are only available from a restricted set of IP addresses: you usally can only use the recursive resolver of the FooBar ISP from the FooBar network.

In you are using a VPN, two variants of this problem might happen:

• you are trying to use resolver of your ISP from an IP address of your VPN;
• you trying to use the resolver of your VPN resolver from an IP address of your ISP.

This can be a problem:

• either of the configuration of the resolvers;
• or the configuration of the routing table.

You should check:

• which resolvers are used (see resolv.conf, resolvectl, etc.);
• which route is used to reach them (ip route get $nameserver).

Internal (intranet) services not available over a VPN

When using a VPN to connect to some intranet, if the internal (intranet) host names are not resolved (but other external host names are correctly resolved), this is probably because you need to use the intranet resolvers for resolving these host names (split DNS). You should check which resolvers are configured.

If the internal resolvers are correctly configured, the problem might be that your are reaching the resolver with the wrong network interface (from the wrong IP address). You should check which route is used to reach the resolvers:

ip route get $nameserver

Useful troubleshooting commands

NSS plug-in modules

Check which NSS plug-in modules are enabled:

cat /etc/nsswitch.conf  | grep ^hosts:

Check which NSS plug-in modules are installed:

ls -l /lib/libnss_* /lib/*/libnss_*

ls: cannot access '/lib/libnss_*': No such file or directory
-rw-r--r-- 1 root root  38696 10 oct.  21:54 /lib/i386-linux-gnu/libnss_compat-2.31.so
lrwxrwxrwx 1 root root     38 10 oct.  21:54 /lib/i386-linux-gnu/libnss_compat.so -> /lib/i386-linux-gnu/libnss_compat.so.2
lrwxrwxrwx 1 root root     21 10 oct.  21:54 /lib/i386-linux-gnu/libnss_compat.so.2 -> libnss_compat-2.31.so
-rw-r--r-- 1 root root  22060 10 oct.  21:54 /lib/i386-linux-gnu/libnss_dns-2.31.so
lrwxrwxrwx 1 root root     35 10 oct.  21:54 /lib/i386-linux-gnu/libnss_dns.so -> /lib/i386-linux-gnu/libnss_dns.so.2
lrwxrwxrwx 1 root root     18 10 oct.  21:54 /lib/i386-linux-gnu/libnss_dns.so.2 -> libnss_dns-2.31.so
-rw-r--r-- 1 root root  50812 10 oct.  21:54 /lib/i386-linux-gnu/libnss_files-2.31.so
lrwxrwxrwx 1 root root     37 10 oct.  21:54 /lib/i386-linux-gnu/libnss_files.so -> /lib/i386-linux-gnu/libnss_files.so.2
lrwxrwxrwx 1 root root     20 10 oct.  21:54 /lib/i386-linux-gnu/libnss_files.so.2 -> libnss_files-2.31.so
-rw-r--r-- 1 root root  22080 10 oct.  21:54 /lib/i386-linux-gnu/libnss_hesiod-2.31.so
lrwxrwxrwx 1 root root     38 10 oct.  21:54 /lib/i386-linux-gnu/libnss_hesiod.so -> /lib/i386-linux-gnu/libnss_hesiod.so.2
lrwxrwxrwx 1 root root     21 10 oct.  21:54 /lib/i386-linux-gnu/libnss_hesiod.so.2 -> libnss_hesiod-2.31.so
-rw-r--r-- 1 root root  17852 29 sept. 11:31 /lib/i386-linux-gnu/libnss_mdns4_minimal.so.2
-rw-r--r-- 1 root root  17852 29 sept. 11:31 /lib/i386-linux-gnu/libnss_mdns4.so.2
-rw-r--r-- 1 root root  17852 29 sept. 11:31 /lib/i386-linux-gnu/libnss_mdns6_minimal.so.2
-rw-r--r-- 1 root root  17852 29 sept. 11:31 /lib/i386-linux-gnu/libnss_mdns6.so.2
-rw-r--r-- 1 root root  17852 29 sept. 11:31 /lib/i386-linux-gnu/libnss_mdns_minimal.so.2
-rw-r--r-- 1 root root  17852 29 sept. 11:31 /lib/i386-linux-gnu/libnss_mdns.so.2
lrwxrwxrwx 1 root root     23 18 oct.  10:56 /lib/i386-linux-gnu/libnss_nisplus.so.2 -> libnss_nisplus.so.2.0.0
-rw-r--r-- 1 root root  58912 18 oct.  10:56 /lib/i386-linux-gnu/libnss_nisplus.so.2.0.0
lrwxrwxrwx 1 root root     19 18 oct.  10:48 /lib/i386-linux-gnu/libnss_nis.so.2 -> libnss_nis.so.2.0.0
-rw-r--r-- 1 root root  58924 18 oct.  10:48 /lib/i386-linux-gnu/libnss_nis.so.2.0.0
-rw-r--r-- 1 root root  39736 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_compat-2.31.so
lrwxrwxrwx 1 root root     40 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_compat.so -> /lib/x86_64-linux-gnu/libnss_compat.so.2
lrwxrwxrwx 1 root root     21 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_compat.so.2 -> libnss_compat-2.31.so
-rw-r--r-- 1 root root  26952 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_dns-2.31.so
lrwxrwxrwx 1 root root     37 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_dns.so -> /lib/x86_64-linux-gnu/libnss_dns.so.2
lrwxrwxrwx 1 root root     18 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_dns.so.2 -> libnss_dns-2.31.so
-rw-r--r-- 1 root root  51696 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_files-2.31.so
lrwxrwxrwx 1 root root     39 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_files.so -> /lib/x86_64-linux-gnu/libnss_files.so.2
lrwxrwxrwx 1 root root     20 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_files.so.2 -> libnss_files-2.31.so
-rw-r--r-- 1 root root  27000 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_hesiod-2.31.so
lrwxrwxrwx 1 root root     40 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_hesiod.so -> /lib/x86_64-linux-gnu/libnss_hesiod.so.2
lrwxrwxrwx 1 root root     21 10 oct.  21:54 /lib/x86_64-linux-gnu/libnss_hesiod.so.2 -> libnss_hesiod-2.31.so
-rw-r--r-- 1 root root  18504 29 sept. 11:31 /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
-rw-r--r-- 1 root root  18504 29 sept. 11:31 /lib/x86_64-linux-gnu/libnss_mdns4.so.2
-rw-r--r-- 1 root root  18504 29 sept. 11:31 /lib/x86_64-linux-gnu/libnss_mdns6_minimal.so.2
-rw-r--r-- 1 root root  18504 29 sept. 11:31 /lib/x86_64-linux-gnu/libnss_mdns6.so.2
-rw-r--r-- 1 root root  18504 29 sept. 11:31 /lib/x86_64-linux-gnu/libnss_mdns_minimal.so.2
-rw-r--r-- 1 root root  18504 29 sept. 11:31 /lib/x86_64-linux-gnu/libnss_mdns.so.2
lrwxrwxrwx 1 root root     23 18 oct.  10:56 /lib/x86_64-linux-gnu/libnss_nisplus.so.2 -> libnss_nisplus.so.2.0.0
-rw-r--r-- 1 root root  51568 18 oct.  10:56 /lib/x86_64-linux-gnu/libnss_nisplus.so.2.0.0
lrwxrwxrwx 1 root root     19 18 oct.  10:48 /lib/x86_64-linux-gnu/libnss_nis.so.2 -> libnss_nis.so.2.0.0
-rw-r--r-- 1 root root  55664 18 oct.  10:48 /lib/x86_64-linux-gnu/libnss_nis.so.2.0.0
-rw-r--r-- 1 root root 291544 15 oct.  23:48 /lib/x86_64-linux-gnu/libnss_resolve.so.2

Check which NSS plug-in modules are installed using the package manager (for Debian-based systems):

aptitude search '^libnss-!dbgsym~i'

i   libnss-mdns           - module NSS pour la résolution de nom Multicast DNS
i   libnss-mdns:i386      - module NSS pour la résolution de nom Multicast DNS
i A libnss-nis            - NSS module for using NIS as a naming service
i A libnss-nis:i386       - NSS module for using NIS as a naming service
i A libnss-nisplus        - NSS module for using NIS+ as a naming service
i A libnss-nisplus:i386   - NSS module for using NIS+ as a naming service
i   libnss-resolve        - module NSS pour la résolution de nom avec systemd-resolved

Tip: you might want to check that they are all installed for all the available architectures.

Check whether any host aliases are defined:

echo $HOSTALIASES
if test -n "$HOSTALIASES"; then cat "$HOSTALIASES"; fi

Check which NSS plug-in modules are called for a given lookup:

ltrace -e "*gethostbyname*@libnss*" getent hosts www.example.com

libnss_files.so.2->_nss_files_gethostbyname3_r(0x7ffeece904fa, 10, 0x7fbbb7a6e040, 0x5622738d2a00)                                = 0
libnss_mdns4_minimal.so.2->_nss_mdns4_minimal_gethostbyname3_r(0x7ffeece904fa, 10, 0x7fbbb7a6e040, 0x5622738d2a00)                = 0xffffffff
libnss_resolve.so.2->_nss_resolve_gethostbyname3_r(0x7ffeece904fa, 10, 0x7fbbb7a6e040, 0x5622738d2a00)                            = 1
2606:2800:220:1:248:1893:25c8:1946 www.example.com
+++ exited (status 0) +++

Check socket communications for a given lookup:

strace -e "connect,sendmsg" getent hosts www.example.com

connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (Aucun fichier ou dossier de ce type)
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (Aucun fichier ou dossier de ce type)
connect(3, {sa_family=AF_UNIX, sun_path="/run/dbus/system_bus_socket"}, 30) = 0
sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\0AUTH EXTERNAL\r\nDATA\r\n", iov_len=22}, {iov_base="NEGOTIATE_UNIX_FD\r\n", iov_len=19}, {iov_base="BEGIN\r\n", iov_len=7}], msg_iovlen=3, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 48
sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\1\0\1\0\0\0\0\1\0\0\0m\0\0\0\1\1o\0\25\0\0\0/org/freedesktop/DBus\0\0\0\3\1s\0\5\0\0\0Hello\0\0\0\2\1s\0\24\0\0\0org.freedesktop.DBus\0\0\0\0\6\1s\0\24\0\0\0org.freedesktop.DBus\0\0\0\0", iov_len=128}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 128
sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\1\2\1(\0\0\0\2\0\0\0\242\0\0\0\1\1o\0\31\0\0\0/org/freedesktop/resolve1\0\0\0\0\0\0\0\3\1s\0\17\0\0\0ResolveHostname\0\2\1s\0 \0\0\0org.freedesktop.resolve1.Manager\0\0\0\0\0\0\0\0\6\1s\0\30\0\0\0org.freedesktop.resolve1\0\0\0\0\0\0\0\0\10\1g\0\4isit\0\0\0\0\0\0\0", iov_len=184}, {iov_base="\0\0\0\0\17\0\0\0www.example.com\0\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=40}], msg_iovlen=2, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 224
2606:2800:220:1:248:1893:25c8:1946 www.example.com
+++ exited with 0 ++

Check resolv.conf files

Check the content of /etc/resolv.conf:

cat /etc/resolv.conf

Check if resolvconf is a symlink and where it is pointing to (this will tell you which software manages it):

ls -l /etc/resolv.conf

Check other resolv.conf files:

cat /run/resolv.conf/resolv.conf
cat /run/systemd/resolve/stub-resolv.conf
cat /run/systemd/resolve/resolv.conf
cat /var/run/NetworkManager/resolv.conf
cat /var/run/NetworkManager/no-stub-resolv.conf

Check systemd-resolved

Check for systemd-resolved configuration;

systemd-resolve --status
# same as:
resolvectl

Test name resolution

Test host name resolution:

getent hosts www.example.com
dig A www.example.com
dig AAAA www.example.com
systemd-resolve -4 www.example.com
systemd-resolve -6 www.example.com

Test mDNS name resolution:

avahi-resolve-host-name -4 foo.local
avahi-resolve-host-name -6 foo.local
systemd-resolve -p mdns -4 foo.local
systemd-resolve -p mdns -6 foo.local

Test whether the resolvers configured in resolv.conf are answering:

for nameserver in $(cat /etc/resolv.conf | sed 's/#.*//' | grep ^nameserver | sed "s/^ *nameserver//"); do
  echo "Testing $nameserver"
  dig "@$nameserver" www.example.com
  echo
done

Check routes for resolvers configured in resolv.conf:

for nameserver in $(cat /etc/resolv.conf | sed 's/#.*//' | grep ^nameserver | sed "s/^ *nameserver//"); do
  ip route get "$nameserver"
done

Test whether the resolvers configured in systemd-resolved are answering:

for nameserver in $(resolvectl dns | sed "s/^[^:]*://"); do
  echo "Testing $nameserver"
  dig "@$nameserver" www.example.com
  echo
done

Check routes for resolvers configured in systemd-resolved:

for nameserver in $(resolvectl dns | sed "s/^[^:]*://"); do
  ip route get "$nameserver"
done

Monitor network traffic

Monitor DNS traffic^[10]:

sudo tcpdump -i any "port 53"

Monitor mDNS traffic:

sudo tcpdump -i any "udp port 5353"

Monitor LLMNR traffic:

sudo tcpdump -i any "udp port 5355"

Monitor DNS, mDNS and LLMNR traffic:

sudo tcpdump -i any "port 53 or udp port 5353 or udp port 5355"

You might want to run these commands while you are trying some name resolutions (eg. getent hosts www.example.com, dig www.esample.com, etc.).

References

References:

• The Sisyphean Task Of DNS Client Config on Linux
• Anatomy of a Linux DNS Lookup

Backlinks:

• What Does It Take To Resolve A Hostname
• domain-nameベースのルーティング(インタフェース選択) on Linux