{"version": "https://jsonfeed.org/version/1", "title": "/dev/posts/ - Tag index - systemd", "home_page_url": "https://www.gabriel.urdhr.fr", "feed_url": "/tags/systemd/feed.json", "items": [{"id": "http://www.gabriel.urdhr.fr/2020/03/17/systemd-revolved-dns-configuration-for-vpn/", "title": "Systemd-resolved DNS configuration for VPN", "url": "https://www.gabriel.urdhr.fr/2020/03/17/systemd-revolved-dns-configuration-for-vpn/", "date_published": "2020-03-17T00:00:00+01:00", "date_modified": "2020-03-17T00:00:00+01:00", "tags": ["computer", "vpn", "dns", "systemd", "covid-19"], "content_html": "

Some guidance about configuring/fixing domain name resolution\nwith a corporate VPN (especially OpenVPN) with systemd-based Linux\nsystems.\nThis configuration uses the internal/private corporate resolvers\nfor resolving internal/private domain names\nwhile using the ISP resolver for general domain names.\nThis might help if your VPN is struggling these days\nbecause of the COVID-19 threat \"\ud83d\ude37\".

\n

I've been helping some people to make corporate VPNs (OpenVPN-based)\nwork correctly, since yesterday.\nThe problem is the resolution of domain names:

\n
    \n
  1. \n

    either private internal domain names are not resolved because the\n internal corporate resolver is used;

    \n
  2. \n
  3. \n

    or general domain name resolution is slowish because the internal corporate\n resolver is used and the VPN is currently struggling with\n the new load.

    \n
  4. \n
\n

A good solution is to use the internal coporate resolver for internal domain\nnames but another (eg. the ISP) resolver for other domain names.\nDepending on your configuration and how you are running the VPN\n(CLI, NetworkManager, etc.), this might be already what is happening\n(you might find some information in this post in order to check this).

\n

Below are some instructions about one solution to make this work on\nsystemd-based Linux systems using systemd-resolved.\nThere are solution to achieve this without systemd-resolved\n(I think it's possible to do that with dnsmasq).

\n

Warning:\ndon't break your internet access right now\nif you don't think you are going to be able to fix it.

\n

Manual configuration

\n

Summary

\n

If you are using a Linux system with systemd-resolved, a quick fix is\nsomething like this (your need to adjust these values):

\n
# Configure internal corporate domain name resolvers:\nresolvectl dns tun0 192.0.2.53 192.0.2.54\n\n# Only use the internal corporate resolvers for domain names under these:\nresolvectl domain tun0 \"~foo.example.com\" \"~bar.example.com\"\n\n# Not super nice, but might be needed:\nresolvectl dnssec tun0 off\n
\n\n\n

If you don't have the resolvectl commands, you might use instead:

\n
systemd-resolve -i tun0 \\\n  --set-dns=192.0.2.53 --set-dns=192.0.2.54 \\\n  --set-domain=foo.example.com --set-domain=bar.example.com \\\n  --set-dnssec=off  # <- Not super nice, but might be needed.\n
\n\n\n

Prerequisites

\n

First check if you have systemd-resolved installed and running:

\n
systemctl status systemd-resolved\n
\n\n\n

Check if you have the resolvectl too:

\n
resolvectl\n
\n\n\n

An alternative is to use systemd-resolve:

\n
systemd-resolve --status\n
\n\n\n

Check if have the resolve mechanism enabled in nssswitch.conf:

\n
cat /etc/nsswitch.conf  | grep ^hosts: | grep resolve\n
\n\n\n

Install the needed libraries. For example on Debian or Ubuntu AMD64 systems:

\n
dpkg -l | grep libnss-resolve\n
\n\n\n

Note: I'm installing both the 64 bit and 32 bit libraries.\nIf you do not install the 32 bit libraries, proper DNS resolution might fail\nfor 32 programs.

\n

Configure DNS resolution

\n

If all these prerequisites are satisfied, you can configure the DNS resolution\nwith something like:

\n
resolvectl dns tun0 192.0.2.53 192.0.2.54\nresolvectl domain tun0 \"~foo.example.com\" \"~bar.example.com\"\n
\n\n\n

Alternatively, if you don't have resolvectl:

\n
systemd-resolve -i tun0 \\\n  --set-dns=192.0.2.53 --set-dns=192.0.2.54 \\\n  --set-domain=~foo.example.com --set-domain=~bar.example.com\n
\n\n\n

This configures systemd-resolved to use the corporate internal resolvers 192.0.2.53 and 192.0.2.54\nfor resolving all domain names under foo.example.com and bar.example.com\n(eg. some-service.foo.example.com). tun0 is network interface of the VPN.\nYou have to adjust these parameters (IP address, domain names).

\n

Note: the ~ in front of a domain prevents the domain to\nbe added to the search list (i.e. trying to resolve the test host name\nwill not search test.foo.example.com if a ~ was used in front\nof foo.example.com).

\n

You can check your system-resolved configuration with:

\n
resolvectl\n
\n\n\n

If you don't have resolvectl:

\n
systemd-resolve --status\n
\n\n\n

You can now test domain name resolution (assuming some-service.foo.example.com\nactually exists):

\n
resolvectl query some-service.foo.example.com\n
\n\n\n

If you don't have resolvectl:

\n
systemd-resolve some-service.foo.example.com\n
\n\n\n

If this works, you should have:

\n
\nsome-service.foo.example.com: 192.0.2.22    -- link: tun0\n\n-- Information acquired via protocol DNS in 1.7ms.\n-- Data is authenticated: no\n
\n\n

In my case, this failed because of:

\n
\nsome-service.foo.example.com: resolve call failed: DNSSEC validation failed: failed-auxiliary\n
\n\n

A quick/dirty fix is to disabled DNSSEC:

\n
resolvectl dnssec tun0 off\n
\n\n\n

If you don't have resolvectl:

\n
systemd-resolve -i tun0 \\\n  --set-dns=192.0.2.53 --set-dns=192.0.2.54 \\\n  --set-domain=foo.example.com --set-domain=bar.example.com \\\n  --set-dnssec=off\n
\n\n\n

\n

Hooking with OpenVPN (CLI)

\n

If you are using OpenVPN directly from the CLI (not through Network Manager),\nyou can automate this with the --up options.

\n

For example:

\n
openvpn --config client.ovpn --script-security 2 --up ./manual-config\n
\n\n\n

where ./manual-config is a shell script such as:

\n
#!/bin/sh\nset -e\nresolvectl dns $dev 192.0.2.53 192.0.2.54\nresolvectl domain $dev \"~foo.example.com\" \"~bar.example.com\"\nresolvectl dnssec $dev off\n
\n\n\n

or

\n
#!/bin/sh\nsystemd-resolve -i $dev \\\n  --set-dns=192.0.2.53 --set-dns=192.0.2.54 \\\n  --set-domain=foo.example.com --set-domain=bar.example.com \\\n  --set-dnssec=off  # <- Not super nice, but might be needed.\n
\n\n\n

OpenVPN sets the dev\nenvironment variable\nto the name of the VPN TUN/TAP\n(i.e. virtual) network device (eg. tun0) when calling the hooks.

\n

Don't forget to make this script executable.

\n

Using update-systemd-resolved

\n

An alternative is to use the\nupdate-systemd-resolved\nscript (on Debian and Ubuntu it is in the openvpn-systemd-resolved package):

\n
openvpn \\\n  --config client.ovpn \\\n  --up /etc/openvpn/update-systemd-resolved \\\n  --down /etc/openvpn/update-systemd-resolved \\\n  --down-pre \\\n
\n\n\n

This automatically pulls configuration advertised by the VPN server:

\n\n

If this fails, the previous section might help you pinpoint to the problem:

\n
    \n
  1. check the configuration (resolvectl);
  2. \n
  3. test internal/external names resolution (resolvectl query some-service.foo.example.com)
  4. \n
  5. disable DNSSEC if needed (resolvectl dnssec tun0 off).
  6. \n
"}, {"id": "http://www.gabriel.urdhr.fr/2015/04/29/journald-workflow/", "title": "Logging message workflow with journald", "url": "https://www.gabriel.urdhr.fr/2015/04/29/journald-workflow/", "date_published": "2015-04-29T00:00:00+02:00", "date_modified": "2020-07-30T23:00:00+02:00", "tags": ["computer", "system", "log", "syslog", "systemd", "journald"], "content_html": "

A short summary of the logging message workflow with\nsystemd-journald\n(and the different formats and sockets involved).

\n

Summary

\n
\n       /run/systemd/journal/dev-log\n       a.k.a. /dev/log (syslog format)\n      .--------------->------------------.\n      |                                  |\n      |                                  |\n      |      /run/systemd/journal/socket v\n.-----------.(native format)            .----------.  .--------------------.\n| processes |---------------------------| journald |->| /var/log/journal/* |\n'-----------'                           '----------'  '--------------------'\n      |  /run/systemd/journal/stdout    ^ ^  |^ |\n      |  (stream format)                | |  || |/run/systemd/journal/syslog\n      '--------------->-----------------' |  || |(syslog format)\n                                          |  || |\n .--------.  /dev/kmsg (kmsg format)      |  || |  .---------.  .------------.\n | kernel |-------------------------------'  || '->| rsyslog |->| /var/log/* |\n '--------'                                  ||    '---------'  '------------'\n                             (Journal Export ||       | ^\n                             Format)         v|       v |(syslog format)\n                               .-----------------.  .----------------------.\n                               | Remote journald |  | Remote syslog daemon |\n                               '-----------------'  '----------------------'\n
\n\n

UPDATE 2020-07-30: since systemd 216,\nthe /run/systemd/journal/syslog connection does not exist anymore.\nThe syslog daemon is expected to pull the data from journald instead.

\n

Inputs

\n

Kernel messages

\n

/dev/kmsg is a device used to receive logging messages from the\nLinux\nkernel\nusing a specific (kmsg) format.

\n

Syslog format input

\n

/run/systemd/journal/dev-log is a SOCK_DGRAM socket which can be\nused by processes to send syslog-compatible messages to journald. It\nis symlinked in /dev/log which means that all processes trying to\nuse the system syslog will in fact send their messages to journald.

\n

It is used:

\n\n

Stream format input

\n

/run/systemd/journal/stdout is a SOCK_STREAM socket which can be used by\nprocesses to send logging messages in the simple \u201cstream\u201d format (one\nline per message with an optional severity prefix).

\n

Usage:

\n\n

Just after opening, the file sd_journal_stream_fd() send the\nfollowing information:

\n
    \n
  1. \n

    the self-proclaimed name of the program (optional);

    \n
  2. \n
  3. \n

    unit name (optional);

    \n
  4. \n
  5. \n

    default severity (when no prefix severity is used);

    \n
  6. \n
  7. \n

    whether to parse severity prefixes in the messages;

    \n
  8. \n
  9. \n

    whether to forward messages to /run/systemd/journal/syslog (0 or\n1);

    \n

    This happens as well if ForwardToSyslog=true in the systemd\nconfiguration (enabled by default).

    \n
  10. \n
  11. \n

    whether to forward messages to /dev/kmsg (0 or 1);

    \n

    This happens as well if ForwardToKMsg=true in the systemd\nconfiguration (disabled by default). with the MaxLevelKMsg option.

    \n
  12. \n
  13. \n

    whether to forward messages to console (0 or 1).

    \n

    This happens as well if ForwardToConsole=true in the systemd\nconfiguration (disabled by default).

    \n
  14. \n
\n

Each line after this prolog is a logging message optionally prefixed\nwith a severity.

\n

Example:

\n
foo\nfoo.service\n5\n1\n0\n0\n0\n<7>Debug 1\n<7>Debug 2\n
\n\n\n

We can create log entries with:

\n
echo \"foo\nfoo.service\n5\n1\n0\n0\n0\n<7>Debug 1\n<7>Debug 2\" | socat STDIN UNIX-CONNECT:/run/systemd/journal/stdout\n
\n\n\n

Native format input

\n

/run/systemd/journal/socket is a SOCK_DGRAM socket used for\nlogging data to syslog using the native format. AFAIK, this is the\nsame format as Journal Export\nFormat but\nwith one logging message per datagram:

\n
PRIORITY=7\nMESSAGE=First (debug) message\nSYSLOG_IDENTIFIER=foo\n
\n\n\n

We can create a message with:

\n
echo \"PRIORITY=7\nMESSAGE=Debug 1\nSYSLOG_IDENTIFIER=foo\nFOO=bar\n\" | socat STDIN UNIX-SENDTO:/run/systemd/journal/socket\n
\n\n\n

It is used by the C\nAPI:\nsd_journal_print, sd_journal_send, etc.

\n

Output

\n

Journal files

\n

If the /var/log/journal directory exists and has the proper\npermissions, journald will use it to store the logging information in\na binary\nformat.

\n

Syslog output

\n

/run/systemd/journal/syslog is a SOCK_DGRAM socket used to send\nthe messages to a syslog daemon (rsyslog).

"}]}