Logging message workflow with journald

A short summary of the logging message workflow with systemd-journald (and the different formats and sockets involved).

Summary

       /run/systemd/journal/dev-log
       a.k.a. /dev/log (syslog format)
      .--------------->------------------.
      |                                  |
      |                                  |
      |      /run/systemd/journal/socket v
.-----------.(native format)            .----------.  .--------------------.
| processes |--------->-----------------| journald |->| /var/log/journal/* |
'-----------'                           '----------'  '--------------------'
      |  /run/systemd/journal/stdout    ^ ^  |^ |
      |  (stream format)                | |  || |/run/systemd/journal/syslog
      '--------------->-----------------' |  || |(syslog format)
                                          |  || |
 .--------.  /dev/kmsg (kmsg format)      |  || |  .---------.  .------------.
 | kernel |-------------------------------'  || '->| rsyslog |->| /var/log/* |
 '--------'                                  ||    '---------'  '------------'
                             (Journal Export ||       | ^
                             Format)         v|       v |(syslog format)
                               .-----------------.  .----------------------.
                               | Remote journald |  | Remote syslog daemon |
                               '-----------------'  '----------------------'

UPDATE 2020-07-30: since systemd 216, the /run/systemd/journal/syslog connection does not exist anymore. The syslog daemon is expected to pull the data from journald instead.

Inputs

Kernel messages

/dev/kmsg is a device used to receive logging messages from the Linux kernel using a specific (kmsg) format.

Syslog format input

/run/systemd/journal/dev-log is a SOCK_DGRAM socket which can be used by processes to send syslog-compatible messages to journald. It is symlinked in /dev/log which means that all processes trying to use the system syslog will in fact send their messages to journald.

It is used:

• by the openlog() function;
• by the loggger command.

Stream format input

/run/systemd/journal/stdout is a SOCK_STREAM socket which can be used by processes to send logging messages in the simple “stream” format (one line per message with an optional severity prefix).

Usage:

• the stdout/stderr of the services managed by systemd is connected to this socket by default (see the StandardOutput and StandardError options);
• it is opened by the sd_journal_stream_fd() function,
• it is used by the systemd-cat command.

Just after opening the file, sd_journal_stream_fd() sends the following information:

1. the self-proclaimed name of the program (optional);
2. unit name (optional);
3. default severity (when no prefix severity is used);
4. whether to parse severity prefixes in the messages;
5. whether to forward messages to /run/systemd/journal/syslog (0 or 1). This happens as well if ForwardToSyslog=true in the systemd configuration (enabled by default).
6. whether to forward messages to /dev/kmsg (0 or 1). This happens as well if ForwardToKMsg=true in the systemd configuration (disabled by default). with the MaxLevelKMsg option.
7. whether to forward messages to console (0 or 1). This happens as well if ForwardToConsole=true in the systemd configuration (disabled by default).

Each line after this prolog is a logging message optionally prefixed with a severity.

Example:

foo
foo.service
5
1
0
0
0
<7>Debug 1
<7>Debug 2

We can create log entries with:

echo "foo
foo.service
5
1
0
0
0
<7>Debug 1
<7>Debug 2" | socat STDIN UNIX-CONNECT:/run/systemd/journal/stdout

Native format input

/run/systemd/journal/socket is a SOCK_DGRAM socket used for logging data to syslog using the native format. AFAIK, this is the same format as Journal Export Format but with one logging message per datagram:

PRIORITY=7
MESSAGE=First (debug) message
SYSLOG_IDENTIFIER=foo

We can create a message with:

echo "PRIORITY=7
MESSAGE=Debug 1
SYSLOG_IDENTIFIER=foo
FOO=bar
" | socat STDIN UNIX-SENDTO:/run/systemd/journal/socket

It is used by the C API: sd_journal_print, sd_journal_send, etc.

Output

Journal files

If the /var/log/journal directory exists and has the proper permissions, journald will use it to store the logging information in a binary format.

Syslog output

/run/systemd/journal/syslog is a SOCK_DGRAM socket used to send the messages to a syslog daemon (rsyslog).