Docker offers quite a few options for storing log files. However, the traditional syslog format remains one of the most flexible to this day. Chances are you're familiar with the concept of a central logging server. It is an indispensable component in any infrastructure that is incredibly convenient and powerful. It allows for easier audits, managing and archiving of log files in a very controlled and secure manner.

We're going to explore and show how to configure Docker to write container logs in syslog format on Docker hosts as well as how to forward to and store those log files on a centrally managed logs server.

The Big Picture

Before we go into the details, let's take a moment to look at what kind of logging system we want to implement.

On a Docker host, processes in containers send their logging output to STDOUT and STDERR. A Docker daemon routes logs to /dev/log, which is a symlink to AF_UNIX socket of datagram-oriented (SOCK_DGRAM) type /run/systemd/journal/dev-log. Rsyslog running on the same Docker host listens on /dev/log and collects, parses and writes Docker containers logs in a structured format. Each log entry is tagged with container name. Each container gets an individual log file under /var/log/docker directory. Rsyslog also sends the logs to a logs host via RELP protocol.

On Rsyslog host, the logs are received via the RELP protocol (uses TCP for transmitting data), processed, sorted and stored in the central logs archive directory under /var/log/central. Each Docker host gets its own directory named after the Docker host hostname. Each container also has an individual log file.

Logrotate is configured on the Docker host and logs host to compress and prune old logs.

Default Docker Logging Configuration

How Docker captures and where it writes out logs depends on what logging driver is used. Logging drivers cover a range of options from local files to remote logging destinations. Currently, the options are:

• local file (file-based storage in binary format)
• Logentries
• local files in JSON format
• Greylog Extended Format (GELF)
• Syslog
• Amazon CloudWatch
• ETW (Event Tracing in Windows)
• Fluentd
• Google Cloud
• Journald (systemd) and
• Splunk.
  

By default, Docker writes container logs in JSON format using the "json-file" logging driver. The log files generated by this logging driver are stored locally in container directories.

The Two-Tier Logging Configuration

Docker has a two-tier logging configuration. There's Docker daemon (dockerd) logging configuration and containers logging configuration. The Docker daemon logging settings define default logging configuration for any new container. Newly created containers inherit Docker daemon logging driver settings. However, the default settings can be overridden when a new container is created.

Let's take a closer look at how this works.

To determine which logging driver is currently used as a default for Docker daemon we need to run docker info command:

admin@dockerhost:~$ sudo docker info | grep Logging
WARNING: No swap limit support
Logging Driver: json-file

If we create a new container and generate some output to STDOUT:

admin@dockerhost:~$ sudo docker run -td ubuntu echo "Test log entry"
7317241f66bd56b5d9c35440ee68a66b29eaa4b88efb1615bcc231153632c6dc

We'll see that the newly created container inherited Docker daemon logging settings and it is configured to write logs via the json-file logging driver:

admin@dockerhost:~$ sudo docker inspect 7317241f66bd | grep -A 3 LogConfig
"LogConfig": {
"Type": "json-file",
"Config": {}
},

As it was mentioned earlier, the default json-file logging driver stores log files locally on Docker host in container directories.

admin@dockerhost:~$ sudo cat /var/lib/docker/containers/7317241f66bd56b5d9c35440ee68a66b29eaa4b88efb1615bcc231153632c6dc/7317241f66bd56b5d9c35440ee68a66b29eaa4b88efb1615bcc231153632c6dc-json.log
{"log":"Test log entry\r\n","stream":"stdout","time":"2021-03-22T10:52:48.128860524Z"}

One can override default Docker hostwide logging settings and create a new container that will use a different logging driver.

admin@dockerhost:~$ sudo docker run --log-driver syslog --log-opt syslog-address=unixgram:///dev/log -td ubuntu echo "Test log entry"
594112caf3edb97711fe460e87b6de2a3dd2c65634b2d74e44e627da424af47d

Inspect the new container logging configuration and we can see that it uses syslog logging driver with options all set from the command line.

admin@dockerhost:~$ sudo docker inspect 594112caf3ed | grep -A 5 LogConfig
"LogConfig": {
"Type": "syslog",
"Config": {
"syslog-address": "unixgram:///dev/log"
}
},

As expected, there's no log file in the container directory now.

admin@dockerhost:~$ sudo ls -la /var/lib/docker/containers/594112caf3edb97711fe460e87b6de2a3dd2c65634b2d74e44e627da424af47d
total 44
drwx-----x 4 root root 4096 Mar 22 11:27 .
drwx-----x 5 root root 4096 Mar 22 11:27 ..
drwx------ 2 root root 4096 Mar 22 11:27 checkpoints
-rw------- 1 root root 2343 Mar 22 11:27 config.v2.json
-rw-r----- 1 root root 43 Mar 22 11:27 container-cached.log
-rw-r--r-- 1 root root 1507 Mar 22 11:27 hostconfig.json
-rw-r--r-- 1 root root 13 Mar 22 11:27 hostname
-rw-r--r-- 1 root root 174 Mar 22 11:27 hosts
drwx-----x 2 root root 4096 Mar 22 11:27 mounts
-rw-r--r-- 1 root root 585 Mar 22 11:27 resolv.conf
-rw-r--r-- 1 root root 71 Mar 22 11:27 resolv.conf.hash

The test log entry was sent to a system socket at /dev/log, which on a systemd-based Linux distribution is a symlink to journald AF_UNIX socket of datagram-oriented (SOCK_DGRAM) type /run/systemd/journal/dev-log. journald uses this socket for receiving syslog messages and for forwarding them to any other syslog daemon.

It means that our test log entry made via the syslog logging-driver should appear in journald records and one of the system logs created by rsyslogd on Docker host.

admin@dockerhost:~$ sudo journalctl -u docker.service | grep "594112caf3ed\[.*\]"
Mar 22 11:27:02 dockerhost 594112caf3ed[9447]: Test log entry

admin@dockerhost:~$ sudo grep -r "594112caf3ed\[.*\]" /var/log/
/var/log/syslog:Mar 22 11:27:02 dockerhost 594112caf3ed[9447]: Test log entry

The logging-driver settings provided on the command line are permanent. You can stop and start containers and the logging settings will persist.

If you delete a container and recreate it, default Docker daemon settings will be used unless you override them again with --log-driver and --log-opt command line flags.

Unfortunately, logging driver settings cannot be changed for existing containers. A container must be deleted and recreated with desired logging configuration settings.

Logging With Rsyslog. Docker Host Configuration.

Docker

Now that we understand how the Docker syslog logging driver works, let's build a logging system where Docker logs are routed to and processed by rsyslog daemon.

rsyslogd running on Docker host will collect, process and write Docker containers logs in a structured format that we'll define to meet our requirements. Each log entry will be tagged with container name to allow for easier identification, differentiation, analysis, sorting and grouping of log entries. Each container will have an individual log file under /var/log/docker directory. Finally, rsyslogd on Docker host will send all logs generated on docker host to a logs host via Reliable Event Logging Protocol (RELP) protocol.

We start by defining global logging configuration settings for Docker daemon and thus effectively all new containers.

admin@dockerhost:~$ cat /etc/docker/daemon.json
{
"log-driver": "syslog",
"log-opts": {
"syslog-address": "unixgram:///dev/log",
"tag" : "docker/{{.Name}}"
}
}

Here we use the syslog logging driver and tell dockerd explicitly to use /dev/log system socket. We also specify with unixgram:// that the /dev/log socket type is datagram-oriented as opposed to unix://, which is used for stream-oriented type of sockets.

Then, we define the format for our syslog message tag. The {{.Name}} will be replaced by a container name. Thus, the "docker/{{.Name}}" tag will become, for example, "docker/bold_mendeleev".

This docker "tag" log option matches rsyslog syslogtag property and can be used in logical expressions as well as for customizing filesystem paths in rsyslog configuration. There are other template markups similar to {{.Name}} that can be used to build a unique syslogtag property. However, keep in mind that by default size for syslogtag is limited (by RFC standards) to 32 characters. The syslog tag suggested in our examples requires less than that and makes the tag property both functional and compliant with the standards.

It is possible to overcome the 32 character limit in case you really need to have longer container names or use full-length 64-character container ID's. You can read more about this here.

If daemons.json is already present, add or modify the logging configuration settings as necessary. If daemon.json does not exist yet, simply create the file and copy-paste the block of JSON code shown above. Make sure the file is owned by root:root and permissions mode is set to 0644.

This is all you need to do to configure Docker daemon.

Before we move on to rsyslog configuration, let us first create a new docker container and execute a command that will generate one log entry every 60 seconds. This way we can easily confirm if logging works as we work on setting up our logging system.

admin@dockerhost:~$ sudo docker run --hostname container0 --name container0 -it ubuntu
root@container0:/# c=0; while true; do echo "$c: $(date) $HOSTNAME"; c=$((c+1)); sleep 60; done
0: Mon Mar 22 14:19:19 UTC 2021 container0
...

Press Ctrl+P+Q to detach from the container.

Confirm that the container is still running:

admin@dockerhost:~$ sudo docker ps | grep container0
container0 ubuntu "/bin/bash" 2 minutes ago Up About a minute container0

The "Up About a minute" field tells us that the container is up and has been running for that long.

Verify that the while loop in the container is generating log messages and they're written to system log files:

admin@dockerhost:~$ sudo grep -r "container0\[.*\]:" /var/log/
...
/var/log/syslog:Mar 22 14:19:19 dockerhost docker/container0[10742]: 0: Mon Mar 22 14:19:19 UTC 2021 container0
/var/log/syslog:Mar 22 14:20:19 dockerhost docker/container0[10742]: 1: Mon Mar 22 14:20:19 UTC 2021 container0
/var/log/syslog:Mar 22 14:21:19 dockerhost docker/container0[10742]: 2: Mon Mar 22 14:21:19 UTC 2021 container0
/var/log/syslog:Mar 22 14:22:19 dockerhost docker/container0[10742]: 3: Mon Mar 22 14:22:19 UTC 2021 container0

Rsyslog

To configure Rsyslog on Docker host we will need to update /etc/rsyslog.conf and create two new configuration include files in /etc/rsyslog.d/ directory: 00-logshost.conf and 10-docker.conf.

Open /etc/rsyslog.conf and add the following configuration statement to "GLOBAL DIRECTIVES" section:

$PreserveFQDN on

The $PreserveFQDN statement accepts either on or off. When set to on, it ensures that fully-qualified domain names are used in log messages.

Now create /etc/rsyslog.d/10-docker.conf with the following contents:

admin@dockerhost:~$ cat /etc/rsyslog.d/10-docker.conf
$FileCreateMode 0644
$template DockerDaemonLogFileName,"/var/log/docker/docker.log"
$template DockerContainerLogFileName,"/var/log/docker/%SYSLOGTAG:R,ERE,1,FIELD:docker/(.*)\[--end:secpath-replace%.log"
if $programname == 'dockerd' then {
?DockerDaemonLogFileName
stop
}
if $programname == 'containerd' then {
?DockerDaemonLogFileName
stop
}
if $programname == 'docker' then {
if $syslogtag contains 'docker/' then {
?DockerContainerLogFileName
stop
}
}
$FileCreateMode 0600

In this configuration file we make sure that Docker container log files will have permissions mode set to 0644. The DockerLogFileName template defines location and filenames for the containers log files.

If you're not familiar with rsyslog property replacement syntax the "%SYSLOGTAG:R,ERE,1,FIELD:docker/(.*)\[--end:secpath-replace%.log" bit probably looks quite confusing and intimidating.

What it means is that the template filename path will be set to /var/log/docker/{{.Name}}.log. Here {{.Name}} is equivalent to dockerd log option "tag" we saw a bit earlier in the example /etc/docker/daemon.json file.

The following if ... then expressions test if programname property matches 'dockerd', 'containerd' or 'docker' string for each syslog message that is being processed by rsyslog. The first two expressions ensure that docker daemon logs are written to the /var/log/docker/docker.log file.

The last if ... then expression is a little different. It compares if the programname property matches 'docker' string. If there is a match, one more test is done to see if syslogtag, another syslog message property, contains 'docker/' string. This is where our custom tag "docker/{{.Name}}" comes into play. If the test evaluates to True, rsyslog will use the DockerLogFileName template to create a log file named /var/log/docker/{{.Name}}.log and write the syslog message that is being processed to this file.

Finally, the "stop" word is a discard action that tells rsyslog to stop processing the current syslog message and move on to the next one. Effectively, this ensures that docker log messages are only written to the files referenced by the DockerDaemonLogFileName and DockerContainerLogFileName templates.

To validate the new configuration file run

admin@dockerhost:~$ sudo rsyslogd -f /etc/rsyslog.d/10-docker.conf -N1
rsyslogd: version 8.32.0, config validation run (level 1), master config /etc/rsyslog.d/10-docker.conf
rsyslogd: End of config validation run. Bye.

If rsyslog encounters any errors when parsing the configuration file, it will issue a warning and give you a hint as to what the problem might be and where it is located. For example, here's what happens if you intentionally break the first if ... then clause by writing 'if' statement as 'xxif':

admin@dockerhost:~$ sudo rsyslogd -f /etc/rsyslog.d/10-docker.conf -N1
rsyslogd: version 8.32.0, config validation run (level 1), master config /etc/rsyslog.d/10-docker.conf
rsyslogd: error during parsing file /etc/rsyslog.d/10-docker.conf, on or before line 11: warnings occured in file '/etc/rsyslog.d/10-docker.conf' around line 11 [v8.32.0 try http://www.rsyslog.com/e/2207 ]
rsyslogd: invalid or yet-unknown config file command 'programname' - have you forgotten to load a module? [v8.32.0 try http://www.rsyslog.com/e/3003 ]
rsyslogd: error during parsing file /etc/rsyslog.d/10-docker.conf, on or before line 20: syntax error on token '}' [v8.32.0 try http://www.rsyslog.com/e/2207 ]
rsyslogd: CONFIG ERROR: could not interpret master config file '/etc/rsyslog.d/10-docker.conf'. [v8.32.0 try http://www.rsyslog.com/e/2207 ]

As soon as you restart rsyslogd on Docker host, you should see Docker containers log files in /var/log/docker directory:

admin@dockerhost:~$ sudo ls -la /var/log/docker/
total 20
drwxr-xr-x 2 syslog syslog 4096 Mar 22 14:24 .
drwxrwxr-x 9 root syslog 4096 Mar 22 06:25 ..
-rw-r--r-- 1 syslog adm 110 Mar 22 14:24 container0.log
-rw-r--r-- 1 syslog adm 2833 Mar 22 10:48 container1.log
-rw-r--r-- 1 syslog adm 2815 Mar 22 10:48 container2.log
-rw-r--r-- 1 syslog adm 595 Mar 22 14:25 docker.log

Log messages printed to STDOUT and STDERR in containers are sent by Docker daemon to /dev/log system socket. Rsyslog collects these logs from the socket, processes them and writes to /var/log/docker/{{.Name}}.log files. Each container has a dedicated log file.

Now that that's been taken care of, we can also send these log messages to a remote logs host.

Create /etc/rsyslog.d/00-logshost.conf.

admin@dockerhost:~$ cat /etc/rsyslog.d/00-ship.conf
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName forward # file name; enables disk mode
$ActionResumeRetryCount -1. # infinite retries on insert failures
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
$ModLoad omrelp
*.* :omrelp:10.8.7.52:2514 # IP address of the logs host

Restart rsyslog on the Docker host.

Logging With Rsyslog. Logs Host Configuration.

On logs host, create /etc/rsyslog.d/00-remote.conf with the following contents.

module(load="imrelp")
input(type="imrelp" port="2514" ruleset="RemoteLogProcess")
$template PerHostAuth,"/var/log/central/%HOSTNAME%/auth.log"
$template PerHostCron,"/var/log/central/%HOSTNAME%/cron.log"
$template PerHostDaemon,"/var/log/central/%HOSTNAME%/daemon.log"
$template PerHostDebug,"/var/log/central/%HOSTNAME%/debug.log"
$template PerHostKern,"/var/log/central/%HOSTNAME%/kern.log"
$template PerHostLpr,"/var/log/central/%HOSTNAME%/lpr.log"
$template PerHostMail,"/var/log/central/%HOSTNAME%/mail.log"
$template PerHostMailInfo,"/var/log/central/%HOSTNAME%/mail.info.log"
$template PerHostMailWarn,"/var/log/central/%HOSTNAME%/mail.warn.log"
$template PerHostMailErr,"/var/log/central/%HOSTNAME%/mail.err.log"
$template PerHostMessages,"/var/log/central/%HOSTNAME%/messages.log"
$template PerHostNewsCrit,"/var/log/central/%HOSTNAME%/news.crit.log"
$template PerHostNewsErr,"/var/log/central/%HOSTNAME%/news.err.log"
$template PerHostNewsNotice,"/var/log/central/%HOSTNAME%/news.notice.log"
$template PerHostSyslog,"/var/log/central/%HOSTNAME%/syslog.log"
$template PerHostUser,"/var/log/central/%HOSTNAME%/user.log"
$template PerHostDockerDaemonLogFileName,"/var/log/central/%HOSTNAME%/docker/docker.log"
$template PerHostDockerContainerLogFileName,"/var/log/central/%HOSTNAME%/docker/%SYSLOGTAG:R,ERE,1,FIELD:docker/(.*)\[--end:secpath-replace%.log"
ruleset(name="RemoteLogProcess" queue.type="fixedarray") {
$FileCreateMode 0644
if $programname == 'dockerd' then {
?PerHostDockerDaemonLogFileName
stop
}
if $programname == 'containerd' then {
?PerHostDockerDaemonLogFileName
stop
}
if $programname == 'docker' then {
if $syslogtag contains 'docker/' then {
?PerHostDockerLogFileName
stop
}
}
auth,authpriv.* ?PerHostAuth
*.*;auth,authpriv.none -?PerHostSyslog
cron.* ?PerHostCron
daemon.* -?PerHostDaemon
kern.* -?PerHostKern
lpr.* -?PerHostLpr
mail.* -?PerHostMail
user.* -?PerHostUser
mail.info -?PerHostMailInfo
mail.warn -?PerHostMailWarn
mail.err ?PerHostMailErr
news.crit ?PerHostNewsCrit
news.err ?PerHostNewsErr
news.notice -?PerHostNewsNotice
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -?PerHostDebug
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -?PerHostMessages
}
$FileCreateMode 0640

This configuration does two main things.

It tells rsyslog how to process general system logs (/var/log/auth.log, /var/log/syslog, etc.) received from any host that is sending its logs to this logs host.

There are also configuration statements that tell rsyslog how to process Docker logs. You can see that configuration statements are almost identical to those we used on Docker host as we want to have a similar filesystem layout, file names and permissions on the logs host. We also use the same property replacement expressions to extract Docker container names and use that as part of log files names.

In the now familiar-looking recursive if ... then logical expressions we subject each log message to the same tests as on the Docker host and take essentially the same actions: write matching log messages to /var/log/central/%HOSTNAME%/docker/docker.log and /var/log/central/%HOSTNAME%/docker/{{.Name}}.log files and stop processing of a syslog message thus ensuring that rsyslog writes Docker logs only to the log files that we want and not any other files.

One minor difference is the use of %HOSTNAME% property which is replaced by a fully-qualified domain name of the host that generated and/or sent a log message to the logs host.

The imrelp module listens on TCP port 2514 for a stream of data generated by omrelp module on our example Docker host and any other host that's configured to send log stream data to the logs host. The RELP protocol extends the functionality of the syslog protocol and provides a more reliable method for sending and receiving log messages.

The general system logs will be created in /var/log/central/%HOSTNAME%/ directories for each host that sends its logs to the logs host. Traditional syslog facilities (daemon.*, cron.*, kern.*, etc.) are used to filter (select) log messages and corresponding templates are used to instruct rsyslogd where to write those messages.

Now, save the changes and restart rsyslog:

admin@logshost:~$ sudo systemctl restart rsyslog

Wait for a few minutes and confirm that general system log files are being written to the correct destination:

admin@logshost:~$ sudo ls -la /var/log/central/dockerhost/
total 360
drwxr-xr-x 3 syslog syslog 4096 Mar 22 09:23 .
drwxr-xr-x 3 syslog syslog 4096 Mar 16 18:15 ..
-rw-r--r-- 1 syslog adm 95493 Mar 22 18:17 auth.log
-rw-r--r-- 1 syslog adm 27829 Mar 22 18:17 cron.log
-rw-r--r-- 1 syslog adm 54804 Mar 22 18:26 daemon.log
drwxr-xr-x 2 syslog syslog 4096 Mar 22 17:19 docker
-rw-r--r-- 1 syslog adm 15476 Mar 22 17:20 kern.log
-rw-r--r-- 1 syslog adm 20882 Mar 22 17:20 messages.log
-rw-r--r-- 1 syslog adm 104135 Mar 22 18:26 syslog.log
-rw-r--r-- 1 syslog adm 1272 Mar 21 07:00 user.log

and that Docker logs are being received and written out as well:

admin@logshost:~$ sudo ls -la /var/log/central/dockerhost/docker/
[sudo] password for admin:
total 52
drwxr-xr-x 2 syslog syslog 4096 Mar 22 17:19 .
drwxr-xr-x 3 syslog syslog 4096 Mar 22 09:23 ..
-rw-r--r-- 1 syslog adm 29254 Mar 22 17:19 container0.log
-rw-r--r-- 1 syslog adm 3053 Mar 22 10:48 container1.log
-rw-r--r-- 1 syslog adm 2925 Mar 22 10:48 container2.log
-rw-r--r-- 1 syslog adm 605 Mar 22 14:32 docker.log

That is all that you have to do to configure rsyslogd on the logs host.

Of course, there are quite a few more things to do to make the logging system more secure and performant. Communications of rsyslogd daemons via RELP protocol could benefit from TLS encryption. You may need to fine-tune rsyslogd and Docker configurations to ensure optimal performance in your environment and so on.

Managing Log Files

There is perhaps one more thing left to do. A finishing touch. We need to compress and rotate older log files both on our Docker host as well as the logs host.

The old trusty logrotate is one way to do it.

On the Docker host you might want to use a logrotate config like this:

admin@dockerhost:~$ sudo cat /etc/logrotate.d/rsyslog-docker
/var/log/docker/*.log
{
daily
rotate 10
minsize 200M
missingok
notifempty
compress
sharedscripts
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}

And on the logs host like this:

admin@logshost:~$ cat /etc/logrotate.d/rsyslog-central
/var/log/central/*/*.log /var/log/central/*/*/*.log
{
daily
rotate 10
minsize 200M
missingok
notifempty
compress
sharedscripts
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}

These are pretty standard configs. Note that they take care of Docker logs and general system logs in the /var/log/central directories.

The general system logs on both Docker host and the logs host are already managed by /etc/logrotate.d/rsyslog which is installed by default on, for example, Ubuntu 18 LTS systems.

Conclusion

You can think of this particular setup as a foundation for perhaps most of your logging needs. Modern logs collection and analysis systems such as ELK, Fluentd and others can now be easily integrated into your logging infrastructure by configuring rsyslogd to send any and all logs from a single central location to those systems.

And there you have it. Docker logs fully managed by Rsyslog.

Let us know if you found this tutorial useful or if you have any questions!