Why journalctl is cool and syslog will survive for another decade

There was a recent discussion going on if Fedora 20 should drop rsyslog and just using systemd journal. A lot of people are afraid of systemd and its journal, this a pity.

Well, there are pros and cons about this kind of logging. For System administrators daily use, journalctl is a powerful tool simplifying the hunt for log file entries.

On the other hand, there are AFAIK no monitoring tools (yet) that can work with journalctl. Those first need to be developed. A Nagios plug-in should be implemented quite quickly.

Why makes journalctl the life easier?
Instead of grepping trough thousands of lines in /var/log/messages you simply can filter the messages and work on them.

journalctl has auto completion (just hit the tab key) showing you the options to use. I.e.

fedora:~# journalctl  < TAB > 
_AUDIT_LOGINUID=             __MONOTONIC_TIMESTAMP=
_AUDIT_SESSION=              _PID=
_BOOT_ID=                    PRIORITY=
_CMDLINE=                    __REALTIME_TIMESTAMP=
CODE_FILE=                   _SELINUX_CONTEXT=
CODE_FUNC=                   _SOURCE_REALTIME_TIMESTAMP=
CODE_LINE=                   SYSLOG_FACILITY=
_COMM=                       SYSLOG_IDENTIFIER=
COREDUMP_EXE=                SYSLOG_PID=
__CURSOR=                    _SYSTEMD_CGROUP=
ERRNO=                       _SYSTEMD_OWNER_UID=
_EXE=                        _SYSTEMD_SESSION=
_GID=                        _SYSTEMD_UNIT=
_HOSTNAME=                   _TRANSPORT=
_KERNEL_DEVICE=              _UDEV_DEVLINK=
_KERNEL_SUBSYSTEM=           _UDEV_DEVNODE=
_MACHINE_ID=                 _UDEV_SYSNAME=
MESSAGE=                     _UID=
MESSAGE_ID= 
fedora:~# journalctl 

Quite some filtering options available here. Most of this options are self-explanatory.

If you just want to see the entries made by a particular command, issue journalctl _COMM= and the TAB key.

fedora:~# journalctl _COMM=
abrtd            dnsmasq          mtp-probe        sh               tgtd
anacron          gnome-keyring-d  network          smartd           udisksd
avahi-daemon     hddtemp          polkit-agent-he  smbd             umount
bash             journal2gelf     polkitd          sshd             userhelper
blueman-mechani  kdumpctl         pulseaudio       sssd_be          yum
chronyd          krb5_child       qemu-system-x86  su               
colord           libvirtd         sealert          sudo             
crond            logger           sendmail         systemd          
dbus-daemon      mcelog           setroubleshootd  systemd-journal  
fedora:~# journalctl _COMM=

If you enter journalctl _COMM=sshd you will just see the messages created by sshd.

fedora:~# journalctl _COMM=sshd 
-- Logs begin at Tue 2013-07-23 08:46:28 CEST, end at Wed 2013-07-24 11:10:01 CEST. --
Jul 23 09:48:45 fedora.example.com sshd[2172]: Server listening on 0.0.0.0 port 22.
Jul 23 09:48:45 fedora.example.com sshd[2172]: Server listening on :: port 22.
fedora:~#

Usually one is just interested in messages within a particular time range.

fedora:~# journalctl _COMM=crond --since "10:00" --until "11:00"
-- Logs begin at Tue 2013-07-23 08:46:28 CEST, end at Wed 2013-07-24 11:23:25 CEST. --
Jul 24 10:20:01 fedora.example.com CROND[28305]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jul 24 10:50:01 fedora.example.com CROND[28684]: (root) CMD (/usr/lib64/sa/sa1 1 1)
fedora:~#   

And why will rsyslog stay another decade or even longer?

There are a lot of tools and scripts which are in place since a long time, some of them even come from a time before Linux was born.

Most of those scripts must be rewritten or at least change its behaviour. I.e taking input from STDIN instead of a log file, so those tools can digest the output from journalctl|your-super-duper-scipt.pl

For log digesting tools that are needed to be compatible between different Unix and Linux Systems they probably wont be changed. In this case syslogd will survive until the last of those systems is decommissioned.

Further reading

Tags: , ,

6 Responses to “Why journalctl is cool and syslog will survive for another decade”

  1. Like most of Lennart’s project, its needlessly complex in its management, config and use.

    `journalctl _COMM=sshd` seriously? not `journal sshd` ? or `grep sshd /var/log/messages`

    I’m fully aware that it contains useful features and it’ll be the future, but having to learn all this daft stuff because no one has sat down and thought about common use cases and RSI…

    • Luc de Louw says:

      What complexity? Journald is much easier to configure than rsyslog. The command “journalctl _COMM=sshd” was entirely composed with auto-completation. If you dont like filtering, just use “journalctl |grep sshd”

      I propose you should try it out 🙂

  2. Klaus says:

    Hi,
    I can’t find any notion of this, but how about syslog forwarding? There are entities that need forwarding all syslog to a central syslog server

    Klaus

    • Luc de Louw says:

      AFIK there is no native implementation to send it to a remote server via syslog protocol. What you can do is setting ForwardToSyslog=yes in /etc/systemd/journald.conf. So you can use rsyslogd to remote forward the logs.

      You can find a helper on https://github.com/systemd/journal2gelf for sending journals to Greylog2.

  3. Jubal says:

    How does this scale on hundreds or thousands of hosts?