For the past years, all Linux kernel logs were handled by a daemon called syslogd. This daemon has past been dropped as this daemon somewhat gives no proper management of logs. For so reason, many Linux distros have switched to a new system daemon which comes with a new log daemon. This is a massive improvement in the Linux world as systemd tries to handle this improper log management by providing a centralized management solution that handles all kernel logs and user’s processes. This issue was solved by the introduction of a new daemon called the journald daemon.
The journald daemon stores log data in a structured and indexed binary file which includes extra information about log events. The content of the systemd journal can be query using the journalctl command. When the journalctl command is entered on the terminal without any parameter, it shows the full system journal starting with the oldest log entry. If one or more arguments are passed, the output is filtered according to the argument set.
In this article, you will learn how to interpret and find systemd journals to troubleshoot problems in your Linux systems
HOW TO USE JOURNALCTL TO READ ALL SYSTEM LOGS
Running journalctl only on the command line will display the full system journal starting from the oldest. The messages that are highlighted in bold text are of priority notice or warning. Those shown in red are of priority error.
Let’s see how this works in the command line
You will get to see the red coloration when we are dealing with how logs can be filtered. Pay attention to every command.
HOW TO LIMIT THE OUTPUT OF JOURNAL SEARCHES
One of the important features of the journalctl is the ability to filter outputs to ease troubleshooting. By default, the journalctl –n displays the last 10 outputs. It can also take numbers as parameters to limit the number of outputs to display.
As you can see from the command above, the journalctl –n displays the last 10 logs. When specified with a number 3, it displays the last three log entries.
HOW TO FILTER OUT ERROR MESSAGES USING JOURNALCTL
They are sometimes you don’t want to go through all the junks the journalctl command will show to you. You need it to display all log entries by priority. The journalctl with the –p option will handle that. It can take the number or the name of the priority level as an argument.
A quick flashback of the well-known priority levels. We have the emerg, alert, crit, err, warning, notice, info, debug with numbers ranging from 0-7 respectively.
You can see the red coloration we talked about initially which signifies error priority. By default, it displays all log entries with error priority. You can also limit the numbers to display as shown below
HOW TO DISPLAY JOURNAL OUTPUT IN REAL TIME
You can also display the output of journal entry in the realtime mode to see all logs as they come into the system by specifying with the –f option just like using tail –f command.
You can see the prompt awaiting for new log entries to be displayed
HOW TO LIMIT JOURNAL ENTRY TO A SPECIFIC TIME FRAME
The journalctl command can also display journal entries in a specific time frame. It can take two options to limit the output of the journal entries to a particular time frame. The – -since and the – -until options. Both options can take a time parameter of YYY-MM-DD and hh:mm:ss. If the date is not specified, the system assumes it to be today. If the time is omitted, the system looks up with the 00:00:00. The – -since and – -until options can also take today, yesterday and tomorrow as an argument.
Let’s execute a command to list all journal entries from yesterday
The command above will list all the log entries from yesterday till date
let’s try listing entries from a specific date and time
This will display all journal entries from 24th of July 2019 at 16:23:45 to 25th of August 2019 at 18:23:35. Please note that the date and time specified should be enclosed with a quote
HOW TO PERMANENTLY STORE JOURNAL ENTRIES
By default, all journal logs are stored in the /run/log/journal, and these logs are wiped out immediately the system reboots. However, there is a way to go about this to make the logs available even after a reboot.
If a directory of /var/log/journal exists, the system will log all messages there instead and this will make log entries available after a reboot. Even when this is assumed to be persistent, it is not persistent to some extent. The journal has an inbuilt logrotate utility that rotates the log files when it is getting 10% filled with the available space. These values can be configured in the /etc/system/journal.conf file. You can specify the size at which you want your log entries to occupy, and after that, the logrotate utility can perform its operation based on the specification.
Let’s take a look at our current journal entry size limit. It can be seen on the first two lines of our log messages
Our journal entries should not use more than 48.6 MB, and we have currently used up 6 MB from the available space
Now let’s try to make our system journal persistent. Please follow the process
First, we have to create a directory /var/log/journal as root using our mkdir command
Now change the ownership of the directory to root and group ownership to systemd-journal and set permission 2755. A better interpretation of the permission; setting 2755 permission means setting sgid for the group, read, write, and execute permission for the user, read and execute permission for the group and others. This means that only the root user will have full access to the file
Now you are done. You can choose to reboot your system or send a special signal USR1 to systemd-journald as root. As an administrator, a reboot is not an option since Linux has all robust tool to keep your server running without a reboot.
When you are done, your system journal is now persistent across reboot, and you can use journalctl –b to filter out only the option since after the last reboot.
Now, you good to go.
HOW TO MAINTAIN JOURNAL
For better maintenance of your system and reduction of storage cost, you might need to do some clean up on your systems old journal entries.
They are various ways in which you can achieve that. However, I will be taking you into some basic steps while working on maintaining your Linux system.
First, you can check the total disk size your journal entries is currently occupying in the system by running journalctl and the – -disk-usage as an option.
After checking the disk usage, you can also choose to delete old logs from the system if you feel the journal entries are using more space than required. They are various ways available in deleting old logs in the system. You can choose to specify the – -vacuum-size as an option to shrink the size of the journal by indicating the size. This will remove all journal entries to the specified size. The sizes can be specified in K, M or G for Kilobyte, Megabyte and Gigabyte respectively
You can also choose to delete log files based on the time they are logged by specifying the time limit. Any log entries entered between that time range will be deleted.
You can also make persistent entry into the /etc/systemd/journal.conf configuration file to set how journal entries are treated in your server. As seen below, you can uncomment and edit the file to suit how you want your journal to be handled.
The following items from the command above can be used to limit the journal growth.
SystemMaxUse: It specifies the maximum space that can be used by the system journal for storing files persistently.
SystemKeepFree: It specifies the amount of space the systemd-journald should leave free while logging journal entries.
SystemMaxFileSize: It specifies the maximum size a particular journal entry is allowed to stay in the persistent storage before been rotated
RuntimeMaxUse: It specifies the maximum amount of space the systemd-journald can use in the volatile memory. That is in the /run directory
RunTimeKeepFree: It specifies the amount of reserved memory for other use
RunTimeMaxFileSize: It specifies the maximum amount of space a particular journal entry can stay in the volatile memory.
Please note that
HOW TO SET LOCAL TIME AND TIMEZONES
For proper management of your log messages, it is necessary you set the correct synchronized system time. Correct and accurate time information can be obtained over the internet via the Network Time Protocol (NTP).
The timedatectl is the command used to get an overview of the current time-related system settings, which includes the time zone, current time, and NTP synchronization of the system.
To list all available well-known timezones, you use the timedatectl list-timezones command
You can also use the tzselect command to identify the correct zone information of names of timezones. It will prompt the user of the location of the system and display the correct time zone
To set your system time, you use the timedatectl command, as seen below. Please note that before you can set your system time manually, you have to disable the automatic time synchronization. Which means you have to set the NTP server to false
As you can see from the command, I tried setting the time to 09:23 but the command was denied simply because my NTP server is still enabled. To go about this, you have to disable the NTP server by setting it to “false” using the following command timedatectl set-ntp false
CONCLUSION
Please note that when there is a system crash with a persistent journal. It is required to limit the output of your query to the reboot after the system crashed. You can use the –b option followed by a negative number. For example, you can use the journal –b -1 to limit the output of the systemd journal to the last boot.
I firmly believe you can now monitor your log files using journalctl command and you can as well set up a systemd-journal that is persistent across reboot.
REFERENCES:
Journalctl man page
Red Hat System Administration I Course Module