How can we help you today?

How to Configure Automatic Logfile Rotation

Created by: Sophie van Vrijberghe de Coningh

Modified on: Fri, 28 May, 2021 at 1:47 PM


We have a hypernode-auto-logrotate command as part of our hypernode- system utilities. This is a command line tool that automatically rotates log files, which helps avoid disk space issues and decreases time the required to restore back-ups.

TABLE OF CONTENTS

Log Rotation

A hypernode contains files that applications write their logs to. For example, Magento writes to /data/web/magento2/var/log/debug.log. These log files can get quite large (we’ve seen examples of over 30GB), so for example when doing a backup, you end up transferring obscenely large files over the internet.

You can avoid this problem by rotating the log files. This means that when a log file /data/web/example.log gets too big, you compress it, move it to /data/web/example.log.1.gz, and give the application an empty log file to write to. The hypernode-auto-logrotate command automates this process.

Example Usage

For a log file ~/example.log, the tool can be used as follows:

app@levkd4-example-magweb-cmbl:~$ hypernode-auto-logrotate example.log 
Adding logrotate cron job 
Adding logrotate config entry for "/data/web/example.log"

This will cause the log file to be rotated on a daily basis, at around midnight. If the file is over 50MB, the next day the old logs will have been moved to ~/example.log.1.gz.

app@levkd4-example-magweb-cmbl:~$ ls example.log*
example.log  example.log.1.gz

The tool also has a --detect option, with which it will search for log files and rotate them:

app@levkd4-example-magweb-cmbl:~$ hypernode-auto-logrotate --detect
Searching for logfiles...
Logfile found: /data/web/example.log
Logfile found: /data/web/magento2/var/log/system.log
Logfile found: /data/web/magento2/var/log/connector.log
Logfile found: /data/web/magento2/var/log/debug.log
Logfile found: /data/web/magento2-sample-data/dev/tools/exclude.log
Logfile "/data/web/example.log" is already being rotated
Adding logrotate config entry for "/data/web/magento2/var/log/system.log"
Adding logrotate config entry for "/data/web/magento2/var/log/connector.log"
Adding logrotate config entry for "/data/web/magento2/var/log/debug.log"
Adding logrotate config entry for "/data/web/magento2-sample-data/dev/tools/exclude.log"

The tool has a --dry-run option which will print what the tool will do without actually doing it. For more detailed output, run it with the --verbose option.

Configuration

The hypernode-auto-logrotate command comes with a sensible out-of-the-box configuration. Among other things, this means that it will rotate log files daily, only when they exceed 50MB, and that it will keep four files in rotation (meaning you end up with example.log.1.gz up to example.log.4.gz). These configurations are stored in /data/web/hypernode_logrotate.conf, and can be customized per log by editing this file. For example, the entry added for example.log reads

/data/web/example.log {
    rotate 4
    daily
    compress
    # do not raise an error if the logfile does not exist
    missingok
    # file size at which rotation is triggered
    size 50M
    # copy the file and truncate the original, in case
    # another process cannot be told to close the logfile
    copytruncate
}

The logrotate main page contains instructions on how to customize this.

S
Sophie is the author of this solution article.

Did you find it helpful? Yes No

Send feedback
Sorry we couldn't be helpful. Help us improve this article with your feedback.