Improved disk usage with rsyslog
Running out of disk space can have detrimental impacts on your production servers like causing downtime until someone manually cleans up files on the server.
Today, we're introducing a change to also limit syslog's disk usage. By default, Ubuntu forwards all journald logs to rsyslog. This means logs are stored in journald's binary format and syslog's plain text format.
Ubuntu's default is for logrotate to clean up syslog files once a week, and save 4 weeks' worth of log files. As you can imagine, this could become quite large.
Hatchbox configures journald to use a maximum of 1G of disk for logs. This works great, but it doesn't apply to syslog meaning that noisy applications could still eat up disk space.
This change modifies two things in the rsyslog logrotate configuration.
Hatchbox now changes syslog to:
- Rotate logs daily instead of weekly
- Sets a maximum size to 1G
This means logrotate will ensure syslog's disk usage stays under 4GB.
Need to persist logs for longer? Services like AppSignal, PostHog, BetterStack, Grafana Loki, etc have features to forward logs from journald, rsyslog, or your application directly to their log aggregation systems. By saving logs externally, you can set whatever retention period you want and also search logs across multiple servers.
To apply these changes, update your server(s) to the latest configuration.