Dealing with Log Files
Log files are collected on ClearOS in the /var/log directory and its subdirectories. There are situations in which you may need to address issues with log files on ClearOS as a part of general maintenance. These situations may include:
- Monitoring logs for current activity as opposed to past activity
- Searching through log files for certain information to create custom reports
- Dealing with log files that may be filling up your hard disk
- Retaining log files beyond the current limits or reducing their retention
Monitoring Log Files
Active log files are dynamically created and retired on the system. You can distinguish between active logs that are current and activity logs in the past by looking at the files in the various directories. Find the files which do NOT have a date-stamp on the filename. That file is the active file and has changes made to it all the time. To look at this file dynamically, launch the console or SSH into your ClearOS server and tail the file. For example, to watch the system log to see why a service is not starting, you might run the following:
tail -f /var/log/messages
This would show you all the active entries to this file. In another session or in Webconfig, attempt to start the service. That service will then spit out messages to the file in real time. This will show you what is going on.
You can even search the output for specific information. For example, if you want to limit the output to show only messages from systemd in this file you might use the command:
tail -f /var/log/messages | grep systemd
To cancel a tail, hit 'CTRL+c', the cancel keystroke in Linux.
Searching Log Files
As mentioned above, you can use the 'grep' command for search for specific data in your log files. To search a specific logfile directly run the grep command with your search term on the file:
grep searchterm /var/log/logfile.name
For example, kernel messages may indicate issues with hardware. The following command would show that:
grep kernel /var/log/messages
You can even search this and PAST logs with wildcards
grep kernel /var/log/messages*
Command line is not your only resource for generating this information. The Log Viewer module found in the 'Reports' section will allow you to search specific files for specific or wildcard terms. Navigate to the Log Viewer under Reports » Performance and Resources » Log Viewer.
The following image shows searching for kernel entries in the messages log file:
Large Log Files
Log files can get big in ClearOS when things are going wrong or on systems that have to deal with a lot of activity. Under ClearBOX or if you followed our guide for partitioning and using bind mounts your log files will be sequestered into a partition where it cannot utterly crash your system through run-away logging.
Dealing with log files is an important function of administration of ClearOS because it can exist as a way to deny service by causing a system to log too much. Hackers know this and dealing with attempts to fill up a server with log files is an important strategy. The default configuration of ClearOS will help mitigate this by cleaning up log files on a regular basis.
To find out how much space your log files are using on your system, you can use the “Disk Usage Report” from the ClearCenter Marketplace or you can use command line.
In the Disk Usage Report, drill down to /var and then ./log. This will show the files and subdirectories involved and which one are using a lot of space.
From command line you can accomplish similar results. Navigate to the /var/log directory and run the 'du' command in order to assess the sizes of directories involved:
cd /var/log du --max-depth=1 -h
You may get a result similar to this:
0 ./iptraf-ng 8.0K ./samba 6.0M ./webconfig 0 ./ppp 0 ./ntpstats 30M ./audit 36K ./tuned 0 ./chrony 712K ./anaconda 1.1M ./clamav 0 ./cups 884K ./httpd 0 ./BackupPC 236K ./snort 0 ./mariadb 0 ./proftpd 313M .
You can see from this report the sizes of the various subdirectories and the TOTAL of the present directory on the last line (313 Megabytes of total log file usage). If your run-away log files is in one of these subdirectories, change to that directory now and run the second part of the discovery which is to list the files. To list files in a directory from smallest to largest, run the following from the directory you are investigating:
ls -lahSr
Your last results may look similar to this:
-rw-r--r-- 1 root root 371K May 22 03:50 cron-20160522 -rw------- 1 root root 402K Feb 7 03:39 messages-20160207 -rw-r--r-- 1 root root 428K Feb 14 03:49 cron-20160214 -rw-r--r-- 1 root root 492K May 16 03:37 cron-20160516 -rw-r--r-- 1 root root 1.4M Feb 5 14:40 syswatch-20160205 -rw------- 1 root root 1.8M May 8 03:36 messages-20160508 -rw-------. 1 root root 1.9M Feb 5 14:45 messages-20160205 -rw-r--r-- 1 root root 4.1M Feb 14 03:48 syswatch-20160214 -rw------- 1 root root 4.1M May 3 17:10 messages-20160503 -rw-r--r-- 1 root root 4.7M May 3 17:04 syswatch-20160503 -rw------- 1 root root 4.8M May 26 10:09 messages -rw------- 1 root root 5.0M Feb 14 03:48 messages-20160214 -rw------- 1 root root 6.2M May 22 03:50 messages-20160522 -rw------- 1 root root 8.7M May 16 03:37 messages-20160516 -rw------- 1 root root 12M May 26 09:57 secure -rw------- 1 root root 22M May 8 03:32 secure-20160508 -rw------- 1 root root 28M May 22 03:48 secure-20160522 -rw------- 1 root root 45M May 16 03:36 secure-20160516 -rw------- 1 root utmp 123M May 26 07:37 btmp
Now I Know What Files Are Large, How Do I Get Rid of Them
You can delete some files here. The safest ones to remove are those that have older date-stamps on them. Before removing these files, read on. But to remove it now run something similar to this:
rm messages-20160522
The reason why I ask that you read on is because you have an opportunity here to find out WHY your log files are getting big and to deal with this situation. Your log system will regularly purge files after a month so the system SHOULD be self-sustaining. But if your system has a problem, you may want to at least record what is going on by copying the files to another resource before just purging them. After you have a copy or after you have addressed the issue, you can deal with the log file issue by removing just enough to handle the full drive issue.
Retaining Log Files
The logrotate functions of ClearOS will archive files with date stamps indicating the END of the log file. Such logs will be purged off of the system on a monthly basis. You can tweak these value by modifying the /etc/logrotate.conf file. A daily process will run the command '/usr/sbin/logrotate' against this configuration file to know what you want to do with the log files. To learn more about the parameters of this file, consult the manual file:
man logrotate
Some things you can do to trim the size of log files is to compress them. You can uncomment the #compress directive by removing the # symbol. The affect on the system will be that it will take up less space. The downside is that it will make the log files unreadable by certain subsystems. For example, you'd only be able to review files in the Webconfig reports for the current file. This is detrimental to the Proxy and Bypass Report module, for example but this might be exactly what you want. You can also limit or extend retention on the rotate line. Here are some of the typical values manipulated:
- weekly - rotate files that are older than one week
- rotate - the number of weeks to retain a rotated file before it is deleted. The default in ClearOS is 10 weeks
- compress - Whether to compress the rotated files. This will save space but affect reports