content:en_us:kb_o_creating_granular_reports_using_log_viewer

Creating Granular Reports Using Log Viewer

This guide will help you use the Log Viewer app to make relevant reports for various services. It will offer tips as to search terms that can be used to discover what is happening on your system. In some cases, this will augment the data you are getting from other Report tools. For very granular reporting you can use the log viewer with searches to get more detailed information.

How to use the Log Viewer App

To begin, the log viewer will list the various files in the system's '/var/log/' directory and subdirectories. It will parse these results in the pulldown menu. The amount of logs listed may be daunting to new users but you can use the following key to help you find the log file you want:

  • system
    • The system log will show you messages relevant to Webconfig, ClearOS specific warnings and errors, and the Marketplace
  • messages
    • The messages log will show you data relevant to the underlying operating system. This is one of the most useful logs.
  • squid/access.log
    • This log is very useful for tracking user authentication issues at the proxy. If you are using NTLM or other user authentication. This is quite useful for showing which user was on which IP address or if a service running on a workstation is NOT using the underlying NTLM to authenticate and is therefore not working.
  • dansguardian/access.log
    • This log is very, very useful. Using this tool you can see traffic from IP addresses and which content filter policy is being applied. You can also see if NO filter policy is being applied if the user hits the 'Default' policy. This will also show block codes for sites that are blocked.

The filter is the search field. You will want to use this frequently because most logs are quite chatty.

Explanation of Logs

System

Say you are getting an error in Webconfig. The app you are using is encountering a situation that produces an error. This log will show you what is happening behind the scenes and can be useful to produce data that is relevant to a bug report or a trouble ticket.

Messages

This log has so much going on. It tracks minutia detail about the various underlying services that are running on the operating system as well as kernel messages themselves. This log can be a bit big and contain lots of information. You'll be grateful for the filter option here because you can use it to track many, many problems.

Squid and DansGuardian

Say for instance you see some suspicious activity to a certain site that shows up in the general report, like a spike in blacklist hit activity. You can run the log viewer and search for the block codes.

There are two relevant logs for Content Filter. First there is the squid log which will show you bad authentications or software that is not properly configured to authenticate. These will show up as TCP_DENIED requests. This log will show the event and the relevant USER. Next is the dansguardian log which will show you the the disposition of the request through the relevant policy. It will show you the group, the IP and the state of the traffic through the filter (ie. whether it was blocked or passed.)

Anonymous

A '-' in the squid log where users would normally follow represents unauthenticated users. This can be a foreign or unaccounted device, like a mobile device on your network, or it can be a service on a machine that otherwise is incapable of pushing the user authentication. In cases where you need to have a 3rd party service processed through the proxy without authentication, you will need to manually add a 'whitelist' of those sites via command line.

In dansguardian, if a user or requested object is not categorized into a policy, it will go to the Default policy. Many administrators set this up as a walled garden. For example, they will blanket block ALL traffic in their Default policy and exempt things in a whitelist that are safe or otherwise ok to surf. This can include 3rd party apps that use web services and are unaware of the proxy authentication or fail the authentication. It can also include a website that the user can use to register in order to get better privilege on your server, or to issue a support ticket.

Targeted hits with Log Viewer

dansguardian example

So let's say I want to find the banned hits on a particular user. Go into the Log viewer app and choose the dansguardian/access.log. In the filter type the following:

192.168.1.101.*DENIED

Where 192.168.1.101 is the address you are looking for and you want to return any results that have the word DENIED in them after the IP address.

Since I block some add URLS like doubleclick.net on my Banned Site list, I get several results, including the following:

2014.5.11 7:18:20 - 192.168.1.101 http://ad.doubleclick.net/ad/N7096.8427.TRIBALFUSION.COM/B8037885.107265990;sz=1x1;ord=188406716? *DENIED* Banned site: ad.doubleclick.net GET 0 0 1 403 - Default -

The delimiter here for all the stuff in between is the '.*' characters (without the single quotes.) You can get more granular by limiting the results to a certain day or even time frame:

2014.5.11 [7-9].*192.168.1.101.*DENIED

The above would search for results only between 7:00 AM and 9:59 AM.

messages example

The entries in the messages log range from useless to extremely important. So let's say that you want to review any messages that the running kernel of the operating system has to say.

Select the messages log in the Log Viewer app. Then type 'kernel' in the search field. Here is an example of a result:

May 11 03:15:04 mysystemname kernel: device eth0 entered promiscuous mode
May 11 03:17:20 mysystemname kernel: EXT4-fs error (device dm-2): ext4_lookup: deleted inode referenced: 172885703

Uh oh, I have some filesystem issues I didn't know existed. Time to investigate and find out why that is happening.

Other messages may be to a specific service during a particular time. Let us say you are troubleshooting an issue with dnsmasq. A cursory search reveals pages and pages of too much information. I can use the filter to limit the results:

May 11 [01][0,2-9].*dnsmasq.*192.169.1.101

This will return all entries of DNSMasq involving 192.168.1.101 on May 11 from Midnight to 00:59 AM and from 2:00 AM to 7:59 PM.

content/en_us/kb_o_creating_granular_reports_using_log_viewer.txt · Last modified: 2014/12/23 21:05 (external edit)