✎ Content Filter [ClearOS Documentation]

You've loaded an old revision of the document! If you save it, you will create a new version with this data.

===== Content Filter =====
The content filtering software blocks inappropriate websites from the end user. The software can also be used to enforce company policies; for instance, blocking personal webmail sites like Hotmail can decrease lost productivity at the office.

The filter engine uses a variety of methods including phrase matching, URL filtering, and black/white lists.  Although the filter works effectively 'out-of-the-box', we recommend subscribing to a service level the includes the 'Content Filter Update' service (see Services link below) for best results.  By keeping your blacklist up-to-date, you will be providing your LAN with the most effective blocking solution against the 'churn' of sites that change daily.

You can find more information about the underlying technology in the  [[:content:en_us:kb_o_content_filtering_ins_and_outs|Content Filtering Ins and Outs]] document.

If you are new to ClearOS and/or setting up a content filter policies, you may want to refer to the [[:content:en_us:kb_o_clearos_guides_setting_up_content_filter_policies|Guide to Setting up Web Proxy, Content Filter and Access Control Guide]].

<note>To use the Content Filter in non-transparent mode, please configure your browser to use proxy port 8080 or set up [[content:en_us:kb_o_web_proxy_auto-detection_configuration|Web Proxy Auto-Discovery (WPAD)]] to do it for you.</note>

<note warning>Please note that the Proxy/Content Filter and Gateway Management apps are mutually exclusive. If you have the Proxy/Content Filter running, you should not use Gateway Management and vice-versa</note>
===== Installation =====
If your system does not have this app available, you can install it via the [[:content:en_us:6_marketplace|Marketplace]].

===== Menu =====
You can find this feature in the menu system at the following location:
 
<navigation>Gateway|Proxy and Filtering|Content Filter</navigation>

===== Content Filter Updates - Blacklists =====
The ClearCenter [[http://www.clearcenter.com/Services/clearsdn-content-filter-updates-7.html|Content Filter Updates]] service provides regular blacklist updates to improve the effectiveness of the content filter system, including blocking HTTPS sites (e.g. <nowiki>https://facebook.com</nowiki>).  These blacklists are compiled from third party organizations as well as internal engineering resources from ClearCenter.  We keep tabs on the latest available updates and fine tune the system so you can focus on more important things.

The Content Filter also hooks into the [[:content:en_us:7_ug_antivirus|Gateway Antivirus]] and [[:content:en_us:7_ug_antiphishing|Gateway Antiphishing]] engines in ClearOS.  You may also want to subscribe to the [[http://www.clearcenter.com/Services/clearsdn-antimalware-updates-7.html|Antimalware Updates]] service to keep your content filter running at its optimum.

===== Blocking HTTPS, Facebook, etc =====
If you need to block web sites that provide access via secure HTTP (HTTPS) - for example facebook.com - then you need to enable non-transparent mode for your web proxy / content filter setup.  In transparent mode, it is not possible to properly filter secure web pages since the connection to the web server is already encrypted (and unreadable) by the time it gets to the ClearOS gateway.  You can change the transparent/non-transparent mode setting from the [[:content:en_us:7_ug_web_proxy|Web Proxy]] app.

As an alternative, if you know the IP or IP range that you would like to block, you can block connections to a particular site using the [[:content:en_us:7_ug_incoming_firewall|Incoming Firewall]] and the [[:content:en_us:7_ug_egress_firewall|Egress Firewall]].  The user will not see a helpful warning page with this alternative method, just a failed connection message in their web browser.

<note warning>By design, the HTTPS protocol encrypts the payload to ensure the web browser and HTTPS server have a secure channel.  This is what makes it possible to do online banking securely, for example.  An HTTPS page cannot be scanned for keywords or other content.

However, HTTPS addresses/URLs are not encrypted.  If filtering HTTPS sites is important, then the regularly updated blacklists from the [[:content:en_us:7_ug_content_filter_updates|Content Filter Updates]] paid app are recommended.</note>

===== Configuration =====
The web-based administration tool gives you access to a number of configuration settings.  The filter **must** be run in parallel with the [[:content:en_us:6_web_proxy|Web Proxy]] server.

==== Enable/Disable Service ====
{{ :omedia:ss_content_filter_status.png?250|Content Filter Status}}

The content filter service is enabled when both the content filter service is running **and** the proxy server.  To determine if these services are running, look to the App Status bar on the right-hand-side of the web-based interface.  You will find a status field along with start/stop controls to toggle the service.

==== Global Settings ====
The global settings apply to all users, regardless of the content filter group being applied to the user browsing websites from the Local Area Network (LAN).  In fact, the settings explicitly exempt or ban a device on the LAN from using the proxy/content filter service.  A device is identified by its source IP address.

<note important>If you are using global exemption or banned IP addresses, it is good practice to ensure these systems are assigned static IP addresses.</note>

==== Group Policies ====
Group policy settings allow an administrator to 'fine-tune' how the content filter applies policies to different users.  To do this, group policies (not to be confused with Windows AD Group Policy) are created and configured.  Users are assigned to a group which dictates what policies are enforced on their browsing habits.

The content filter engine supports up to 9 different group policies - each of which can be configured to the administrators preference.

<note important>Be descriptive in naming your group policies.  For example, a school might have policies named for the types of individuals that might be accessing the Internet from the school - Support Staff, Teachers, Students, Parents, Visitors etc.</note>

When you first configure the content filter engine, one group policy is already created - named "Default".  By default, this group policy applies to all users.  If a user is assigned a content filter group policy, these settings will supersede the default group policy.

The **Default** Group Policy cannot be deleted, nor can the default setting of 'allusers' be modified.  You can (and should) modify the settings of this group to comply with the desired filtration of any user not falling into a higher up policy.

==== Adding a Group Policy ====
As previously mentioned, up to 9 (including the Default Policy) group policies can be created for the content filter.  To add a policy, click **Add** in the Group Policy table list.

The only configuration setting to be made after you create a group policy is to assign an actual group to it.  Users can be assigned to groups - this is how the filter will apply specific settings.

<note tip>Filtering based on device IP addresses is not currently supported via webconfig.</note>

==== Editing a Group Policy ====
Once you have added a group policy (or if you need to edit the default policy) it is time to edit/configure it.

Click on the "Configure Policy" link next to the group policy you wish to edit.  You will see a summary of parameters/categories to edit, similar to that in the screenshot below.

=== General Settings ===
== Sensitivity Level ==
The sensitivity level is an arbitrary scale that allows 'coarse' adjustment of the phrase filter sensitivity.  Increasing the sensitivity level means that fewer bad phrases/words will cause the filter to block the page.

== PICS Level ==
An Internet standard for rating web content.  This setting will prove to be of minor significance as sites self-administrate this parameter.  As a general rule, the recommendation is to disable this setting.

== Reporting Level ==
Several options are available to customize what a user sees when the filter blocks a page:

* Stealth Mode - Site is not blocked...User's IP and site is logged

* Access Denied - User's browser will receive an 'Access Denied' in place of the web page.

* Short Report - A short error message 'bubble' will be displayed like the one below

* Full Report - Same as above, but the weighted limit and actual value will be displayed (useful for fine-tuning the system).

* Custom Report - Uses the customizable HTML template

== Block IP Domains ==
Used to prevent users from circumnavigating the URL-based portion of the filter by using IP addresses instead of URL's.  Pages will still be filtered based on the other filtering mechanisms: weightedphrases, mime types, file extensions etc.

== Blanket Block ==
Most restrictive setting.  All sites will be blocked with the exception of those listed in the exempt list.  Useful for kiosks/public terminals where a browser is used to access a company site etc.

=== Blacklists ===
The content filter system uses black lists to block specific web sites. You can fine tune your content filter black lists by specifying which lists to use.  Note that these lists are updated weekly by the //**Content Filter Update Service** // if you have subscribed to that service.

=== Phrase Lists ===
The content filter system uses phrase lists to calculate a score for every web page. You can fine tune your content filter scoring by specifying which phrase lists to use.

In general you will want the phrase lists you select here to correspond with the blacklists you are using.  At a minimum you will want to include the //proxies// phrase list to prevent your users from bypassing the filter.

<note info>Note that more weighted phrases activated for the content filter mean that the filter will take more time to look at each page.  It is recommended that if you are using a low powered server, you limit the number of weighted phrase lists you use and instead use more blacklists. 
</note>
=== MIME Types ===
MIME types instruct a browser to utilize certain applications in order to display content encoding.  Security exploits in the applications themselves can be used to infiltrate a computer.  MIME types checked in the "Banned MIME Types" form will not be allowed to pass through the firewall and to the computer making the request on the LAN, providing a more secure environment.

=== File Extensions ===
Banning specific file extensions is a useful tool for limiting content available to users on the LAN.  It can also greatly decrease the chances of users unwittingly downloading and running 'arbitrary' code downloaded from the Internet which could potentially contain viruses, spyware of other malicious code.

By checking a box next to an extension, you are **disallowing** filtered users from accessing this file type.  If you wish an extension to be blocked and it is not listed in the available list, add it to the list using the "Add a new extension type" form.

=== Banned Sites ===
Sites entered in the "Banned Site List" will be banned, regardless of the site's content, or whether the site is on one of the blacklists.

=== Gray Sites ===
Sites entered in the "Grey Site List" will not be blocked by the blacklists but will still be checked for content.  For example, you may have the **news** blacklist enabled to prevent people from wasting time during the business day.  However, you may have also decided to allow just BBC news. If you add bbc.co.uk to the exception list, all web pages will be allowed.  If you add bbc.co.uk to the greylist, then most pages will pass through just fine, but [[http://news.bbc.co.uk/2/hi/health/1346542.stm|this mildly racy page]] and other might get blocked by the phrase list system.

=== Exception Sites ===
Sites entered in the "Exempt Site List" will be allowed, regardless of the site's content.  Use this form if content on a site triggers a 'false positive' that you wish to override.

===== Microsoft Active Directory Integration =====
To proxy/filter when there is a MS Active Directory on the server, please see [[:content:en_us:kb_bestpractices_content_filtering_with_active_directory|Content Filtering with Active Directory]].

===== Tips =====
If you find your cache files building in /var/cache/dansguardian then you should be able to delete the old ones. You can add a job to /etc/cron.daily with some thing like:
<code>find /var/cache/dansguardian -mtime +7 -delete</code>in it to clear out cache files older than 7 days.
===== Links =====
  * [[:content:en_us:kb_o_content_filtering_ins_and_outs|Content Filtering Ins and Outs]]
  * [[:content:en_us:kb_bestpractices_content_filtering_with_active_directory|Content Filtering with Active Directory]]
  * [[:content:en_us:6_web_proxy|Web Proxy]]
  * [[:content:en_us:6_content_filter_updates|Content Filter Updates]]
  * [[:content:en_us:kb_o_clearos_guides_setting_up_content_filter_policies|Guide to Setting up Web Proxy, Content Filter and Access Control Guide]]
{{keywords>clearos, clearos content, Content Filter, app-content-filter, clearos7, userguide, categorygateway, subcategorycontentfilterandproxy, maintainer_dloper}}

Edit summary

Note: By editing this page you agree to license your content under the following license: CC Attribution-Noncommercial-Share Alike 4.0 International

CLEAROS DOCUMENTATION

Sitemap

Foundation

Company

Partners

Purchase

Contact Us