net/adblock/files/README.md

   1 # dns based ad/abuse domain blocking
   2
   3 ## Description
   4 A lot of people already use adblocker plugins within their desktop browsers, but what if you are using your (smart) phone, tablet, watch or any other wlan gadget...getting rid of annoying ads, trackers and other abuse sites (like facebook ;-) is simple: block them with your router. When the dns server on your router receives dns requests, you will sort out queries that ask for the resource records of ad servers and return a simple 'NXDOMAIN'. This is nothing but **N**on-e**X**istent Internet or Intranet domain name, if domain name is unable to resolved using the dns server, a condition called the 'NXDOMAIN' occurred.
   5
   6 ## Main Features
   7 * support of the following domain blocklist sources (free for private usage, for commercial use please check their individual licenses):
   8     * [adaway](https://adaway.org)
   9     * => infrequent updates, approx. 400 entries (enabled by default)
  10     * [adguard](https://adguard.com)
  11     * => numerous updates on the same day, approx. 12.000 entries
  12     * [bitcoin](https://github.com/hoshsadiq/adblock-nocoin-list)
  13     * => infrequent updates, approx. 15 entries
  14     * [blacklist]()
  15     * => static local blacklist, located by default in '/etc/adblock/adblock.blacklist'
  16     * [disconnect](https://disconnect.me)
  17     * => numerous updates on the same day, approx. 6.500 entries (enabled by default)
  18     * [dshield](http://dshield.org)
  19     * => daily updates, approx. 4.500 entries
  20     * [feodotracker](https://feodotracker.abuse.ch)
  21     * => daily updates, approx. 0-10 entries
  22     * [hphosts](https://hosts-file.net)
  23     * => monthly updates, approx. 50.000 entries
  24     * [malwaredomains](http://malwaredomains.com)
  25     * => daily updates, approx. 16.000 entries
  26     * [malwaredomainlist](http://www.malwaredomainlist.com)
  27     * => daily updates, approx. 1.500 entries
  28     * [openphish](https://openphish.com)
  29     * => numerous updates on the same day, approx. 1.800 entries
  30     * [ransomware tracker](https://ransomwaretracker.abuse.ch)
  31     * => daily updates, approx. 150 entries
  32     * [reg_cn](https://easylist-downloads.adblockplus.org/easylistchina+easylist.txt)
  33     * => regional blocklist for China, daily updates, approx. 1.600 entries
  34     * [reg_id](https://easylist-downloads.adblockplus.org/abpindo+easylist.txt)
  35     * => regional blocklist for Indonesia, daily updates, approx. 800 entries
  36     * [reg_nl](https://easylist-downloads.adblockplus.org/easylistdutch+easylist.txt)
  37     * => regional blocklist for the Netherlands, weekly updates, approx. 1300 entries
  38     * [reg_pl](http://adblocklist.org)
  39     * => regional blocklist for Poland, daily updates, approx. 50 entries
  40     * [reg_ro](https://easylist-downloads.adblockplus.org/rolist+easylist.txt)
  41     * => regional blocklist for Romania, weekly updates, approx. 600 entries
  42     * [reg_ru](https://code.google.com/p/ruadlist)
  43     * => regional blocklist for Russia, weekly updates, approx. 2.000 entries
  44     * [securemecca](http://www.securemecca.com)
  45     * => infrequent updates, approx. 25.000 entries
  46     * [shallalist](http://www.shallalist.de) (categories "adv" "costtraps" "spyware" "tracker" "warez" enabled by default)
  47     * => daily updates, approx. 32.000 entries (a short description of all shallalist categories can be found [online](http://www.shallalist.de/categories.html))
  48     * [spam404](http://www.spam404.com)
  49     * => infrequent updates, approx. 5.000 entries
  50     * [sysctl/cameleon](http://sysctl.org/cameleon)
  51     * => weekly updates, approx. 21.000 entries
  52     * [whocares](http://someonewhocares.org)
  53     * => weekly updates, approx. 12.000 entries
  54     * [winhelp](http://winhelp2002.mvps.org)
  55     * => infrequent updates, approx. 15.000 entries
  56     * [winspy](https://github.com/crazy-max/WindowsSpyBlocker)
  57     * => infrequent updates, approx. 120 entries
  58     * [yoyo](http://pgl.yoyo.org/adservers)
  59     * => weekly updates, approx. 2.500 entries (enabled by default)
  60     * [zeus tracker](https://zeustracker.abuse.ch)
  61     * => daily updates, approx. 440 entries
  62 * zero-conf like automatic installation & setup, usually no manual changes needed
  63 * simple but yet powerful adblock engine: adblock does not use error prone external iptables rulesets, http pixel server instances and things like that
  64 * supports five different dns backends / blocklist formats: dnsmasq, unbound, named (bind), kresd and dnscrypt-proxy
  65 * automatically selects uclient-fetch or wget as download utility (other tools like curl or aria2c are supported as well)
  66 * provides 'http only' mode without installed ssl library for all non-SSL blocklist sources
  67 * supports a wide range of router modes, even AP modes are supported
  68 * full IPv4 and IPv6 support
  69 * provides top level domain compression ('tld compression'), this feature removes thousands of needless host entries from the blocklist and lowers the memory footprint for the dns backends
  70 * blocklist source parsing by fast & flexible regex rulesets
  71 * overall duplicate removal in central blocklist 'adb_list.overall'
  72 * additional whitelist for manual overrides, located by default in /etc/adblock/adblock.whitelist
  73 * quality checks during blocklist update to ensure a reliable dns backend service
  74 * minimal status & error logging to syslog, enable debug logging to receive more output
  75 * procd based init system support (start/stop/restart/reload/suspend/resume/query/status)
  76 * procd network interface trigger support or classic time based startup
  77 * suspend & resume adblock actions temporarily without blocklist reloading
  78 * output comprehensive runtime information via LuCI or via 'status' init command
  79 * query function to quickly identify blocked (sub-)domains, e.g. for whitelisting
  80 * strong LuCI support
  81 * optional: force dns requests to local resolver
  82 * optional: force overall sort / duplicate removal for low memory devices (handle with care!)
  83 * optional: automatic blocklist backup & restore, they will be used in case of download errors or during startup in backup mode
  84 * optional: 'backup mode' to re-use blocklist backups during startup, get fresh lists only via reload or restart action
  85 * optional: 'whitelist mode' to block access to all domains except those explicitly listed in the whitelist file
  86 * optional: add new adblock sources on your own via uci config
  87
  88 ## Prerequisites
  89 * [LEDE project](https://www.lede-project.org), tested with latest stable release (LEDE 17.01) and with current LEDE snapshot
  90 * a usual setup with an enabled dns backend at minimum - dump AP modes without a working dns backend are _not_ supported
  91 * a download utility:
  92     * to support all blocklist sources a full version (with ssl support) of 'wget', 'uclient-fetch' with one of the 'libustream-*' ssl libraries, 'aria2c' or 'curl' is required
  93     * for limited devices with real memory constraints, adblock provides also a 'http only' option and supports wget-nossl and uclient-fetch (without libustream-ssl) as well
  94     * for more configuration options see examples below
  95
  96 ## Installation & Usage
  97 * install 'adblock' (_opkg install adblock_)
  98 * at minimum configure the appropriate dns backend ('dnsmasq' by default) and enable the adblock service in _/etc/config/adblock_
  99 * control the adblock service manually with _/etc/init.d/adblock_ start/stop/restart/reload/suspend/resume/status or use the LuCI frontend
 100
 101 ## LuCI adblock companion package
 102 * for easy management of the various blocklist sources and all other adblock options you should use the provided LuCI frontend
 103 * install 'luci-app-adblock' (_opkg install luci-app-adblock_)
 104 * the application is located in LuCI under 'Services' menu
 105
 106 ## Tweaks
 107 * **runtime information:** the adblock status is available via _/etc/init.d/adblock status_ (see example below)
 108 * **debug logging:** for script debugging please set the config option 'adb\_debug' to '1' and check the runtime output with _logread -e "adblock"_
 109 * **storage expansion:** to process and store all blocklist sources at once it might helpful to enlarge your temp directory with a swap partition => see [openwrt wiki](https://wiki.openwrt.org/doc/uci/fstab) for further details
 110 * **add white- / blacklist entries:** add domain white- or blacklist entries to always-allow or -deny certain (sub) domains, by default both lists are empty and located in _/etc/adblock_. Please add one domain per line - ip addresses, wildcards & regex are _not_ allowed (see example below)
 111 * **backup & restore blocklists:** enable this feature, to restore automatically the latest compressed backup of your blocklists in case of any processing error (e.g. a single blocklist source is not available during update). Please use an (external) solid partition and _not_ your volatile router temp directory for this
 112 * **scheduled list updates:** for a scheduled call of the adblock service add an appropriate crontab entry (see example below)
 113 * **change startup behaviour:** by default the startup will be triggered by the 'wan' procd interface trigger. Choose 'none' to disable automatic startups, 'timed' to use a classic timeout (default 30 sec.) or select another trigger interface.
 114 * **suspend & resume adblocking:** to quickly switch the adblock service 'on' or 'off', simply use _/etc/init.d/adblock [suspend|resume]_
 115 * **domain query:** to query the active blocklist for a specific domain, please run _/etc/init.d/adblock query `<DOMAIN>`_ (see example below)
 116 * **add new list sources:** you could add new blocklist sources on your own via uci config, all you need is a source url and an awk one-liner (see example below)
 117 * **disable active dns probing in windows 10:** to prevent a yellow exclamation mark on your internet connection icon (which wrongly means connected, but no internet), please change the following registry key/value from "1" to "0" _HKLM\SYSTEM\CurrentControlSet\Services\NlaSvc\Parameters\Internet\EnableActiveProbing_
 118
 119 ## Further adblock config options
 120 * usually the pre-configured adblock setup works quite well and no manual config overrides are needed, all listed options apply to the 'global' config section:
 121     * adb\_enabled => main switch to enable/disable adblock service (default: '0', disabled)
 122     * adb\_debug => enable/disable adblock debug output (default: '0', disabled)
 123     * adb\_dns => select the dns backend for your environment: 'dnsmasq', 'unbound', 'named', 'kresd' or 'dnscrypt-proxy' (default: 'dnsmasq')
 124     * adb\_dnsdir => target directory for the generated blocklist 'adb_list.overall' (default: not set, use dns backend default)
 125     * adb\_trigger => set the startup trigger to a certain interface, to 'timed' or to 'none' (default: 'wan')
 126     * adb\_triggerdelay => additional trigger delay in seconds before adblock processing begins (default: '1')
 127     * adb\_fetch => full path to a dedicated download utility, see example below (default: not set, use wget default)
 128     * adb\_fetchparm => options for the download utility, see example below (default: not set, use wget default options)
 129     * adb\_forcedns => force dns requests to local resolver (default: '0', disabled)
 130     * adb\_forcesrt => force overall sort on low memory devices with less than 64 MB RAM (default: '0', disabled)
 131     * adb\_backup_mode => do not automatically update blocklists during startup, use backups instead (default: '0', disabled)
 132     * adb\_whitelist_mode => block access to all domains except those explicitly listed in the whitelist file (default: '0', disabled)
 133
 134 ## Examples
 135 **change default dns backend to 'unbound':**
 136
 137 Adblock deposits the final blocklist 'adb_list.overall' in '/var/lib/unbound' where unbound can find them in its jail. If you use manual configuration for unbound, then just include the following line in your 'server' clause:
 138 <pre><code>
 139   include: "/var/lib/unbound/adb_list.overall"
 140 </code></pre>
 141
 142 **change default dns backend to 'named' (bind):**
 143
 144 Adblock deposits the final blocklist 'adb_list.overall' in '/var/lib/bind'. To use the blocklist please modify '/etc/bind/named.conf':
 145 <pre><code>
 146 in the 'options' namespace add:
 147   response-policy { zone "rpz"; };
 148
 149 and at the end of the file add:
 150   zone "rpz" {
 151     type master;
 152     file "/var/lib/bind/adb_list.overall";
 153     allow-query { none; };
 154     allow-transfer { none; };
 155   };
 156 </code></pre>
 157
 158 **change default dns backend to 'kresd':**
 159
 160 The knot-resolver (kresd) is only available on turris omnia devices. Adblock deposits the final blocklist 'adb_list.overall' in '/etc/kresd'. To use the blocklist please modify '/etc/config/resolver':
 161 <pre><code>
 162     list rpz_file '/etc/kresd/adb_list.overall'
 163 </code></pre>
 164
 165 **change default dns backend to 'dnscrypt-proxy':**
 166
 167 The required 'blacklist' option of dnscrypt-proxy is not enabled by default, because the package will be compiled without plugins support. Take a custom LEDE build with plugins support to use this feature. Adblock deposits the final blocklist 'adb_list.overall' in '/tmp'. To use the blocklist please modify '/etc/config/dnscrypt-proxy' per instance:
 168 <pre><code>
 169   list blacklist 'domains:/tmp/adb_list.overall'
 170 </code></pre>
 171
 172 **configuration for different download utilities:**
 173 <pre><code>
 174 wget (default):
 175   option adb_fetch '/usr/bin/wget'
 176   option adb_fetchparm '--quiet --no-cache --no-cookies --max-redirect=0 --timeout=10 --no-check-certificate -O'
 177
 178 aria2c:
 179   option adb_fetch '/usr/bin/aria2c'
 180   option adb_fetchparm '-q --timeout=10 --allow-overwrite=true --auto-file-renaming=false --check-certificate=false -o'
 181
 182 uclient-fetch:
 183   option adb_fetch '/bin/uclient-fetch'
 184   option adb_fetchparm '-q --timeout=10 --no-check-certificate -O'
 185
 186 curl:
 187   option adb_fetch '/usr/bin/curl'
 188   option adb_fetchparm '-s --connect-timeout 10 --insecure -o'
 189 </code></pre>
 190
 191 **receive adblock runtime information:**
 192 <pre><code>
 193 /etc/init.d/adblock status
 194 ::: adblock runtime information
 195   + adblock_status  : enabled
 196   + adblock_version : 3.1.0
 197   + overall_domains : 5117
 198   + fetch_utility   : wget (built-in)
 199   + dns_backend     : kresd (/etc/kresd)
 200   + last_rundate    : 03.11.2017 22:57:41
 201   + system_release  : Turris Omnia, OpenWrt omnia 15.05/3.8.4
 202 </code></pre>
 203
 204 **cronjob for a regular block list update (/etc/crontabs/root):**
 205 <pre><code>
 206 0 06 * * *    /etc/init.d/adblock reload
 207 </code></pre>
 208
 209 **blacklist entry (/etc/adblock/adblock.blacklist):**
 210 <pre><code>
 211 ads.example.com
 212
 213 This entry blocks the following (sub)domains:
 214   http://ads.example.com/foo.gif
 215   http://server1.ads.example.com/foo.gif
 216   https://ads.example.com:8000/
 217
 218 This entry does not block:
 219   http://ads.example.com.ua/foo.gif
 220   http://example.com/
 221 </code></pre>
 222
 223 **whitelist entry (/etc/adblock/adblock.whitelist):**
 224 <pre><code>
 225 here.com
 226
 227 This entry removes the following (sub)domains from the blocklist:
 228   maps.here.com
 229   here.com
 230
 231 This entry does not remove:
 232   where.com
 233   www.adwhere.com
 234 </code></pre>
 235
 236 **query the active blocklist for a certain (sub-)domain, e.g. for whitelisting:**
 237 The query function checks against the submitted (sub-)domain and recurses automatically to the upper top level domain. For every (sub-)domain it returns the first ten relevant results.
 238 <pre><code>
 239 /etc/init.d/adblock query www.example.google.com
 240 ::: max. ten results for domain 'www.example.google.com'
 241   - no match
 242 ::: max. ten results for domain 'example.google.com'
 243   - no match
 244 ::: max. ten results for domain 'google.com'
 245   + analytics.google.com
 246   + googleadapis.l.google.com
 247   + pagead.l.google.com
 248   + partnerad.l.google.com
 249   + ssl-google-analytics.l.google.com
 250   + video-stats.video.google.com
 251   + www-google-analytics.l.google.com
 252 </code></pre>
 253
 254 **add a new blocklist source:**
 255 1. the easy way ...
 256 example: https://easylist-downloads.adblockplus.org/rolist+easylist.txt
 257 Adblock already supports an easylist source, called 'reg_ru'. To add the additional local easylist as a new source, copy the existing config source section and change only
 258 the source name, the url and the description - that's all!
 259 <pre><code>
 260 config source 'reg_ro'
 261   option enabled '0'
 262   option adb_src 'https://easylist-downloads.adblockplus.org/rolist+easylist.txt'
 263   option adb_src_rset 'BEGIN{FS=\"[|^]\"}\$0~/^\|\|([[:alnum:]_-]+\.){1,}[[:alpha:]]+\^("\\\$third-party")?$/{print tolower(\$3)}'
 264   option adb_src_desc 'focus on romanian ads plus generic easylist additions, weekly updates, approx. 9.400 entries'
 265 </code></pre>
 266
 267 2. a bit harder ...
 268 To add a really new source with different domain/host format you have to write a suitable awk one-liner on your own, so basic awk skills are needed. As a starting point check the already existing awk rulesets 'adb_src_rset' in the config file, probably you need only small changes for your individual list. Download the desired list and test your new awk string locally. The output result should be a sequential list with one domain/host per line - nothing more. If your awk one-liner works quite well, add a new source section to the adblock config file and test the new source.
 269
 270 ## Support
 271 Please join the adblock discussion in this [forum thread](https://forum.lede-project.org/t/adblock-2-x-support-thread/507) or contact me by mail <dev@brenken.org>
 272
 273 ## Removal
 274 * stop all adblock related services with _/etc/init.d/adblock stop_
 275 * optional: remove the adblock package (_opkg remove adblock_)
 276
 277 Have fun!
 278 Dirk