net/adblock/files/README.md

   1 # dns based ad/abuse domain blocking
   2
   3 ## Description
   4 A lot of people already use adblocker plugins within their desktop browsers, but what if you are using your (smart) phone, tablet, watch or any other wlan gadget...getting rid of annoying ads, trackers and other abuse sites (like facebook ;-) is simple: block them with your router. When the dns server on your router receives dns requests, you will sort out queries that ask for the resource records of ad servers and return a simple 'NXDOMAIN'. This is nothing but **N**on-e**X**istent Internet or Intranet domain name, if domain name is unable to resolved using the dns server, a condition called the 'NXDOMAIN' occurred.
   5
   6 ## Main Features
   7 * support of the following domain block list sources (free for private usage, for commercial use please check their individual licenses):
   8     * [adaway](https://adaway.org)
   9     * => infrequent updates, approx. 400 entries (enabled by default)
  10     * [adguard](https://adguard.com)
  11     * => numerous updates on the same day, approx. 12.000 entries
  12     * [blacklist]()
  13     * => static local blacklist, located by default in '/etc/adblock/adblock.blacklist'
  14     * [disconnect](https://disconnect.me)
  15     * => numerous updates on the same day, approx. 6.500 entries (enabled by default)
  16     * [dshield](http://dshield.org)
  17     * => daily updates, approx. 4.500 entries
  18     * [feodotracker](https://feodotracker.abuse.ch)
  19     * => daily updates, approx. 0-10 entries
  20     * [hphosts](https://hosts-file.net)
  21     * => monthly updates, approx. 50.000 entries
  22     * [malwaredomains](http://malwaredomains.com)
  23     * => daily updates, approx. 16.000 entries
  24     * [malwaredomainlist](http://www.malwaredomainlist.com)
  25     * => daily updates, approx. 1.500 entries
  26     * [openphish](https://openphish.com)
  27     * => numerous updates on the same day, approx. 1.800 entries
  28     * [palevo tracker](https://palevotracker.abuse.ch)
  29     * => daily updates, approx. 15 entries
  30     * [ransomware tracker](https://ransomwaretracker.abuse.ch)
  31     * => daily updates, approx. 150 entries
  32     * [rolist/easylist](https://easylist-downloads.adblockplus.org/rolist+easylist.txt)
  33     * => weekly updates, approx. 600 entries
  34     * [ruadlist/easylist](https://code.google.com/p/ruadlist)
  35     * => weekly updates, approx. 2.000 entries
  36     * [securemecca](http://www.securemecca.com)
  37     * => infrequent updates, approx. 25.000 entries
  38     * [shallalist](http://www.shallalist.de) (categories "adv" "costtraps" "spyware" "tracker" "warez" enabled by default)
  39     * => daily updates, approx. 32.000 entries (a short description of all shallalist categories can be found [online](http://www.shallalist.de/categories.html))
  40     * [spam404](http://www.spam404.com)
  41     * => infrequent updates, approx. 5.000 entries
  42     * [sysctl/cameleon](http://sysctl.org/cameleon)
  43     * => weekly updates, approx. 21.000 entries
  44     * [whocares](http://someonewhocares.org)
  45     * => weekly updates, approx. 12.000 entries
  46     * [winhelp](http://winhelp2002.mvps.org)
  47     * => infrequent updates, approx. 15.000 entries
  48     * [winspy](https://github.com/crazy-max/WindowsSpyBlocker)
  49     * => infrequent updates, approx. 120 entries
  50     * [yoyo](http://pgl.yoyo.org/adservers)
  51     * => weekly updates, approx. 2.500 entries (enabled by default)
  52     * [zeus tracker](https://zeustracker.abuse.ch)
  53     * => daily updates, approx. 440 entries
  54 * zero-conf like automatic installation & setup, usually no manual changes needed
  55 * simple but yet powerful adblock engine: adblock does not use error prone external iptables rulesets, http pixel server instances and things like that
  56 * automatically selects dnsmasq or unbound as dns backend
  57 * automatically selects uclient-fetch or wget as download utility (other tools like curl or aria2c are supported as well)
  58 * automatically supports a wide range of router modes, even AP modes are supported
  59 * full IPv4 and IPv6 support
  60 * each block list source will be updated and processed separately
  61 * block list source parsing by fast & flexible regex rulesets
  62 * overall duplicate removal in separate block lists
  63 * additional whitelist for manual overrides, located by default in /etc/adblock/adblock.whitelist
  64 * quality checks during block list update to ensure a reliable dns backend service
  65 * minimal status & error logging to syslog, enable debug logging to receive more output
  66 * procd based init system support (start/stop/restart/reload/suspend/resume)
  67 * procd based hotplug support, the adblock start will be triggered by interface triggers
  68 * suspend & resume adblock actions temporarily without block list reloading
  69 * runtime statistics via ubus service call
  70 * query function to quickly identify blocked (sub-)domains, e.g. for whitelisting
  71 * optional: automatic block list backup & restore, backups will be (de-)compressed and restored on the fly in case of any runtime error
  72 * optional: add new adblock sources on your own via uci config
  73
  74 ## Prerequisites
  75 * [openwrt](https://openwrt.org), tested with latest stable release (Chaos Calmer) and with current trunk (Designated Driver)
  76 * [LEDE project](https://www.lede-project.org), tested with trunk > r98
  77 * a usual setup with an enabled dns backend at minimum - dump AP modes without a working dns backend are _not_ supported
  78 * a download utility: full versions (with ssl support) of 'wget', 'uclient-fetch', 'aria2c' or 'curl' are supported - the Chaos Calmer built-in busybox wget is not
  79     * Chaos Calmer: download & install the external 'wget' package
  80     * Designated Driver/Trunk: use built-in 'uclient-fetch' or download & install the external 'wget' package
  81     * for more configuration options see examples below
  82
  83 ## OpenWrt / LEDE trunk Installation & Usage
  84 * install 'adblock' (_opkg install adblock_) and that's it - the adblock start will be automatically triggered by procd interface triggers
  85 * start/stop/restart/reload/suspend/resume the adblock service manually with _/etc/init.d/adblock_
  86 * enable/disable your favored block list sources in _/etc/config/adblock_ - 'adaway', 'disconnect' and 'yoyo' are enabled by default
  87
  88 ## LuCI adblock companion package
  89 * for easy management of the various block list sources and options you can also use a nice & efficient LuCI frontend
  90 * install 'luci-app-adblock' (_opkg install luci-app-adblock_)
  91 * the application is located in LuCI under 'Services' menu
  92 * _Thanks to Hannu Nyman for this great adblock LuCI frontend!_
  93
  94 ## Chaos Calmer installation notes
  95 * 'adblock' and 'luci-app-adblock' are _not_ available as .ipk packages in the Chaos Calmer download repository
  96 * download both packages from a development snapshot package directory:
  97     * for 'adblock' look [here](https://downloads.lede-project.org/snapshots/packages/x86_64/packages/)
  98     * for 'luci-app-adblock' look [here](https://downloads.lede-project.org/snapshots/packages/x86_64/luci/)
  99 * manually transfer the packages to your routers temp directory (with tools like _sshfs_ or _winscp_)
 100 * install the packages with _opkg install <...>_ as described above
 101
 102 ## Tweaks
 103 * **status/runtime statistics:** the adblock status and runtime statistics are available via ubus service call (see example below)
 104 * **debug logging:** for script debugging please set the config option 'adb\_debug' to '1' and check the runtime output with _logread -e "adblock"_
 105 * **storage expansion:** to process and store all block list sources at once it might helpful to enlarge your temp directory with a swap partition => see [openwrt wiki](https://wiki.openwrt.org/doc/uci/fstab) for further details
 106 * **add white-/blacklist entries:** add domain white- or blacklist entries to always-allow or -deny certain (sub) domains, by default both lists are empty and located in _/etc/adblock_. Please add one domain per line - ip addresses, wildcards & regex are _not_ allowed (see example below)
 107 * **backup & restore block lists:** enable this feature, to restore automatically the latest compressed backup of your block lists in case of any processing error (e.g. a single block list source is not available during update). Please use an (external) solid partition and _not_ your volatile router temp directory for this
 108 * **scheduled list updates:** for a scheduled call of the adblock service add an appropriate crontab entry (see example below)
 109 * **restrict/disable procd interface trigger:** to restrict the procd interface trigger to a (list of) certain wan interface(s) or to disable it at all, set 'adb\_iface' to an existing interface like 'wan' or to a non-existing like 'false'
 110 * **suspend & resume adblocking:** to quickly switch the adblock service 'on' or 'off', simply use _/etc/init.d/adblock [suspend|resume]_
 111 * **domain query:** to query the active block lists for a specific domain, please run _/etc/init.d/adblock query `<DOMAIN>`_ (see example below)
 112 * **divert dns requests:** to force dns requests to your local dns resolver add an appropriate firewall rule (see example below)
 113 * **add new list sources:** you could add new block list sources on your own via uci config, all you need is a source url and an awk one-liner (see example below)
 114 * **disable active dns probing in windows 10:** to prevent a yellow exclamation mark on your internet connection icon (which wrongly means connected, but no internet), please change the following registry key/value from "1" to "0" _HKLM\SYSTEM\CurrentControlSet\Services\NlaSvc\Parameters\Internet\EnableActiveProbing_
 115
 116 ## Further adblock config options
 117 * usually the pre-configured adblock setup works quite well and no manual config overrides are needed, all listed options apply to the 'global' config section:
 118     * adb\_enabled => main switch to enable/disable adblock service (default: '1', enabled)
 119     * adb\_debug => enable/disable adblock debug output (default: '0', disabled)
 120     * adb\_iface => restrict the procd interface trigger to a (list of) certain wan interface(s) or disable it at all (default: not set, disabled)
 121     * adb\_fetch => full path to a different download utility, see example below (default: not set, use wget)
 122     * adb\_fetchparm => options for the download utility, see example below (default: not set, use wget options)
 123
 124 ## Examples
 125
 126 **change default dns backend to 'unbound':**
 127 <pre><code>
 128 Adblock detects the presence of an active unbound dns backend and the block lists will be automatically pulled in by unbound.
 129 The adblock script deposits the sorted and filtered block lists in '/var/lib/unbound' where unbound can find them in its jail.
 130 If you use manual configuration for unbound, then just include the following line in your 'server:' clause:
 131
 132   include: "/var/lib/unbound/adb_list.*"
 133 </code></pre>
 134
 135 **configuration for different download utilities:**
 136 <pre><code>
 137 wget (default):
 138   option adb_fetch="/usr/bin/wget"
 139   option adb_fetchparm="--no-config --quiet --tries=1 --no-cache --no-cookies --max-redirect=0 --timeout=5 --no-check-certificate -O"
 140
 141 aria2c:
 142   option adb_fetch '/usr/bin/aria2c'
 143   option adb_fetchparm '-q --max-tries=1 --timeout=5 --allow-overwrite=true --auto-file-renaming=false --check-certificate=false -o'
 144
 145 uclient-fetch:
 146   option adb_fetch '/bin/uclient-fetch'
 147   option adb_fetchparm '-q --timeout=5 --no-check-certificate -O'
 148
 149 curl:
 150   option adb_fetch '/usr/bin/curl'
 151   option adb_fetchparm '-s --retry 1 --connect-timeout 5 --insecure -o'
 152 </code></pre>
 153
 154 **receive adblock statistics via ubus:**
 155 <pre><code>
 156 ubus call service list '{"name":"adblock_stats"}'
 157 This will output the active block lists and other runtime information as JSON, e.g.:
 158 {
 159     "adblock_stats": {
 160         "instances": {
 161             "statistics": {
 162                 "running": false,
 163                 "command": [
 164                     ""
 165                 ],
 166                 "data": {
 167                     "active_lists": [
 168                         {
 169                             "palevo": "14",
 170                             "blacklist": "144",
 171                             "winspy": "168",
 172                             "zeus": "422",
 173                             "adaway": "408",
 174                             "rolist": "649",
 175                             "malwarelist": "1219",
 176                             "ransomware": "1495",
 177                             "ruadlist": "1791",
 178                             "yoyo": "2304",
 179                             "openphish": "2139",
 180                             "dshield": "154",
 181                             "disconnect": "3176",
 182                             "spam404": "6251",
 183                             "adguard": "11081",
 184                             "whocares": "11575",
 185                             "winhelp": "10574",
 186                             "malware": "13854",
 187                             "sysctl": "8539",
 188                             "securemecca": "9262",
 189                             "shalla": "25358",
 190                             "hphosts": "36256"
 191                         }
 192                     ],
 193                     "adblock_version": "2.3.0",
 194                     "blocked_domains": "146833",
 195                     "dns_backend": "dnsmasq",
 196                     "last_rundate": "04.02.2017 21:10:31",
 197                     "system": "LEDE Reboot SNAPSHOT r3286-c980147527"
 198                 }
 199             }
 200         }
 201     }
 202 }
 203 </code></pre>
 204
 205 **cronjob for a regular block list update (/etc/crontabs/root):**
 206 <pre><code>
 207 0 06 * * *    /etc/init.d/adblock start
 208 </code></pre>
 209
 210 **blacklist entry (/etc/adblock/adblock.blacklist):**
 211 <pre><code>
 212 ads.example.com
 213
 214 This entry blocks the following (sub)domains:
 215   http://ads.example.com/foo.gif
 216   http://server1.ads.example.com/foo.gif
 217   https://ads.example.com:8000/
 218
 219 This entry does not block:
 220   http://ads.example.com.ua/foo.gif
 221   http://example.com/
 222 </code></pre>
 223
 224 **whitelist entry (/etc/adblock/adblock.whitelist):**
 225 <pre><code>
 226 here.com
 227
 228 This entry removes the following (sub)domains from the block lists:
 229   maps.here.com
 230   here.com
 231
 232 This entry does not remove:
 233   where.com
 234   www.adwhere.com
 235 </code></pre>
 236
 237 **query active block lists for a certain (sub-)domain, e.g. for whitelisting:**
 238 <pre><code>
 239 /etc/init.d/adblock query example.www.doubleclick.net
 240 ::: distinct results for domain 'example.www.doubleclick.net'
 241  no match
 242 ::: distinct results for domain 'www.doubleclick.net'
 243  adb_list.sysctl      : www.doubleclick.net
 244 ::: distinct results for domain 'doubleclick.net'
 245  adb_list.adaway      : ad-g.doubleclick.net
 246  adb_list.securemecca : 1168945.fls.doubleclick.net
 247  adb_list.sysctl      : 1435575.fls.doubleclick.net
 248  adb_list.whocares    : 3ad.doubleclick.net
 249
 250 The query function checks against the submitted (sub-)domain and recurses automatically to the upper top level domain(s).
 251 For every domain it returns the overall count plus a distinct list of active block lists with the first relevant result.
 252 In the example above whitelist "www.doubleclick.net" to free the submitted domain.
 253 </code></pre>
 254
 255 **divert dns requests to local dns resolver in /etc/config/firewall:**
 256 <pre><code>
 257 config redirect
 258     option name 'Divert DNS'
 259     option src 'lan'
 260     option proto 'tcp udp'
 261     option src_dport '53'
 262     option dest_port '53'
 263     option target 'DNAT'
 264 </code></pre>
 265
 266 **add a new block list source:**
 267 <pre><code>
 268 1. the easy way ...
 269 example: https://easylist-downloads.adblockplus.org/rolist+easylist.txt
 270 adblock already supports an easylist source, called 'ruadlist'. To add the additional local easylist
 271 as a new source, copy the existing config source 'ruadlist' section and change only
 272 the source name, the url and the description - that's all!
 273
 274 config source 'rolist'
 275   option enabled '0'
 276   option adb_src 'https://easylist-downloads.adblockplus.org/rolist+easylist.txt'
 277   option adb_src_rset '{FS=\"[|^]\"} \$0 ~/^\|\|([A-Za-z0-9_-]+\.){1,}[A-Za-z]+\^$/{print tolower(\$3)}'
 278   option adb_src_desc 'focus on romanian ad related domains plus generic easylist additions, weekly updates, approx. 600 entries'
 279
 280 2. a bit harder ...
 281 to add a really new source with different domain/host format you have to write a suitable
 282 awk one-liner on your own, so basic awk skills are needed. As a starting point check the already
 283 existing awk strings (adb_src_rset) in adblock config, maybe you need only small changes for your individual list.
 284 Download the desired list and test your new awk string locally with:
 285   cat new.list | awk 'fs__individual search__search core__result'
 286   'fs' => field separator (optional)
 287   'individual search' => individual search part to filter out needless list information
 288   'search core' => always '([A-Za-z0-9_-]+\.){1,}[A-Za-z]+', this is part of all list sources and should be unchanged
 289   'result' => always '{print tolower(\$n)}', only the output column 'n' may vary
 290 the output result should be a sequential list with one domain/host per line - nothing more.
 291
 292 If your awk one-liner works quite well, add a new source section in adblock config and test your new source
 293 </code></pre>
 294
 295 ## Support
 296 Please join the adblock discussion in this [forum thread](https://forum.openwrt.org/viewtopic.php?id=59803) or contact me by mail <dev@brenken.org>
 297
 298 ## Removal
 299 * stop all adblock related services with _/etc/init.d/adblock stop_
 300 * optional: remove the adblock package (_opkg remove adblock_)
 301
 302 Have fun!
 303 Dirk