net/adblock/files/README.md

   1 # dns based ad/abuse domain blocking
   2
   3 ## Description
   4 A lot of people already use adblocker plugins within their desktop browsers, but what if you are using your (smart) phone, tablet, watch or any other wlan gadget...getting rid of annoying ads, trackers and other abuse sites (like facebook ;-) is simple: block them with your router. When the dns server on your router receives dns requests, you will sort out queries that ask for the resource records of ad servers and return the local ip address of your router and the internal web server delivers a transparent pixel instead.
   5
   6 ## Main Features
   7 * support of the following domain blocklist sources (free for private usage, for commercial use please check their individual licenses):
   8     * [adaway](https://adaway.org)
   9     * => infrequent updates, approx. 400 entries (enabled by default)
  10     * [blacklist]()
  11     * => static local blacklist, located by default in '/etc/adblock/adblock.blacklist'
  12     * [disconnect](https://disconnect.me)
  13     * => numerous updates on the same day, approx. 6.500 entries (enabled by default)
  14     * [dshield](http://dshield.org)
  15     * => daily updates, approx. 4.500 entries
  16     * [feodotracker](https://feodotracker.abuse.ch)
  17     * => daily updates, approx. 0-10 entries
  18     * [malwaredomains](http://malwaredomains.com)
  19     * => daily updates, approx. 16.000 entries
  20     * [malwaredomainlist](http://www.malwaredomainlist.com)
  21     * => daily updates, approx. 1.500 entries
  22     * [openphish](https://openphish.com)
  23     * => numerous updates on the same day, approx. 1.800 entries
  24     * [palevo tracker](https://palevotracker.abuse.ch)
  25     * => daily updates, approx. 15 entries
  26     * [ransomware tracker](https://ransomwaretracker.abuse.ch)
  27     * => daily updates, approx. 150 entries
  28     * [rolist/easylist](https://easylist-downloads.adblockplus.org/rolist+easylist.txt)
  29     * => weekly updates, approx. 600 entries
  30     * [ruadlist/easylist](https://code.google.com/p/ruadlist)
  31     * => weekly updates, approx. 2.000 entries
  32     * [shallalist](http://www.shallalist.de) (categories "adv" "costtraps" "spyware" "tracker" "warez" enabled by default)
  33     * => daily updates, approx. 32.000 entries (a short description of all shallalist categories can be found [online](http://www.shallalist.de/categories.html))
  34     * [spam404](http://www.spam404.com)
  35     * => infrequent updates, approx. 5.000 entries
  36     * [sysctl/cameleon](http://sysctl.org/cameleon)
  37     * => weekly updates, approx. 21.000 entries
  38     * [whocares](http://someonewhocares.org)
  39     * => weekly updates, approx. 12.000 entries
  40     * [winhelp](http://winhelp2002.mvps.org)
  41     * => infrequent updates, approx. 15.000 entries
  42     * [winspy](https://github.com/crazy-max/WindowsSpyBlocker)
  43     * => infrequent updates, approx. 120 entries
  44     * [yoyo](http://pgl.yoyo.org/adservers)
  45     * => weekly updates, approx. 2.500 entries (enabled by default)
  46     * [zeus tracker](https://zeustracker.abuse.ch)
  47     * => daily updates, approx. 440 entries
  48 * zero-conf like automatic installation & setup, usually no manual changes needed (i.e. ip address, network devices etc.)
  49 * supports a wide range of router modes (incl. AP mode), as long as the firewall and the DNS server are enabled & in use
  50 * full IPv4 and IPv6 support
  51 * each blocklist source will be updated and processed separately
  52 * timestamp check to download and process only updated adblock list sources
  53 * overall duplicate removal in separate adblock lists (will be automatically disabled on low memory systems)
  54 * adblock source list parsing by fast & flexible regex rulesets
  55 * additional whitelist for manual overrides, located by default in /etc/adblock/adblock.whitelist
  56 * quality checks during & after update of adblock lists to ensure a reliable dnsmasq service
  57 * adblock statistics, last runtime and list states/counts/update times will be stored in uci config for LuCI frontend
  58 * status & error logging to stdout and syslog
  59 * use two dynamic uhttpd instances as adblock pixel server, separated for ads delivered on port 80 and on port 443
  60 * use dynamic iptables chains/rulesets for adblock related redirects/rejects
  61 * init system support (start/stop/restart/reload/toggle/stats/cfgup)
  62 * hotplug support, the adblock start will be triggered by wan 'ifup' event
  63 * adblock toggle to quickly switch adblocking 'on' or 'off'
  64 * optional: automatic adblock list backup/restore, backups will be (de-)compressed on the fly (disabled by default)
  65 * optional: add new adblock sources via uci config (see example below)
  66
  67 ## Prerequisites
  68 * [openwrt](https://openwrt.org), tested with latest stable release (Chaos Calmer) and with current trunk (Designated Driver)
  69 * [LEDE project](https://www.lede-project.org), tested with trunk > r98
  70 * usual setup with enabled 'iptables', 'dnsmasq' and 'uhttpd' - dump AP modes without these basics are _not_ supported!
  71 * additional required software packages:
  72     * a download utility: 'uclient-fetch' and 'wget' (full versions with ssl support) are supported. Normally you should use 'wget', it's quite stable and supports the online timestamp checks. If you need a smaller memory footprint try 'uclient-fetch' without openssl dependency. The default ustream ssl backend 'libustream-polarssl' has issues with certain https sites and is currently not supported. To change the ssl backend see example below.
  73     * optional: 'kmod-ipt-nat6' for IPv6 support
  74 * the above dependencies and requirements will be checked during package installation & script runtime
  75
  76 ## OpenWrt / LEDE trunk Installation & Usage
  77 * install 'adblock' (_opkg install adblock_)
  78 * adblock starts automatically during boot, triggered by wan-ifup event, check _logread -e "adblock"_ for adblock related information
  79 * optional: start/restart/stop the adblock service manually with _/etc/init.d/adblock_
  80 * optional: enable/disable your required adblock list sources in _/etc/config/adblock_ - 'adaway', 'disconnect' and 'yoyo' are enabled by default
  81 * optional: maintain the adblock service in LuCI under 'System => Startup'
  82
  83 ## LuCI adblock companion package
  84 * for easy management of the various blocklist sources and adblock options there is also a nice & efficient LuCI frontend available
  85 * install 'luci-app-adblock' (_opkg install luci-app-adblock_)
  86 * the application is located in LuCI under 'Services' menu
  87 * _Thanks to Hannu Nyman for this great adblock LuCI frontend!_
  88
  89 ## Chaos Calmer installation notes
  90 * 'adblock' and 'luci-app-adblock' are _not_ available as .ipk packages in the Chaos Calmer download repository
  91 * download both packages from a development snapshot package directory:
  92     * for 'adblock' look [here](https://downloads.lede-project.org/snapshots/packages/x86_64/packages/)
  93     * for 'luci-app-adblock' look [here](https://downloads.lede-project.org/snapshots/packages/x86_64/luci/)
  94 * manually transfer the packages to your routers temp directory (with tools like _sshfs_ or _winscp_)
  95 * install the packages with _opkg install <...>_ as described above
  96
  97 ## Tweaks
  98 * **storage:** to process & store all blocklist sources at once it might helpful to enlarge your temp directory with a swap partition => see [openwrt wiki](https://wiki.openwrt.org/doc/uci/fstab) for further details
  99 * **white-/blacklist:** add domain white- or blacklist entries to always-allow or -deny certain (sub) domains, by default both lists are located in _/etc/adblock_. Please add one domain per line - ip addresses, wildcards & regex are _not_ allowed (see example below)
 100 * **backup/restore:** enable the backup/restore feature, to restore automatically the latest compressed backup of your adblock lists in case of any processing error (i.e. a single blocklist source is down). Please use an (external) solid partition and _not_ your volatile router temp directory for this
 101 * **list updates:** for a scheduled call of the adblock service add an appropriate crontab entry (see example below)
 102 * **new list sources:** you could add new blocklist sources on your own via uci config, all you need is a source url and an awk one-liner (see example below)
 103 * **AP mode:** in 'AP mode' adblock uses automatically the local router ip as nullip address. To make sure that your LuCI interface will be still accessible, you have to change the local uhttpd instance to ports <> 80/443 (see example below)
 104 * **restricted mode:** to disable flash writes with adblock status information to the adblock config file (used by LuCI frontend), please set 'adb\_restricted' to '1'
 105 * **adblock toggle:** to quickly switch adblocking 'on' or 'off', simply use _/etc/init.d/adblock toggle_
 106 * **adblock statistics:** to update only the adblock statistics (without updating the block lists as well), please run _/etc/init.d/adblock stats_
 107 * **configuration update:** to update an outdated adblock config file with the current default version, please run _/etc/init.d/adblock cfgup_, make your individual changes and start the adblock service again
 108 * **debugging:** for script debugging please set the 'adb\_debug' variable in the header of _/etc/init.d/adblock_ to '1'
 109 * **disable active dns probing in windows:** to prevent a possible yellow exclamation mark on your internet connection icon (which wrongly means connected, but no internet), please change the following registry key/value from "1" to "0" _HKLM\SYSTEM\CurrentControlSet\Services\NlaSvc\Parameters\Internet\EnableActiveProbing_
 110
 111 ## Further adblock config options
 112 * usually the adblock autodetection works quite well and no manual config overrides are needed, all options apply to the 'global' config section:
 113     * adb\_enabled => main switch to enable/disable adblock service (default: '1', enabled)
 114     * adb\_cfgver => config version string (do not change!) - adblock will check this entry during startup
 115     * adb\_lanif => name of the logical lan interface (default: 'lan')
 116     * adb\_nullport => port of the adblock uhttpd instance used for ads delivered on port 80 (default: '65534')
 117     * adb\_nullportssl => port of the adblock uhttpd instance used for ads delivered on port 443 (default: '65535')
 118     * adb\_nullipv4 => IPv4 blackhole ip address (default: '198.18.0.1', in AP mode: local router ip)
 119     * adb\_nullipv6 => IPv6 blackhole ip address (default: '::ffff:c612:0001', in AP mode: local router ip)
 120     * adb\_forcedns => redirect all local DNS queries to the local dnsmasq resolver (default: '1', enabled / always disabled in 'AP mode')
 121     * adb\_fetchttl => set the timeout for list downloads (default: '5' seconds)
 122     * adb\_restricted => disable updates of the adblock config file (no flash writes) during runtime (default: '0', disabled)
 123
 124 ## Examples
 125
 126 **example to change the ssl backend for 'uclient-fetch':**
 127 <pre><code>
 128 opkg update
 129 opkg remove --force-depends libustream-polarssl
 130 opkg install libustream-mbedtls
 131 </code></pre>
 132
 133 **example cronjob for a regular block list update:**
 134 <pre><code>
 135 # configuration found in /etc/crontabs/root
 136 # start adblock script once a day at 6 a.m.
 137 #
 138 0 06 * * *    /etc/init.d/adblock start
 139 </code></pre>
 140
 141 **example blacklist entry (/etc/adblock/adblock.blacklist):**
 142 <pre><code>
 143 ads.example.com
 144
 145 This entry blocks the following (sub)domains:
 146   http://ads.example.com/foo.gif
 147   http://server1.ads.example.com/foo.gif
 148   https://ads.example.com:8000/
 149
 150 This entry does not block:
 151   http://ads.example.com.ua/foo.gif
 152   http://example.com/
 153 </code></pre>
 154
 155 **example whitelist entry (/etc/adblock/adblock.whitelist):**
 156 <pre><code>
 157 here.com
 158
 159 This entry removes the following (sub)domains from the blocklists:
 160   maps.here.com
 161   here.com
 162
 163 This entry does not remove:
 164   where.com
 165   www.adwhere.com
 166 </code></pre>
 167
 168 **example uhttpd configuration in AP mode:**
 169 <pre><code>
 170 # configuration found in /etc/config/uhttpd
 171 # change default http/https ports <> 80/443
 172 #
 173 config uhttpd 'main'
 174     list listen_http '0.0.0.0:88'
 175     list listen_https '0.0.0.0:445'
 176 </code></pre>
 177
 178 **example grep for blocked (sub-)domains in adblock source files:**
 179 <pre><code>
 180 grep "google-analytics.com" "/tmp/dnsmasq.d/adb_list"*
 181
 182 This will output all matches with corresponding source files:
 183   /tmp/dnsmasq.d/adb_list.winhelp:address=/ssl.google-analytics.com/198.18.0.1
 184   /tmp/dnsmasq.d/adb_list.winhelp:address=/www.google-analytics.com/198.18.0.1
 185   /tmp/dnsmasq.d/adb_list.yoyo:address=/google-analytics.com/198.18.0.1
 186 </code></pre>
 187
 188 **example to find blocked domains on certain sites for whitelisting:**
 189 <pre><code>
 190 1. the easy way ...
 191 enable the network analysis builtins in chrome or firefox to identify domains
 192 which are redirected to the adblock null-ip (default 198.18.0.1), add these domains to your whitelist
 193
 194 2. a bit harder ...
 195 enable 'Log queries' in the dnsmasq configuration (via LuCI Network => DHCP/DNS),
 196 ssh to your router and start tracing with 'logread -f -e "dnsmasq" -e "198.18.0.1"'
 197 switch to your client, access the relevant site and check all domains
 198 that are blocked/listed in logread, add these domains to your whitelist
 199
 200 => finally restart the adblock service (/etc/init.d/adblock restart) in both variants
 201 </code></pre>
 202
 203 **example to add a new blocklist source:**
 204 <pre><code>
 205 1. the easy way ...
 206 example: https://easylist-downloads.adblockplus.org/rolist+easylist.txt
 207 adblock already supports an easylist source, called 'ruadlist'. To add the additional local easylist
 208 as a new source, copy the existing config source 'ruadlist' section and change only
 209 the source name, the url and the description - that's all!
 210
 211 config source 'rolist'
 212   option enabled '0'
 213   option adb_src 'https://easylist-downloads.adblockplus.org/rolist+easylist.txt'
 214   option adb_src_rset '{FS=\"[|^]\"} \$0 ~/^\|\|([A-Za-z0-9_-]+\.){1,}[A-Za-z]+\^$/{print tolower(\$3)}'
 215   option adb_src_desc 'focus on romanian ad related domains plus generic easylist additions, weekly updates, approx. 600 entries'
 216
 217 2. a bit harder ...
 218 to add a really new source with different domain/host format you have to write a suitable
 219 awk one-liner on your own, so basic awk skills are needed. As a starting point check the already
 220 existing awk strings (adb_src_rset) in adblock config, maybe you need only small changes for your individual list.
 221 Download the desired list and test your new awk string locally with:
 222   cat new.list | awk 'fs__individual search__search core__result'
 223   'fs' => field separator (optional)
 224   'individual search' => individual search part to filter out needless list information
 225   'search core' => always '([A-Za-z0-9_-]+\.){1,}[A-Za-z]+', this is part of all list sources and should be unchanged
 226   'result' => always '{print tolower(\$n)}', only the output column 'n' may vary
 227 the output result should be a sequential list with one domain/host per line - nothing more.
 228
 229 If your awk one-liner works quite well, add a new source section in adblock config and test your new source
 230 </code></pre>
 231
 232 ## Background
 233 This adblock package is a dns/dnsmasq based adblock solution.
 234 Queries to ad/abuse domains are never forwarded and always replied with a local IP address which may be IPv4 or IPv6. For that purpose adblock uses an ip address from the private 'Benchmark Test' subnet (198.18.0.1 / ::ffff:c612:0001) by default (in AP mode the local router ip address will be used). Furthermore all ad/abuse queries will be filtered by ip(6)tables and redirected to two uhttpd instances, separated for ads delivered on port 80 and on port 443 (in PREROUTING chain) or rejected (in FORWARD or OUTPUT chain). In 'AP mode' only the uhttpd related rules in PREROUTING chain are enabled.
 235
 236 All iptables and uhttpd related adblock additions are non-destructive, no hard-coded changes in 'firewall.user', 'uhttpd' config or any other system related config files. There is _no_ adblock background daemon running, the (scheduled) start of the adblock service keeps only the adblock lists up-to-date.
 237
 238 ## Support
 239 Please join the adblock discussion in this [openwrt forum thread](https://forum.openwrt.org/viewtopic.php?id=59803) or contact me by mail <dev@brenken.org>
 240
 241 ## Removal
 242 * stop all adblock related services with _/etc/init.d/adblock stop_
 243 * optional: remove the adblock package (_opkg remove adblock_)
 244
 245 Have fun!
 246 Dirk