Skip to content

Downloading IP Lists

Jump to bottom Edit New page
Costa Tsaousis edited this page Nov 10, 2015 · 62 revisions

You need to have FireHOL's update-ipsets installed for downloading IP Lists directly from their maintainers. Follow Installing update-ipsets to install it.

Using update-ipsets

update-ipsets can be run as root or as any normal user.

To run it as root either sudo su before you start, or prefix with sudo all commands given below.

Let's enable an IP list, e.g. the dshield list:

update-ipsets enable dshield

We can enable multiple IP lists at once:

update-ipsets enable dshield blocklist_de spamhaus_drop spamhaus_edrop

Let's download all the enabled lists:

update-ipsets

We can run this command repeatedly. It will not harm. update-ipsets is smart enough to download IP lists only when they are expected to be updated.

For example, it knows that dshield is updated every 10 minutes, so it will not download it again unless there have been 10 minutes since the date/time of the last download (it does not care when you downloaded it, but what date/time the downloaded file had when downloaded). It will attempt to download it again, only if 10 minutes have passed since the modification date/time of the downloaded file.

Where are the ipsets?

When run as root, update-ipsets keeps its files in /etc/firehol/ipsets. Otherwise they will be in ~/ipsets.

In this directory you will have, for each IP list:

  • NAME.source - the file downloaded from the IP list maintainer
  • NAME.ipset (or .netset depending on the IP list type) - the final processed and normalized file

You can set BASE_DIR to change the path where files are saved.

Periodic Updates - The Cron Job

This is how I run update-ipsets via cron:

# crontab -e   or   nano /etc/crontab

*/9 * * * * root update-ipsets >/dev/null 2>&1

I run it every 9 minutes. You can choose 8, 11, 12, etc, it does not matter. Avoid using 5, 10, 15, 20, etc. If all of us choose, let's say 10, the maintainers' sites will get all the requests concurrently. I suggest to pick a random number between 5 and 20 avoiding 5, 10, 15 and 20.

Download all IP Lists

To download all the IP lists known by update-ipsets, use this:

update-ipsets --enable-all

Then, sit back and watch it download all IP lists (it will take some time).

This option enables most, but not all the IP lists known by update-ipsets:

  1. a few IP lists are exact duplicates of others - example php_bad.

  2. a few IP lists are converted to IPs from hostnames - example all hphosts_* lists. These can be converted to IPs by update-ipsets directly, however converting them needs some time, so we decided to exclude them from the --enable-all option. You can enable them manually, and update-ipsets will convert them to IPs by itself.

  3. a few IP lists are updated using external programs - example all sorbs_* and dronebl_* lists. There is no point to enable these lists by default, since update-ipsets is not capable of downloading them.

    I have developed a few scripts that can convert dnsbl zone files (like the ones used by sorbs.net or dronebl.org) to IP lists, however this work is not yet standardized or finalized for inclusion in update-ipsets.

Updating the ipsets in kernel

When running as root, update-ipsets will do this by itself, automatically!

If it finds an ipset named with the name of an IP list, it will update it automatically when the IP List is updated.

update-ipsets goes through the following procedure for each IP list:

  1. download the IP list from the maintainer
the downloaded file is saved in `/etc/firehol/ipsets` as `NAME.source` - it has the modification date/time of the same file at the maintainer's site
  1. process it to extract / normalize its content
the result is saved in `/etc/firehol/ipsets` as `NAME.ipset` or `NAME.netset` depending on its contents

  1. if an ipset with the same NAME already exists in kernel, then do all the following:

    1. create a temporary ipset in kernel

    2. load the contents of NAME.ipset or NAME.netset to the temporary ipset

    3. swap the temporary and the production ipsets in kernel

    this swap is an atomic operation that is handled by the kernel itself - either it succeeds (and the new ipset is used) or it fails (and the old ipset is left untouched)

If at any step there is a failure, the operation is aborted (all temporary resources used are released) and a message is sent to syslog.

The whole operation has been designed so that there will be no disruption at all at your firewall.

Keep in mind that the above are not FireHOL specific. update-ipsets will update the ipsets in kernel (when running as root) even if they are created by hand or another netfilter based firewall solution. The only condition is to have the same NAME and of course ipset type.

Since update-ipsets uses iprange, it also optimizes hash:net ipsets for optimal kernel performance. For more information about this optimization check iprange: optimizing ipsets for iptables.

You can disable the automatic kernel updating of the downloaded ipsets by setting the IPSETS_APPLY option.

update-ipsets configuration

The default configuration file for root is /etc/firehol/update-ipsets.conf. For all other users is ~/.update-ipsets/update-ipsets.conf.

You can override this path using the option -f, like this:

update-ipsets -f /path/to/other/configuration/file

update-ipsets supports these configuration file options:

BASE_DIR

BASE_DIR states the directory ipset files should be stored.

  • When run as root, the default is /etc/firehol/ipsets.
  • When run as any other user, the default is ~/ipsets.

RUN_PARENT_DIR

RUN_PARENT_DIR states the directory to save the lock file. The lock file prevents running multiple update-ipsets concurrently.

  • When run as root, the default is /var/run.
  • When run as any other user, the default is ~/.update-ipsets.

Different users can run update-ipsets at the same time, without problems. The locking prevents only the same user from running update-ipsets more than once (if there was no locking, the same file could be downloaded multiple times, processed more than once, or applied in kernel multiple times - the locking prevents all these).

TMP_DIR

TMP_DIR states the directory to save temporary files. A new directory named update-ipsets-XXXXXXX will be created in TMP_DIR during each run.

The default is /tmp.

HISTORY_DIR

HISTORY_DIR is used for storing the aggregation log for IP lists that need one.

  • When run as root, the default is /etc/firehol/ipsets/history.
  • When run as any other user, the default is ~/ipsets/history.

ERRORS_DIR

ERRORS_DIR is used for storing downloaded IP list files that failed to be applied to kernel.

  • When run as root, the default is /etc/firehol/ipsets/errors.
  • When run as any other user, the default is ~/ipsets/errors.

If you set it to empty, this feature will be disabled.

LIB_DIR

LIB_DIR is primarily used for storing data that calculate age and retention policy of IP lists. update-ipsets may also store other state information in this directory.

The age and retention calculations do not take place, unless you also enable Monitoring.

  • When run as root, the default is /var/lib/update-ipsets.
  • When run as any other user, the default is ~/.update-ipsets/lib.

CACHE_DIR

CACHE_DIR is used by update-ipsets download managers to cache data they need.

  • When run as root, the default is /var/cache/update-ipsets.
  • When run as any other user, the default is ~/.update-ipsets/cache.

MAX_CONNECT_TIME

MAX_CONNECT_TIME gives the time in seconds to connect to maintainer's site. The default is 10.

MAX_DOWNLOAD_TIME

MAX_DOWNLOAD_TIME gives the time in seconds to wait for a download to complete. The default is 300 (5 minutes).

IGNORE_REPEATING_DOWNLOAD_ERRORS

IGNORE_REPEATING_DOWNLOAD_ERRORS is the number consecutive download errors to be ignored and favoured. After these many consecutive download errors, the refresh frequency of the IP list will be incremented.

Let's take an example:

IP list X has a refresh frequency of 20 minutes. The web server of the maintainer has issues and force us to timeout, slowing down the whole process.

We have set IGNORE_REPEATING_DOWNLOAD_ERRORS=5.

This is what will happen:

failed try next try in minutes
1 10 = 20 / 2
2 10 = 20 / 2
3 10 = 20 / 2
4 10 = 20 / 2
5 10 = 20 / 2
6 20 = 20 * 1
7 40 = 20 * 2
8 60 = 20 * 3
9 80 = 20 * 4
10 100 = 20 * 5

So, while the fail count is below IGNORE_REPEATING_DOWNLOAD_ERRORS, update-ipsets favours the ipset, hopping the maintainer will fix the problem soon. If the fail count goes above IGNORE_REPEATING_DOWNLOAD_ERRORS then update-ipsets delays the update every time more, to avoid stalling the whole process due to poorly maintained download sites.

PARALLEL_DNS_QUERIES

PARALLEL_DNS_QUERIES gives the number of threads update-ipsets will spawn for resolving hostnames found in downloaded IP lists. The default is 10.

(this is a feature of iprange, but the parameter is used by update-ipsets to configure iprange properly).

Keep in mind that increasing this too much may hit a quota on the name server, while lowering it too much may delay the whole process.

IPSET_REDUCE_FACTOR and IPSET_REDUCE_ENTRIES

IPSET_REDUCE_FACTOR and IPSET_REDUCE_ENTRIES are used by update-ipsets to optimize hash:net ipsets when they are loaded into the kernel. These values are not used when generating the .netset files on disk. They are only used when ipsets are updated in kernel.

The defaults are: IPSET_REDUCE_FACTOR=20 and IPSET_REDUCE_ENTRIES=65536.

Check iprange: optimizing ipsets for iptables for more information and a demonstration of how this works.

IPSETS_APPLY

IPSETS_APPLY enables or disables the automatic ipsets update in the kernel.

  • The default for root is 1.
  • The default for any other user is 0. Keep in mind that enabling this for non root users will not work. The kernel will not allow any other user to update the ipsets.
Add a custom footer
Add a custom sidebar
Clone this wiki locally