Skip to content

Home

Jump to bottom Edit New page
Costa Tsaousis edited this page Dec 3, 2018 · 3 revisions

iprange

iprange is a tool capable of managing sets of IPs.

Features:

  • can read any kind of IP address (one per line):

    • a.b.c.d (like 1.2.3.4)
    • a.b.c.d/prefix (like 1.2.3.0/24)
    • a.b.c.d/netmask (like 1.2.3.0/255.255.255.0)

  • it also supports abbreviations like:

    • a.b (like 10.1 which is 10.0.0.1)
    • a.b.c (like 10.1.1 which is 10.1.0.1)
    • the above can also include /prefix or /netmask

  • can read any kind of IP ranges (one per line):

    • a.b.c.d - e.f.g.h (like 1.2.3.0 - 1.2.3.255)
    • a.b.c.d/prefix - e.f.g.h/prefix (like 1.2.3.0/24 - 1.2.4.0/24)
    • a.b.c.d/netmask - e.f.g.h/netmask (like 1.2.3.0/255.255.255.0 - 1.2.4.0/255.255.255.0)

  • can read hostnames (one per line), and it does parallel, asynchronous queries to resolve them to IPs.

  • IP address conversion is done with inet_aton() (check the man page), which also accepts octal (be careful: IPs with components starting with zero, are considered octal) or hex (starting with 0x).

  • abbreviations, prefixes and masks may also appear on the same line (like 1.2.2.0/24 - 1.2.3.0/255.255.255.0)

  • IPs can also be given in numeric format (as integers)

  • all of the above formats can coexist in the same file

  • it can read from stdin and / or any number of files given on the command line

  • it is super fast, for 1 million lines, it will need less than a second

  • it considers as comments anything after a # or a ;

  • it supports these modes of operation

    • merge or optimize, all the files are merged and the result is printed (union in sets theory)
    • common, finds all the IPs common to all files given - this is not a text diff - it finds all IPs matched by the IPs, the IP ranges or the CIDRs in all files given (intersection in sets theory)
    • exclude, merge a set of files and exclude all IPs that are matched by another set of files (complement is sets theory)
    • reduce, all files are merged, but the result is processed in order to reduce the number of unique subnets generated. This is very useful of netfilter/iptables ipsets where the number of different subnets in an ipset affect the kernel performance.
    • compare, where all files are compared with one another and the result is printed as CSV
    • compare first, where the first file is compared with each other file given, and the result is printed as CSV
    • compare next, where a set of files is compared to each of another set of files, and the result is printed as CSV
    • count, where all the files are merged, and just a few statistics are printed
    • count all, like count but the files are not merged, one line of statistics is printed per file

You can find all the options it supports by running it with -h:

# iprange -h

iprange
manage IP ranges
version: development ($Id: a2d5195567cf1ad0e461c42e16a1d4175cea22f1 $)

Original,   Copyright (C) 2003 Gabriel L. Somlo
Adapted,    Copyright (C) 2004 Paul Townsend
Refactored, Copyright (C) 2015 Costa Tsaousis for FireHOL
License: GPL

Usage: iprange [options] file1 file2 file3 ...

options (multiple options are aliases):

        --------------------------------------------------------------
        CIDR OUTPUT MODES

        --optimize
        --combine
        --merge
        --union
        --union-all
        -J
                > UNION mode (the default)
                returns all IPs found on all files
                the resulting set is sorted

        --common
        --intersect
        --intersect-all
                > INTERSECT mode
                intersect all files to find their common IPs
                the resulting set is sorted

        --exclude-next
        --complement
        --complement-next
                > COMPLEMENT mode
                here is how it works:
                1. union all files before this parameter (ipset A)
                2. remove all IPs found in the files after this
                   parameter, from ipset A
                the resulting set is sorted

        --ipset-reduce PERCENT
        --reduce-factor PERCENT
                > IPSET REDUCE mode
                union all files and print the merged set
                but try to reduce the number of prefixes (subnets)
                found, while allowing some increase in entries
                the PERCENT is how much percent to allow
                increase on the number of entries in order to reduce
                the prefixes (subnets)
                (the internal default PERCENT is 20)
                (use -v to see exactly what it does)
                the resulting set is sorted

        --ipset-reduce-entries ENTRIES
        --reduce-entries ENTRIES
                > IPSET REDUCE mode
                allow increasing the entries above PERCENT, if
                they are below ENTRIES
                (the internal default ENTRIES is 16384)


        --------------------------------------------------------------
        CSV OUTPUT MODES

        --compare
                > COMPARE ALL mode (CSV output)
                compare all files with all other files
                add --header to get the CSV header too

        --compare-first
                > COMPARE FIRST mode (CSV output)
                compare the first file with all other files
                add --header to get the CSV header too

        --compare-next
                > COMPARE NEXT mode (CSV output)
                compare all the files that appear before this
                parameter, to all files that appear after this
                parameter
                add --header to get the CSV header too

        --count-unique
        -C
                > COUNT UNIQUE mode (CSV output)
                merge all files and print its counts
                add --header to get the CSV header too

        --count-unique-all
                > COUNT UNIQUE ALL mode (CSV output)
                print counts for each file
                add --header to get the CSV header too


        --------------------------------------------------------------
        OPTIONS THAT AFFECT INPUT

        --dont-fix-network
                by default, the network address of all CIDRs
                is used (i.e. 1.1.1.17/24 is read as 1.1.1.0/24)
                this option disables this feature
                (i.e. 1.1.1.17/24 is read as 1.1.1.17-1.1.1.255)

        --default-prefix PREFIX
        -p PREFIX
                Set the default prefix for all IPs without mask
                the default is 32


        --------------------------------------------------------------
        OPTIONS THAT AFFECT CIDR OUTPUT

        --min-prefix N
                do not generate prefixes larger than N
                i.e. if N is 24 then /24 to /32 entries will be
                     generated (a /16 network will be generated
                     using multiple /24 networks)
                this is useful to optimize netfilter/iptables
                ipsets, where each different prefix increases the
                lookup time for each packet, but the number of
                entries in the ipset do not affect its performance
                with this setting more entries will be produced
                to accomplish the same match
                warning: misuse of this parameter can create a large
                         number of entries in the generated set

        --prefixes N,N,N, ...
                enable only the given prefixes to express all CIDRs
                prefix 32 is always enabled
                warning: misuse of this parameter can create a large
                         number of entries in the generated set

        --print-ranges
        -j
                print IP ranges (A.A.A.A-B.B.B.B)
                the default is to print CIDRs (A.A.A.A/B)
                it only applies when the output is not CSV

        --print-single-ips
        -1
                print single IPs
                this can produce large output
                the default is to print CIDRs (A.A.A.A/B)
                it only applies when the output is not CSV

        --print-binary
                print binary data
                this is the fastest way to print a large ipset
                the result can be read by iprange on the same
                architecture (no conversion of endianess)

        --print-prefix STRING
                print STRING before each IP, range or CIDR
                this sets both --print-prefix-ips and
                --print-prefix-nets

        --print-prefix-ips STRING
                print STRING before each single IP
                useful for entering single IPs to a different
                ipset than the networks

        --print-prefix-nets STRING
                print STRING before each range or CIDR
                useful for entering sunbets to a different
                ipset than single IPs

        --print-suffix STRING
                print STRING after each IP, range or CIDR
                this sets both --print-suffix-ips and
                --print-suffix-nets

        --print-suffix-ips STRING
                print STRING after each single IP
                useful for giving single IPs different
                ipset options

        --print-suffix-nets STRING
                print STRING after each range or CIDR
                useful for giving subnets different
                ipset options


        --------------------------------------------------------------
        OPTIONS THAT AFFECT CSV OUTPUT

        --header
                when the output is CSV, print the header line
                the default is to not print the header line


        --------------------------------------------------------------
        OPTIONS THAT AFFECT DNS RESOLUTION

        --dns-threads NUMBER
                the number of parallel DNS queries to execute
                when the input files contain hostnames
                the default is 5

        --dns-silent
                do not print DNS resolution errors
                the default is to print all DNS related errors

        --dns-progress
                print DNS resolution progress bar


        --------------------------------------------------------------
        OTHER OPTIONS

        --has-compare
        --has-reduce
                exits with 0
                other versions of iprange will exit with 1
                use this option in scripts to find if this
                version of iprange is present in a system

        -v
                be verbose on stderr

        --help
        -h
                print this message


        --------------------------------------------------------------
        INPUT FILES

        fileN
                a filename or - for stdin
                each filename can be followed by [as NAME]
                to change its name in the CSV output

                if no filename is given, stdin is assumed

                files may contain any or all of the following:
                - comments starting with # or ;
                - one IP per line (without mask)
                - a CIDR per line (A.A.A.A/B)
                - an IP range per line (A.A.A.A - B.B.B.B)
                - a CIDR range per line (A.A.A.A/B - C.C.C.C/D)
                  the range is calculated as the network address of
                  A.A.A.A/B to the broadcast address of C.C.C.C/D
                  (this is affected by --dont-fix-network)
                - CIDRs can be given in either prefix or netmask
                  format in all cases (including ranges)
                - one hostname per line, to be resolved with DNS
                  (if the IP resolves to multiple IPs, all of them
                  will be added to the ipset)
                  hostnames cannot be given as ranges
                - spaces and empty lines are ignored

                any number of files can be given

why to use it over any other aggregate

  • simpler
  • supports many input formats, all together in the same file
  • faster, actually a lot faster (thanks to the original design by Gabriel Somlo)
  • can reduce the subnets/prefixes to produce high-performing netfilter/iptables ipsets
  • can compare files with IPs to find if they overlap and to what degree
  • can find the IPs common to a set of files
  • can exclude IPs (merge a set of files while excluding all IPs matched by another set of files)

Reducing ipsets

This is the more interesting, so a small demonstration is needed.

netfilter/iptables ipsets are a very effective way of managing lists of IPs for firewall use. ipsets are super fast. Actually the number of entries in an ipset does not affect its performance.

You can also create ipsets that can contain IP subnets (hash:net - I call them netsets). These netsets have a little overhead over the plain hash:ip: For each different subnet mask in the netset, another lookup is made. So if you have a netset that has entries in all possible IPv4 subnets, up to 31 lookups have to be made for a single check.

iprange solves this problem by reducing the subnets in any given netset. For example, instead for one entry with /23, it can produce two entries with /24. The result is the same. The final unique number of IPs the netset matches, is exactly the same. But the subnets are reduced, thus improving the performance of the firewall.

Let's see it.

I'll use the netset with all the IPs in Greece, as updated by FireHOL program update-ipsets.

These are the metadata for this netset:

# iprange /etc/firehol/ipsets/geolite2_country/country_gr.netset -C --header
entries,unique_ips
406,6304132

So, it has 406 entries and 6.3 million unique IPs.

Let's see a break down per subnet. The following command requests a merge but we added -v to get some more info:

# iprange -v /etc/firehol/ipsets/geolite2_country/country_gr.netset >/dev/null
iprange: Loading from /etc/firehol/ipsets/geolite2_country/country_gr.netset
iprange: Optimizing combined ipset
iprange: Printing combined ipset

Break down by prefix:
        - prefix /13 counts 1 entries
        - prefix /14 counts 3 entries
        - prefix /15 counts 7 entries
        - prefix /16 counts 42 entries
        - prefix /17 counts 19 entries
        - prefix /18 counts 17 entries
        - prefix /19 counts 21 entries
        - prefix /20 counts 21 entries
        - prefix /21 counts 30 entries
        - prefix /22 counts 50 entries
        - prefix /23 counts 50 entries
        - prefix /24 counts 98 entries
        - prefix /25 counts 4 entries
        - prefix /27 counts 2 entries
        - prefix /28 counts 7 entries
        - prefix /29 counts 25 entries
        - prefix /31 counts 3 entries
        - prefix /32 counts 6 entries

totals: 406 lines loaded, 362 distinct IP ranges, 18 CIDR prefixes,
        406 CIDRs printed, 6304132 unique IPs
completed in 0.00234 seconds (load 0.00059 + think 0.00026 + print 0.00150)

It uses 18 prefixes. Adding this to your firewall and you will have your kernel do 18 lookups for every packet it checks against it.

Let's optimize it:

# iprange -v --ipset-reduce 20 /etc/firehol/ipsets/geolite2_country/country_gr.netset >/dev/null

Eliminated 15 out of 18 prefixes (3 remain in the final set).

iprange: Printing combined ipset

Break down by prefix:
        - prefix /21 counts 3028 entries
        - prefix /24 counts 398 entries
        - prefix /32 counts 900 entries

totals: 406 lines loaded, 362 distinct IP ranges, 3 CIDR prefixes,
        4326 CIDRs printed, 6304132 unique IPs
completed in 0.01468 seconds (load 0.00060 + think 0.00416 + print 0.00991)

Now it has only 3 different subnets, so the kernel will only do 3 lookups for each check. iprange generated 4.326 entries to match all 6.3 million IPs, but this is not a problem for the kernel.

There are 2 options for you to tweak:

  • --ipset-reduce PERCENT
  • --ipset-reduce-entries ENTRIES

You enable ipset reduce mode if you give any of these options. The default PERCENT is 20, the default ENTRIES is 16384.

This is what these options do:

iprange before it attempts to reduce a netset, tries to find the maximum acceptable entries the resulting netset should have. This is necessary, because otherwise, the optimal would be to generate 6.3 million entries with /32, which is too big (the kernel will probably handle it, but every operation on it will be very slow).

To figure out the maximum acceptable entries, iprange takes the maximum value of these two:

  1. The current size of the ipset, increased by PERCENT %
  2. The absolute number of ENTRIES given

The maximum of the two is the maximum acceptable entries of the netset.

You can give both options at the same time. If you don't give one of the two the internal default will be used (PERCENT = 20%, ENTRIES = 16384).

These options have been designed in such a way that you will get acceptable results for all sizes of netsets.

For example, you can always call iprange like this:

# iprange --ipset-reduce 20 --ipset-reduce-entries 50000 FILENAME

A small netset, like the country IPs we saw before, will be scaled up to 50.000 entries, while a huge netset with let's say 200.000 entries will be scaled up to 240.000 entries.

Let's try this on the Greece netset we used before:

# iprange -v --ipset-reduce 20 --ipset-reduce-entries 50000 /etc/firehol/ipsets/geolite2_country/country_gr.netset >/dev/null

# lots of output due to -v (verbose)

Eliminated 16 out of 18 prefixes (2 remain in the final set).

iprange: Printing combined ipset

Break down by prefix:
        - prefix /24 counts 24622 entries
        - prefix /32 counts 900 entries

totals: 406 lines loaded, 362 distinct IP ranges, 2 CIDR prefixes,
        25522 CIDRs printed, 6304132 unique IPs
completed in 0.03334 seconds (load 0.00035 + think 0.00113 + print 0.03186)

Now that we increased the maximum entries to 50.000, iprange eliminated another subnet, by increasing the entries to 25.522

Let's now check a big netset:

# iprange -v /etc/firehol/ipsets/ib_bluetack_level1.netset >/dev/null
iprange: Loading from /etc/firehol/ipsets/ib_bluetack_level1.netset
iprange: Optimizing combined ipset
iprange: Printing combined ipset

Break down by prefix:
        - prefix /7 counts 5 entries
        - prefix /8 counts 15 entries
        - prefix /9 counts 1 entries
        - prefix /10 counts 4 entries
        - prefix /11 counts 10 entries
        - prefix /12 counts 24 entries
        - prefix /13 counts 52 entries
        - prefix /14 counts 143 entries
        - prefix /15 counts 484 entries
        - prefix /16 counts 1674 entries
        - prefix /17 counts 191 entries
        - prefix /18 counts 311 entries
        - prefix /19 counts 676 entries
        - prefix /20 counts 1092 entries
        - prefix /21 counts 1308 entries
        - prefix /22 counts 2326 entries
        - prefix /23 counts 4767 entries
        - prefix /24 counts 17416 entries
        - prefix /25 counts 4856 entries
        - prefix /26 counts 8190 entries
        - prefix /27 counts 18123 entries
        - prefix /28 counts 32637 entries
        - prefix /29 counts 94802 entries
        - prefix /31 counts 4370 entries
        - prefix /32 counts 24830 entries

totals: 218307 lines loaded, 195416 distinct IP ranges, 25 CIDR prefixes,
        218307 CIDRs printed, 764993634 unique IPs
completed in 0.55851 seconds (load 0.16760 + think 0.06466 + print 0.32625)

It is big. 218.307 entries, using 25 subnets.

Let's reduce it:

# iprange -v --ipset-reduce 20 --ipset-reduce-entries 50000 /etc/firehol/ipsets/ib_bluetack_level1.netset >/dev/null

# lots of output due to -v (verbose)

Eliminated 17 out of 25 prefixes (8 remain in the final set).

iprange: Printing combined ipset

Break down by prefix:
        - prefix /16 counts 11118 entries
        - prefix /20 counts 5216 entries
        - prefix /24 counts 46718 entries
        - prefix /26 counts 17902 entries
        - prefix /27 counts 18123 entries
        - prefix /28 counts 32637 entries
        - prefix /29 counts 94802 entries
        - prefix /32 counts 33570 entries

totals: 218307 lines loaded, 195416 distinct IP ranges, 8 CIDR prefixes,
        260086 CIDRs printed, 764993634 unique IPs
completed in 0.62254 seconds (load 0.12429 + think 0.12056 + print 0.37769)

It eliminated 17 of the 25 subnets, by increasing the entries from 218.307 to 260.086. Quite acceptable I think.

If you increase PERCENT to 50%, it will leave only 6 subnets for 311.767 entries. At 100% it will leave 5 subnets for 389.977 entries, etc.

If you read the verbose output I cut, you will find out how it works. It is optimal. It tries to find the subnet with the smallest effect on the number of entries, it merges it to the next available and it continues until the maximum acceptable entries is reached. So the result is maximum effectiveness on the number of subnets, with the fewest number of entries added.

Keep in mind you can always get the original netset with iprange. Example:

iprange --ipset-reduce 100 --ipset-reduce-entries 50000 \
   /etc/firehol/ipsets/ib_bluetack_level1.netset |\
   iprange -v >/dev/null

# this is the output of the second iprange:

iprange: Loading from stdin
iprange: Optimizing combined ipset
iprange: Printing combined ipset

Break down by prefix:
        - prefix /7 counts 5 entries
        - prefix /8 counts 15 entries
        - prefix /9 counts 1 entries
        - prefix /10 counts 4 entries
        - prefix /11 counts 10 entries
        - prefix /12 counts 24 entries
        - prefix /13 counts 52 entries
        - prefix /14 counts 143 entries
        - prefix /15 counts 484 entries
        - prefix /16 counts 1674 entries
        - prefix /17 counts 191 entries
        - prefix /18 counts 311 entries
        - prefix /19 counts 676 entries
        - prefix /20 counts 1092 entries
        - prefix /21 counts 1308 entries
        - prefix /22 counts 2326 entries
        - prefix /23 counts 4767 entries
        - prefix /24 counts 17416 entries
        - prefix /25 counts 4856 entries
        - prefix /26 counts 8190 entries
        - prefix /27 counts 18123 entries
        - prefix /28 counts 32637 entries
        - prefix /29 counts 94802 entries
        - prefix /31 counts 4370 entries
        - prefix /32 counts 24830 entries

totals: 388865 lines loaded, 195416 distinct IP ranges, 25 CIDR prefixes,
        218307 CIDRs printed, 764993634 unique IPs
completed in 1.30543 seconds (load 0.88931 + think 0.08936 + print 0.32676)

The output of the above, is exactly the same with the source file.

If you are looking for maintained IP blacklists/blocklists, we have a large collection of IP lists tracking abuse, attacks, malware, botnets, command and control hosts, open proxies, anonymizers, etc. at FireHOL IP Lists.

Add a custom footer
Add a custom sidebar
Clone this wiki locally