Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] iptables rules (iptables.insert with save:True) added with each salt run, resulting in duplicate entries in /etc/iptables/rules.v* #60467

Closed
mirko opened this issue Jun 30, 2021 · 13 comments · Fixed by #61736
Assignees
Labels
Bug broken, incorrect, or confusing behavior Confirmed Salt engineer has confirmed bug/feature - often including a MCVE help-wanted Community help is needed to resolve this
Milestone

Comments

@mirko
Copy link

mirko commented Jun 30, 2021

Description
Having iptables rules defined via salt's iptables states, e.g.:

port_fw_rule:
  iptables.insert:
    - position: 1
    - chain: PREROUTING
    - table: nat
    - in-interface: eth0
    - proto: tcp
    - dport: 4242
    - jump: DNAT
    - to: 10.0.0.42:42
    - save: True

results in correct entries in /etc/iptables/rules.v4 after the first run.
With each further salt run, though, the same line gets appended to the existing lines in /etc/iptables/rules.v4.
After 42 runs I have 42 lines of above defined rule listed in /etc/iptables/rules.v4.
Running salt in this case means running salt-ssh.

Setup
salt-ssh on Debian/Bullseye (currently testing)

Steps to Reproduce the behavior
see above

Expected behavior
The respective iptables rule, derived from the above defined iptables salt state, being present only once in /etc/iptables/rules.v4 - no matter how often salt was run.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
          Salt: 3003.1
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.6.1
        pygit2: Not Installed
        Python: 3.7.3 (default, Jan 22 2021, 20:04:44)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.1.2
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.1
 
System Versions:
          dist: debian 10 buster
        locale: UTF-8
       machine: x86_64
       release: 4.19.0-14-amd64
        system: Linux
       version: Debian GNU/Linux 10 buster

Additional context
Potentially relevant issues / commits:

@mirko mirko added Bug broken, incorrect, or confusing behavior needs-triage labels Jun 30, 2021
@waynew
Copy link
Contributor

waynew commented Jul 29, 2021

blarp:
----------
          ID: port_fw_rule
    Function: iptables.insert
      Result: True
     Comment: Set and saved iptables rule port_fw_rule for ipv4
              /usr/sbin/iptables --wait -t nat -I PREROUTING 1 -p tcp --dport 4242 --in-interface eth0 --jump DNAT --to 10.0.0.42:42                                                                              
              Wrote 1 lines to "/etc/iptables/rules.v4"                                                                                                                                                           
     Started: 21:35:52.866638
    Duration: 237.233 ms
     Changes:   
              ----------                                                                                                                                                                                          
              locale:
                  port_fw_rule

Summary for blarp
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time: 237.233 ms
root@552b989792d1:/# salt-ssh blarp state.apply fun
blarp:
----------
          ID: port_fw_rule
    Function: iptables.insert
      Result: True
     Comment: iptables rule for port_fw_rule already set for ipv4 (/usr/sbin/iptables --wait -t nat -I PREROUTING 1 -p tcp --dport 4242 --in-interface eth0 --jump DNAT --to 10.0.0.42:42)
     Started: 21:35:58.095930
    Duration: 28.386 ms
     Changes:   

Summary for blarp
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time:  28.386 ms
root@552b989792d1:/# salt-ssh --versions-report
Salt Version:
          Salt: 3003.1
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.0.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.10.1
        pygit2: Not Installed
        Python: 3.7.3 (default, Jan 22 2021, 20:04:44)
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 22.1.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: debian 10 buster
        locale: utf-8
       machine: x86_64
       release: 5.13.5-arch1-1
        system: Linux
       version: Debian GNU/Linux 10 buster

I am just not able to repro. iptables v1.8.2 here 🤔 Not sure why that would make a difference though.

@waynew
Copy link
Contributor

waynew commented Jul 29, 2021

Try running check: https://docs.saltproject.io/en/latest/ref/modules/all/salt.modules.iptables.html#salt.modules.iptables.check

I think it would look like

salt-ssh my-minion iptables.check nat PREROUTING rule="-j DNAT -i eth0 -p tcp --dport 99 --to 10.0.0.42:42"

@mirko
Copy link
Author

mirko commented Jul 29, 2021

I'm on a very slim, clean and vanilla debian bullseye (installed via virt-install).

test.sls:

port_fw_rule:
  iptables.insert:
    - position: 1
    - chain: PREROUTING
    - table: nat
    - in-interface: enp1s0
    - proto: tcp
    - dport: 55555
    - jump: DNAT
    - to: 10.0.3.11
    - save: True

Output of two runs:

mirko@conf:/data/salt/states$ salt-ssh -v 'HOST' 'state.sls' 'test'
Executing job with jid 20210729221426986186
-------------------------------------------


debug1: client_input_channel_open: ctype auth-agent@openssh.com rchan 2 win 65536 max 16384
debug1: channel 1: new [authentication agent connection]
debug1: confirm auth-agent@openssh.com
debug1: channel 1: FORCE input drain
debug1: channel 1: free: authentication agent connection, nchannels 2
debug1: client_input_channel_open: ctype auth-agent@openssh.com rchan 2 win 65536 max 16384
debug1: channel 1: new [authentication agent connection]
debug1: confirm auth-agent@openssh.com
debug1: channel 1: FORCE input drain
debug1: channel 1: free: authentication agent connection, nchannels 2
debug1: client_input_channel_open: ctype auth-agent@openssh.com rchan 2 win 65536 max 16384
debug1: channel 1: new [authentication agent connection]
debug1: confirm auth-agent@openssh.com
debug1: channel 1: FORCE input drain
debug1: channel 1: free: authentication agent connection, nchannels 2
HOST:
----------
          ID: port_fw_rule
    Function: iptables.insert
      Result: True
     Comment: Set and saved iptables rule port_fw_rule for ipv4
              /usr/sbin/iptables --wait -t nat -I PREROUTING 1 -p tcp --dport 55555 --in-interface enp1s0 --jump DNAT --to 10.0.3.11
              Wrote 1 lines to "/etc/iptables/rules.v4"
     Started: 22:14:52.230583
    Duration: 223.894 ms
     Changes:   
              ----------
              locale:
                  port_fw_rule

Summary for HOST
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time: 223.894 ms
mirko@conf:/data/salt/states$ salt-ssh -v 'HOST' 'state.sls' 'test'
Executing job with jid 20210729221523296293
-------------------------------------------


debug1: client_input_channel_open: ctype auth-agent@openssh.com rchan 2 win 65536 max 16384
debug1: channel 1: new [authentication agent connection]
debug1: confirm auth-agent@openssh.com
debug1: channel 1: FORCE input drain
debug1: channel 1: free: authentication agent connection, nchannels 2
debug1: client_input_channel_open: ctype auth-agent@openssh.com rchan 2 win 65536 max 16384
debug1: channel 1: new [authentication agent connection]
debug1: confirm auth-agent@openssh.com
debug1: channel 1: FORCE input drain
debug1: channel 1: free: authentication agent connection, nchannels 2
debug1: client_input_channel_open: ctype auth-agent@openssh.com rchan 2 win 65536 max 16384
debug1: channel 1: new [authentication agent connection]
debug1: confirm auth-agent@openssh.com
debug1: channel 1: FORCE input drain
debug1: channel 1: free: authentication agent connection, nchannels 2
HOST:
----------
          ID: port_fw_rule
    Function: iptables.insert
      Result: True
     Comment: Set and saved iptables rule port_fw_rule for ipv4
              /usr/sbin/iptables --wait -t nat -I PREROUTING 1 -p tcp --dport 55555 --in-interface enp1s0 --jump DNAT --to 10.0.3.11
              Wrote 1 lines to "/etc/iptables/rules.v4"
     Started: 22:15:40.136389
    Duration: 325.749 ms
     Changes:   
              ----------
              locale:
                  port_fw_rule

Summary for HOST
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time: 325.749 ms
mirko@conf:/data/salt/states$

@mirko
Copy link
Author

mirko commented Jul 29, 2021

Calling
salt-ssh HOST iptables.check nat PREROUTING rule="-p tcp --dport 55555 --in-interface enp1s0 --jump DNAT --to 10.0.3.11"
results in

HOST:
    DNAT  tcp opt -- in enp1s0 out *  0.0.0.0/0  -> 0.0.0.0/0   tcp dpt:55555 to:10.0.3.11

@waynew
Copy link
Contributor

waynew commented Aug 2, 2021

Okay that's heckin interesting! And also confusing. Because the iptables.check is how the state determines that it needs to re-apply... yet it's succeeding here :suspect: so I'm thinking that it might be incorrectly comparing the outputs.

Do you have a copy of the settings that you used to spin up the vm?

@c1pherx
Copy link

c1pherx commented Aug 21, 2021

I can confirm that this is still happening with a Debian 11 salt master, and a Debian 11 salt minion. Version reports below:

Master

Salt Version:
          Salt: 3003.2

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.5
     gitpython: 3.1.14
        Jinja2: 2.11.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.9.7
        pygit2: Not Installed
        Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 22.2.1
         smmap: 4.0.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: aarch64
       release: 5.9.0-5-arm64
        system: Linux
       version: Debian GNU/Linux 11 bullseye

Minion:

Salt Version:
          Salt: 3003.2

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.9.7
        pygit2: Not Installed
        Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 20.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: aarch64
       release: 5.10.0-8-arm64
        system: Linux
       version: Debian GNU/Linux 11 bullseye

@waynew
Copy link
Contributor

waynew commented Sep 13, 2021

@c1pherx was this docker images or virt, or...?

@waynew waynew added Confirmed Salt engineer has confirmed bug/feature - often including a MCVE and removed needs-triage labels Sep 14, 2021
@waynew waynew added this to the Approved milestone Sep 14, 2021
@waynew
Copy link
Contributor

waynew commented Sep 14, 2021

Okay, I was able to repro this using a vagrant box from generic/debian11. I spun up the box, did an apt install of python3-pip and iptables, then did a pip install of salt.

port_fw_rule:
  iptables.insert:
    - position: 1
    - chain: PREROUTING
    - table: nat
    - in-interface: eth0
    - proto: tcp
    - dport: 4242
    - jump: DNAT
    - to: 10.0.0.42:42
    - save: True

Is the state that I used, and I set the root password to asdf. This roster:

localhost: 
  host: 127.0.0.1
  user: root
  passwd: asdf

(followed by ssh 127.0.0.1 to accept the host key) allowed me to run salt-ssh localhost state.apply mystate. It will continually add entries to /etc/iptables/rules.v4, as reported by Salt.

salt-ssh localhost iptables.check nat PREROUTING rule="-p tcp --dport 4242 --in-interface eth0 --jump DNAT --to 10.0.0.42:42"

Will report that the rule exists, so apparently somewhere in the iptables code when we do the check we're not doing it correctly. I didn't dig into that part, but when someone fixes this issue, it's going to probably be within the iptables module where it does check -- I remember when looking at this earlier there was some code where it was trying to do normalization of the rules. I'm pretty sure that it's maybe mis-parsing this because the iptables.check correctly reports the presence of the rule.

Anyway, this is confirmed 👍

@mirko
Copy link
Author

mirko commented Sep 30, 2021

Since this is confirmed and Debian bullseye (11) is released as stable meanwhile, can we push that a little tiny bit on the agenda? :)

@waynew
Copy link
Contributor

waynew commented Oct 21, 2021

@mirko added to our list to consider for Phosphorus. No promises, but we'll at least consider it. Of course, if you're willing to add some tests/fix, we welcome PRs!

If you're not sure about writing tests, I run the Test Clinics on Twitch every Tues/Thurs, and am there to help folks get started writing tests!

@waynew waynew added the help-wanted Community help is needed to resolve this label Nov 19, 2021
@raphael-walther
Copy link

raphael-walther commented Jan 21, 2022

I had this issue on Debian bullseye and this is a fix that works with bullseye but not with buster. This bug is due to a change in the return of iptables --check in bullseye compare to buster.

Following servers have this rule active in firewall:
-A INPUT -i lo -j ACCEPT

With buster:


root@buster:~#  salt-call iptables.check filter INPUT rule="-i lo -j ACCEPT"
local:
    True
root@buster:~# echo $?
0
root@buster:~#  salt-call iptables.check filter INPUT rule="-i lo -j DROP"
local:
    iptables: Bad rule (does a matching rule exist in that chain?).
root@buster:~# echo $?
0

With bullseye:

root@bullseye:~# salt-call iptables.check filter INPUT rule="-i lo -j ACCEPT"
local:
    ACCEPT  all opt -- in lo out *  0.0.0.0/0  -> 0.0.0.0/0
root@bullseye:~# echo $?
0
root@bullseye:~# salt-call iptables.check filter INPUT rule="-i lo -j DROP"
local:
    iptables: Bad rule (does a matching rule exist in that chain?).
root@bullseye:~# echo $?
1

One way to make iptables.append work on bullseye is to replace following line by next two:

if __salt__["iptables.check"](table, kwargs["chain"], rule, family) is True:

__salt__["iptables.check"](table, kwargs["chain"], rule, family)
if __context__["retcode"] == 0:

There are two other lines in the code with "iptables.check" that need attention.

Hope it helps.

@TheDJVG
Copy link
Contributor

TheDJVG commented Feb 28, 2022

Also hit this problem testing upgrade to bullseye.

Looking at the code (salt & iptables) it's unclear to me it this actually might be an iptables bug. The reason is that iptables in stretch and buster by default do not return the rule when running iptables --check :

root@efd6228692fa:/# iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
root@efd6228692fa:/# iptables --check INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
root@efd6228692fa:/# echo $?
0
root@efd6228692fa:/# iptables --version; cat /etc/debian_version
iptables v1.6.0
9.13

But it does when you pass the --verbose flag:

root@6e08a26b44b8:/# iptables --verbose --check INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
ACCEPT  all opt -- in * out *  0.0.0.0/0  -> 0.0.0.0/0   ctstate RELATED,ESTABLISHED
root@6e08a26b44b8:/# iptables --version; cat /etc/debian_version
iptables v1.6.0
9.13

For some reason iptables under bullseye (v1.8.7) prints the rule, even when verbose is not enable (or is it implied with --check?):

root@c186cb51b76e:/# iptables --check INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
ACCEPT  all opt -- in * out *  0.0.0.0/0  -> 0.0.0.0/0   ctstate RELATED,ESTABLISHED
root@c186cb51b76e:/# echo $?
0
root@c186cb51b76e:/# iptables --version; cat /etc/debian_version
iptables v1.8.7 (nf_tables)
11.2

When this option was introduced (in iptables commit d59b9db031abee37a9aa9776662dd15370faabf4) it states:

[..] it can detect and indicate the existence of the
    specified rule by modifying the exit code. [...]

Based on this I think it's safe to just check on the return code from iptables --check and we do not have to focus on checking the output of the command. Adding this to the iptables module should all that's needed:

--- iptables.py     2022-02-28 13:24:48.719229701 +0000
+++ iptables.py     2022-02-28 13:24:42.698220497 +0000
@@ -744,6 +744,7 @@
     if _has_option("--check", family):
         cmd = "{} -t {} -C {} {}".format(ipt_cmd, table, chain, rule)
         out = __salt__["cmd.run"](cmd, output_loglevel="quiet")
+        return not __context__["retcode"]
     else:
         _chain_name = hex(uuid.getnode())

The only downside to this is if someone actually uses the return of this command for something, but I don't think so, and I can't find a use case for the actual output in the code.

@TheHolm
Copy link

TheHolm commented Aug 31, 2022

As it not yet fixed in Salt: 3002.6 for Debian 11 bullseye

I have honour to present ugliest workaround ever:
Snapshot bellow will apply TheDJVG patch to modules/iptables.py on salt minions. Number of leading spaces is essential.

{% if grains['osfinger'] == 'Debian-11' %}
Patch salt minion iptables module to work on debian 11:
  file.blockreplace:
    - name: /usr/lib/python3/dist-packages/salt/modules/iptables.py
    - marker_start: '        cmd = "{} -t {} -C {} {}".format(ipt_cmd, table, chain, rule)'
    - marker_end:   '    else:'
    - content: |2
                __salt__["cmd.run_stderr"](cmd, output_loglevel="quiet")
                return not __context__["retcode"]
{% endif %}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug broken, incorrect, or confusing behavior Confirmed Salt engineer has confirmed bug/feature - often including a MCVE help-wanted Community help is needed to resolve this
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants