Deletion of individual NodePortLocal (NPL) mapping rules not working correctly

[antoninbas]

Describe the bug
It is possible to create a situation where the NPL controller in the Antrea Agent cannot delete NPL rules appropriately when needed (on Linux Nodes). The DNAT rule gets deleted correctly from iptables, but the Pod annotation is not updated, and we get stuck in an error loop when trying to handle Pod updates.

To Reproduce
On a cluster running Antrea with NPL enabled, do the following:

1. Apply the following YAML manifest. You will get a Pod with 2 containers, both running web servers on different ports. You will also get 2 Services, one for each web server, with NPL enabled.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-server
  template:
    metadata:
      labels:
        app: web-server
    spec:
      containers:
      - name: web-server-1
        image: registry.k8s.io/e2e-test-images/agnhost:2.29
        command: ["/agnhost", "test-webserver", "--port=9090"]
        ports:
        - containerPort: 9090
      - name: web-server-2
        image: registry.k8s.io/e2e-test-images/agnhost:2.29
        command: ["/agnhost", "test-webserver", "--port=9000"]
        ports:
        - containerPort: 9000

---
apiVersion: v1
kind: Service
metadata:
  name: web-service-9090
  annotations:
    nodeportlocal.antrea.io/enabled: "true"
spec:
  selector:
    app: web-server
  ports:
  - protocol: TCP
    port: 80
    targetPort: 9090

---
apiVersion: v1
kind: Service
metadata:
  name: web-service-9000
  annotations:
    nodeportlocal.antrea.io/enabled: "true"
spec:
  selector:
    app: web-server
  ports:
  - protocol: TCP
    port: 80
    targetPort: 9000

2. Check the Pod NPL annotation, it should have 2 mappings. For example:

$ kubectl get pod/web-deployment-7fd7879d77-b96jf -o=jsonpath='{.metadata.annotations}'
{"nodeportlocal.antrea.io":"[{\"podPort\":9090,\"nodeIP\":\"172.18.0.3\",\"nodePort\":61001,\"protocol\":\"tcp\"},{\"podPort\":9000,\"nodeIP\":\"172.18.0.3\",\"nodePort\":61000,\"protocol\":\"tcp\"}]"}

3. Edit one of the Services to remove the NPL annotation and disable NPL.

Expected

After a short while, the Pod's NPL annotation should be updated to show a single mapping.

Actual behavior

The Pod's NPL annotation is never updated, and the antrea-agent logs (on the Node where the Pod is running) show the following errors:

I0502 23:03:21.694528       1 iptable_rule.go:124] "Successfully deleted DNAT rule" podAddr="10.10.1.2:9090" nodePort=61001 protocol="tcp"
E0502 23:03:21.694730       1 npl_controller.go:375] Error syncing Pod default/web-deployment-7fd7879d77-b96jf, requeuing. Error: failed to delete rule for Pod IP 10.10.1.2, Pod Port 9090, Protocol tcp: protocol tcp is still in use, cannot release socket
I0502 23:03:23.705246       1 iptable_rule.go:124] "Successfully deleted DNAT rule" podAddr="10.10.1.2:9090" nodePort=61001 protocol="tcp"
E0502 23:03:23.705867       1 npl_controller.go:375] Error syncing Pod default/web-deployment-7fd7879d77-b96jf, requeuing. Error: failed to delete rule for Pod IP 10.10.1.2, Pod Port 9090, Protocol tcp: protocol tcp is still in use, cannot release socket
I0502 23:03:27.714051       1 iptable_rule.go:124] "Successfully deleted DNAT rule" podAddr="10.10.1.2:9090" nodePort=61001 protocol="tcp"
E0502 23:03:27.714315       1 npl_controller.go:375] Error syncing Pod default/web-deployment-7fd7879d77-b96jf, requeuing. Error: failed to delete rule for Pod IP 10.10.1.2, Pod Port 9090, Protocol tcp: protocol tcp is still in use, cannot release socket
I0502 23:03:35.719107       1 iptable_rule.go:124] "Successfully deleted DNAT rule" podAddr="10.10.1.2:9090" nodePort=61001 protocol="tcp"
E0502 23:03:35.719317       1 npl_controller.go:375] Error syncing Pod default/web-deployment-7fd7879d77-b96jf, requeuing. Error: failed to delete rule for Pod IP 10.10.1.2, Pod Port 9090, Protocol tcp: protocol tcp is still in use, cannot release socket
I0502 23:03:51.724275       1 iptable_rule.go:124] "Successfully deleted DNAT rule" podAddr="10.10.1.2:9090" nodePort=61001 protocol="tcp"

This issue only exists when we (the NPL controller in the Agent) need to edit an existing NPL Pod annotation to remove one of the rules / mappings. When the Pod is deleted, or when NPL is completely "disabled" for a Pod (all mappings are removed), the issue does not arise. This is because we use different functions to handle these 2 different situations:

• delete all NPL rules for the Pod (this works):

  antrea/pkg/agent/nodeportlocal/portcache/port_table_others.go

  Lines 155 to 170 in f9345da

  	func (pt *PortTable) DeleteRulesForPod(podIP string) error {
  	pt.tableLock.Lock()
  	defer pt.tableLock.Unlock()
  	podEntries := pt.getDataForPodIP(podIP)
  	for _, podEntry := range podEntries {
  	protocolSocketData := podEntry.Protocol
  	if err := pt.PodPortRules.DeleteRule(podEntry.NodePort, podIP, podEntry.PodPort, protocolSocketData.Protocol); err != nil {
  	return err
  	}
  	if err := protocolSocketData.socket.Close(); err != nil {
  	return fmt.Errorf("error when releasing local port %d with protocol %s: %v", podEntry.NodePort, protocolSocketData.Protocol, err)
  	}
  	pt.deletePortTableCache(podEntry)
  	}
  	return nil
  	}

• delete a single NPL rule for the Pod (this does not work):

  antrea/pkg/agent/nodeportlocal/portcache/port_table_others.go

  Lines 136 to 153 in f9345da

  	func (pt *PortTable) DeleteRule(podIP string, podPort int, protocol string) error {
  	pt.tableLock.Lock()
  	defer pt.tableLock.Unlock()
  	data := pt.getEntryByPodIPPortProto(podIP, podPort, protocol)
  	if data == nil {
  	// Delete not required when the PortTable entry does not exist
  	return nil
  	}
  	if err := pt.PodPortRules.DeleteRule(data.NodePort, podIP, podPort, protocol); err != nil {
  	return err
  	}
  	if err := data.CloseSockets(); err != nil {
  	return err
  	}
  	// We don't need to delete cache from different indexes repeatedly because they map to the same entry.
  	pt.deletePortTableCache(data)
  	return nil
  	}

Versions:
All "recent" Antrea versions.
This issue was reported for Antrea v1.11.3, and I confirmed that the issue is still present in Antrea v2.0.0.