Antrea should take into account changes to the configmap without requiring agent restart

[antoninbas]

Describe the bug
When updating the antrea-agent.conf in the Antrea YAML manifest and re-applying the manifest, all changes are ignored. One possible solution is to update the ConfigMap name (or update build/yamls/base/conf/antrea-agent.conf and run generate-manifest.sh again, which will take care of consistently updating the ConfigMap name); however, this requires an Agent restart.

I believe that we should have the following:

• for a configuration parameter like enablePrometheusMetrics, we should be able to update the value without having to restart Agents. On a related note, should we have the ability to turn metrics on/off easily on a per-Agent basis (e.g. using antctl)?
• for configuration parameters that cannot be changed at runtime, we should still detect the change and log an error.

To Reproduce
Apply "vanilla" Antrea manifest, uncomment enablePrometheusMetrics in the configuration and set it to true, then apply the manifest again: metrics will not be enabled.

Versions:
Confirmed with Antrea v0.6.0

[srikartati]

@antoninbas Currently Antrea agent loads its config map (antrea-agent.conf) as subpath volume.
From k8s documentation, kubelet will not update subpath volumes.

Note: A container using a ConfigMap as a subPath volume will not receive ConfigMap updates.

Does it make sense to have a separate config map called antrea-agent-updatable.conf and mount it as a volume with no subpath? In Antrea agent, we need to create a poll that keeps on checking this file and various keys for updates and act accordingly.
In this way, we do not have to worry about metrics that cannot be changed as they are in antrea-agent.conf.

In addition, I am trying to understand why antrea-agent.conf is considered as subPath volume. Do you have any insights?

[antoninbas]

I think we can stick with a single ConfigMap, but use different volumes, instead of a single volume with subPaths:

...
      volumeMounts:
        - name: antrea-config-agent
          mountPath: /etc/antrea/antrea-agent.conf
...
  volumes:
    - name: antrea-config-agent
      configMap:
        name: antrea-config-agent
        items:
          - key: antrea-agent.conf
            path: antrea-agent.conf
...

Then we can have another volume for the other files (controller, cni).

I haven't tried it, but I feel like it should work. Maybe @tnqn can also provide some insight.

[tnqn]

@antoninbas @srikartati I don't know a real reason why we used subPath volume, guess it was just straightforward to use subPath to mount a single key of a ConfigMap and we didn't consider dynamic config reloading at that time.
If the approach @antoninbas raised can work, I think it would be perfect.

[jayunit100]

Ya that would work i guess - from my perspective the reason i was thinking it would be awesome to have this,

• Calico allows you to quickly toggle a protocol w/o restarting things, so gives you an instant read out of the new routing tables -- helps users in having the same feature parity
• Without this change, there is an inconsitency in the state of the antrea configmap and the actual state of the routing tables
• makes for really cool demos :)

[jayunit100]

You also could just have the agent crash immediately on configmap hash changes probably without having to make any other major changes....

[mrporcles]

@antoninbas Any idea what release this is now planned for?

[antoninbas]

@mrporcles I am not actively working on this at the moment and it is not scoped for any release.
I did have a PR out to automatically restart the Antrea containers when the ConfigMap changes, but we have found a couple of drawbacks with that approach. At this time, my favorite solution would be treat the ConfigMap as immutable and offer support in the Antrea CLI (antctl) to edit configuration parameters. These edits would cause a new ConfigMap to be generated (with a new map), and trigger a rolling update of the Antrea components. I believe this is a very clean approach but a rolling update can be a bit "expensive" depending on which configuration parameter you want to modify. Specifically, in the context of this issue, it would still require a restart of the Antrea components, but you wouldn't have to do it manually any more.

[antoninbas]

The "immutable ConfigMap approach" is described here: #1427 (comment)

[luolanzone]

Hi @antoninbas @jianjuns @tnqn I would like propose a new way to detect configmap change:
add a new deployment with single replica to watch configmap change and update the agent/controller Daemonset/Deployment annotation which is similar like Helm chart: https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments, then we can rely on K8s rolling update strategy to update agent one by one. Could you help to check and give your feedback? thanks.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days