k3s implement clean automated maintenance

[dpausp]

k3s: automated maintenance

• Add fc-kubernetes agent command to be used manually and in maintenance
  enter/leave commands.
• Only allow one k3s-agent in maintenance at the same time
• Drain nodes before running maintenance requests.

@flyingcircusio/release-managers

Release process

Impact:

Changelog:

• k3s/kubernetes: implement clean automated maintenance. Agent nodes are drained (pods move to other nodes if possible) before executing maintenance requests and are uncordoned after maintenance is finished. Only one agent node can be in maintenance at any given time (PL-131525).

Security implications

• Security requirements defined? (WHERE)
  • kubernetes maintenance commands must be limited to sudo-srv and admins groups
• Security requirements tested? (EVIDENCE)
  • checked on the test k3s cluster that
    • automated maintenance drains and uncordons agent nodes properly
    • drain timeouts cause an agent tempfail
    • server-only nodes cannot be drained (which makes only sense for agents)
    • sudo rules are correct
  • automated tests cover new functionality