feat: add csi-secrets-store addon

[aramase]

Reason for Change:

• Adds csi-secrets-store addon
• Enabled by default instead of keyvault-flexvolume for 1.16+ clusters
• If both keyvault-flexvolume and csi-secrets-store addon enabled simultaneously, then will result in an error

Error: validating API model after populating values: Both keyvault-flexvolume and csi-secrets-store addons are enabled, only one of these may be enabled on a cluster

• If keyvault-flexvolume addon enabled explicitly for 1.16 clusters, a warning message will be displayed and csi-secrets-store addon will not be enabled.

WARN[0000] keyvault-flexvolume add-on will be DEPRECATED in favor of csi-secrets-store addon for 1.16+ 

• Disable keyvault-flexvol addon for 1.16+ clusters
• Failure to enable csi-secrets-store addon for < 1.16 clusters

Error: validating API model after populating values: csi-secrets-store add-on can only be used in 1.16+

Issue Fixed:

fixes #2695

Requirements:

• uses conventional commit messages
• includes documentation
• adds unit tests
• tested upgrade from previous version

Notes:

[aramase]

/cc @ritazh

[codecov]

Codecov Report

Merging #2936 into master will decrease coverage by 0.47%.
The diff coverage is 94.87%.

@@            Coverage Diff            @@
##           master   #2936      +/-   ##
=========================================
- Coverage   72.58%   72.1%   -0.48%     
=========================================
  Files         141     141              
  Lines       25893   26251     +358     
=========================================
+ Hits        18794   18929     +135     
- Misses       6005    6228     +223     
  Partials     1094    1094

[jackfrancis]

Because we're installing this by default and it's so much larger in terms of its yaml byte payload compared to keyvault-flexvol, we may run into cloud-init data limits. We'll want to test this with other data-heavy configurations (e.g., calico).

If we do find that this touches the limit we can schedule some cloud-init payload optimization refactor work, but that would become a blocker to this landing.

@ritazh FYI

[aramase]

Going to test it now with calico enabled.

[aramase]

@jackfrancis I was able to successfully deploy 1.16 cluster with csi-secrets-store and calico enabled. Any other heavy config that you are aware of we should test with?

[jackfrancis]

Thanks for validating! I'm testing across the 1.17 matrix, will report back if any cluster configs complain about customData limits.

[jackfrancis]

Validation passed, so we don't have to worry about this (for now!)

[jackfrancis]

/lgtm

[acs-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aramase, jackfrancis

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jackfrancis]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment