K8s extension/release 1.3.8 (#5798)

* Kubernetes Data Protection Extension CLI (#173) * First draft for Data Protection K8s backup extension (Pending internal review) * Removing tracing * Minor changes to improve azdev style * Internal PR review feedback Co-authored-by: Rishabh Raj <rishraj@microsoft.com> * {AKS - ARC} fix: Update DCR creation to Clusters resource group instead of workspace (#175) * fix: Update DCR creation to Clusters resource group instead of workspace * . * . * casing check * Add self-signed cert to fix PR gate for azureml extension * adding the api version to the operation definition in the client factory * bump k8s-extension version to 1.3.6 * adding tests for all 4 extension types calls * adding to test config file * updating the api version for extension types to be the correct version expected by the service * add test case for flux extension (#184) * bump k8s-extension version to 1.3.6 * bump k8s-extension version to 1.3.6 * adding upstream test for extension types * updating history.rst * [Dapr] Prompt user for existing Dapr installation during extension create (#188) * Add more validations and user prompt for existing installation scenario Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Add Dapr test' Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Handle stateful set Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Update default handling Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Fix HA handling Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Add placement service todo Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Add non-interactive mode Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Fix lint Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Update tests Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Reset configuration for StatefulSet during k8s upgrade Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Fix lint Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Retrigger tests Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Add changes to manage ha and placement params Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Update message Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * nits Signed-off-by: Shubham Sharma <shubhash@microsoft.com> Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * bump k8s-extension version to 1.3.7 * [Dapr] Disable applying CRDs during a downgrade (#193) * Add logging Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Lint Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Update log Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Revert applyCrds when not downgrading Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Update logic for removing hooks.applyCrds Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Revert logic Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Handle explicit hooks configuration Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Update comment Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * re-trigger pipeline Signed-off-by: Shubham Sharma <shubhash@microsoft.com> Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * ContainerInsights extension - Add dataCollectionSettings configuration settings (#200) * data collection settings * add support for dataCollectionSettings * fix indention * avoid duplicate use of json loads * remove whitespaces * fix pr feedback * Upgrade Python version from 3.6 to 3.7 (#203) * Upgrade Python version from 3.6 to 3.10 Upgrade to 3.10 for the job that runs Wheel, PyLint, Flake, etc., since 3.6 is not supported anymore by hosted-agent-software. * Upgrade to Python 3.10 from 3.6 Upgrade to 3.10 as 3.6 is not supported * Switch PyLink to 1.9.4 Switch PyLink to 1.9.4 from 1.9.5, as 1.9.5 is not supported with Python 3.10 * Use Python 3.7 for Static Analysis Use 3.7, as 3.10 does not support certain properties used by astpeephole.py that is used by Static Analysis tools * Try unpinned version of PyLint PyLint 1.9.5 doesn't work with Python 3.7. So, trying to see if it automatically pulls the latest compatible version. * Run pylint as a separate command * Update pylintrc (#204) * Update pylintrc * Update k8s-custom-pipelines.yml * Disable PyLint (#205) Disable PyLint for now, as the new version has breaking changes and requires lot more fixes * Disable PyLint on CI scripts * Fixes for script errors * Upgrade Static Analysis Python version Upgrade the Python version for Static Analysis to 3.10, from 3.7, now that PyLint is disabled * Try 3.9, as 3.10 has breaking changes for Flake8 * Remove version pinning for flake8 Try Python 3.10, without pinning flake8 to a version * Update k8s-custom-pipelines.yml * Use Python 3.8.1 & flake8 6.0.0 * Use Python 3.8 instead of 3.8.1 * Update k8s-custom-pipelines.yml * Update .flake8 Update to reflect breaking change in flake8 6.0 * Update source_code_static_analysis.py Scope static analysis tools to only k8s-extension module's source in our branch. * Update k8s-custom-pipelines.yml * Update k8s-custom-pipelines.yml * Update k8s-custom-pipelines.yml * Update pool name in StaticAnalysis To mirror what is in main of azure-cli-extensions * Update k8s-custom-pipelines.yml * Fix indentation * Update k8s-custom-pipelines.yml * Update k8s-custom-pipelines.yml * Revert changes * Revert changes * Revert changes to source_code_static_analysis.py * Update source_code_static_analysis.py * Revert changes * Use Ubuntu 20.4 for BuiltTestPublish stage * Switch to ubuntu-20.04 from latest Co-authored-by: Rishik Hombal <hombalrishik@gmail.com> * [Dapr] Do not apply CRD hook when version is unchanged or auto-upgrade is being disabled (#201) * Update logic Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * re-trigger pipeline Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * re-trigger pipeline Signed-off-by: Shubham Sharma <shubhash@microsoft.com> Signed-off-by: Shubham Sharma <shubhash@microsoft.com> Co-authored-by: NarayanThiru <nanthi@microsoft.com> * add dummy key for amalogs as well * bump k8s-extension version to 1.3.8 * Adding GA api version 2022-11-01 exposing isSystemExtension and support for plan info * Seperate args for plan name, product and publisher * updating cassete file * updating HISTORY.rst * k8s-extension release 1.3.8 --------- Signed-off-by: Shubham Sharma <shubhash@microsoft.com> Co-authored-by: Rishabh Raj <rishabhstpaul@gmail.com> Co-authored-by: Rishabh Raj <rishraj@microsoft.com> Co-authored-by: bragi92 <kadubey@microsoft.com> Co-authored-by: Yue Yu <yuyu3@microsoft.com> Co-authored-by: Deeksha Sharma <deesharma@microsoft.com> Co-authored-by: Bavneet Singh <33008256+bavneetsingh16@users.noreply.github.com> Co-authored-by: Shubham Sharma <shubhash@microsoft.com> Co-authored-by: Bavneet Singh <bavneetsingh@microsoft.com> Co-authored-by: Ganga Mahesh Siddem <gangams@microsoft.com> Co-authored-by: NarayanThiru <nanthi@microsoft.com> Co-authored-by: Rishik Hombal <hombalrishik@gmail.com> Co-authored-by: Amol Agrawal <amagraw@microsoft.com> Co-authored-by: Amol Agrawal <pfrcks@gmail.com> Co-authored-by: Arif Lakhani <ariflakhani@microsoft.com> Co-authored-by: Arif-lakhani <ariflakhani7786@gmail.com>
Azure · Jan 28, 2023 · d847792 · d847792
1 parent 055c2cf
commit d847792
Show file tree

Hide file tree

Showing 82 changed files with 22,673 additions and 94 deletions.
diff --git a/src/k8s-extension/HISTORY.rst b/src/k8s-extension/HISTORY.rst
@@ -3,6 +3,13 @@
 Release History
 ===============
 
+1.3.8
+++++++++++++++++++
+* Fixes to address the bug with msi auth mode for azuremonitor-containers extension version >= 3.0.0
+* microsoft.dapr: disable apply-CRDs hook if auto-upgrade is disabled
+* microsoft.azuremonitor.containers: ContainerInsights Extension add dataCollectionSettings to configuration settings
+* k8s-extension Adding GA api version 2022-11-01 exposing isSystemExtension and support
+
 1.3.7
 ++++++++++++++++++
 * microsoft.dapr: prompt user for existing dapr installation during extension create

diff --git a/src/k8s-extension/README.rst b/src/k8s-extension/README.rst
@@ -31,6 +31,10 @@ az k8s-extension create \
     --version versionNumber \
     --auto-upgrade-minor-version autoUpgrade \
     --configuration-settings exampleSetting=exampleValue \
+    --plan-name examplePlanName \
+    --plan-publisher examplePublisher \
+    --plan-product exampleOfferId \
+
 ```
 
 ##### Get a KubernetesExtension

diff --git a/src/k8s-extension/azext_k8s_extension/_format.py b/src/k8s-extension/azext_k8s_extension/_format.py
@@ -15,12 +15,21 @@ def k8s_extension_show_table_format(result):
 
 
 def __get_table_row(result):
+    plan_name, plan_publisher, plan_product = '', '', ''
+    if result['plan']:
+        plan_name = result['plan']['name']
+        plan_publisher = result['plan']['publisher']
+        plan_product = result['plan']['product']
     return OrderedDict([
         ('name', result['name']),
         ('extensionType', result.get('extensionType', '')),
         ('version', result.get('version', '')),
         ('provisioningState', result.get('provisioningState', '')),
         ('lastModifiedAt', result.get('systemData', {}).get('lastModifiedAt', '')),
+        ('plan_name', plan_name),
+        {'plan_publisher', plan_publisher},
+        ('plan_product', plan_product),
+        ('isSystemExtension', result.get('isSystemExtension', '')),
     ])
 
 

diff --git a/src/k8s-extension/azext_k8s_extension/_help.py b/src/k8s-extension/azext_k8s_extension/_help.py
@@ -15,13 +15,19 @@
 
 helps[f'{consts.EXTENSION_NAME} create'] = f"""
     type: command
-    short-summary: Create a Kubernetes Extension.
+    short-summary: Create a Kubernetes Cluster Extension, including purchasing an extension Offer from Azure Marketplace (AKS only). Please refer to the example at the end to see how to create an extension or purchase an extension offer.
     examples:
       - name: Create a Kubernetes Extension
         text: |-
           az {consts.EXTENSION_NAME} create --resource-group my-resource-group \
 --cluster-name mycluster --cluster-type connectedClusters --name myextension \
 --extension-type microsoft.openservicemesh --scope cluster --release-train stable
+      - name: Create a Kubernetes Marketplace Extension
+        text: |-
+          az {consts.EXTENSION_NAME} create --resource-group my-resource-group \
+--cluster-name mycluster --cluster-type managedClusters --name myextension \
+--extension-type Contoso.AzureVoteKubernetesAppTest --scope cluster --release-train stable \
+--plan-name testplan --plan-product kubernetest_apps_demo_offer --plan-publisher test_test_mix3pptest0011614206850774
 """
 
 helps[f'{consts.EXTENSION_NAME} list'] = f"""
@@ -67,9 +73,9 @@
 --cluster-name mycluster --cluster-type connectedClusters \
 --name myextension --auto-upgrade true/false --version extension-version \
 --release-train stable --configuration-settings settings-key=settings-value \
---configuration-protected-settings protected-settings-key=protected-value \
---configuration-settings-file=config-settings-file \
---configuration-protected-settings-file=protected-settings-file
+--config-protected-settings protected-settings-key=protected-value \
+--config-settings-file=config-settings-file \
+--config-protected-file=protected-settings-file
 """
 
 helps[f'{consts.EXTENSION_NAME} extension-types'] = """

diff --git a/src/k8s-extension/azext_k8s_extension/_params.py b/src/k8s-extension/azext_k8s_extension/_params.py
@@ -12,7 +12,7 @@
 
 from .action import (
     AddConfigurationSettings,
-    AddConfigurationProtectedSettings
+    AddConfigurationProtectedSettings,
 )
 
 
@@ -52,30 +52,41 @@ def load_arguments(self, _):
                    help='Specify the release train for the extension type.')
         c.argument('configuration_settings',
                    arg_group="Configuration",
-                   options_list=['--configuration-settings', '--config-settings', '--config'],
+                   options_list=['--configuration-settings', '--config'],
                    action=AddConfigurationSettings,
                    nargs='+',
                    help='Configuration Settings as key=value pair.  Repeat parameter for each setting')
         c.argument('configuration_protected_settings',
                    arg_group="Configuration",
-                   options_list=['--configuration-protected-settings', '--config-protected-settings', '--config-protected'],
+                   options_list=['--config-protected-settings', '--config-protected'],
                    action=AddConfigurationProtectedSettings,
                    nargs='+',
                    help='Configuration Protected Settings as key=value pair.  Repeat parameter for each setting')
         c.argument('configuration_settings_file',
                    arg_group="Configuration",
-                   options_list=['--configuration-settings-file', '--config-settings-file', '--config-file'],
+                   options_list=['--config-settings-file', '--config-file'],
                    help='JSON file path for configuration-settings')
         c.argument('configuration_protected_settings_file',
                    arg_group="Configuration",
-                   options_list=['--configuration-protected-settings-file', '--config-protected-settings-file', '--config-protected-file'],
+                   options_list=['--config-protected-file', '--protected-settings-file'],
                    help='JSON file path for configuration-protected-settings')
         c.argument('release_namespace',
                    help='Specify the namespace to install the extension release.')
         c.argument('target_namespace',
                    help='Specify the target namespace to install to for the extension instance. This'
                    ' parameter is required if extension scope is set to \'namespace\'')
-
+        c.argument('plan_name',
+                   arg_group="Marketplace",
+                   options_list=['--plan-name'],
+                   help='The plan name is referring to the Plan ID of the extension that is being taken from Marketplace portal under Usage Information + Support')
+        c.argument('plan_product',
+                   arg_group="Marketplace",
+                   options_list=['--plan-product'],
+                   help='The plan product is referring to the Product ID of the extension that is being taken from Marketplace portal under Usage Information + Support. An example of this is the name of the ISV offering used.')
+        c.argument('plan_publisher',
+                   arg_group="Marketplace",
+                   options_list=['--plan-publisher'],
+                   help='The plan publisher is referring to the Publisher ID of the extension that is being taken from Marketplace portal under Usage Information + Support')
     with self.argument_context(f"{consts.EXTENSION_NAME} update") as c:
         c.argument('yes',
                    options_list=['--yes', '-y'],

diff --git a/src/k8s-extension/azext_k8s_extension/consts.py b/src/k8s-extension/azext_k8s_extension/consts.py
@@ -21,7 +21,7 @@
 PROVISIONED_CLUSTER_TYPE = "provisionedclusters"
 
 CONNECTED_CLUSTER_API_VERSION = "2021-10-01"
-MANAGED_CLUSTER_API_VERSION = "2021-10-01"
+MANAGED_CLUSTER_API_VERSION = "2022-11-01"
 APPLIANCE_API_VERSION = "2021-10-31-preview"
 HYBRIDCONTAINERSERVICE_API_VERSION = "2022-05-01-preview"
 

diff --git a/src/k8s-extension/azext_k8s_extension/custom.py b/src/k8s-extension/azext_k8s_extension/custom.py
@@ -109,6 +109,9 @@ def create_k8s_extension(
     configuration_settings_file=None,
     configuration_protected_settings_file=None,
     no_wait=False,
+    plan_name=None,
+    plan_publisher=None,
+    plan_product=None
 ):
     """Create a new Extension Instance."""
 
@@ -182,6 +185,9 @@ def create_k8s_extension(
         config_protected_settings,
         configuration_settings_file,
         configuration_protected_settings_file,
+        plan_name,
+        plan_publisher,
+        plan_product
     )
 
     # Common validations

diff --git a/src/k8s-extension/azext_k8s_extension/partner_extensions/AzureDefender.py b/src/k8s-extension/azext_k8s_extension/partner_extensions/AzureDefender.py
@@ -21,7 +21,8 @@ class AzureDefender(DefaultExtension):
     def Create(self, cmd, client, resource_group_name, cluster_name, name, cluster_type, cluster_rp,
                extension_type, scope, auto_upgrade_minor_version, release_train, version, target_namespace,
                release_namespace, configuration_settings, configuration_protected_settings,
-               configuration_settings_file, configuration_protected_settings_file):
+               configuration_settings_file, configuration_protected_settings_file, plan_name,
+               plan_publisher, plan_product):
 
         """ExtensionType 'microsoft.azuredefender.kubernetes' specific validations & defaults for Create
            Must create and return a valid 'Extension' object.

diff --git a/src/k8s-extension/azext_k8s_extension/partner_extensions/AzureMLKubernetes.py b/src/k8s-extension/azext_k8s_extension/partner_extensions/AzureMLKubernetes.py
@@ -114,7 +114,8 @@ def __init__(self):
     def Create(self, cmd, client, resource_group_name, cluster_name, name, cluster_type, cluster_rp,
                extension_type, scope, auto_upgrade_minor_version, release_train, version, target_namespace,
                release_namespace, configuration_settings, configuration_protected_settings,
-               configuration_settings_file, configuration_protected_settings_file):
+               configuration_settings_file, configuration_protected_settings_file, plan_name,
+               plan_publisher, plan_product):
 
         logger.warning("Troubleshooting: {}".format(self.TSG_LINK))
 

diff --git a/src/k8s-extension/azext_k8s_extension/partner_extensions/ContainerInsights.py b/src/k8s-extension/azext_k8s_extension/partner_extensions/ContainerInsights.py
@@ -7,6 +7,7 @@
 
 import datetime
 import json
+import re
 
 from ..utils import get_cluster_rp_api_version
 from .. import consts
@@ -36,7 +37,8 @@ class ContainerInsights(DefaultExtension):
     def Create(self, cmd, client, resource_group_name, cluster_name, name, cluster_type, cluster_rp,
                extension_type, scope, auto_upgrade_minor_version, release_train, version, target_namespace,
                release_namespace, configuration_settings, configuration_protected_settings,
-               configuration_settings_file, configuration_protected_settings_file):
+               configuration_settings_file, configuration_protected_settings_file,
+               plan_name, plan_publisher, plan_product):
         """ExtensionType 'microsoft.azuremonitor.containers' specific validations & defaults for Create
            Must create and return a valid 'Extension' object.
 
@@ -453,6 +455,7 @@ def _get_container_insights_settings(cmd, cluster_resource_group_name, cluster_r
     subscription_id = get_subscription_id(cmd.cli_ctx)
     workspace_resource_id = ''
     useAADAuth = False
+    extensionSettings = {}
 
     if configuration_settings is not None:
         if 'loganalyticsworkspaceresourceid' in configuration_settings:
@@ -473,6 +476,26 @@ def _get_container_insights_settings(cmd, cluster_resource_group_name, cluster_r
             logger.info("provided useAADAuth flag is : %s", useAADAuthSetting)
             if (isinstance(useAADAuthSetting, str) and str(useAADAuthSetting).lower() == "true") or (isinstance(useAADAuthSetting, bool) and useAADAuthSetting):
                 useAADAuth = True
+        if useAADAuth and ('dataCollectionSettings' in configuration_settings):
+            dataCollectionSettingsString = configuration_settings["dataCollectionSettings"]
+            logger.info("provided dataCollectionSettings  is : %s", dataCollectionSettingsString)
+            dataCollectionSettings = json.loads(dataCollectionSettingsString)
+            if 'interval' in dataCollectionSettings.keys():
+                intervalValue = dataCollectionSettings["interval"]
+                if (bool(re.match(r'^[0-9]+[m]$', intervalValue))) is False:
+                    raise InvalidArgumentValueError('interval format must be in <number>m')
+                intervalValue = int(intervalValue.rstrip("m"))
+                if intervalValue <= 0 or intervalValue > 30:
+                    raise InvalidArgumentValueError('interval value MUST be in the range from 1m to 30m')
+            if 'namespaceFilteringMode' in dataCollectionSettings.keys():
+                namespaceFilteringModeValue = dataCollectionSettings["namespaceFilteringMode"].lower()
+                if namespaceFilteringModeValue not in ["off", "exclude", "include"]:
+                    raise InvalidArgumentValueError('namespaceFilteringMode value MUST be either Off or Exclude or Include')
+            if 'namespaces' in dataCollectionSettings.keys():
+                namspaces = dataCollectionSettings["namespaces"]
+                if isinstance(namspaces, list) is False:
+                    raise InvalidArgumentValueError('namespaces must be an array type')
+            extensionSettings["dataCollectionSettings"] = dataCollectionSettings
 
     workspace_resource_id = workspace_resource_id.strip()
 
@@ -502,7 +525,7 @@ def _get_container_insights_settings(cmd, cluster_resource_group_name, cluster_r
     if is_ci_extension_type:
         if useAADAuth:
             logger.info("creating data collection rule and association")
-            _ensure_container_insights_dcr_for_monitoring(cmd, subscription_id, cluster_resource_group_name, cluster_rp, cluster_type, cluster_name, workspace_resource_id)
+            _ensure_container_insights_dcr_for_monitoring(cmd, subscription_id, cluster_resource_group_name, cluster_rp, cluster_type, cluster_name, workspace_resource_id, extensionSettings)
         elif not _is_container_insights_solution_exists(cmd, workspace_resource_id):
             logger.info("Creating ContainerInsights solution resource, since it doesn't exist and it is using legacy authentication")
             _ensure_container_insights_for_monitoring(cmd, workspace_resource_id).result()
@@ -520,6 +543,7 @@ def _get_container_insights_settings(cmd, cluster_resource_group_name, cluster_r
 
     # workspace key not used in case of AAD MSI auth
     configuration_protected_settings['omsagent.secret.key'] = "<not_used>"
+    configuration_protected_settings['amalogs.secret.key'] = "<not_used>"
     if not useAADAuth:
         shared_keys = log_analytics_client.shared_keys.get_shared_keys(
             workspace_rg_name, workspace_name)
@@ -570,7 +594,7 @@ def get_existing_container_insights_extension_dcr_tags(cmd, dcr_url):
     return tags
 
 
-def _ensure_container_insights_dcr_for_monitoring(cmd, subscription_id, cluster_resource_group_name, cluster_rp, cluster_type, cluster_name, workspace_resource_id):
+def _ensure_container_insights_dcr_for_monitoring(cmd, subscription_id, cluster_resource_group_name, cluster_rp, cluster_type, cluster_name, workspace_resource_id, extensionSettings):
     from azure.core.exceptions import HttpResponseError
 
     cluster_region = ''
@@ -665,6 +689,7 @@ def _ensure_container_insights_dcr_for_monitoring(cmd, subscription_id, cluster_
                                 "Microsoft-ContainerInsights-Group-Default"
                             ],
                             "extensionName": "ContainerInsights",
+                            "extensionSettings": extensionSettings
                         }
                     ]
                 },