Add preview Log Analytics query extension #250

alexeldeib · 2018-07-26T20:31:19Z

This checklist is used to make sure that common guidelines for a pull request are followed.

General Guidelines

Have you run ./scripts/ci/test_static.sh locally? (pip install pylint flake8 required)
Have you run python scripts/ci/test_index.py -q locally?

For new extensions:

My extension description/summary conforms to the Extension Summary Guidelines.

azuresdkci · 2018-07-26T20:31:34Z

If this PR is for a new extension or change to an existing extension, use the following to try out the changes in this PR:

docker run -it microsoft/azure-cli:latest
export EXT=<NAME>
pip install --upgrade --target ~/.azure/cliextensions/$EXT "git+https://github.com/alexeldeib/azure-cli-extensions.git@loganalytics#subdirectory=src/$EXT&egg=$EXT"

alexeldeib · 2018-07-26T20:59:36Z

Hi -- Per offline discussion with @mayurid @williexu, reintroducing Log Analytics query functionality as an extension.

I had some issues running/recording the tests after migrating from module-style to extension-style. Specifically I can't import azure.cli.testsdk package. I assumed this was due to the namespacing issue referenced in the faq, but I see many other packages using this in their tests? Do I need to set up my environment somehow differently to make this work properly?

williexu · 2018-07-26T22:22:33Z

@alexeldeib You should be able to import from azure.cli.testsdk. Many of the other extensions do the same. Are you following these instructions to test your extension?

Also, can you introduce the extension commands under the monitor command-group (i.e. 'az monitor log-analytics query')? and address CI

alexeldeib · 2018-07-26T23:02:05Z

Yes, I'm using separate virtualenvs to test. Is there possibly a related package I have installed globally that could be messing with the namespacing? For good measure I ran pytest and the same issues occurs with all the extensions, so it's definitely something in my env..

I'll fix the other issues.

williexu

Looks good overall, rebase to get updates from master, address comments and add yourself as codeowner :)

williexu · 2018-07-27T00:18:45Z

src/index.json

@@ -951,6 +951,52 @@
                    "version": "0.9.0"
                }
            }
+        ],
+        "loganalytics": [


Given the command will be az monitor log-analytics(with the dash), you may want to rename to log-analytics for more clarity.

I'm hosting on PyPi and we already have the azure-loganalytics python sdk there, so I was going to follow the azure batch style naming and go with azure-loganalytics-cli-extension. I see that test_index.py validates the package name against the .dist-info metadata.json, so would that (azure-loganalytics-cli-extension) be fine here?

Please leave the name as log-analytics as this is the name users will see when they run the az extension list-available command. This being an extension for azure-cli is implied implicitly.

Batch and others were added before these naming conventions were made. The hosting location does not matter for this.

williexu · 2018-07-27T00:19:30Z

src/index.json

+                    "metadata_version": "2.0",
+                    "name": "loganalytics",
+                    "summary": "Support for Azure Log Analytics query functionality.",
+                    "version": "1.0.0-Preview-1"


Preview version will start with 0.1.0, please change the version.

williexu · 2018-07-27T00:19:43Z

src/loganalytics/HISTORY.rst

+1.0.0-Preview-1
++++++++++++++++++
+
+* Initial release.


start with version 0.1.0

edit: replied to stale comment

williexu · 2018-07-27T00:22:05Z

src/index.json

+                              "description": "DESCRIPTION.rst"
+                            },
+                            "project_urls": {
+                                "Home": "https://github.com/Azure/azure-cli-extensions"


You may want to point to "https://github.com/Azure/azure-cli-extensions/tree/master/src/log-analytics" so that people will be directed straight towards your readme.

williexu · 2018-07-27T00:25:53Z

src/loganalytics/azext_loganalytics/_help.py

+
+# pylint: disable=line-too-long
+
+helps['loganalytics query'] = """


change to reflect the command

williexu · 2018-07-27T00:26:48Z

src/loganalytics/azext_loganalytics/commands.py

+        client_factory=loganalytics_data_plane_client
+    )
+
+    with self.command_group('monitor loganalytics', loganalytics_sdk) as g:


Command must be monitor log-analytics for consistency with CLI commands.

williexu · 2018-07-27T00:33:39Z

src/loganalytics/azext_loganalytics/commands.py

+    )
+
+    with self.command_group('monitor loganalytics', loganalytics_sdk) as g:
+        g.command('query', 'execute_query')


Use g.custom_command() to point directly at the custom namespace defined in your loader, no need to reference the same namespace again and will open up the possibility for specifying other sdks if needed.

williexu · 2018-07-27T00:36:33Z

src/loganalytics/setup.py

+from codecs import open
+from setuptools import setup, find_packages
+
+VERSION = "1.0.0-Preview-2"


alexeldeib · 2018-07-27T21:00:13Z

Fixed up my python environment and most of the remaining issues, I'll keep an eye on CI. Let me know if you see anything else that sticks out.

williexu · 2018-07-27T21:04:26Z

src/loganalytics/azext_loganalytics/_help.py

+
+# pylint: disable=line-too-long
+
+helps['monitor loganalytics query'] = """


log-analytics

williexu · 2018-07-27T21:05:39Z

src/loganalytics/azext_loganalytics/_params.py

+
+
+def load_arguments(self, _):
+    with self.argument_context('loganalytics') as c:


monitor log-analytics query

williexu · 2018-07-27T21:07:27Z

src/loganalytics/azext_loganalytics/_params.py

+def load_arguments(self, _):
+    with self.argument_context('loganalytics') as c:
+        c.argument('workspace', options_list=['--workspace', '-w'], help='GUID of the Log Analytics Workspace', required=True)
+        c.argument('kql', help='Query to execute over Log Analytics data.', required=True)


you don't need the required=True for workspace or kql; they will be implied from the command signature in execute_query

williexu · 2018-07-27T21:08:51Z

src/loganalytics/azext_loganalytics/tests/latest/test_loganalytics_commands.py

+            self.check('tables[0].rows[0][0]', 'TenantId')
+        ])
+        query_result = self.cmd('az loganalytics query -w cab864ad-d0c1-496b-bc5e-4418315621bf --kql "Heartbeat | getschema"').get_output_in_json()
+        assert len(query_result['tables'][0]['rows']) == 29


Test looks stale. Can you run your test again? that should also pick up some of the other problems i saw.

williexu · 2018-07-27T21:10:05Z

src/loganalytics/setup.py

+    license='MIT',
+    author='Ace Eldeib',
+    author_email='aleldeib@microsoft.com',
+    url='https://github.com/Azure/azure-cli-extensions/tree/master/src/log-analytics',


'https://github.com/Azure/azure-cli-extensions/tree/master/src/loganalytics' to match your directory name.

Why don't I just change the directory to match? Since we're adding the dash everywhere, may as well be consistent.

williexu

One last thing, remove the indicated files leftover from testing

williexu · 2018-07-27T22:43:42Z

src/log-analytics/.pytest_cache/v/cache/nodeids

@@ -0,0 +1,3 @@
+[
+  "azext_loganalytics/tests/latest/test_loganalytics_commands.py::LogAnalyticsDataClientTests::test_query"
+]


remove this

williexu · 2018-07-27T22:43:53Z

src/log-analytics/.pytest_cache/v/cache/lastfailed

@@ -0,0 +1 @@
+{}


remove this

alexeldeib · 2018-07-27T22:56:36Z

Thanks for all the help cleaning this up! One minor thing, I want to see if I can come up with a better name for --kql parameter. Normally we use "query" but it's used globally 😄 How about --analytics-query?

williexu · 2018-07-27T23:00:18Z

@alexeldeib --analytics-query sounds good to me. Do you want to include that with this PR?

williexu · 2018-07-27T23:41:30Z

@alexeldeib for futures PRs, I'd recommend resolving all issues with the extension before updating the index. That will save you from having to build a new wheel file, upload to pypi and update the index/history/version multiple times.

…zure#251) This reverts commit 584815d.

* Kubernetes Data Protection Extension CLI (#173) * First draft for Data Protection K8s backup extension (Pending internal review) * Removing tracing * Minor changes to improve azdev style * Internal PR review feedback Co-authored-by: Rishabh Raj <rishraj@microsoft.com> * {AKS - ARC} fix: Update DCR creation to Clusters resource group instead of workspace (#175) * fix: Update DCR creation to Clusters resource group instead of workspace * . * . * casing check * Add self-signed cert to fix PR gate for azureml extension * adding the api version to the operation definition in the client factory * bump k8s-extension version to 1.3.6 * adding tests for all 4 extension types calls * adding to test config file * updating the api version for extension types to be the correct version expected by the service * add test case for flux extension (#184) * bump k8s-extension version to 1.3.6 * bump k8s-extension version to 1.3.6 * adding upstream test for extension types * updating history.rst * [Dapr] Prompt user for existing Dapr installation during extension create (#188) * Add more validations and user prompt for existing installation scenario Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Add Dapr test' Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Handle stateful set Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Update default handling Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Fix HA handling Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Add placement service todo Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Add non-interactive mode Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Fix lint Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Update tests Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Reset configuration for StatefulSet during k8s upgrade Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Fix lint Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Retrigger tests Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Add changes to manage ha and placement params Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Update message Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * nits Signed-off-by: Shubham Sharma <shubhash@microsoft.com> Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * bump k8s-extension version to 1.3.7 * [Dapr] Disable applying CRDs during a downgrade (#193) * Add logging Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Lint Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Update log Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Revert applyCrds when not downgrading Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Update logic for removing hooks.applyCrds Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Revert logic Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Handle explicit hooks configuration Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Update comment Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * re-trigger pipeline Signed-off-by: Shubham Sharma <shubhash@microsoft.com> Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * ContainerInsights extension - Add dataCollectionSettings configuration settings (#200) * data collection settings * add support for dataCollectionSettings * fix indention * avoid duplicate use of json loads * remove whitespaces * fix pr feedback * Upgrade Python version from 3.6 to 3.7 (#203) * Upgrade Python version from 3.6 to 3.10 Upgrade to 3.10 for the job that runs Wheel, PyLint, Flake, etc., since 3.6 is not supported anymore by hosted-agent-software. * Upgrade to Python 3.10 from 3.6 Upgrade to 3.10 as 3.6 is not supported * Switch PyLink to 1.9.4 Switch PyLink to 1.9.4 from 1.9.5, as 1.9.5 is not supported with Python 3.10 * Use Python 3.7 for Static Analysis Use 3.7, as 3.10 does not support certain properties used by astpeephole.py that is used by Static Analysis tools * Try unpinned version of PyLint PyLint 1.9.5 doesn't work with Python 3.7. So, trying to see if it automatically pulls the latest compatible version. * Run pylint as a separate command * Update pylintrc (#204) * Update pylintrc * Update k8s-custom-pipelines.yml * Disable PyLint (#205) Disable PyLint for now, as the new version has breaking changes and requires lot more fixes * Disable PyLint on CI scripts * Fixes for script errors * Upgrade Static Analysis Python version Upgrade the Python version for Static Analysis to 3.10, from 3.7, now that PyLint is disabled * Try 3.9, as 3.10 has breaking changes for Flake8 * Remove version pinning for flake8 Try Python 3.10, without pinning flake8 to a version * Update k8s-custom-pipelines.yml * Use Python 3.8.1 & flake8 6.0.0 * Use Python 3.8 instead of 3.8.1 * Update k8s-custom-pipelines.yml * Update .flake8 Update to reflect breaking change in flake8 6.0 * Update source_code_static_analysis.py Scope static analysis tools to only k8s-extension module's source in our branch. * Update k8s-custom-pipelines.yml * Update k8s-custom-pipelines.yml * Update k8s-custom-pipelines.yml * Update pool name in StaticAnalysis To mirror what is in main of azure-cli-extensions * Update k8s-custom-pipelines.yml * Fix indentation * Update k8s-custom-pipelines.yml * Update k8s-custom-pipelines.yml * Revert changes * Revert changes * Revert changes to source_code_static_analysis.py * Update source_code_static_analysis.py * Revert changes * Use Ubuntu 20.4 for BuiltTestPublish stage * Switch to ubuntu-20.04 from latest Co-authored-by: Rishik Hombal <hombalrishik@gmail.com> * [Dapr] Do not apply CRD hook when version is unchanged or auto-upgrade is being disabled (#201) * Update logic Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * re-trigger pipeline Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * re-trigger pipeline Signed-off-by: Shubham Sharma <shubhash@microsoft.com> Signed-off-by: Shubham Sharma <shubhash@microsoft.com> Co-authored-by: NarayanThiru <nanthi@microsoft.com> * add dummy key for amalogs as well * bump k8s-extension version to 1.3.8 * Adding GA api version 2022-11-01 exposing isSystemExtension and support for plan info * Seperate args for plan name, product and publisher * updating cassete file * updating HISTORY.rst * Deprecate longer parameter names when accepting config settings (#213) Co-authored-by: deeksha345 <34255011+deeksha345@users.noreply.github.com> * Release 1.3.9 * make containerinsights dcr name consistent (#211) Co-authored-by: Bavneet Singh <33008256+bavneetsingh16@users.noreply.github.com> * [Dapr] Update version comparison logic to use semver based comparison (#219) * Update semver comparison Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Add log Signed-off-by: Shubham Sharma <shubhash@microsoft.com> --------- Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * bump k8s-extension version to 1.4.0 (#220) * Revert "bump k8s-extension version to 1.4.0 (#220)" (#222) This reverts commit ffb8a95. * [k8s-extension] Update extension CLI to v1.4.0 * update release history * fix openservice mesh cli testcase issue * Zetia/fix ssl secret flag (#224) * fix bug: update operation doesn't respect sslSecret parameter * fix bug: update operation doesn't respect sslSecret parameter * fix typo * feat: public preview support for microsoft.azuremonitor.containers.metrics in ARC clusters (managed prometheus) (#227) * remove redundant extension test (#230) * ci MSI default for arc cluster (#231) * bump k8s-extension version to 1.4.2 * ContainerInsights extension - Extend dataCollectionSettings config settings with streams field (#232) * extend containerinsights datacollection settings with streams field * bug fix * fix lint issues * fix pr feedback * fix pr feedback * fix lint error * Generated files for 2023-05-01-preview * Support for 2023-05-01-preview * Rename get to show * Added ExtensionType api test cases * ContainerInsights extension - Extend dataCollectionSettings with containerlogv2 (#237) * Fix for Liniting issues * Fixing test cases * comment failing test cases * [k8s-extension] add kind tag in DCR creation (#240) * Use semver package (#241) Signed-off-by: Shubham Sharma <shubhash@microsoft.com> * Reverting commented test cases * Add support to skip provisioning of prerequisites for Azure Monitor K8s extensions (#234) * {ARC} fix: update logic to sanitize cluster name for dc* objects (#242) * Fix osm-arc version check for CI tags (#244) Signed-off-by: nshankar <nshankar@microsoft.ghe.com> Co-authored-by: nshankar <nshankar@microsoft.ghe.com> * New cassette file * Remove unused propeties from table format * bump k8s-extension version 1.4.3 * Add old commands back with deprecated status * Fix linting issues * Reverting changes for extensions type api * change the location for test runs and arc clusters * [k8s-extension] create new cli release - v1.4.3 (#250) * Revert "[k8s-extension] create new cli release - v1.4.3 (#250)" (#251) This reverts commit 584815d. * [k8s-extension] Update extension CLI to v1.4.3 * Drop relay sdk (#254) * update readme * remove useless snippets (#256) * [k8s-extension] Update extension CLI to v1.4.4 --------- Signed-off-by: Shubham Sharma <shubhash@microsoft.com> Signed-off-by: nshankar <nshankar@microsoft.ghe.com> Co-authored-by: Rishabh Raj <rishabhstpaul@gmail.com> Co-authored-by: Rishabh Raj <rishraj@microsoft.com> Co-authored-by: bragi92 <kadubey@microsoft.com> Co-authored-by: Yue Yu <yuyu3@microsoft.com> Co-authored-by: Deeksha Sharma <deesharma@microsoft.com> Co-authored-by: deeksha345 <34255011+deeksha345@users.noreply.github.com> Co-authored-by: Shubham Sharma <shubhash@microsoft.com> Co-authored-by: Bavneet Singh <bavneetsingh@microsoft.com> Co-authored-by: Ganga Mahesh Siddem <gangams@microsoft.com> Co-authored-by: NarayanThiru <nanthi@microsoft.com> Co-authored-by: Rishik Hombal <hombalrishik@gmail.com> Co-authored-by: Amol Agrawal <amagraw@microsoft.com> Co-authored-by: Amol Agrawal <pfrcks@gmail.com> Co-authored-by: Arif Lakhani <ariflakhani@microsoft.com> Co-authored-by: Arif-lakhani <ariflakhani7786@gmail.com> Co-authored-by: Zeliang Tian <83852443+zetiaatgithub@users.noreply.github.com> Co-authored-by: Long Wan <wanlonghenry@gmail.com> Co-authored-by: ms-hujia <48512395+ms-hujia@users.noreply.github.com> Co-authored-by: Niranjan Shankar <nshankar@microsoft.com> Co-authored-by: nshankar <nshankar@microsoft.ghe.com> Co-authored-by: necusjz <necusjz@gmail.com>

alexeldeib requested review from tjprescott and williexu as code owners July 26, 2018 20:31

alexeldeib changed the title ~~Add preview Log Analytics extension~~ Add preview Log Analytics query extension Jul 26, 2018

williexu suggested changes Jul 27, 2018

View reviewed changes

alexeldeib added 7 commits July 26, 2018 18:25

Initial extension commit

72d3286

Add initial readme, history

005bccc

vendor sdk, change imports, update text, and update index

05a84d4

Fix CI, update version from regen with wheel==0.30.0

073f0fd

add codeowner

c02dfca

fix versions, ci, wheel issues

8f40bb4

rename loganalytics -> log-analytics

46eea30

alexeldeib force-pushed the loganalytics branch from 46345d2 to 46eea30 Compare July 27, 2018 01:25

alexeldeib added 5 commits July 26, 2018 18:29

switch command -> custom command

9417f43

unused imports

6844dd0

fix after python environment mess

edcb154

naming

7fa4cbe

fix homepage and bump minor version

7a1454a

fix src tests

531d1d4

williexu suggested changes Jul 27, 2018

View reviewed changes

alexeldeib added 6 commits July 27, 2018 14:11

loganalytics -> log-analytics

37d1574

rename directory

5c9dbdc

params

6d0288f

accidental deletion

5b826d7

codeowners

ba14a67

fix CI test namespacing issue(?)

87db7df

williexu suggested changes Jul 27, 2018

View reviewed changes

alexeldeib added 2 commits July 27, 2018 15:53

Delete lastfailed

fd5356b

Delete nodeids

345c3db

williexu approved these changes Jul 27, 2018

View reviewed changes

alexeldeib added 2 commits July 27, 2018 16:08

update parameter name

53fddc9

change --kql to --analytics-query, bump version/index/history

9dd23c4

williexu merged commit 89518d0 into Azure:master Jul 27, 2018

necusjz pushed a commit to necusjz/azure-cli-extensions that referenced this pull request Sep 1, 2023

[k8s-extension] create new cli release - v1.4.3 (Azure#250)

584815d

necusjz pushed a commit to necusjz/azure-cli-extensions that referenced this pull request Sep 1, 2023

Revert "[k8s-extension] create new cli release - v1.4.3 (Azure#250)" (A…

aa77ff2

…zure#251) This reverts commit 584815d.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add preview Log Analytics query extension #250

Add preview Log Analytics query extension #250

alexeldeib commented Jul 26, 2018 •

edited

Loading

azuresdkci commented Jul 26, 2018

alexeldeib commented Jul 26, 2018

williexu commented Jul 26, 2018 •

edited

Loading

alexeldeib commented Jul 26, 2018

williexu left a comment •

edited

Loading

williexu Jul 27, 2018

alexeldeib Jul 27, 2018

williexu Jul 27, 2018

alexeldeib Jul 27, 2018

williexu Jul 27, 2018

williexu Jul 27, 2018

alexeldeib Jul 27, 2018

williexu Jul 27, 2018

williexu Jul 27, 2018

williexu Jul 27, 2018

williexu Jul 27, 2018

williexu Jul 27, 2018

alexeldeib commented Jul 27, 2018

williexu Jul 27, 2018

williexu Jul 27, 2018

williexu Jul 27, 2018

williexu Jul 27, 2018

williexu Jul 27, 2018

alexeldeib Jul 27, 2018

williexu Jul 27, 2018

williexu left a comment •

edited

Loading

williexu Jul 27, 2018

williexu Jul 27, 2018

alexeldeib commented Jul 27, 2018 •

edited

Loading

williexu commented Jul 27, 2018 •

edited

Loading

williexu commented Jul 27, 2018 •

edited

Loading


		# pylint: disable=line-too-long

		helps['loganalytics query'] = """


		# pylint: disable=line-too-long

		helps['monitor loganalytics query'] = """



		def load_arguments(self, _):
		with self.argument_context('loganalytics') as c:

Add preview Log Analytics query extension #250

Add preview Log Analytics query extension #250

Conversation

alexeldeib commented Jul 26, 2018 • edited Loading

General Guidelines

azuresdkci commented Jul 26, 2018

alexeldeib commented Jul 26, 2018

williexu commented Jul 26, 2018 • edited Loading

alexeldeib commented Jul 26, 2018

williexu left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexeldeib commented Jul 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

williexu left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexeldeib commented Jul 27, 2018 • edited Loading

williexu commented Jul 27, 2018 • edited Loading

williexu commented Jul 27, 2018 • edited Loading

alexeldeib commented Jul 26, 2018 •

edited

Loading

williexu commented Jul 26, 2018 •

edited

Loading

williexu left a comment •

edited

Loading

williexu left a comment •

edited

Loading

alexeldeib commented Jul 27, 2018 •

edited

Loading

williexu commented Jul 27, 2018 •

edited

Loading

williexu commented Jul 27, 2018 •

edited

Loading