add snippet update step (#26899)

* add snippet update step

* Get latest main (#26934)

* Change default local storage location (#26891)

* code and test (#26908)

Co-authored-by: PythonSdkPipelines <PythonSdkPipelines>

* Update swagger_to_sdk_config_dpg.json (#26913)

* feat: enable internal components in pipeline yaml (#26800)

* feat: enable internal components in pipeline yaml

* refactor: add some gate logic

* fix: fix test_pipeline_job_create_with_registries

* fix: fix pylint

* Cadl automation pipeline (#26912)

* cdoe

* fix

* code

* fix

* code

* meta.json

* meta.json

* compatible

* refactor: divide pipeline related tests (#26886)

* fix: sdk.ml.azure-ai-ml.tests.datastore.e2etests.test_datastore

* feat: component with default label

* refactor: divide test_dsl_pipeline.py

* feat: load labelled arm id

* feat: support name@default
use name@label for node component in pipeline yaml

* [TA ] improve per-document error message when all documents fail per action (#26902)

* fix bad Document error message when all actions fail

* update changelog

* update wording

* skip cosmos emulator (#26930)

* try 3.7

* skip cosmos emulator tests until fixed

* revert change

* comment out emulator

* fix ci for core (#26923)

* fix ci for core

* updates

* updates

* updates

* updates

* updates

* adding additional version override support

* update

* update

* update

* update conda test-requirements to include azure-mgmt-storage that was previously coming dependency tools/azure-sdk-tools

Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com>

Co-authored-by: Leighton Chen <lechen@microsoft.com>
Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com>
Co-authored-by: Yuchao Yan <yuchaoyan@microsoft.com>
Co-authored-by: Xingzhi Zhang <37076709+elliotzh@users.noreply.github.com>
Co-authored-by: Krista Pratico <krpratic@microsoft.com>
Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com>

* update

* update

* update

* update

* update

* Update update_snippet.yml

* Update eng/pipelines/templates/steps/analyze.yml

Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>

* Update update_snippet.yml

* update

* update

* update

* updates

* update

* Updates

* updates

* updates

* update

* Add comments

Co-authored-by: Leighton Chen <lechen@microsoft.com>
Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com>
Co-authored-by: Yuchao Yan <yuchaoyan@microsoft.com>
Co-authored-by: Xingzhi Zhang <37076709+elliotzh@users.noreply.github.com>
Co-authored-by: Krista Pratico <krpratic@microsoft.com>
Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>

eng/pipelines/templates/steps/analyze.yml

@@ -115,6 +115,11 @@ steps:
       TestMarkArgument: ${{ parameters.TestMarkArgument }}
       AdditionalTestArgs: ${{parameters.AdditionalTestArgs}}
 
+  - template: /eng/pipeines/templates/steps/update_snippet.yml
+    parameters:
+      ScanPath: $(Build.SourcesDirectory)/sdk/${{ parameters.ServiceDirectory }}
+      AdditionalTestArgs: ${{parameters.AdditionalTestArgs}}
+
   - template: ../steps/run_breaking_changes.yml
     parameters:
       ServiceDirectory: ${{ parameters.ServiceDirectory }}

eng/pipelines/templates/steps/update_snippet.yml

@@ -0,0 +1,19 @@
+parameters:
+  ServiceDirectory: ''
+  ValidateFormatting: false
+  EnvVars: {}
+
+steps:
+  - task: UsePythonVersion@0
+    displayName: 'Use Python 3.9'
+    inputs:
+     versionSpec: '3.9'
+    condition: succeededOrFailed()
+
+  - task: PythonScript@0
+    displayName: 'Update Snippets'
+    inputs:
+      scriptPath: 'tools/azure-sdk-tools/ci_tools/snippet_update/python_snippet_updater.py'
+      arguments: >-
+        ${{ parameters.ScanPath }}
+    condition: and(succeededOrFailed(), ne(variables['Skip.UpdateSnippet'],'true'))

tools/azure-sdk-tools/ci_tools/snippet_update/python_snippet_updater.py

@@ -0,0 +1,125 @@
+import sys
+import logging
+from pathlib import Path
+import argparse
+import re
+from typing import Dict
+
+_LOGGER = logging.getLogger(__name__)
+
+snippets = {}
+not_up_to_date = False
+
+target_snippet_sources = ["samples/*.py", "samples/*/*.py"]
+target_md_files = ["README.md"]
+
+def check_snippets() -> Dict:
+    return snippets
+
+def check_not_up_to_date() -> bool:
+    return not_up_to_date
+
+def get_snippet(file: str) -> None:
+    file_obj = Path(file)
+    with open(file_obj, 'r') as f:
+        content = f.read()
+    pattern = "# \\[START(?P<name>[A-Z a-z0-9_]+)\\](?P<body>[\\s\\S]+?)# \\[END[A-Z a-z0-9_]+\\]"
+    matches = re.findall(pattern, content)
+    for match in matches:
+        s = match
+        name = s[0].strip()
+        snippet = s[1]
+        # Remove extra spaces
+        # A sample code snippet could be like:
+        # \n
+        #         # [START trio]
+        #         from azure.core.pipeline.transport import TrioRequestsTransport
+
+        #         async with AsyncPipeline(TrioRequestsTransport(), policies=policies) as pipeline:
+        #             return await pipeline.run(request)
+        #         # [END trio]
+        # \n
+        # On one hand, the spaces in the beginning of the line may vary. e.g. If the snippet 
+        # is in a class, it may have more spaces than if it is not in a class.
+        # On the other hand, we cannot remove all spaces because indents are part of Python syntax.
+        # Here is our algorithm:
+        # We firstly count the spaces of the # [START snippet] line.
+        # And for every line, we remove this amount of spaces in the beginning of the line.
+        # To only remove the spaces in the beginning and to make sure we only remove it once per line,
+        # We use replace('\n' + spaces, '\n').
+        spaces = ""
+        for char in snippet[1:]:
+            if char == " ":
+                spaces += char
+            else:
+                break
+        snippet = snippet.replace("\n" + spaces, "\n")
+        # Remove first newline
+        snippet = snippet[1:].rstrip()
+        if snippet[-1] == "\n":
+            snippet = snippet[:-1]
+
+        file_name = str(file_obj.name)[:-3]
+        identifier = ".".join([file_name, name])
+        if identifier in snippets.keys():
+            _LOGGER.warning(f'Found duplicated snippet name "{identifier}".')
+            _LOGGER.warning(file)
+        _LOGGER.debug(f"Found: {file_obj.name}.{name}")
+        snippets[identifier] = snippet
+
+
+def update_snippet(file: str) -> None:
+    file_obj = Path(file)
+    with open(file_obj, 'r') as f:
+        content = f.read()
+    pattern = "(?P<content>(?P<header><!-- SNIPPET:(?P<name>[A-Z a-z0-9_.]+)-->)\\n```python\\n[\\s\\S]*?\\n<!-- END SNIPPET -->)"
+    matches = re.findall(pattern, content)
+    for match in matches:
+        s = match
+        body = s[0].strip()
+        header = s[1].strip()
+        name = s[2].strip()
+        _LOGGER.debug(f"Found name: {name}")
+        if name not in snippets.keys():
+            _LOGGER.error(f'In {file}, failed to found snippet name "{name}".')
+            exit(1)
+        target_code = "".join([header, "\n```python\n", snippets[name], "\n```\n", "<!-- END SNIPPET -->"])
+        if body != target_code:
+            _LOGGER.warning(f'Snippet "{name}" is not up to date.')
+            global not_up_to_date
+            not_up_to_date = True
+            content = content.replace(body, target_code)
+    with open(file_obj, 'w') as f:
+        f.write(content)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "path",
+        nargs="?",
+        help=(
+            "The targeted path for update."
+        ),
+    )
+    args = parser.parse_args()
+    path = sys.argv[1]
+    _LOGGER.info(f"Path: {path}")
+    for source in target_snippet_sources:
+        for py_file in Path(path).rglob(source):
+            try:
+                get_snippet(py_file)
+            except UnicodeDecodeError:
+                pass
+    for key in snippets.keys():
+        _LOGGER.debug(f"Found snippet: {key}")
+    for target in target_md_files:
+        for md_file in Path(path).rglob(target):
+            try:
+                update_snippet(md_file)
+            except UnicodeDecodeError:
+                pass
+    if not_up_to_date:
+        _LOGGER.error(f'Error: code snippets are out of sync. Please run Python PythonSnippetUpdater.py "{path}" to fix it.')
+        exit(1)
+    _LOGGER.info(f"README.md under {path} is up to date.")