[Package mode] add new flag `--package-mode` #1154

msyyc · 2022-02-15T06:37:08Z

#1053

…into package-mode

autorest/codegen/__init__.py

autorest/codegen/serializers/__init__.py

autorest/codegen/templates/templates_package/CHANGELOG.md.jinja2

autorest/codegen/templates/templates_package/MANIFEST.in.jinja2

autorest/codegen/templates/templates_package/setup.py.jinja2

iscai-msft · 2022-02-15T17:13:10Z

overall looks much better, thank you @msyyc!

…into package-mode

autorest/codegen/templates/templates_package/README.md.jinja2

autorest/codegen/serializers/__init__.py

autorest/codegen/templates/templates_package/CHANGELOG.md.jinja2

autorest/codegen/templates/templates_package/setup.py.jinja2

docs/samples/specification/management/generated/setup.py

test/azure/legacy/Expected/AcceptanceTests/PackageModeDataPlane/CHANGELOG.md

test/azure/legacy/Expected/AcceptanceTests/PackageModeDataPlane/README.md

autorest/codegen/__init__.py

autorest/codegen/serializers/__init__.py

autorest/codegen/templates/dev_requirements.txt.jinja2

autorest/codegen/templates/setup.py.jinja2

autorest/codegen/templates/MANIFEST.in.jinja2

autorest/codegen/templates/dev_requirements.txt.jinja2

autorest/codegen/templates/setup.py.jinja2

...egacy/Expected/AcceptanceTests/PackageModeCustomize/azure/package/mode/aio/_configuration.py

test/azure/legacy/Expected/AcceptanceTests/PackageModeCustomize/template.txt

autorest/codegen/serializers/__init__.py

test/azure/legacy/Expected/AcceptanceTests/Paging/paging/operations/_paging_operations.py

test/vanilla/legacy/Expected/AcceptanceTests/PackageModeDataPlane/packagemode/_vendor.py

msyyc · 2022-02-28T07:38:34Z

@iscai-msft
(1) I think we could disable py27 CI check now that we have not supported it yet.
(2) azure-report.json is deleted in latest autorest.testserver: https://github.com/Azure/autorest.testserver/pull/358/files. We need a pr to fix it:

autorest.python/tasks.py

Line 76 in c9f5f14

'AzureReport': 'azure-report.json',

iscai-msft

def getting closer @lmazuel can you take a look, left some comments ccing you. Thanks!

iscai-msft · 2022-03-03T15:36:38Z

autorest/codegen/__init__.py

@@ -76,6 +77,12 @@ def _validate_code_model_options(options: Dict[str, Any]) -> None:
            "If you want operation files, pass in flag --show-operations"
        )

+    if options["package_mode"] not in ("mgmtplane", "dataplane", None) and not Path(options["package_mode"]).exists():


shouldn't it just be not in ("mgmtplane", "dataplane")? Not sure why None is a valid entry

Path(None).exist() will raise exception so we have to avoid that.

we don't really do None checks for any other flag, it's still a little weird because it looks like we're making None a valid entry (by including it with mgmtplane and dataplane)

I could check None first and it will make the logic more clear

iscai-msft · 2022-03-03T15:39:40Z

autorest/codegen/serializers/__init__.py

+            for template_name in package_files:
+                file = template_name.replace(".jinja2", "")
+                output_name = out_path / file
+                if not self._autorestapi.read_file(output_name) or file == 'setup.py':


yay looks good, just want to confirm with @lmazuel : are you good with us not overriding any of the files except setup.py during regeneration? This way if, for example, users update the README, we won't override when regenerating

We should override the MANIFEST as well (though it's unlikely it changes, this should not be changed manually by anyone)

Got it and will make a override-list. Now it contains 'setup.py' and 'MAINFEST.in'

autorest/codegen/templates/CHANGELOG.md.jinja2

iscai-msft · 2022-03-03T15:41:32Z

autorest/codegen/templates/README.md.jinja2

+# {{ package_pprint_name }} client library for Python
+<!-- write necessary description of service -->
+
+## _Disclaimer_


Can remove the disclaimer, we've already droppd support for 2.7

Yes, I think it's fair to remove it we're in March already

iscai-msft · 2022-03-03T15:41:50Z

autorest/codegen/templates/README.md.jinja2

+### Installating the package
+
+```bash
+python -m pip install {{ package_name }}


ideally just do pip install {{ package_name }}

That's debatable, we saw people confused because they had several Python installed, and python and pip were not referencing the same python interpreter. I would suggest to talk about venv instead, since this is a more efficient to ensure consistency between python and pip

docs/samples/specification/management/generated/setup.py

iscai-msft · 2022-03-03T15:52:28Z

...ected/AcceptanceTests/PackageModeCustomize/azure/packagemode/customize/models/_models_py3.py

+
+    def __init__(self, *, status: Optional[Union[str, "OperationResultStatus"]] = None, **kwargs):
+        """
+        :keyword status: The status of the request. Possible values include: "Succeeded", "Failed",


Can you also run at least one acceptance test on each package mode package? This way we can know if the package can be run. So for this package, you can run a paging test, and so on and so forth for the other two package mode packages

add test:
test_packagemode.py for dataplane,
test_package_mode_mgmt.py for mgmtplane,
test_package_mode_customize.py for customize

iscai-msft · 2022-03-03T15:54:22Z

test/azure/legacy/specification/packagemodecustomize/README.md

+
+```yaml $(package-mode)
+package-configuration:
+    min_python_version: 3.6


nit: it's a little weird that this test includes package_name as an entry in the custom template, since we already have flag --package-name, maybe we could switch the packge_name customization to a classifier customization or something

test/azure/legacy/specification/packagemodecustomize/template/setup.py

test/azure/legacy/specification/packagemodemgmtplane/README.md

autorest/codegen/templates/setup.py.jinja2

iscai-msft

ok, i think with the following comments, we should be good to merge!

iscai-msft · 2022-03-07T20:40:20Z

autorest/codegen/__init__.py

@@ -76,6 +77,12 @@ def _validate_code_model_options(options: Dict[str, Any]) -> None:
            "If you want operation files, pass in flag --show-operations"
        )

+    if options["package_mode"] not in ("mgmtplane", "dataplane", None) and not Path(options["package_mode"]).exists():


we don't really do None checks for any other flag, it's still a little weird because it looks like we're making None a valid entry (by including it with mgmtplane and dataplane)

iscai-msft · 2022-03-07T20:42:52Z

autorest/codegen/templates/setup.py.jinja2

+{% set author_email = "azpysdkhelp@microsoft.com" %}
+{% set url = "https://github.com/Azure/azure-sdk-for-python/tree/main/sdk" %}
+{% else %}
+version = "{{ code_model.options.get('package_version', '0.0.0') }}"


thanks @msyyc. with that check thought, we don't have to set a default value when we get package_version from options. I think you set default value here and at least one more place, so can you change those just to code_model.options["package_version"]?

autorest/codegen/templates/setup.py.jinja2

iscai-msft · 2022-03-07T22:48:59Z

test/azure/legacy/specification/packagemodecustomize/template/setup.py

@@ -12,7 +12,7 @@
 from setuptools import setup, find_packages


-PACKAGE_NAME = "{{ package_name }}"
+PACKAGE_NAME = "azure-packagemode-customize"


this file should be setup.py.jinja2 because it's formatted as a jinja template. Endings without .jinja2 means "keep as is" inside the custom package-mode folder, so we should be able to handle both

this file should be setup.py.jinja2 because it's formatted as a jinja template. Endings without .jinja2 means "keep as is" inside the custom package-mode folder, so we should be able to handle both

sure

some comments are same with up and they are resolved

autorest/codegen/__init__.py

iscai-msft · 2022-03-07T22:53:33Z

autorest/codegen/serializers/__init__.py

@@ -93,14 +98,17 @@ def _serialize_and_write_package_files_proc(**kwargs: Any):
            for template_name in package_files:
                file = template_name.replace(".jinja2", "")
                output_name = out_path / file
-                if not self._autorestapi.read_file(output_name) or file == 'setup.py':
+                if not self._autorestapi.read_file(output_name) or file in _REGENERATE_FILES:


just to confirm: if a user passes any template / file inside --package-mode=my-custom-folder/, that will override any of the templates we have in autorest?

If user passes folder, autorest will use those files instead of built-in template files. This logic is to permit to override specific files when rerun autorest.

…into package-mode

iscai-msft

yay looks great @msyyc!

Want to thank you again for dealing with all of the comments and your patience, going to go ahead and merge!

…into inherit_customization * 'autorestv3' of https://github.com/Azure/autorest.python: dpg service-driven-test (#1180) prepare for release (#1181) [Package mode] add new flag `--package-mode` (#1154) hide operation group docs (#1179)

…into mypy_code * 'autorestv3' of https://github.com/Azure/autorest.python: (26 commits) dpg service-driven-test (#1180) prepare for release (#1181) [Package mode] add new flag `--package-mode` (#1154) hide operation group docs (#1179) add coverage var (#1178) improve logging (#1177) don't reformat query params for dpg next link calls (#1168) Update package.json Update ChangeLog.md default optional const parameter to none (#1171) add storage to ci checks (#1176) fail on regeneration diff (#1173) Drop Python 2.7 (#1175) Add secrets test (#1174) bump testserver version (#1172) make content type keyword only if multiple (#1167) add default value info for params to docs (#1164) update release tests to hopefully get coverage report green (#1152) fix windows: clean the outputFolderUri (#1135) Update CODEOWNERS ...

msyyc and others added 5 commits February 10, 2022 16:14

init

c11542f

add test

1d5f306

add test for package-mode

a330b23

Merge branch 'autorestv3' of https://github.com/Azure/autorest.python …

7fb023d

…into package-mode

regenerate and pylint and mypy

e16267f

msyyc marked this pull request as ready for review February 15, 2022 06:59

msyyc requested a review from iscai-msft February 15, 2022 06:59

msyyc mentioned this pull request Feb 15, 2022

add --package-mode for new service to package #1112

Closed

iscai-msft reviewed Feb 15, 2022

View reviewed changes

msyyc added 5 commits February 16, 2022 13:41

review

27df09c

review

ca60360

Merge branch 'autorestv3' of https://github.com/Azure/autorest.python …

0a294f7

…into package-mode

review

eb42ea4

regenerate

c5c8203

iscai-msft reviewed Feb 16, 2022

View reviewed changes

msyyc and others added 3 commits February 17, 2022 13:02

review and regenerate

f4a6134

pylint

575071f

Merge branch 'autorestv3' into package-mode

47e131b

iscai-msft reviewed Feb 18, 2022

View reviewed changes

msyyc added 2 commits February 21, 2022 15:15

review and regenerate

653c253

install package for test

e54afe0

iscai-msft reviewed Feb 26, 2022

View reviewed changes

msyyc added 2 commits February 28, 2022 15:04

review

c302e25

merge and regenerate

52ac6dc

msyyc and others added 4 commits February 28, 2022 17:06

regenerate

15349fa

fix test

460d2a2

regenerate

3a9ada9

Merge branch 'autorestv3' into package-mode

8dc3cfb

review and regenerate

ba4ace6

msyyc mentioned this pull request Mar 2, 2022

Drop Python 2.7 #1175

Merged

iscai-msft and others added 2 commits March 2, 2022 14:10

Merge branch 'autorestv3' into package-mode

ae59026

Merge branch 'autorestv3' into package-mode

158fdc4

msyyc requested a review from changlong-liu as a code owner March 3, 2022 03:02

regenerate

482c0b8

iscai-msft reviewed Mar 3, 2022

View reviewed changes

lmazuel reviewed Mar 3, 2022

View reviewed changes

autorest/codegen/templates/setup.py.jinja2 Outdated Show resolved Hide resolved

lmazuel reviewed Mar 3, 2022

View reviewed changes

autorest/codegen/templates/setup.py.jinja2 Outdated Show resolved Hide resolved

msyyc added 2 commits March 7, 2022 14:40

review

8c426f0

regenerate

e20e50e

iscai-msft reviewed Mar 7, 2022

View reviewed changes

msyyc added 5 commits March 8, 2022 11:29

review

7f45e93

Merge branch 'autorestv3' of https://github.com/Azure/autorest.python …

25d5cb5

…into package-mode

changelog

4705474

review

a14b58c

regenerate

b54e494

iscai-msft approved these changes Mar 8, 2022

View reviewed changes

iscai-msft merged commit c151a1e into Azure:autorestv3 Mar 8, 2022

msyyc deleted the package-mode branch March 9, 2022 00:07

msyyc mentioned this pull request Mar 9, 2022

[pipeline] update configuration file for all mgmt services Azure/azure-sdk-for-python#23423

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Package mode] add new flag `--package-mode` #1154

[Package mode] add new flag `--package-mode` #1154

msyyc commented Feb 15, 2022 •

edited

Loading

iscai-msft commented Feb 15, 2022

msyyc commented Feb 28, 2022 •

edited

Loading

iscai-msft left a comment

iscai-msft Mar 3, 2022

msyyc Mar 4, 2022

iscai-msft Mar 7, 2022

msyyc Mar 8, 2022

iscai-msft Mar 3, 2022

lmazuel Mar 3, 2022

msyyc Mar 4, 2022

iscai-msft Mar 3, 2022

lmazuel Mar 3, 2022

msyyc Mar 4, 2022 •

edited

Loading

iscai-msft Mar 3, 2022

lmazuel Mar 3, 2022

iscai-msft Mar 3, 2022

msyyc Mar 7, 2022

iscai-msft Mar 3, 2022

msyyc Mar 7, 2022

iscai-msft left a comment

iscai-msft Mar 7, 2022

iscai-msft Mar 7, 2022

iscai-msft Mar 7, 2022

msyyc Mar 8, 2022

msyyc Mar 8, 2022

iscai-msft Mar 7, 2022

msyyc Mar 8, 2022

iscai-msft left a comment

[Package mode] add new flag --package-mode #1154

[Package mode] add new flag --package-mode #1154

Conversation

msyyc commented Feb 15, 2022 • edited Loading

iscai-msft commented Feb 15, 2022

msyyc commented Feb 28, 2022 • edited Loading

iscai-msft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msyyc Mar 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iscai-msft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iscai-msft left a comment

Choose a reason for hiding this comment

[Package mode] add new flag `--package-mode` #1154

[Package mode] add new flag `--package-mode` #1154

msyyc commented Feb 15, 2022 •

edited

Loading

msyyc commented Feb 28, 2022 •

edited

Loading

msyyc Mar 4, 2022 •

edited

Loading