[PR] - Add JSON dataset support in ADF #6739

shawnxzq · 2019-07-26T08:41:19Z

Latest improvements:

MSFT employees can try out our new experience at OpenAPI Hub - one location for using our validation tools and finding your workflow.

Contribution checklist:

I have reviewed the documentation for the workflow.
Validation tools were run on swagger spec(s) and have all been fixed in this PR.
The OpenAPI Hub was used for checking validation status and next steps.

ARM API Review Checklist

Service team MUST add the "WaitForARMFeedback" label if the management plane API changes fall into one of the below categories.
adding/removing APIs.
adding/removing properties.
adding/removing API-version.
adding a new service in Azure.

Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs.

If you are blocked on ARM review and want to get the PR merged urgently, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.
Please follow the link to find more details on API review process.

openapi-sdkautomation · 2019-07-26T08:41:21Z

SDK Automation [Logs] (Generated from d49776e, Iteration 15)

Python: Azure/azure-sdk-for-python [Logs] [Diff]

azure-mgmt-datafactory [Logs] [Instructions] [Generation PR]
- azure-mgmt-datafactory-0.7.0.zip
- azure_mgmt_datafactory-0.7.0-py2.py3-none-any.whl

Java: Azure/azure-sdk-for-java [Logs] [Diff]

datafactoryv2/resource-manager/v2017_09_01_preview [Logs]
datafactoryv2/resource-manager/v2018_06_01 [Logs]

Go: test-repo-billy/azure-sdk-for-go [Logs] [Diff]

datafactory/mgmt/2018-06-01 [Logs] [Instructions] [Generation PR]
- datafactory.mgmt.2018-06-01.zip
preview/datafactory/mgmt/2017-09-01-preview [Logs]

JavaScript: Azure/azure-sdk-for-js [Logs] [Diff]

@azure/arm-datafactory [Logs] [Instructions]
- azure-arm-datafactory-5.3.0.tgz

msftclas · 2019-07-26T08:41:33Z

All CLA requirements met.

AutorestCI · 2019-07-26T08:43:57Z

Automation for azure-sdk-for-python

Encountered an unknown error: (azure-sdk-for-python)

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/azure_devtools/ci_tools/github_tools.py", line 33, in exception_to_github
    yield context
  File "/usr/local/lib/python3.6/dist-packages/swaggertosdk/restapi/github.py", line 170, in rest_handle_action
    return rest_pull_close(body, restapi_repo, sdk_pr_target_repo, sdkbase, sdk_tag)
  File "/usr/local/lib/python3.6/dist-packages/swaggertosdk/restapi/github.py", line 185, in rest_pull_close
    rest_pr_management(rest_pr, sdk_pr_target_repo, sdk_tag, sdk_default_base)
  File "/usr/local/lib/python3.6/dist-packages/swaggertosdk/restapi/github_handler.py", line 151, in rest_pr_management
    sdk_tag=sdk_tag
  File "/usr/local/lib/python3.6/dist-packages/swaggertosdk/SwaggerToSdkNewCLI.py", line 254, in generate_sdk_from_git_object
    with manage_git_folder(gh_token, Path(temp_dir) / Path("rest"), branched_rest_api_id, pr_number=pr_number) as restapi_git_folder, \
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.6/dist-packages/azure_devtools/ci_tools/github_tools.py", line 272, in manage_git_folder
    clone_to_path(gh_token, temp_dir, split_git_id[0], branch_or_commit=branch, pr_number=pr_number)
  File "/usr/local/lib/python3.6/dist-packages/azure_devtools/ci_tools/github_tools.py", line 212, in clone_to_path
    repo.git.checkout(branch_or_commit)
  File "/usr/local/lib/python3.6/dist-packages/git/cmd.py", line 548, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/git/cmd.py", line 1014, in _call_process
    return self.execute(call, **exec_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/git/cmd.py", line 825, in execute
    raise GitCommandError(command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git checkout f58024b5cbf0800f8f9a9b7b1179e9d01ce47347
  stderr: 'fatal: reference is not a tree: f58024b5cbf0800f8f9a9b7b1179e9d01ce47347'

azuresdkci · 2019-07-26T09:04:52Z

Can one of the admins verify this patch?

AutorestCI · 2019-07-26T09:18:08Z

Automation for azure-sdk-for-go

Encountered an unknown error: (azure-sdk-for-go)

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/azure_devtools/ci_tools/github_tools.py", line 33, in exception_to_github
    yield context
  File "/usr/local/lib/python3.6/dist-packages/swaggertosdk/restapi/github.py", line 170, in rest_handle_action
    return rest_pull_close(body, restapi_repo, sdk_pr_target_repo, sdkbase, sdk_tag)
  File "/usr/local/lib/python3.6/dist-packages/swaggertosdk/restapi/github.py", line 185, in rest_pull_close
    rest_pr_management(rest_pr, sdk_pr_target_repo, sdk_tag, sdk_default_base)
  File "/usr/local/lib/python3.6/dist-packages/swaggertosdk/restapi/github_handler.py", line 151, in rest_pr_management
    sdk_tag=sdk_tag
  File "/usr/local/lib/python3.6/dist-packages/swaggertosdk/SwaggerToSdkNewCLI.py", line 254, in generate_sdk_from_git_object
    with manage_git_folder(gh_token, Path(temp_dir) / Path("rest"), branched_rest_api_id, pr_number=pr_number) as restapi_git_folder, \
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.6/dist-packages/azure_devtools/ci_tools/github_tools.py", line 272, in manage_git_folder
    clone_to_path(gh_token, temp_dir, split_git_id[0], branch_or_commit=branch, pr_number=pr_number)
  File "/usr/local/lib/python3.6/dist-packages/azure_devtools/ci_tools/github_tools.py", line 212, in clone_to_path
    repo.git.checkout(branch_or_commit)
  File "/usr/local/lib/python3.6/dist-packages/git/cmd.py", line 548, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/git/cmd.py", line 1014, in _call_process
    return self.execute(call, **exec_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/git/cmd.py", line 825, in execute
    raise GitCommandError(command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git checkout f58024b5cbf0800f8f9a9b7b1179e9d01ce47347
  stderr: 'fatal: reference is not a tree: f58024b5cbf0800f8f9a9b7b1179e9d01ce47347'

amarzavery · 2019-07-29T18:06:32Z

...atafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Dataset.json

@@ -651,6 +651,43 @@
        }
      }
    },
+    "JsonDataset": {


Please update examples that showcase the new models that have been added.

The dataset model itself is not new, and it already have examples for the dataset model. The JsonDataset is just one type of the dataset, and we usually don't have examples to cover all types of datasets since it has 80+ types. However, we will definitely add serialization/deserialization tests in SDK to cover all dataset types.

amarzavery

Adding new child model to an existing polymorphic hierarchy in an existing stable api version will cause breaking changes to the generated SDKs.

shawnxzq

@hvermis I think this is a common practice in data factory. We don't assume customer with lower version of SDK to be able to consume everything generated by a higher version of SDK, right?

shawnxzq · 2019-07-30T12:58:10Z

...atafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Dataset.json

@@ -651,6 +651,43 @@
        }
      }
    },
+    "JsonDataset": {


The dataset model itself is not new, and it already have examples for the dataset model. The JsonDataset is just one type of the dataset, and we usually don't have examples to cover all types of datasets since it has 80+ types. However, we will definitely add serialization/deserialization tests in SDK to cover all dataset types.

hvermis · 2019-08-02T01:22:21Z

@KrisBash Can you please review this PR? It is again very similar to other ADF PRs that you reviewed in the past - we are adding new Dataset types again.

ravbhatnagar · 2019-08-08T07:51:12Z

Same concern as with the other PR from DataFactory RP. writable properties being added to existing API version is against Azure API guidelines. https://github.com/microsoft/api-guidelines/blob/vNext/Guidelines.md#123-definition-of-a-breaking-change

shawnxzq · 2019-08-08T14:21:54Z

Same concern as with the other PR from DataFactory RP. writable properties being added to existing API version is against Azure API guidelines. https://github.com/microsoft/api-guidelines/blob/vNext/Guidelines.md#123-definition-of-a-breaking-change

@hvermis Could you please help clarify ADF definitions for breaking changes to the whole team, otherwise we can't do anything to add new ADF connectors, thanks!

shawnxzq added 4 commits July 16, 2019 19:31

Add json dataset

29e561c

Added JSON v2 dataset

55333d2

Fixed description messages

76fa69d

Improved description message according to review comments

d49776e

azuresdkci assigned amarzavery Jul 26, 2019

azuresdkci requested a review from amarzavery July 26, 2019 08:41

AutorestCI mentioned this pull request Jul 26, 2019

[AutoPR datafactory/resource-manager] [PR] - Add JSON dataset support in ADF Azure/azure-sdk-for-python#6506

Closed

This was referenced Jul 26, 2019

[AutoPR datafactoryv2/resource-manager/v2017_09_01_preview] [PR] - Add JSON dataset support in ADF Azure/azure-sdk-for-java#4615

Closed

[AutoPR datafactoryv2/resource-manager/v2018_06_01] [PR] - Add JSON dataset support in ADF Azure/azure-sdk-for-java#4616

Closed

AutorestCI mentioned this pull request Jul 26, 2019

[AutoPR datafactory/resource-manager] [PR] - Add JSON dataset support in ADF Azure/azure-sdk-for-go#5351

Closed

openapi-sdkautomation bot mentioned this pull request Jul 26, 2019

[AutoPR @azure/arm-datafactory] [PR] - Add JSON dataset support in ADF Azure/azure-sdk-for-js#4452

Closed

amarzavery reviewed Jul 29, 2019

View reviewed changes

amarzavery added potential-sdk-breaking-change WaitForARMFeedback <valid label in PR review process> add this label when ARM review is required labels Jul 29, 2019

shawnxzq commented Jul 31, 2019

View reviewed changes

hvermis approved these changes Aug 2, 2019

View reviewed changes

amarzavery added the Reassign label Aug 5, 2019

azuresdkci removed the Reassign label Aug 5, 2019

azuresdkci assigned michaeljqzq and unassigned amarzavery Aug 5, 2019

azuresdkci requested a review from michaeljqzq August 5, 2019 18:24

ravbhatnagar added the ARMChangesRequested <valid label in PR review process>add this label when require changes after ARM review label Aug 8, 2019

shawnxzq closed this Aug 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PR] - Add JSON dataset support in ADF #6739

[PR] - Add JSON dataset support in ADF #6739

shawnxzq commented Jul 26, 2019 •

edited

Loading

openapi-sdkautomation bot commented Jul 26, 2019 •

edited

Loading

msftclas commented Jul 26, 2019 •

edited

Loading

AutorestCI commented Jul 26, 2019 •

edited

Loading

azuresdkci commented Jul 26, 2019

AutorestCI commented Jul 26, 2019 •

edited

Loading

amarzavery Jul 29, 2019

shawnxzq Jul 30, 2019

amarzavery left a comment

shawnxzq left a comment

shawnxzq Jul 30, 2019

hvermis commented Aug 2, 2019

ravbhatnagar commented Aug 8, 2019

shawnxzq commented Aug 8, 2019

[PR] - Add JSON dataset support in ADF #6739

[PR] - Add JSON dataset support in ADF #6739

Conversation

shawnxzq commented Jul 26, 2019 • edited Loading

Latest improvements:

Contribution checklist:

ARM API Review Checklist

openapi-sdkautomation bot commented Jul 26, 2019 • edited Loading

SDK Automation [Logs] (Generated from d49776e, Iteration 15)

msftclas commented Jul 26, 2019 • edited Loading

AutorestCI commented Jul 26, 2019 • edited Loading

Automation for azure-sdk-for-python

azuresdkci commented Jul 26, 2019

AutorestCI commented Jul 26, 2019 • edited Loading

Automation for azure-sdk-for-go

amarzavery Jul 29, 2019

Choose a reason for hiding this comment

shawnxzq Jul 30, 2019

Choose a reason for hiding this comment

amarzavery left a comment

Choose a reason for hiding this comment

shawnxzq left a comment

Choose a reason for hiding this comment

shawnxzq Jul 30, 2019

Choose a reason for hiding this comment

hvermis commented Aug 2, 2019

ravbhatnagar commented Aug 8, 2019

shawnxzq commented Aug 8, 2019

shawnxzq commented Jul 26, 2019 •

edited

Loading

openapi-sdkautomation bot commented Jul 26, 2019 •

edited

Loading

msftclas commented Jul 26, 2019 •

edited

Loading

AutorestCI commented Jul 26, 2019 •

edited

Loading

AutorestCI commented Jul 26, 2019 •

edited

Loading