Change for Synapse pipeline to pass parameters to SparkJobDefinition/Notebook #17375

wonner · 2022-01-17T08:48:15Z

MSFT employees can try out our new experience at OpenAPI Hub - one location for using our validation tools and finding your workflow.

Changelog

Add a changelog entry for this PR by answering the following questions:

What's the purpose of the update?
- new service onboarding
- new API version
- update existing version for new feature
- update existing version to fix swagger quality issue in s360
- Other, please clarify
When are you targeting to deploy the new service/feature to public regions? Please provide the date or, if the date is not yet available, the month.
When do you expect to publish the swagger? Please provide date or, the the date is not yet available, the month.
If updating an existing version, please select the specific langauge SDKs and CLIs that must be refreshed after the swagger is published.
- SDK of .NET (need service team to ensure code readiness)
- SDK of Python
- SDK of Java
- SDK of Js
- SDK of Go
- PowerShell
- CLI
- Terraform
- No refresh required for updates in this PR

Contribution checklist:

I commit to follow the Breaking Change Policy of "no breaking changes"
I have reviewed the documentation for the workflow.
Validation tools were run on swagger spec(s) and errors have all been fixed in this PR. How to fix?

If any further question about AME onboarding or validation tools, please view the FAQ.

ARM API Review Checklist

Applicability: ⚠️

If your changes encompass only the following scenarios, you should SKIP this section, as these scenarios do not require ARM review.

Change to data plane APIs

Adding new properties

All removals

Otherwise your PR may be subject to ARM review requirements. Complete the following:

Check this box if any of the following apply to the PR so that label "WaitForARMFeedback" will be added automatically to begin ARM API Review. Failure to comply may result in delays to the manifest.
- Adding a new service
- Adding new API(s)
- Adding a new API version
  -[ ] To review changes efficiently, ensure you are using OpenAPIHub to initialize the PR for adding a new version. More details, refer to the wiki.
Ensure you've reviewed following guidelines including ARM resource provider contract and REST guidelines. Estimated time (4 hours). This is required before you can request review from ARM API Review board.
If you are blocked on ARM review and want to get the PR merged with urgency, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.

Breaking Change Review Checklist

If any of the following scenarios apply to the PR, request approval from the Breaking Change Review Board as defined in the Breaking Change Policy.

Removing API(s) in a stable version
Removing properties in a stable version
Removing API version(s) in a stable version
Updating API in a stable or public preview version with Breaking Change Validation errors
Updating API(s) in public preview over 1 year (refer to Retirement of Previews)

Action: to initiate an evaluation of the breaking change, create a new intake using the template for breaking changes. Addition details on the process and office hours are on the Breaking change Wiki.

Please follow the link to find more details on PR review process.

openapi-workflow-bot · 2022-01-17T08:48:18Z

Hi, @wonner Thanks for your PR. I am workflow bot for review process. Here are some small tips.

Please ensure to do self-check against checklists in first PR comment.

PR assignee is the person auto-assigned and responsible for your current PR reviewing and merging.

For specs comparison cross API versions, Use API Specs Comparison Report Generator

If there is CI failure(s), to fix CI error(s) is mandatory for PR merging; or you need to provide justification in PR comment for explanation. How to fix?

Any feedback about review process or workflow bot, pls contact swagger and tools team. vsswagger@microsoft.com

openapi-workflow-bot · 2022-01-17T08:48:18Z

[Call for Action] To better understand Azure service dev/test scenario, and support Azure service developer better on Swagger and REST API related tests in early phase, please help to fill in with this survey https://aka.ms/SurveyForEarlyPhase. It will take 5 to 10 minutes. If you already complete survey, please neglect this comment. Thanks.

openapi-pipeline-app · 2022-01-17T08:48:19Z

Swagger Validation Report

️❌BreakingChange: 18 Errors, 2 Warnings failed [Detail]

Compared Swaggers (Based on Oad v0.9.1)
- current:Microsoft.Synapse/stable/2020-12-01 compared with base:Microsoft.Synapse/stable/2020-12-01
- current:2020-12-01/entityTypes/Pipeline.json compared with base:2020-12-01/entityTypes/Pipeline.json

Rule	Message
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'file' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7250:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7246:7
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'className' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7250:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7246:7
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'files' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7250:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7246:7
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'targetBigDataPool' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7250:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7246:7
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'executorSize' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7250:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7246:7
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'conf' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7250:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7246:7
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'driverSize' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7250:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7246:7
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'numExecutors' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7250:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7246:7
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'sparkPool' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7206:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7206:7
⚠️ 1026 - TypeChanged	The new version has a different type 'object' than the previous one 'string'. New: Microsoft.Synapse/stable/2020-12-01/artifacts.json#L778:9 Old: Microsoft.Synapse/stable/2020-12-01/artifacts.json#L778:9

Rule	Message
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'sparkPool' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7206:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7206:7
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'file' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7250:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7246:7
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'className' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7250:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7246:7
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'files' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7250:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7246:7
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'targetBigDataPool' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7250:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7246:7
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'executorSize' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7250:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7246:7
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'conf' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7250:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7246:7
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'driverSize' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7250:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7246:7
`❌` 1045 - AddedOptionalProperty	The new version has a new optional property 'numExecutors' that was not found in the old version. New: stable/2020-12-01/entityTypes/Pipeline.json#L7250:7 Old: stable/2020-12-01/entityTypes/Pipeline.json#L7246:7
⚠️ 1026 - TypeChanged	The new version has a different type 'object' than the previous one 'string'. New: Microsoft.Synapse/stable/2020-12-01/artifacts.json#L778:9 Old: Microsoft.Synapse/stable/2020-12-01/artifacts.json#L778:9

️⚠️LintDiff: 0 Warnings warning [Detail]

Linted configuring files (Based on source branch, openapi-validator v1.10.1 , classic-openapi-validator v1.1.10 )
- synapse/data-plane/readme.md tag:package-artifacts-composite-v3
Linted configuring files (Based on target branch, openapi-validator v1.10.1 , classic-openapi-validator v1.1.10 )
- synapse/data-plane/readme.md tag:package-artifacts-composite-v3

The following errors/warnings exist before current PR submission:

Only 30 items are listed, please refer to log for more details.

Rule	Message
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/preview/2021-11-01-preview/kqlScripts.json#L259
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/preview/2021-06-01-preview/sparkConfigurations.json#L285
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/stable/2020-12-01/artifacts.json#L692
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/stable/2020-12-01/artifacts.json#L723
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/stable/2020-12-01/dataflows.json#L501
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/stable/2020-12-01/dataflows.json#L540
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/stable/2020-12-01/dataflows.json#L710
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/stable/2020-12-01/dataflows.json#L975
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/stable/2020-12-01/dataflows.json#L993
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/stable/2020-12-01/linkedServices.json#L283
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/stable/2020-12-01/datasets.json#L283
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/preview/2019-06-01-preview/integrationRuntime.json#L2861
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/stable/2020-12-01/library.json#L428
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/stable/2020-12-01/notebooks.json#L379
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/stable/2020-12-01/sparkJobDefinitions.json#L381
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/stable/2020-12-01/sqlScripts.json#L297
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/stable/2020-12-01/triggers.json#L535
⚠️ R2001 - AvoidNestedProperties	Consider using x-ms-client-flatten to provide a better end user experience Location: Microsoft.Synapse/stable/2020-12-01/triggers.json#L608
⚠️ R2007 - LongRunningOperationsWithLongRunningExtension	The operation 'Library_GetOperationResult' returns 202 status code, which indicates a long running operation, please enable 'x-ms-long-running-operation. Location: Microsoft.Synapse/stable/2020-12-01/library.json#L103
⚠️ R2007 - LongRunningOperationsWithLongRunningExtension	The operation 'NotebookOperationResult_Get' returns 202 status code, which indicates a long running operation, please enable 'x-ms-long-running-operation. Location: Microsoft.Synapse/stable/2020-12-01/notebooks.json#L289
⚠️ R2007 - LongRunningOperationsWithLongRunningExtension	The operation 'Pipeline_CreatePipelineRun' returns 202 status code, which indicates a long running operation, please enable 'x-ms-long-running-operation. Location: Microsoft.Synapse/stable/2020-12-01/pipelines.json#L253
⚠️ R2010 - LongRunningOperationsOptionsValidator	A LRO Post operation with return schema must have 'x-ms-long-running-operation-options' extension enabled. Location: Microsoft.Synapse/stable/2020-12-01/dataflows.json#L249
⚠️ R2010 - LongRunningOperationsOptionsValidator	A LRO Post operation with return schema must have 'x-ms-long-running-operation-options' extension enabled. Location: Microsoft.Synapse/stable/2020-12-01/dataflows.json#L418
⚠️ R2010 - LongRunningOperationsOptionsValidator	A LRO Post operation with return schema must have 'x-ms-long-running-operation-options' extension enabled. Location: Microsoft.Synapse/stable/2020-12-01/library.json#L62
⚠️ R2010 - LongRunningOperationsOptionsValidator	A LRO Post operation with return schema must have 'x-ms-long-running-operation-options' extension enabled. Location: Microsoft.Synapse/stable/2020-12-01/triggers.json#L206
⚠️ R2010 - LongRunningOperationsOptionsValidator	A LRO Post operation with return schema must have 'x-ms-long-running-operation-options' extension enabled. Location: Microsoft.Synapse/stable/2020-12-01/triggers.json#L282
⚠️ R2018 - XmsEnumValidation	The enum types should have x-ms-enum type extension set with appropriate options. Property name: type Location: Microsoft.Synapse/preview/2019-06-01-preview/integrationRuntime.json#L2954
⚠️ R2063 - OperationIdNounConflictingModelNames	OperationId has a noun that conflicts with one of the model names in definitions section. The model name will be disambiguated to 'KqlScriptModel'. Consider using the plural form of 'KqlScript' to avoid this. Note: If you have already shipped an SDK on top of this spec, fixing this warning may introduce a breaking change. Location: Microsoft.Synapse/preview/2021-11-01-preview/kqlScripts.json#L70
⚠️ R2063 - OperationIdNounConflictingModelNames	OperationId has a noun that conflicts with one of the model names in definitions section. The model name will be disambiguated to 'KqlScriptModel'. Consider using the plural form of 'KqlScript' to avoid this. Note: If you have already shipped an SDK on top of this spec, fixing this warning may introduce a breaking change. Location: Microsoft.Synapse/preview/2021-11-01-preview/kqlScripts.json#L117
⚠️ R2063 - OperationIdNounConflictingModelNames	OperationId has a noun that conflicts with one of the model names in definitions section. The model name will be disambiguated to 'KqlScriptModel'. Consider using the plural form of 'KqlScript' to avoid this. Note: If you have already shipped an SDK on top of this spec, fixing this warning may introduce a breaking change. Location: Microsoft.Synapse/preview/2021-11-01-preview/kqlScripts.json#L151

️️✔️Avocado succeeded [Detail] [Expand]

Validation passes for Avocado.

️️✔️~[Staging] ApiReadinessCheck succeeded [Detail] [Expand]

️️✔️ModelValidation succeeded [Detail] [Expand]

Validation passes for ModelValidation.

️️✔️SemanticValidation succeeded [Detail] [Expand]

Validation passes for SemanticValidation.

️️✔️Cross-Version Breaking Changes succeeded [Detail] [Expand]

There are no breaking changes.

️🔄CredScan inProgress [Detail]

️️✔️SDK Track2 Validation succeeded [Detail] [Expand]

Validation passes for SDKTrack2Validation

The following tags are being changed in this PR

️️✔️PrettierCheck succeeded [Detail] [Expand]

Validation passes for PrettierCheck.

️️✔️SpellCheck succeeded [Detail] [Expand]

Validation passes for SpellCheck.

️️✔️Lint(RPaaS) succeeded [Detail] [Expand]

Validation passes for Lint(RPaaS).

_{Posted by Swagger Pipeline | How to fix these errors?}

openapi-pipeline-app · 2022-01-17T08:48:20Z

Swagger pipeline restarted successfully, please wait for status update in this comment.

openapi-workflow-bot · 2022-01-17T08:51:39Z

NewApiVersionRequired reason:

A service’s API is a contract with customers and is represented by using the api-version query parameter. Changes such as adding an optional property to a request/response or introducing a new operation is a change to the service’s contract and therefore requires a new api-version value. This is critically important for documentation, client libraries, and customer support.

EXAMPLE: if a customer calls a service in the public cloud using api-version=2020-07-27, the new property or operation may exist but if they call the service in a government cloud, air-gapped cloud, or Azure Stack Hub cloud using the same api-version, the property or operation may not exist. Because there is no clear relationship between the service api-version and the new property/operation, customers can’t trust the documentation and Azure customer have difficulty helping customers diagnose issues. In addition, each client library version documents the service version it supports. When an optional property or new operation is added to a service and its Swagger, new client libraries must be produced to expose this functionality to customers. Without updating the api-version, it is unclear to customers which version of a client library supports these new features.

openapi-workflow-bot · 2022-01-17T08:52:03Z

Hi @wonner, one or multiple breaking change(s) is detected in your PR. Please check out the breaking change(s), and provide business justification in the PR comment and @ PR assignee why you must have these change(s), and how external customer impact can be mitigated. Please ensure to follow breaking change policy to request breaking change review and approval before proceeding swagger PR review.
Action: To initiate an evaluation of the breaking change, create a new intake using the template for breaking changes. Addition details on the process and office hours are on the Breaking change Wiki.
If you want to know the production traffic statistic, please see ARM Traffic statistic.
If you think it is false positive breaking change, please provide the reasons in the PR comment, report to Swagger Tooling Team via https://aka.ms/swaggerfeedback.

openapi-workflow-bot · 2022-01-17T08:52:12Z

Hi @wonner, Your PR has some issues. Please fix the CI sequentially by following the order of Avocado, semantic validation, model validation, breaking change, lintDiff. If you have any questions, please post your questions in this channel https://aka.ms/swaggersupport.

Task	How to fix	Priority
Avocado	Fix-Avocado	High
Semantic validation	Fix-SemanticValidation-Error	High
Model validation	Fix-ModelValidation-Error	High
LintDiff	Fix-LintDiff	high

If you need further help, please feedback via swagger feedback.

ghost · 2022-02-13T16:02:46Z

Hi, @wonner. Your PR has no update for 14 days and it is marked as stale PR. If no further update for over 14 days, the bot will close the PR. If you want to refresh the PR, please remove no-recent-activity label.

…Notebook (Azure#17375) * notebook/sparkjobdef activity parametrization * fix lint * update description

notebook/sparkjobdef activity parametrization

40a7226

ghost added the Synapses label Jan 17, 2022

openapi-pipeline-app bot added the NewApiVersionRequired label Jan 17, 2022

openapi-pipeline-app bot added the BreakingChangeReviewRequired label Jan 17, 2022

openapi-pipeline-app bot added the CI-FixRequiredOnFailure label Jan 17, 2022

fix lint

aa0ba93

openapi-pipeline-app bot removed the CI-FixRequiredOnFailure label Jan 17, 2022

update description

c580b18

wonner marked this pull request as ready for review January 24, 2022 05:24

wonner requested a review from idear1203 as a code owner January 24, 2022 05:24

openapi-assignment-bot bot assigned lmazuel Jan 24, 2022

ghost added the no-recent-activity label Feb 13, 2022

wonner added FixS360 and removed no-recent-activity labels Feb 14, 2022

JeffreyRichter added the Approved-BreakingChange DO NOT USE! OBSOLETE label. See https://github.com/Azure/azure-sdk-tools/issues/6374 label Feb 14, 2022

lmazuel merged commit 4d0adaa into Azure:main Feb 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change for Synapse pipeline to pass parameters to SparkJobDefinition/Notebook #17375

Change for Synapse pipeline to pass parameters to SparkJobDefinition/Notebook #17375

wonner commented Jan 17, 2022

openapi-workflow-bot bot commented Jan 17, 2022

openapi-workflow-bot bot commented Jan 17, 2022

openapi-pipeline-app bot commented Jan 17, 2022 •

edited

Loading

openapi-pipeline-app bot commented Jan 17, 2022 •

edited

Loading

openapi-workflow-bot bot commented Jan 17, 2022

openapi-workflow-bot bot commented Jan 17, 2022

openapi-workflow-bot bot commented Jan 17, 2022

ghost commented Feb 13, 2022

Change for Synapse pipeline to pass parameters to SparkJobDefinition/Notebook #17375

Change for Synapse pipeline to pass parameters to SparkJobDefinition/Notebook #17375

Conversation

wonner commented Jan 17, 2022

Changelog

Contribution checklist:

ARM API Review Checklist

Breaking Change Review Checklist

openapi-workflow-bot bot commented Jan 17, 2022

openapi-workflow-bot bot commented Jan 17, 2022

openapi-pipeline-app bot commented Jan 17, 2022 • edited Loading

Swagger Validation Report

openapi-pipeline-app bot commented Jan 17, 2022 • edited Loading

openapi-workflow-bot bot commented Jan 17, 2022

openapi-workflow-bot bot commented Jan 17, 2022

openapi-workflow-bot bot commented Jan 17, 2022

ghost commented Feb 13, 2022

openapi-pipeline-app bot commented Jan 17, 2022 •

edited

Loading

openapi-pipeline-app bot commented Jan 17, 2022 •

edited

Loading