Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move regex validation checks to template schema #108

Merged
merged 11 commits into from
Nov 9, 2023
Merged

Move regex validation checks to template schema #108

merged 11 commits into from
Nov 9, 2023

Conversation

shreyas-goenka
Copy link
Collaborator

@shreyas-goenka shreyas-goenka commented Oct 23, 2023 •
edited
Loading

Changes

This PR moves regex validation checks to the template schema. This has the following advantages:

  1. Mlops-stacks will fail / give feedback early when an invalid value is provided
  2. Allows us to remove existing validation checks

Note: We first need to get databricks/cli#912 in, and set / update the minimum CLI version for mlops-stacks before this PR can be merged.

Tests

Tested manually. Here are the cases:

Case project_name: ab

shreyas.goenka@THW32HFW6T playground % cli bundle init ~/mlops-stack
Welcome to MLOps Stacks. For detailed information on project generation, see the README at https://github.com/databricks/mlops-stacks/blob/main/README.md. 

Project Name [my-mlops-project]: ab
Error: invalid value for input_project_name: "ab". Project name must be at least 3 characters long and cannot contain the following characters: "\", "/", " " and ".".

Case project_name: abcd/

shreyas.goenka@THW32HFW6T playground % cli bundle init ~/mlops-stack
Welcome to MLOps Stacks. For detailed information on project generation, see the README at https://github.com/databricks/mlops-stacks/blob/main/README.md. 

Project Name [my-mlops-project]: abcd/
Error: invalid value for input_project_name: "abcd/". Project name must be at least 3 characters long and cannot contain the following characters: ",", "/", " " and ".".

Case staging URL: abc

URL of staging Databricks workspace, used to run CI tests on PRs and preview config changes before they're deployed to production. Default: 
Azure - https://adb-xxxx.xx.azuredatabricks.net
AWS - https://your-staging-workspace.cloud.databricks.com
: abc
Error: invalid value for input_databricks_staging_workspace_host: "abc". Databricks staging workspace host URLs must start with https. Got invalid workspace host.

arpitjasa-db
arpitjasa-db reviewed Oct 24, 2023
View reviewed changes
Copy link
Collaborator

@arpitjasa-db arpitjasa-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome thanks a ton @shreyas-goenka ! I'll approve once the CLI PR merges and we can test this out with some project generations

library/input_validation.tmpl Outdated Show resolved Hide resolved
arpitjasa-db
arpitjasa-db reviewed Nov 2, 2023
View reviewed changes
Copy link
Collaborator

@arpitjasa-db arpitjasa-db left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shreyas-goenka would you mind updating the tests in tests/test_create_project.py::test_generate_fails_with_invalid_params for the expected error once the CLI update goes out? I believe the tests should pass after that and we can merge this in!

@shreyas-goenka shreyas-goenka mentioned this pull request Nov 7, 2023
Merged
@shreyas-goenka
Copy link
Collaborator Author

This PR requires a CLI side fix before proceeding: databricks/cli#959

github-merge-queue bot pushed a commit to databricks/cli that referenced this pull request Nov 7, 2023
@shreyas-goenka
Remove validation for default value against pattern (#959)
283f241 
## Changes
This PR removes validation for default value against the regex pattern
specified in a JSON schema at schema load time. This is required because
#795 introduces parameterising the
default value as a Go text template impling that the default value now
does not necessarily have to match the pattern at schema load time.

This will also unblock:
databricks/mlops-stacks#108

Note, this does not remove runtime validation for input parameters right
before template initialization, which happens here:
https://github.com/databricks/cli/blob/fb32e78c9b9fb000ce898b8a60b0b47920f487d3/libs/template/materialize.go#L76

## Tests
Changes to existing test.
arpitjasa-db
arpitjasa-db approved these changes Nov 9, 2023
View reviewed changes
Copy link
Collaborator

@arpitjasa-db arpitjasa-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with one minor ask, thanks @shreyas-goenka !

databricks_template_schema.json Show resolved Hide resolved
@shreyas-goenka shreyas-goenka merged commit 1159734 into main Nov 9, 2023
1 check passed
@shreyas-goenka shreyas-goenka deleted the regex-in-schema branch November 9, 2023 22:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@arpitjasa-db arpitjasa-db arpitjasa-db approved these changes

@mingyu89 mingyu89 Awaiting requested review from mingyu89

@pietern pietern Awaiting requested review from pietern

@niall-turbitt niall-turbitt Awaiting requested review from niall-turbitt

@vladimirk-db vladimirk-db Awaiting requested review from vladimirk-db

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@shreyas-goenka @arpitjasa-db