Enabled debug logging for every job task run through a file, which is accessible from both workspace UI and Databricks CLI #426
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nfx
requested changes
Oct 10, 2023
nfx
requested changes
Oct 10, 2023
Codecov Report
@@ Coverage Diff @@
## main #426 +/- ##
==========================================
- Coverage 84.45% 83.41% -1.04%
==========================================
Files 31 31
Lines 2850 2889 +39
Branches 539 542 +3
==========================================
+ Hits 2407 2410 +3
- Misses 323 359 +36
Partials 120 120
|
nfx
requested changes
Oct 10, 2023
FastLee
force-pushed
the
feature/file_logging
branch
from
28ffd07
to
a3130f9
Compare
nfx
requested changes
Oct 10, 2023
FastLee
added
enhancement
nfx
changed the title
nfx
force-pushed
the
feature/file_logging
branch
from
74260e9
to
ee7ee98
Compare
nfx
approved these changes
Oct 11, 2023
Merged
nfx
added a commit
that referenced
this pull request
Oct 12, 2023
# Version changelog ## 0.4.0 * Added exception handling for secret scope not found. ([#418](#418)). * Added a crawler for creating an inventory of Azure Service Principals ([#326](#326)). * Added check if account group already exists during failure recovery ([#446](#446)). * Added checking for index out of range. ([#429](#429)). * Added hyperlink to UCX releases in the main readme ([#408](#408)). * Added integration test to check backup groups get deleted ([#387](#387)). * Added logging of errors during threadpool operations. ([#376](#376)). * Added recovery mode for workspace-local groups from temporary groups ([#435](#435)). * Added support for migrating Legacy Table ACLs from workspace-local to account-level groups ([#412](#412)). * Added detection for installations of unreleased versions ([#399](#399)). * Decoupled `PermissionsManager` from `GroupMigrationToolkit` ([#407](#407)). * Enabled debug logging for every job task run through a file, which is accessible from both workspace UI and Databricks CLI ([#426](#426)). * Ensured that table exists, even when crawlers produce zero records ([#373](#373)). * Extended test suite for HMS->HMS TACL migration ([#439](#439)). * Fixed handling of secret scope responses ([#431](#431)). * Fixed `crawl_permissions` task to respect 'workspace_start_path' config ([#444](#444)). * Fixed broken logic in `parallel` module and applied hardened error handling design for parallel code ([#405](#405)). * Fixed codecov.io reporting ([#403](#403)). * Fixed integration tests for crawlers ([#379](#379)). * Improved README.py and logging messages ([#433](#433)). * Improved cleanup for workspace backup groups by adding more retries on errors ([#375](#375)). * Improved dashboard queries to show unsupported storage types. ([#398](#398)). * Improved documentation for readme notebook ([#257](#257)). * Improved test coverage for installer ([#371](#371)). * Introduced deterministic `env_or_skip` fixture for integration tests ([#396](#396)). * Made HMS & UC fixtures return `CatalogInfo`, `SchemaInfo`, and `TableInfo` ([#409](#409)). * Merge `workspace_access.Crawler` and `workspace_access.Applier` interfaces to `workspace_access.AclSupport` ([#436](#436)). * Moved examples to docs ([#404](#404)). * Properly isolated integration testing for workflows on an existing shared cluster ([#414](#414)). * Removed thread pool for any IAM Group removals and additions ([#394](#394)). * Replace plus char with minus in version tag for GCP dev installation of UCX ([#420](#420)). * Run integration tests on shared clusters for a faster devloop ([#397](#397)). * Show difference between serverless and PRO warehouses during installation ([#385](#385)). * Split `migrate-groups` workflow into three different stages for reliability ([#442](#442)). * Use groups instead of usernames in code owners file ([#389](#389)).
FastLee
added a commit
that referenced
this pull request
Oct 25, 2023
… accessible from both workspace UI and Databricks CLI (#426) This PR improves debuggability of UCX. ## More concise error messaging from job run <img width="1390" alt="image" src="https://github.com/databrickslabs/ucx/assets/259697/be4dc82c-3724-4684-b8cd-a77a05f24d67"> ## Viewing logs via Databricks CLI <img width="929" alt="image" src="https://github.com/databrickslabs/ucx/assets/259697/b1901d03-1c9a-4c63-81c7-501f0f3e4e48"> ## Added `.ucx/logs` folder in the installation <img width="313" alt="image" src="https://github.com/databrickslabs/ucx/assets/259697/a396b637-308a-4791-9cc5-dbd0494431c4"> where every executes workflow has a folder per job run, with ability to get back to Job Run in Workspace UI <img width="1092" alt="image" src="https://github.com/databrickslabs/ucx/assets/259697/1bf60990-eab6-4866-b0b5-a75c8a44674e"> ## Viewing full debug logs via workspace UI <img width="1218" alt="image" src="https://github.com/databrickslabs/ucx/assets/259697/0d1a84bb-0af9-4cbc-b662-77c1715ff4b1"> Closes #86 --------- Co-authored-by: Serge Smertin <serge.smertin@databricks.com>
FastLee
pushed a commit
that referenced
this pull request
Oct 25, 2023
# Version changelog ## 0.4.0 * Added exception handling for secret scope not found. ([#418](#418)). * Added a crawler for creating an inventory of Azure Service Principals ([#326](#326)). * Added check if account group already exists during failure recovery ([#446](#446)). * Added checking for index out of range. ([#429](#429)). * Added hyperlink to UCX releases in the main readme ([#408](#408)). * Added integration test to check backup groups get deleted ([#387](#387)). * Added logging of errors during threadpool operations. ([#376](#376)). * Added recovery mode for workspace-local groups from temporary groups ([#435](#435)). * Added support for migrating Legacy Table ACLs from workspace-local to account-level groups ([#412](#412)). * Added detection for installations of unreleased versions ([#399](#399)). * Decoupled `PermissionsManager` from `GroupMigrationToolkit` ([#407](#407)). * Enabled debug logging for every job task run through a file, which is accessible from both workspace UI and Databricks CLI ([#426](#426)). * Ensured that table exists, even when crawlers produce zero records ([#373](#373)). * Extended test suite for HMS->HMS TACL migration ([#439](#439)). * Fixed handling of secret scope responses ([#431](#431)). * Fixed `crawl_permissions` task to respect 'workspace_start_path' config ([#444](#444)). * Fixed broken logic in `parallel` module and applied hardened error handling design for parallel code ([#405](#405)). * Fixed codecov.io reporting ([#403](#403)). * Fixed integration tests for crawlers ([#379](#379)). * Improved README.py and logging messages ([#433](#433)). * Improved cleanup for workspace backup groups by adding more retries on errors ([#375](#375)). * Improved dashboard queries to show unsupported storage types. ([#398](#398)). * Improved documentation for readme notebook ([#257](#257)). * Improved test coverage for installer ([#371](#371)). * Introduced deterministic `env_or_skip` fixture for integration tests ([#396](#396)). * Made HMS & UC fixtures return `CatalogInfo`, `SchemaInfo`, and `TableInfo` ([#409](#409)). * Merge `workspace_access.Crawler` and `workspace_access.Applier` interfaces to `workspace_access.AclSupport` ([#436](#436)). * Moved examples to docs ([#404](#404)). * Properly isolated integration testing for workflows on an existing shared cluster ([#414](#414)). * Removed thread pool for any IAM Group removals and additions ([#394](#394)). * Replace plus char with minus in version tag for GCP dev installation of UCX ([#420](#420)). * Run integration tests on shared clusters for a faster devloop ([#397](#397)). * Show difference between serverless and PRO warehouses during installation ([#385](#385)). * Split `migrate-groups` workflow into three different stages for reliability ([#442](#442)). * Use groups instead of usernames in code owners file ([#389](#389)).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR improves debuggability of UCX.
More concise error messaging from job run
Viewing logs via Databricks CLI
Added
.ucx/logsfolder in the installationwhere every executes workflow has a folder per job run, with ability to get back to Job Run in Workspace UI
Viewing full debug logs via workspace UI
Closes #86