Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure that USE statements are recognized and apply to table references without a qualifying schema in SQL and pyspark #1433

Merged
merged 12 commits into from
Apr 22, 2024
Merged

Ensure that USE statements are recognized and apply to table references without a qualifying schema in SQL and pyspark #1433

merged 12 commits into from
Apr 22, 2024

Commits on Apr 22, 2024

  1. feat: Implement FromTable changes to account for USE statements in SQL 

    Signed-off-by: Jim.Idle <jimi@idle.ws>
    @jimidle
    jimidle committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    5a1c990 View commit details
    Browse the repository at this point in the history

  2. feat: Ensure that linter does not exclude tables 

    Signed-off-by: Jim.Idle <jimi@idle.ws>
    @jimidle
    jimidle committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    9ab38f5 View commit details
    Browse the repository at this point in the history

  3. feat: use consistent names for catalog.schema.table 

    Signed-off-by: Jim.Idle <jimi@idle.ws>
    @jimidle
    jimidle committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    9b61753 View commit details
    Browse the repository at this point in the history

  4. feat: Setup basis of all linters and migrators to track the current s… 

    …chema

  - All linters now have the ability to track the schema they last saw
  - All migrators also do this

Next step is to add discovery of the schema change in calls such as spark.sql

Signed-off-by: Jim.Idle <jimi@idle.ws>
    @jimidle
    jimidle committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    da94748 View commit details
    Browse the repository at this point in the history

  5. feat: Fix make fmt reports :( 

    Signed-off-by: Jim.Idle <jimi@idle.ws>
    @jimidle
    jimidle committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    5a74250 View commit details
    Browse the repository at this point in the history

  6. feat: All liners and migrators now track USE statements. Notebooks be… 

    …tween cells.

TODO: Implement tests - there are basically no tests for NOtebook migrator actually migrating, only
tests of smaller parts.

Signed-off-by: Jim.Idle <jimi@idle.ws>
    @jimidle
    jimidle committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    16b17e5 View commit details
    Browse the repository at this point in the history

  7. feat: Change implementation to provide a CurrentSessionState class to… 

    … track schema changes

  - Removes the need to change the LInter interface
  - Injects new instance of CurrentSessionState into Languages
  - Adds test of Notebook linting
  - More tests to follow shortly

Signed-off-by: Jim.Idle <jimi@idle.ws>
    @jimidle
    jimidle committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    823daea View commit details
    Browse the repository at this point in the history

  8. feat: Adds extra tests for making sure all spark func calls are cover… 

    …ed and use is tracked

Signed-off-by: Jim.Idle <jimi@idle.ws>
    @jimidle
    jimidle committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    e869167 View commit details
    Browse the repository at this point in the history

  9. feat: Changes to conform to code review 

    Signed-off-by: Jim.Idle <jimi@idle.ws>
    @jimidle
    jimidle committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    7b30306 View commit details
    Browse the repository at this point in the history

  10. fix: Remove redundant reference to DEFAULT_SCHEMA 

    Signed-off-by: Jim.Idle <jimi@idle.ws>
    @jimidle
    jimidle committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    d9c87d1 View commit details
    Browse the repository at this point in the history

  11. fix: Changes NoteBook linter to remove empty string. 

    Reverts back to the how it was, and does not implement Linter, for the same reasons
that NotebookMigrator does not implement Fixer. They are controllers for linting/fixing
rather than Linter/Fixer themselves.

Signed-off-by: Jim.Idle <jimi@idle.ws>
    @jimidle
    jimidle committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    bc541b8 View commit details
    Browse the repository at this point in the history

  12. feat: Rebase branch on main to add an extra advice that was not raise… 

    …d in the feature branch

  - A new Advice was added warning about the default format change from Parquet to Delta.
  - This branch did not have that change, so it causesd the CI to fail
  - A rebase (not merge) from main was needed

Signed-off-by: Jim.Idle <jimi@idle.ws>
    @jimidle
    jimidle committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    5f2d13e View commit details
    Browse the repository at this point in the history