Update databricks-labs-lsql requirement from ~=0.2.2 to >=0.2.2,<0.4.0

[dependabot]

Updates the requirements on databricks-labs-lsql to permit the latest version.

Release notes

Sourced from databricks-labs-lsql's releases.

v0.3.0

• Added support for save_table(..., mode="overwrite") to StatementExecutionBackend (#74). In this release, we've added support for overwriting a table when saving data using the save_table method in the StatementExecutionBackend. Previously, attempting to use the overwrite mode would raise a NotImplementedError. Now, when this mode is specified, the method first truncates the table before inserting the new rows. The truncation is done using the execute method to run a TRUNCATE TABLE SQL command. Additionally, we've added a new integration test, test_overwrite, to the test_deployment.py file to verify the new overwrite mode functionality. A new option, mode="overwrite", has been added to the save_table method, allowing for the existing data in the table to be deleted and replaced with the new data being written. We've also added two new test cases, test_statement_execution_backend_save_table_overwrite_empty_table and test_mock_backend_overwrite, to verify the new functionality. It's important to note that the method signature has been updated to include a default value for the mode parameter, setting it to append by default. This change does not affect the functionality and only provides a more convenient default behavior for users of the method.

Contributors: @​william-conti

Changelog

Sourced from databricks-labs-lsql's changelog.

0.3.0

• Added support for save_table(..., mode="overwrite") to StatementExecutionBackend (#74). In this release, we've added support for overwriting a table when saving data using the save_table method in the StatementExecutionBackend. Previously, attempting to use the overwrite mode would raise a NotImplementedError. Now, when this mode is specified, the method first truncates the table before inserting the new rows. The truncation is done using the execute method to run a TRUNCATE TABLE SQL command. Additionally, we've added a new integration test, test_overwrite, to the test_deployment.py file to verify the new overwrite mode functionality. A new option, mode="overwrite", has been added to the save_table method, allowing for the existing data in the table to be deleted and replaced with the new data being written. We've also added two new test cases, test_statement_execution_backend_save_table_overwrite_empty_table and test_mock_backend_overwrite, to verify the new functionality. It's important to note that the method signature has been updated to include a default value for the mode parameter, setting it to append by default. This change does not affect the functionality and only provides a more convenient default behavior for users of the method.

0.2.5

• Fixed PyPI badge (#72). In this release, we have implemented a fix to the PyPI badge in the README file of our open-source library. The PyPI badge displays the version of the package and serves as a quick reference for users. This fix ensures the accuracy and proper functioning of the badge, without involving any changes to the functionality or methods within the project. Software engineers can be assured that this update is limited to the README file, specifically the PyPI badge, and will not affect the overall functionality of the library.
• Fixed no-cheat check (#71). In this release, we have made improvements to the no-cheat verification process for new code. Previously, the check for disabling the linter was prone to false positives when the string '# pylint: disable' appeared for reasons other than disabling the linter. The updated code now includes an additional filter to exclude the string CHEAT from the search, and the number of characters in the output is counted using the wc -c command. If the count is not zero, the script will terminate with an error message. This change enhances the accuracy of the no-cheat check, ensuring that the linter is being used correctly and that all new code meets our quality standards.
• Removed upper bound on sqlglot dependency (#70). In this update, we have removed the upper bound on the sqlglot dependency version in the project's pyproject.toml file. Previously, the version constraint required sqlglot to be at least 22.3.1 but less than 22.5.0. With this modification, there will be no upper limit, enabling the project to utilize any version greater than or equal to 22.3.1. This change provides the project with the flexibility to take advantage of future bug fixes, performance improvements, and new features available in newer sqlglot package versions. Developers should thoroughly test the updated package version to ensure compatibility with the existing codebase.

0.2.4

• Fixed Builder object is not callable error (#67). In this release, we have made an enhancement to the Backends class in the databricks/labs/lsql/backends.py file. The DatabricksSession.builder() method call in the __init__ method has been changed to DatabricksSession.builder. This update uses the builder attribute to create a new instance of DatabricksSession without calling it like a function. The sdk_config method is then used to configure the instance with the required settings. Finally, the getOrCreate method is utilized to obtain a SparkSession object, which is then passed as a parameter to the parent class constructor. This modification simplifies the code and eliminates the error caused by treating the builder attribute as a callable object. Software engineers may benefit from this change by having a more streamlined and error-free codebase when working with the open-source library.
• Prevent silencing of pylint (#65). In this release, we have introduced a new job, "no-lint-disabled", to the GitHub Actions workflow for the repository. This job runs on the latest Ubuntu version and checks out the codebase with a full history. It verifies that no new instances of code suppressing pylint checks have been added, by filtering the differences between the current branch and the main branch for new lines of code, and then checking if any of those new lines contain a pylint disable comment. If any such lines are found, the job will fail and print a message indicating the offending lines of code, thereby ensuring that the codebase maintains a consistent level of quality by not allowing linting checks to be bypassed.
• Updated _SparkBackend.fetch() to return iterator instead of list (#62). In this release, the fetch() method of the _SparkBackend class has been updated to return an iterator instead of a list, which can result in reduced memory usage and improved performance, as the results of the SQL query can now be processed one element at a time. A new exception has been introduced to wrap any exceptions that occur during query execution, providing better debugging and error handling capabilities. The test_runtime_backend_fetch() unit test has been updated to reflect this change, and users of the fetch() method should be aware that it now returns an iterator and must be consumed to obtain the desired data. Thorough testing is recommended to ensure that the updated method still meets the needs of the application.

0.2.3

• Added support for common parameters in StatementExecutionBackend (#59). The StatementExecutionBackend class in the databricks.labs.lsql package's backends.py file now supports the passing of common parameters through keyword arguments (kwargs). This enhancement allows for greater customization and flexibility in the backend's operation, as the kwargs are passed to the StatementExecutionExt constructor. This change empowers users to control the behavior of the backend, making it more adaptable to various use cases. The key modification in this commit is the addition of the **kwargs parameter in the constructor signature and passing it to StatementExecutionExt, with no changes made to any methods within the class.

0.2.2

• Updating packages. In this update, the dependencies specified in the pyproject.toml file have been updated to more recent versions. The outdated packages "databricks-labs-blueprint~=0.4.0" and "databricks-sdk~=0.21.0" have been replaced with "databricks-labs-blueprint>=0.4.2" and "databricks-sdk>=0.22.0", respectively. These updates are expected to bring new features and bug fixes to the software. The dependency sqlglot remains unchanged, with the same version requirement range of "sqlglot>=22.3.1,<22.5.0". These updates ensure that the software will function as intended, while also taking advantage of the enhancements provided by the more recent versions of the packages.

0.2.1

• Fixed row converter to properly handle nullable values (#53). In this release, the row converter in the databricks.labs.lsql.core module has been updated to handle nullable values correctly. A new method StatementExecutionExt has been added, which manages the handling of nullable values during SQL statement execution. The Row class has also been modified to include nullable values, improving the robustness and flexibility of SQL execution in dealing with various data types, including null values. These enhancements increase the overall reliability of the system, making it more production-ready.
• Improved integration test coverage (#52). In this release, the project's integration test coverage has been significantly improved through several changes. A new function, make_random(), has been added to the conftest.py file to generate a random string of fixed length, aiding in the creation of more meaningful and readable random strings for integration tests. A new file, test_deployment.py, has been introduced, containing a test function for deploying a database schema and verifying successful data retrieval via a view. The test_integration.py file has been renamed to test_core.py, with updates to the test_fetch_one function to test the fetch_one method using a SQL query with an aliased value. Additionally, a new Foo dataclass has been added to the tests/integration/views/__init__.py file, supporting integration test coverage. Lastly, a new SQL query has been added to the integration test suite, located in the some.sql file, which retrieves data from a table named foo in the inventory schema. These changes aim to enhance the overall stability, reliability, and coverage of the project's integration tests. Note: The changes to the .gitignore file and the improvements to the StatementExecutionBackend class in the backends.py file are not included in this summary, as they were described in the opening statement.
• Rely on hatch being present on the build machine (#54). In this release, we have made significant changes to how we manage our build process and toolchain configuration. We have removed the need to manually install hatch version 1.7.0 in the build machine, and instead, rely on its presence, adding it to the list of required tools in the toolchain configuration. The command to create a virtual environment using hatch has also been added, and the pre_setup section no longer includes installing hatch, assuming its availability. We have also updated the hatch package version from 1.7.0 to 1.9.4, which may include bug fixes, performance improvements, or new features. This change may impact the behavior of any existing functionality that relies on hatch. The pyproject.toml file has been modified to update the fmt and verify sections, with ruff check . --fix replacing ruff . --fix and the removal of black --check . and isort . --check-only. A new configuration for isort has also been added to specify the databricks.labs.blueprint package as a known first-party package, enabling more precise management of imports related to that package. These changes simplify the build process and ensure that the project is using a more recent version of the hatch package for packaging and distributing Python projects.
• Updated sqlglot requirement from ~=22.3.1 to >=22.3.1,=22.3.1,<22.5.0. This change enables our project to utilize bug fixes and improvements made in the latest patch versions of sqlglot, while still preventing it from inadvertently using any breaking changes introduced in version 22.5.0 or later versions. This modification allows us to take advantage of the latest features and improvements in sqlglot` while maintaining compatibility and stability in our project.

Dependency updates:

• Updated sqlglot requirement from ~=22.3.1 to >=22.3.1,<22.5.0 (#51).

0.2.0

• Added MockBackend.rows("col1", "col2")[(...), (...)] helper (#49). In this release, we have added a new helper method MockBackend.rows("col1", "col2")[(...), (...)] to simplify testing with MockBackend. This method allows for the creation of rows using a more concise syntax, taking in the column names and a list of values to be used for each column, and returning a list of Row objects with the specified columns and values. Additionally, a __eq__ method has been introduced to check if two rows are equal by converting the rows to dictionaries using the existing as_dict method and comparing them. The __contains__ method has also been modified to improve the behavior of the in keyword when used with rows, ensuring columns can be checked for membership in the row in a more intuitive and predictable manner. These changes make it easier to test and work with MockBackend, improving overall quality and maintainability of the project.

0.1.1

• Updated project metadata (#46). In this release, the project metadata has been updated to reflect changes in the library's capabilities and dependencies. The project now supports lightweight SQL statement execution using the Databricks SDK for Python, setting it apart from other solutions. The library size comparison in the documentation has been updated, reflecting an increase in the compressed and uncompressed size of Databricks Labs LightSQL, as well as the addition of a new direct dependency, SQLglot. The project's dependencies and URLs in the pyproject.toml file have also been updated, including a version update for databricks-labs-blueprint and the removal of a specific range for PyYAML.

Dependency updates:

... (truncated)

Commits

• 073c922 Release v0.3.0 (#76)
• ca137d1 Add support for save_table(..., mode="overwrite") to `StatementExecutionBac...
• 8921e0f Release v0.2.5 (#73)
• a54c330 Fix no-cheat check (#71)
• 1c00b60 Fixed PyPI badge (#72)
• 90279bf Remove upper bound on sqlglot dependency (#70)
• b840229 Release v0.2.4 (#69)
• 13f0e93 Fix 'Builder' object is not callable error (#67)
• 2f3a6aa Prevent silencing of pylint (#65)
• 0e9ac6a Fixed nightly tests (#63)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.73%. Comparing base (86adf16) to head (cc61b1f).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1137      +/-   ##
==========================================
- Coverage   89.73%   89.73%   -0.01%     
==========================================
  Files          61       61              
  Lines        7230     7229       -1     
  Branches     1299     1298       -1     
==========================================
- Hits         6488     6487       -1     
  Misses        478      478              
  Partials      264      264              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.