Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

L044, L045: Handle Exasol VALUES clause #2400

Merged

Conversation

barrywhart
Copy link
Member

@barrywhart barrywhart commented Jan 21, 2022

Brief summary of the change made

Fixes #2362

Enhances SelectCrawler class to interpret VALUES clause in a CTE as an equivalent SELECT.

Are there any other side effects of this change that we should be aware of?

Pull Request checklist

  • Please confirm you have completed any of the necessary steps below.

  • Included test cases to demonstrate any code changes, which may be one or more of the following:

    • .yml rule test cases in test/fixtures/rules/std_rule_cases.
    • .sql/.yml parser test cases in test/fixtures/dialects (note YML files can be auto generated with tox -e generate-fixture-yml).
    • Full autofix test cases in test/fixtures/linter/autofix.
    • Other.
  • Added appropriate documentation for the change.

  • Created GitHub issues for any relevant followup/future enhancements if appropriate.

@barrywhart barrywhart marked this pull request as draft January 21, 2022 13:37
@barrywhart barrywhart marked this pull request as ready for review January 21, 2022 13:39
@codecov
Copy link

codecov bot commented Jan 21, 2022

Codecov Report

Merging #2400 (ddc3e2a) into main (7312879) will increase coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #2400   +/-   ##
=======================================
  Coverage   99.93%   99.93%           
=======================================
  Files         162      162           
  Lines       11522    11529    +7     
=======================================
+ Hits        11514    11521    +7     
  Misses          8        8           
Impacted Files Coverage Δ
src/sqlfluff/core/dialects/common.py 100.00% <100.00%> (ø)
src/sqlfluff/core/rules/analysis/select_crawler.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7312879...ddc3e2a. Read the comment docs.

Comment on lines 518 to 526
test_pass_exasol_values_clause_cte:
pass_str: |
WITH txt AS (
VALUES (1)
AS t (id)
)

SELECT *
FROM txt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't valid snowflake/postgres/mysql
image
so maybe use

configs:
    core:
      dialect: exasol

in these test cases

Copy link
Contributor

@jpy-git jpy-git Jan 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This however is valid snowflake/postgres (although annoyingly doesn't parse in postgres atm) and also raises the same L044 issue:

  • On main branch
    image

  • On your branch:
    image

L044 still there and catastophic failures 😱 🔥

My guess is that L025 and L026 are also rules that use your select_crawler and this values change has messed those up (at least in this case)

Copy link
Contributor

@jpy-git jpy-git Jan 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I'll fix the postgres parsing for my example above since I know exactly where I've accidently introduced this, but that's not related to the above issue 😄 )
EDIT: Addressed in #2401

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Came up with the equivalent MySQL test case but it won't parse so I've added it into issue #2402. I have a feeling if it could parse then it would likely be solved by your fix for the postgres/snowflake example so we can leave it for a later date for now.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, I'll add the exasol config. I intended to do that, but forgot. Will also add the same test for snowflake dialect.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a postgres test case as well.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jpy-git: Think we could create a separate issue for your SELECT * test case? I agree it should be fixed, but I think it's less likely to occur in production code. Happy to look into it, but I'd rather address it later (potentially next release).

Copy link
Contributor

@jpy-git jpy-git Jan 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm it's kind of the same as the existing test case just in a subquery rather than a CTE, and we'd be introducing catastrophic errors in rules L025 + L026. So don't think we should be merging this before at least the L025 + L026 errors are fixed.

Worst case this PR runs into next release so don't think we should rush.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try and take a look this afternoon. I often have a couple of hours to work, then I don't always know when I'll have time again. 👶

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jpy-git: I found a parsing issue with certain uses of VALUES. I created an issue in case you want to look at it: #2406. For now, I'll focus on the ones that parse correctly.

@jpy-git jpy-git mentioned this pull request Jan 21, 2022
barrywhart and others added 2 commits January 21, 2022 15:01
Co-authored-by: Joseph Young <80432516+jpy-git@users.noreply.github.com>
Co-authored-by: Joseph Young <80432516+jpy-git@users.noreply.github.com>
@barrywhart barrywhart marked this pull request as ready for review January 21, 2022 21:03
@barrywhart
Copy link
Member Author

@jpy-git, @tunetheweb: I believe I've addressed the PR feedback, including the additional fixes.

The only failure(s) I'm aware of now are caused by issue #2406 (which I created today). That's apparently a dialect issue which I think we should address separately. No other changes to SelectCrawler or the rules should be necessary once that's fixed; things will "just work".

Copy link
Contributor

@jpy-git jpy-git left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@barrywhart Looks good, I've tested and can confirm those L025/L026 error have gone too 👍 LGTM 🚀 ✨

As you said I think once we get that remaining Postgres/MySQL/Exasol Values grammar parsing, they should be handled by this rule in the exact same way. I'm looking at that in #2404 so will try and add extra unit tests to that to cover these statements

@jpy-git jpy-git merged commit 9f8d8d6 into sqlfluff:main Jan 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

L044 unexpected exception, L045 CTE not used?
3 participants