fix(sql): unable to filter text with quotes #17881

stephenLYZ · 2021-12-28T16:38:32Z

SUMMARY

This PR fixes the problem that the double/single quote will be dropped in SQL query when using filters.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Before

After

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Has associated issue: Fixes Wrong SQL generated when filter is used #11639
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

codecov · 2022-01-01T16:18:47Z

Codecov Report

Merging #17881 (beefae0) into master (ea2d668) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master   #17881   +/-   ##
=======================================
  Coverage   66.05%   66.05%           
=======================================
  Files        1591     1591           
  Lines       62423    62423           
  Branches     6287     6287           
=======================================
  Hits        41233    41233           
  Misses      19568    19568           
  Partials     1622     1622

Flag	Coverage Δ
hive	`52.17% <100.00%> (ø)`
mysql	`81.28% <100.00%> (ø)`
postgres	`81.33% <100.00%> (ø)`
presto	`52.01% <100.00%> (ø)`
python	`81.77% <100.00%> (ø)`
sqlite	`81.03% <100.00%> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/connectors/base/models.py	`88.58% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ea2d668...beefae0. Read the comment docs.

villebro

Thanks for looking into this. I went through the git blame logs, and it appears this change was added in #4651, so a very long time ago. In addition, it was originally prefixed with the comment "backward compatibility with previous components", so I'm not entirely sure we need this anymore - maybe just handle the special cases '' and "", but even those seem dubious. I wonder if @betodealmeida or @hughhhh happen to still remember this case? Also, the unit tests should probably not be implemented using DruidColumn, as the whole native Druid connector is in the process of being deprecated. Ping me if you want to discuss an alternative testing solution.

zhaoyongjie · 2022-01-05T09:10:17Z

Hi @villebro . I have discussed this issue with @stephenLYZ. This issue requires more consideration.

different database dialects
how to escape reserved words

There is a virtual dataset (for Postgresql) for test this issue.

SELECT 
'"text in double quotes"' as name
UNION
SELECT '''text in singlequotes''' as name
UNION
SELECT 'double quotes " in text' as name
UNION
SELECT 'single quotes '' in text' as name

stephenLYZ · 2022-01-05T10:41:15Z

Yeah. I tried to use it locally. When I try to filter name = '"text in double quotes"', the response data seems to be not correct.

superset/connectors/base/models.py

rusackas · 2022-01-25T16:22:20Z

@stephenLYZ are there more tests to be added to this PR before final review? CC @zhaoyongjie

zhaoyongjie · 2022-01-28T11:00:06Z

tests/integration_tests/sqla_models_tests.py

            ),
            database=get_example_database(),
        )
        TableColumn(column_name="foo", type="VARCHAR(255)", table=table)
        SqlMetric(metric_name="count", expression="count(*)", table=table)
+        yield table


table wasn't persisted so we no need delete command.

villebro

LGTM - good test (+115 +61) to functional code change ratio (+1 -1) 😄

serenajiang · 2022-02-08T19:11:21Z

@villebro @stephenLYZ While this fix is definitely a good direction, can we make sure to add this to UPDATING.md as a breaking change, since it changes how string filters work?

After deploying this fix, we found out that a lot of our users had been wrapping strings in quotes, so a ton of important charts broke. I think the previous behavior made some people think that you had to wrap string filter values in quotes.

villebro · 2022-02-08T22:10:19Z

@serenajiang thanks for flagging this and apologies for the inconvenience. This was a good reminder to always flag breaking changes in UPDATING.md, even if they may seem like pure bugfixes from a developer's perspective. I'll open a PR shortly and be more vigilant going forward.

zhaoyongjie · 2022-02-09T03:11:19Z

@serenajiang @villebro
Thanks for the mention and sorry to introduce this broken change. In addition to update UPDATING.md, should we open a backport feature flag to ensure compatibility?

pull-request-size bot added the size/XS label Dec 28, 2021

stephenLYZ marked this pull request as draft January 1, 2022 16:11

stephenLYZ marked this pull request as ready for review January 1, 2022 16:11

pull-request-size bot added size/S and removed size/XS labels Jan 2, 2022

villebro reviewed Jan 4, 2022

View reviewed changes

rusackas requested a review from betodealmeida January 7, 2022 02:17

betodealmeida reviewed Jan 7, 2022

View reviewed changes

superset/connectors/base/models.py Outdated Show resolved Hide resolved

zhaoyongjie reviewed Jan 21, 2022

View reviewed changes

superset/connectors/base/models.py Outdated Show resolved Hide resolved

pull-request-size bot added size/XS and removed size/S labels Jan 21, 2022

pull-request-size bot added size/M and removed size/XS labels Jan 26, 2022

zhaoyongjie mentioned this pull request Jan 27, 2022

fix: null value and empty string in filter #18171

Merged

9 tasks

stephenLYZ added 10 commits January 28, 2022 14:42

fix(sql): missing quotes in sql

65a46c0

fix

486e1e1

add test

fa1ce26

fix test

d558865

rm

26a9d98

rm unused import

d61d629

fix ut

610278c

fix unused ut

bf3dce2

fix ut

0506ddf

add ut

7d23b25

zhaoyongjie force-pushed the fix-wrong-sql branch from ebd49c5 to 7d23b25 Compare January 28, 2022 07:18

add UT

2405701

pull-request-size bot added size/L and removed size/M labels Jan 28, 2022

zhaoyongjie reviewed Jan 28, 2022

View reviewed changes

zhaoyongjie changed the title ~~fix(sql): missing quotes in sql~~ fix(sql): unable to filter text in quotes Jan 28, 2022

zhaoyongjie changed the title ~~fix(sql): unable to filter text in quotes~~ fix(sql): unable to filter text with quotes Jan 28, 2022

minor fix

beefae0

villebro approved these changes Jan 28, 2022

View reviewed changes

zhaoyongjie merged commit 9671384 into apache:master Jan 28, 2022

shcoderAlex pushed a commit to casual-precision/superset that referenced this pull request Feb 7, 2022

fix(sql): unable to filter text with quotes (apache#17881)

aafc12f

ofekisr pushed a commit to nielsen-oss/superset that referenced this pull request Feb 8, 2022

fix(sql): unable to filter text with quotes (apache#17881)

ebde45a

villebro mentioned this pull request Feb 10, 2022

chore(docs): add note about quote handling in filters #18657

Merged

9 tasks

bwang221 pushed a commit to casual-precision/superset that referenced this pull request Feb 10, 2022

fix(sql): unable to filter text with quotes (apache#17881)

8c122f8

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.5.0 labels Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(sql): unable to filter text with quotes #17881

fix(sql): unable to filter text with quotes #17881

stephenLYZ commented Dec 28, 2021 •

edited by zhaoyongjie

Loading

codecov bot commented Jan 1, 2022 •

edited

Loading

villebro left a comment

zhaoyongjie commented Jan 5, 2022 •

edited

Loading

stephenLYZ commented Jan 5, 2022 •

edited

Loading

rusackas commented Jan 25, 2022

zhaoyongjie Jan 28, 2022

villebro left a comment

serenajiang commented Feb 8, 2022

villebro commented Feb 8, 2022 •

edited

Loading

zhaoyongjie commented Feb 9, 2022 •

edited

Loading

fix(sql): unable to filter text with quotes #17881

fix(sql): unable to filter text with quotes #17881

Conversation

stephenLYZ commented Dec 28, 2021 • edited by zhaoyongjie Loading

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Before

After

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

codecov bot commented Jan 1, 2022 • edited Loading

Codecov Report

villebro left a comment

Choose a reason for hiding this comment

zhaoyongjie commented Jan 5, 2022 • edited Loading

stephenLYZ commented Jan 5, 2022 • edited Loading

rusackas commented Jan 25, 2022

zhaoyongjie Jan 28, 2022

Choose a reason for hiding this comment

villebro left a comment

Choose a reason for hiding this comment

serenajiang commented Feb 8, 2022

villebro commented Feb 8, 2022 • edited Loading

zhaoyongjie commented Feb 9, 2022 • edited Loading

stephenLYZ commented Dec 28, 2021 •

edited by zhaoyongjie

Loading

codecov bot commented Jan 1, 2022 •

edited

Loading

zhaoyongjie commented Jan 5, 2022 •

edited

Loading

stephenLYZ commented Jan 5, 2022 •

edited

Loading

villebro commented Feb 8, 2022 •

edited

Loading

zhaoyongjie commented Feb 9, 2022 •

edited

Loading