Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(hogql): null-tolerant concat #15975

Merged
merged 8 commits into from
Jun 9, 2023
Merged

feat(hogql): null-tolerant concat #15975

merged 8 commits into from
Jun 9, 2023

Conversation

mariusandra
Copy link
Collaborator

@mariusandra mariusandra commented Jun 9, 2023

Problem

  1. concat-ing any null value makes everything null
  2. concat-ing any numbers or booleans makes everything crash

We likely want to store null properties as nulls, but concat messes this up.

Changes

Makes concat null-tolerant (sounds the opposite of what it is). Now the following:

concat(null, 'a', 3, toString(4), toString(NULL))

gets automatically converted to

concat('', 'a', toString(3), toString(4), '')

Also adds the piped concat operator:

properties.$screen_width || 'x' || properties.$screen_height

Actual property access inside a ClickHouse SQL string while inside a concat looks kind of dorky, but should be accurate. We can definitely simplify this further if we need to.

-- hogql
concat(properties.$screen_width, 'x', properties.$screen_height)

-- clickhouse
concat(ifNull(toString(replaceRegexpAll(nullIf(nullIf(JSONExtractRaw(events.properties, '$screen_width', ''), 'null'), '^\"|\"$', '')), ''), 'x', ifNull(toString(replaceRegexpAll(nullIf(nullIf(JSONExtractRaw(events.properties, '$screen_height'), ''), 'null'), '^\"|\"$', '')), ''))

Questions

Do we want this? I'd consider it a big UX improvement. Otherwise I need to put the ifNull-s everywhere myself, as I can never be too sure every property is always set.

We can alternatively make properties always take default values, but that has its own issues and is not concluded yet. This fix seems like a happy middle ground bandage.

Alternatives

Alternatively we can implement a new type of non-ANSI compliant string operator, e.g. f'{NULL}-haha-{properties.bla}', but I'd keep the operators standard and be flexible with the types instead.

How did you test this code?

Added tests

@mariusandra mariusandra requested a review from a team June 9, 2023 10:46
Copy link
Collaborator

@Twixes Twixes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Twixes
Copy link
Collaborator

Twixes commented Jun 9, 2023

Well, a test is failing, so maybe not

@mariusandra
Copy link
Collaborator Author

mariusandra commented Jun 9, 2023

It was a pretty nice failure

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants