Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: stream hop window executor should ignore rows with null time value #8130

Closed
st1page opened this issue Feb 22, 2023 · 4 comments · Fixed by #8146
Closed

bug: stream hop window executor should ignore rows with null time value #8130

st1page opened this issue Feb 22, 2023 · 4 comments · Fixed by #8146
Assignees
Labels
component/streaming Stream processing related issue. priority/high type/bug Something isn't working
Milestone

Comments

@st1page
Copy link
Contributor

st1page commented Feb 22, 2023

Because the hop window duplicates the rows and derives stream key with (input_pk, window start). It should not emit the rows with null time value which will generate the same window start value(null!) which can generate duplicate stream key.

@st1page st1page added type/bug Something isn't working component/streaming Stream processing related issue. priority/high labels Feb 22, 2023
@github-actions github-actions bot added this to the release-0.1.18 milestone Feb 22, 2023
@lmatz
Copy link
Contributor

lmatz commented Feb 23, 2023

May need to go through all the table functions once? And udtf in the future

@st1page
Copy link
Contributor Author

st1page commented Feb 23, 2023

May need to go through all the table functions once? And udtf in the future

only for those relational operators which could duplicate multiple rows from one input row. I have thought about other operators(Expand, ProjectSet) and do not find the same issue.
Also, this bug reminds us a too-flexible UDTF is dangerous in our system. I think we can provide a UDTF that can only accept append-only stream(relation) and output append-only stream(relation) so that we can generate row_id for each row.

@st1page st1page self-assigned this Feb 23, 2023
@mergify mergify bot closed this as completed in #8146 Feb 23, 2023
mergify bot pushed a commit that referenced this issue Feb 23, 2023
lmatz pushed a commit that referenced this issue Feb 23, 2023
@chenzl25
Copy link
Contributor

chenzl25 commented Mar 1, 2023

Is it possible to detect the null row at runtime and don't filter out this row?

@st1page
Copy link
Contributor Author

st1page commented Mar 1, 2023

Is it possible to detect the null row at runtime and don't filter out this row?

I am not sure what behavior here is proper.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/streaming Stream processing related issue. priority/high type/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants