Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvement of UDF/UDTF operators #32

Closed
Fanoid opened this issue Dec 27, 2019 · 0 comments
Closed

Improvement of UDF/UDTF operators #32

Fanoid opened this issue Dec 27, 2019 · 0 comments
Assignees

Comments

@Fanoid
Copy link
Collaborator

Fanoid commented Dec 27, 2019

Hi,
I found the current implementation of UDF/UDTF operators has the following limitations:

  • Not aligned with corresponding Flink feature: only 1 selectedCol is used and no joinType supported in UDTF, and case-sensitivity of identifier breaks in UDTFBatchOp by MappingColNameIgnoreCase.
  • No parameter MLEnvrionmentId provided.
  • Incomplete JavaDoc and unit tests.
  • Redundant codes.

Moreover, in current codes, if a same column name is used in selectedCol(s), reservedCols, and outputCols, the related codes are not very clear.
Maybe we can re-phrase codes to 3 rules, and the implementation should align to these rules:

  1. If reservedCols is not set or set to null, then all columns in the input table are automatically reserved except for cases in rule 3. In this way, the behaviors of reservedCols in UDF/UDTF are consistent to other operators.
  2. For UDTF, if a same name is used in selectedCol(s) and outputCols, the usage should be allowed. The conflicts exist due to the cross join or left outer join clause. However, because the implementation details are hidden for users, we should handle such conflicts within the operators.
  3. If a same name is used in reservedCols and outputCols, then the one representing a column in the input table will not exist in the output table.
@Fanoid Fanoid self-assigned this Dec 27, 2019
shaomengwang pushed a commit that referenced this issue Feb 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants