You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi,
I found the current implementation of UDF/UDTF operators has the following limitations:
Not aligned with corresponding Flink feature: only 1 selectedCol is used and no joinType supported in UDTF, and case-sensitivity of identifier breaks in UDTFBatchOp by MappingColNameIgnoreCase.
No parameter MLEnvrionmentId provided.
Incomplete JavaDoc and unit tests.
Redundant codes.
Moreover, in current codes, if a same column name is used in selectedCol(s), reservedCols, and outputCols, the related codes are not very clear.
Maybe we can re-phrase codes to 3 rules, and the implementation should align to these rules:
If reservedCols is not set or set to null, then all columns in the input table are automatically reserved except for cases in rule 3. In this way, the behaviors of reservedCols in UDF/UDTF are consistent to other operators.
For UDTF, if a same name is used in selectedCol(s) and outputCols, the usage should be allowed. The conflicts exist due to the cross join or left outer join clause. However, because the implementation details are hidden for users, we should handle such conflicts within the operators.
If a same name is used in reservedCols and outputCols, then the one representing a column in the input table will not exist in the output table.
The text was updated successfully, but these errors were encountered:
Hi,
I found the current implementation of UDF/UDTF operators has the following limitations:
and no joinType supportedin UDTF, and case-sensitivity of identifier breaks in UDTFBatchOp by MappingColNameIgnoreCase.No parameter MLEnvrionmentId provided.Moreover, in current codes, if a same column name is used in selectedCol(s), reservedCols, and outputCols, the related codes are not very clear.
Maybe we can re-phrase codes to 3 rules, and the implementation should align to these rules:
The text was updated successfully, but these errors were encountered: