Feature: transform during load

[youngsofun]

Summary

todo

• RFC
• support named stage: feat: support copy into table with transform. #10561
• support uri
• support part of columns(maybe reordered): feat: support columns for COPY INTO <TABLE> #10582
• support filter， is it reasonable?
• distributed
• refactor: parquet copy into table all use ParquetTable Instead of StageTable, (parquet unload and streaming load still need StageTable)
  • try to avoid OOM after the refactor

the following is out-of-date, we mainly rely on copy from select

• auto cast

  • for text format, more rules is trivial.
  • for parquet, need some rules or just all built-in cast

• copy from select (much more general)

  • copy into is easier to impl with subquery since we already support select from stage.
    • need Atomicity
  • streaming_load is harder, since it is syntax is special

• insert to view( the view do some transform before insert to table)

[youngsofun]

for streaming load:

what about add a table function stream :
Insert into table 1 from ( select * from stream(format=>’parquet’) )

---

but streaming_load has another issue:

schema of file cannot be inferred during parse...

while as a table to select, we always need some schema in advance.

which means for streaming_load

1. the subquery approach can not works.
2. the cast need todo cannot be determined during parse, while copy into can, which may lead to 2 diff impl