chore: Upgrade arrow/datafusion/deltalake/lance #2429
Conversation
vrongmeal
force-pushed
the
vrongmeal/upgrade
branch
from
42afbeb to
b00c862
Compare
vrongmeal
changed the title
| ast::DataType::Int64 | ||
| | ast::DataType::Float64 | ||
| | ast::DataType::Nvarchar(_) | ||
| | ast::DataType::JSON | ||
| | ast::DataType::Uuid | ||
| | ast::DataType::Binary(_) | ||
| | ast::DataType::Bytes(_) |
There was a problem hiding this comment.
Do we want to add these datatypes? I excluded them since Binary was already excluded…
| ast::DataType::Array(ast::ArrayElemTypeDef::AngleBracket(inner_sql_type)) | ||
| | ast::DataType::Array(ast::ArrayElemTypeDef::SquareBracket(inner_sql_type)) => { |
There was a problem hiding this comment.
Supporting both is OK?
|
Resolving bugs. There are changes in deltalake that remove |
Somehow there’s an extra log that resets the table schema to nothing: {"metaData":{"id":"06d0a73d-b29a-468e-a914-e018559ce305","name":"t1","description":null,"format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[]}","partitionColumns":[],"createdTime":1705658425682,"configuration":{}}}
{"commitInfo":{"timestamp":1705658425687,"operation":"WRITE","operationParameters":{"mode":"Overwrite"},"clientVersion":"delta-rs.0.17.0"}} |
|
Thinking of resetting lance to the latest release (instead of the latest commit) so we can work with existing |
|
So, the |
vrongmeal
force-pushed
the
vrongmeal/upgrade
branch
from
b00c862 to
4b8588f
Compare
Signed-off-by: Vaibhav <vrongmeal@gmail.com>
|
|
||
| # Runs pytest in the tests directory. | ||
| pytest *args: | ||
| poetry -C tests run pytest --rootdir={{invocation_directory()}}/tests {{ if args == "" {'tests'} else {args} }} | ||
| pytest *args: |
There was a problem hiding this comment.
previously this required poetry to be directly installed. Now it properly uses a virtual env.
|
AFAICT, there are 2 main followups for this upgrade.
|
can you open tickets for these just so we don't lose track of them? |
done |
| @@ -296,7 +296,7 @@ jobs: | |||
| tests/.venv/ | |||
| key: ${{ runner.os }}-poetry-${{ hashFiles('**/Cargo.lock') }} | |||
| - name: setup pytest | |||
| run: just pytest-setup | |||
| run: just venv | |||
There was a problem hiding this comment.
I did this because I wanted to keep the virtualenvs separate for the python tests and the bindings.
There was a problem hiding this comment.
I did this because I wanted to keep the virtualenvs separate for the python tests and the bindings.
I have an idea for this one. Ill open up a followup pr for this shortly.
🤞 --------- Signed-off-by: Vaibhav <vrongmeal@gmail.com> Co-authored-by: universalmind303 <cory.grinstead@gmail.com>
🤞