Schema Evolution not working as expected

[pyjads]

Environment

macos
Delta-rs version:
0.17.3

I am getting following error:

ValueError: Schema of data does not match table schema
Data schema:
id: int64
name: string
age: int64
address: string
Table Schema:
id: int64
name: string
age: int64

import pyarrow as pa
from deltalake import DeltaTable, write_deltalake

schema = pa.schema([
    pa.field('id', pa.int64()),
    pa.field('name', pa.string()),
    pa.field('age', pa.int64())
])

new_schema = pa.schema([
        pa.field('id', pa.int64()),
        pa.field('name', pa.string()),
        pa.field('age', pa.int64()),
        pa.field('address', pa.string())
    ])

def data_with_schema():
    return pa.Table.from_pydict({
        'id': [1, 2, 3],
        'name': ['Alice', 'Bob', 'Charlie'],
        'age': [34, 55, 23]
    }, schema=schema)

def data_with_new_schema():
    return pa.Table.from_pydict({
        'id': [1, 2, 3],
        'name': ['Alice', 'Bob', 'Charlie'],
        'age': [34, 55, 23],
        'address': ['123 Main', '456 Elm', '789 Maple']
    }, schema=new_schema)
    
if __name__ == '__main__':
    dd = '/.tmp/some_test'
    
    data = data_with_schema()
    write_deltalake(
        dd,
        data,
        mode='append',
        # partition_by=['id'],
        overwrite_schema=True,
        # schema_mode='overwrite',
    )
    
    data = data_with_new_schema()
    write_deltalake(
        dd,
        data,
        mode='append',
        # partition_by=['id'],
        overwrite_schema=True,
        # schema_mode='overwrite',
    )
    
    # table = DeltaTable(dd)
    # print(table.schema())
    
    # table = DeltaTable.create(dd, schema=new_schema, mode='append')
    # print(table.to_pandas().head())

When there is a new column, DeltaTable throws an error. Schema Evolution is not working as expected. Or am I doing something wrong?

How can I add new column without adding data as well?

[ion-elgreco]

Schema evolution is only supported with engine="rust"

[pyjads]

I then get following error.

_internal.DeltaError: Generic DeltaTable error: Schema overwrite not supported for Append

[ion-elgreco]

Correct, that's not supported. You need to set schema_mode="merge"

[pyjads]

Thank you so much, Could you please also let me know, how can I alter the schema without adding data? Is this possible?
Is it possible to update the datatype of existing column without losing data?

[ion-elgreco]

You can't at the moment