chore(rust): bump arrow v51 and datafusion v37.1

[lasantosr]

Description

Update the arrow and datafusion dependencies.

Related Issue(s)

• closes Update arrow dependency #2328

Documentation

[lasantosr]

@ion-elgreco I've fixed test_expr_sql, but test_delete_nested is still failing

It seems that the expression should have been rewritten, but I'm not sure how or where.

[ion-elgreco]

@lasantosr I dont know where this error is coming from. Likely something needs to be refactored, you could ask the datafusion devs if they have any hints

[lasantosr]

It seems it have something to do with apache/datafusion#9532, but I've just started using delta a few weeks ago, so I'm not sure if I can help here.

[ion-elgreco]

@lasantosr I'll take a look over the weekend

[ion-elgreco]

@lasantosr hey, I have taken a look but I don't see any issue. It seemed to have been a regression in datafusion, see here: apache/datafusion#10029

We have to wait on the 37.1.0 release apache/datafusion#9904

[lasantosr]

@ion-elgreco Thanks! That regression wasn't detected when I made the PR.

I can keep an eye on the 37.1.0 release and update this PR as soon as it lands.

[ion-elgreco]

@lasantosr to get this merged, we need to follow Andrews advice, and manually convert this Expr: apache/datafusion#10181 (comment)

If it's not crucial, we also can wait for datafusion 38

[lasantosr]

@ion-elgreco I've included a temporary fix with a TODO in order to remove it in v38 when the fix is ported back to datafusion core.

[ion-elgreco]

Thanks for sticking with it! 😁