[Feature Request] support for dynamic partition overwrite

[ivoson]

There is no support for dynamic partition overwrite for now. It seems we can only overwrite with replaceWhere option to specify the overwrite part, which is not convenient.
I am not sure why we don't have dynamic overwrite support in delta, is there any limits about it?

[tdas]

Fundamentally, dynamic partition overwrite is not a great API because it is quite scary; you can accidentally overwrite a partition with bad records falling in partitions that shouldnt have been touched. So it is always safer to use replaceWhere to provide guard rails on what partitions should get overwritten.

In the future, we can always add support for dynamic partition overwrite. But I would still recommend using replaceWhere.

[ivoson]

Got it and Thanks for your explain @tdas

[koertkuipers]

note that the risk of bad records overwriting partitions is zero if you do something like

dataframe
  .filter($"x" == 1)
  .write
  .format("delta")
  .option("partitionOverwriteMode", "dynamic")
  .partitionBy("x")
  .save("/some/location")

now you might argue that in such a case i can use replaceWhere and you are right.
but what is the logic is split up across jobs?
i can have one job doing the filtering and writing somewhere, and another job reading this data back in and doing the dynamic partition overwrite into delta. now the replaceWhere is suddenly cumbersome (the filter logic sits in another job, i now have to go repeat it), and this is still safe isn't it? we have this situation pretty frequently.

yes you can do dumb things with dynamic partition overwrite, but also very useful things.
removing it hurts adoption in my opinion.
(we added it back in internally because rewriting logic in many places wasnt an option)

[marmbrus]

It is not really that you cannot do dumb things with replaceWhere. Its that the action being taken is explicit rather than implicit.

More over, up until now, every one who has asked for the feature was happy with replaceWhere. As such its never been a priority to add it.

If you got an implementation you should consider opening a PR.

[koertkuipers]

opened PR here:
#371

[gdoron]

@tdas to be honest, breaking the normal API of spark 2, instead of .mode("overwrite") only overwrites the dynamic partitions in the df, it deletes the entire table. That's scary.

It's so easy to write overwrite when you are used to working with parquets table and forgetting adding the replaceWhere and boom, the table is gone.

Maybe that's why you added the time machine... because you knew it would happen ;)

[mblahay]

Just got bit by this because delta didn't function the same as parquet. Sure it is "dangerous," but it is also a feature that can be turned on and off depending on how one wishes to solve a particular problem. Willing to accept the risk.

[AbderrahmenM]

Any news about this please ?

[lachupacabra]

Does one know how to use the replace where filter when the data is partitioned by multiple columns?

[koertkuipers]

It is not really that you cannot do dumb things with replaceWhere. Its that the action being taken is explicit rather than implicit.

More over, up until now, every one who has asked for the feature was happy with replaceWhere. As such its never been a priority to add it.

If you got an implementation you should consider opening a PR.

@marmbrus as you suggested i opened PR... in april 2020... it would be great if a committer could look at it sometime. thanks

[tdas]

I am reopening this issue. Despite our reservations about the risk of using this feature, by community demand, we have decided to go ahead with implementing this feature.

[tdas]

@koertkuipers interested in continuing to contribute with the PR?

[koertkuipers]

yes. see:
#371

[tdas]

Cool. could you update the PR to latest master and run the tests again. lets hope it still works. further discussions on the PR.

[bkyryliuk]

is it supported now ?

[scottsand-db]

Hi @bkyryliuk - it is included in the preview for delta lake 2.0.0 - see the preview release notes here: https://github.com/delta-io/delta/releases/tag/v2.0.0rc1. It will be included in the official 2.0.0 release.