Support mutating functions

[mohamed82008]

It was brought up on Slack that we may need to think about supporting a mutating API where buffers are stored in a more systematic abstract way. I am opening this issue to track the problem and discuss solutions.

[gdalle]

Bumping this because at the moment it is the main performance limit for ImplicitDifferentiation.jl

[blegat]

Maybe something like MutableArithmetics.operate_to!(::typeof(gradient), ::AbstractBackend, f, xs...)

[ChrisRackauckas]

This also goes with the issue on pre-caching, #41, because if you use in-place but you aren't passing the shadow vectors or the config directly then you still don't get performance. This is the main reason why SciML hasn't been able to adopt AD.jl yet, even though SciMLSensitivity.jl would be a prime candidate it's just too much of a performance hit.

[mohamed82008]

The problem is that most AD packages don't have any special support for mutation as far as I know. I think we need a few MWEs showcasing how mutation would work for each AD package and demonstrating the benefit of doing that. When working with AD, I am generally operating under the assumption that allocations are negligible compared to the cost of the AD. Most AD packages are not even type stable so they allocate a ton already when doing dynamic dispatch. Enzyme changed things a little and is the most greedy AD package in that it can really take advantage of mutation. ForwardDiff too if you know what you are doing and maybe ReverseDiff with a compiled tape. But I think we need a few examples showing the performance difference of pre-allocating a result vs allocating a new result every time.