Representing gradient of constraints as tangents on a power manifold

[mateuszbaran]

It would be nice to support vectorized gradient of constraints, i.e. instead of passing a vector of functions for gradients, pass a single function that returns all gradients as a tangent vector on a power manifold in a user-selected representation. It would likely be faster and make interfacing via Optimization.jl easier. See https://docs.sciml.ai/Optimization/stable/API/optimization_function/ .

[kellertuer]

Currently there are

• a set of functions, each returning a gradient / Tangent vector
• one function returning a vector of tangent vectors, which is what you what I think? The problem of evaluating this if you just need one gradient is the necessary effort. This is also the case if you transfer this to a gradient on the power manifold. So I would always prefer the first one.

Besides that, the second point would just need a change of its representation to be one tangent vector on a power manifold (it is already in the nested sense).

[mateuszbaran]

I mostly meant the array power representation of power manifold but maybe Optimization.jl would accept Jacobians represented by nested arrays, I will have to check.

[kellertuer]

Sure, the reason why it is how it is, is that this way currently both implemented variants (yours would be a third then) yield the same and the one with an array of functions is quite benificial for the Gradient of the Exact Penalty Method, where not all gradients need to be evaluated but only a (maybe even only small) subset. That is when just one large function might be slow.

[mateuszbaran]

I see, I think I will skip such constraints in OptimizationManopt.jl for the first version.

[kellertuer]

To add to this, since I am anyways rewriting parts of the constraint objective for another task, it might also be nice to keep f, g, and h (cost, inequality and equality constraints) as their own objetives within the constrained objective, so that it is a bit more flexible which information we store for them. Most prominently maybe a Hessian of f, but also for g and h second order information might be “nice to have”.