Non-overlapping Distributed Assembly

[termi-official]

In this PR a prototype for distributed assembly on non-overlapping mesh decompositions is proposed

CAUTION - HIGH RAM USAGE Should be under control now.

Open Problems/TODOs

• Non-local interaction entries in distributed matrix missing (part of ghost layer)
• Put distributed code into a abstraction layer
• RHS assembly not working currently (non implemented error in PartitionedArrays when touching the RHS vector)
• Constraint handler integration
• Fix high memory memory usage
• Proper naming of variables 👻
• Manufactured solution test instead of norm
• Generalize to higher order
• Allow partitions to be disconnected from the boundary
• Get vector valued problem running
• Get rid of the just included celldofs/facedofs/vertexdofs/edgedofs dics
• Merge master
• HYPRE assembler
• Fix regression in constraint handler
• Split into 3 extensions (FerriteDistributed, FerriteHypre, FerritePartitionedArrays)
• Proper test coverage
• Merge master
• Changelog entry

Removed TODOs

• Remove unnecessary communication Compression and stuff is future work...
• Ghost layer construction via codim-1 entities (should be close to the lower bound in terms of communication) Wont't work for mixed meshes.
• Optimized exchanger construction (for PartitionedArrays.jl)
• Update to PartitionedArrays 0.3.0 Should be a follow up PR
• Backport to Julia 1.6 (use SciML/DiffEqBase.jl@c2de920 for help) Follow up if we want.

Out of scope (stuff for future PRs)

• Mixed grids
• We do not fully utilize the potential of PartitionedArrays.jl
• Parallel mesh IO
• CSR/CSC assembly
• Reblance operations
• Space filling curve based partitioning
• Optimized ghost layer construction
• Parallel condensation (matrix product infrastructure missing in PartitionedArrays.jl)
• Optimized generate_distributed_grid
• Periodic BC handling
• Distributed Point finder

[termi-official]

System matrix assembly for linear ansatz seems to work for all cases now. I will leave higher order interpolations for later, because there are more some fundamental problems which have to be resolved first. Also note that this is not an AMR ready implementation, as distributed AMR meshes usually work differently from the presented one.

Pulling in @fredrikekre @koehlerson for some help on design decisions and simplifying the algorithms - I will start cleaning up stuff and building proper abstraction layers after we have settled for fundamental design decisions. Please do not push directly into this branch if you want to change something. If you want to try the PR locally, please note that at the time of writing this the MPI.jl master branch is mandatory.

1. Dof management becomes is currently weird, partly because we do not materialize the mesh. I moved some close code to the dof handler https://github.com/Ferrite-FEM/Ferrite.jl/blob/do/distributed-assembly/src/Dofs/DofHandler.jl#L25-L27 . Also, I have an almost duplicate code for the dof assignment https://github.com/Ferrite-FEM/Ferrite.jl/blob/do/distributed-assembly/docs/src/literate/distributed_assembly.jl#L172-L232 .
2. Ghost layer is literally a nightmare right now. The communication overhead is quite big and the algorithmic complexity seems to be higher than necessary.
3. The grid data structure seems to be a main source for suboptimal algorithms, because nothing is materialized or linked from the elements https://github.com/Ferrite-FEM/Ferrite.jl/blob/do/distributed-assembly/src/Grid/DistributedGrid.jl#L10-L55

[termi-official]

This one is blocked until #624 is decided. I will take care of merging the new master afterwards.

We also should discuss whether this should become a separate package with the different assemblers as extensions, or if we should keep this as a Ferrite extension with transient extensions on the assemblers (see JuliaLang/julia#48734 for the corresponding feature request).

[termi-official]

Superseded by Ferrite-FEM/FerriteDistributed.jl#7