Question on OwnAndGhostIndices initialization

[cmhamel]

Hey all, great package!

I'm trying to write a package extension for my package Exodus.jl for reading/writing exodusII FEM files. I'm trying to write a simple helper function to generate a partition from an already decomposed mesh by reading the appropriate communication maps, etc. into OwnAndGhostIndices. This data structure seemed like the most intuitive to use for my use case. I've gotten as far as setting these things up properly and can follow some of the examples to generate a pones vector. The operations consistent! and assemble! worked as expected when initializing a PVector via pones.

My question is in regard to the optional initialization parameter global_to_owner. When I don't have this set, if I try to initialize a PVector via something like b = pvector(II,VV,row_partition) I get the following error:

ERROR: LoadError: Not enough data to perform this operation without communciation.

I've traced this down in PartitionedArrays to not using this optional input. How is this input supposed to be structured? I've tried making a Dict{Int64, Int32} that maps global id to processor and also tried passing that dict through VectorFromDict but I can't seem to get anything to work.

Any help would be greatly appreciated!

[fverdugo]

Hi @cmhamel,

I would recommend not using OwnAndGhostIndices, unless really needed (arbitrary partitions). You can create a partition without ghosts with uniform_partition or variable_partition, and then add ghosts with function union_ghost. In this way, ghost ids are always numbered after own ids at the local level, and global ids are ordered by rank id. This is pretty much what PETSc considers, which should be enough in most of the cases.

If you really need a completely arbitrary partition, then use OwnAndGhostIndices. If you need to automatically discover ghost ids from the global ids like in II in your example, you need to provide global_to_owner. This is a vector containing the owner of ALL global ids. This approach is not scalable (thus to be avoided if you can), but in some applications you already have the vector global_to_owner (e.g., when you work with sequential graph partitioners), and then it is kind of acceptable.

Here you have an example of how to use global_to_owner:

PartitionedArrays.jl/src/p_range.jl

Line 752 in 07381fd

function partition_from_color(ranks,global_to_color;multicast=false,source=MAIN)

[cmhamel]

Thanks @fverdugo
I wound up hacking partition_from_color. My use case is for arbitrary partitions.

Just out of curiosity, is the need to have global_to_owner be a mapping to owner for ALL ids a fundamental weakness in MPI? Or is there some functionality not present in PartitionedArrays? I am no MPI expert so apologies if this is an obvious answer.

[fverdugo]

global_to_owner might be a very large vector and it will be replicated instead of distributed over workers. So, it can take a lot of memory. in 99% of the cases, you don't really need to build a PVector/PSparseMatrix on an arbitrary partition. You can work with 2 partitions, one arbitrary (whithout global_to_owner), and one with sorted global ids by ranks.