Provide more flexible ways to add subset MeshBlockData objects to DataCollections #921

jdolence · 2023-08-11T17:37:34Z

PR Summary

There are lots of examples where it would be useful to have more flexibility in adding new MeshBlockData objects to DataCollections that are slices of other MeshBlockData objects (like "base"). We already sort of support this via the overload to DataCollection::Add that includes a list of names of variables, but in its current form this is of limited use in practice. This PR attempts to make this much more flexible and useful.

The main features included in this PR are:

New overloads to add MeshBlockData or MeshData objects to DataCollections. The preexisting versions that take (label, source) or (label, source, var_names) still exist and function as before. In addition, an AddShallow function (with the same overloads) was added that yields a shallow copy, even for non-Metadata::OneCopy fields.
It now works to Add directly to the Mesh-level DataCollection. When Add is called with a MeshData source object, the new MeshBlockData objects that get created get added to the MeshBlock level DataCollections. The implementation is a bit of a hack, but it works for now and makes things much cleaner downstream.
StateDescriptor now has an overloaded GetVariablesNames member function that returns a std::vector<std::string> of variable names given a list of names, Metadata::FlagCollectoin, and/or a list of sparse ids. This is meant to produce the names that get passed into the DataCollection::Add* calls, so provides the means of creating new Mesh*Data objects that are filtered on, for example, Metadata and/or sparse ids.
GetVariableNames overloads were also added to Mesh, which just forwards the call on to the resolved_state StateDescriptor functions. This again is for convenience so that downstream codes don't have to pull out the resolved_state when they just want a list of names from the simulation state as a whole.
Parthenon will now automatically add a Metadata flag with the same name as the StateDescriptor the field is added to every field, sparse pool, and swarm. StateDescriptor stashes the new flag in a Param and provides a function to retrieve the flag for convenient access later.
Given a list of names, a Metadata::FlagCollection, and/or sparse ids, there are new functions that additionally take two MeshBlockData pointers or two MeshData pointers and return the unique ids of the variables that satisfy the filters and exist in both objects. This is required when trying to work with Mesh*Data objects that may have different fields present. With just the intersection of unique ids, you can write functions that can, for example, add values from one MeshData object into the corresponding fields in another MeshData object quite straightforwardly.
New PackDescriptor code that allows you to produce a SparsePack given a list of unique ids.

With all this stuff in place, you can do things like make registers for the RHSs of subsets of equations. We have integrated this branch into our downstream code and successfully make use of all these features.

PR Checklist

Code passes cpplint
New features are documented.
Adds a test for any bugs fixed. Adds tests for new features.
Code is formatted
Changes are summarized in CHANGELOG.md
CI has been triggered on Darwin for performance regression tests.
Docs build
(@lanl.gov employees) Update copyright on changed files

…ue functions

…sses

…eader

jdolence · 2023-08-15T17:06:51Z

@pdmullen @bprather @Yurlungur @lroberts36 @pgrete I think this one is ready to review. I'm going to leave Draft in place until I do a bit more testing downstream, but so far things look good so I suspect I'll be removing Draft very soon.

lroberts36

LGTM, comments aren't blocking.

src/amr_criteria/amr_criteria.hpp

src/interface/data_collection.cpp

src/interface/meshblock_data.cpp

src/interface/meshblock_data.hpp

src/interface/metadata.hpp

src/interface/state_descriptor.cpp

Co-authored-by: Luke Roberts <lfroberts@lanl.gov>

Yurlungur

LGTM. This will be a nice improvement. Do you need changelog + formatting? Also remove the draft prefix?

src/interface/data_collection.cpp

src/interface/meshblock_data.cpp

Yurlungur · 2023-08-15T20:25:26Z

src/interface/meshblock_data.cpp

-// Constructor for getting sub-containers
-// the variables returned are all shallow copies of the src container.
-// Optionally extract only some of the sparse ids of src variable.
-template <typename T>
-MeshBlockData<T>::MeshBlockData(const MeshBlockData<T> &src,
-                                const std::vector<std::string> &names,
-                                const std::vector<int> &sparse_ids) {
-  CopyFrom(src, true, names, {}, sparse_ids);
-}
-
-// TODO(JMM): Add constructor that takes unique IDs
-template <typename T>
-MeshBlockData<T>::MeshBlockData(const MeshBlockData<T> &src,
-                                const std::vector<MetadataFlag> &flags,
-                                const std::vector<int> &sparse_ids) {
-  CopyFrom(src, true, {}, flags, sparse_ids);
-}
-
-// provides a container that has a single sparse slice
-template <typename T>
-std::shared_ptr<MeshBlockData<T>>
-MeshBlockData<T>::SparseSlice(const std::vector<int> &sparse_ids) const {
-  auto c = std::make_shared<MeshBlockData<T>>();
-  c->CopyFrom(*this, true, {}, {}, sparse_ids);
-  return c;
-}


No constructor overloads, always use copy now? That's cleaner from a self-describing interface point of view I think

src/interface/meshblock_data.hpp

src/utils/unique_id.cpp

pdmullen

LGTM!

src/interface/state_descriptor.hpp

jdolence · 2023-08-16T15:35:32Z

@pgrete I think this one is ready to go. Did you want to look through it first or should I just pull the trigger. I'm pretty sure this won't break anything for you downstream.

bprather · 2023-08-16T15:42:25Z

After pulling and fiddling a bunch I like all this as is too.

It's prompted me to dump some more of the other functionality I was duplicating in KHARMA into Parthenon too, I'll open a separate PR with the additions.

jdolence · 2023-08-16T23:01:15Z

@pgrete didn't give you much time to respond, but I'm going to pull the trigger. We have a least two separate codes that have confirmed backwards compatibility, so I'm pretty confident this won't break anything for you. Obviously, if you have comments on anything in this after it's merged, we can work on getting them addressed.

jdolence added 3 commits August 11, 2023 10:25

add the ability to get a list of vars given constraints

fd6d565

Merge branch 'develop' into jdolence/meshblockdata_add

163d263

expose a mesh-level interface that uses resolved_packages

033baec

jdolence mentioned this pull request Aug 11, 2023

Allowing Metadata to change after construction is unsafe #844

Open

jdolence and others added 22 commits August 11, 2023 13:34

add package metadata to all fields, sparse fields, and swarms

ae64275

refactor Copy and make accessible from DataCollection Add functions

e85c7f9

formatting

7d14eb8

newline for the linter

ade63bf

remove dead code

eeaf23d

sneak in templated versions of the AddField/SparsePool/Swarm/SwarmVal…

3c72945

…ue functions

template SparsePool constructor on variable type

fa1ce4f

that didnt work. try this.

c127627

change name

c58d82e

DataCollection Add functions should returns references

305089b

formatting

56abe2a

fix some things to make adding to the mesh_data collection directly work

3800f48

Add MakePackDescriptor for uids

2c828fd

fix style

7dff9e4

compute uid intersections of variables in different MeshBlockData cla…

097d0bb

…sses

add a MeshData intersection function

19bc17b

a little hack to avoid calling a member of StateDescriptor from the h…

61e345a

…eader

get rid of the hack

346e864

expose Uid_t in the driver/prelude namespace

53e9f06

treat shallow copies differently for sparse allocation

85e0d67

only allocate/deallocate sparse if it exists in the MeshBlockData object

0d286a3

formatting and style

c153a90

jdolence requested review from pdmullen, pgrete and bprather August 15, 2023 17:07

lroberts36 approved these changes Aug 15, 2023

View reviewed changes

jdolence and others added 3 commits August 15, 2023 13:21

add default descriptors/sparse packs for weird usage downstream

811e08e

Update src/interface/state_descriptor.cpp

1da52bb

Co-authored-by: Luke Roberts <lfroberts@lanl.gov>

Update src/interface/state_descriptor.cpp

4e51cc1

Co-authored-by: Luke Roberts <lfroberts@lanl.gov>

Yurlungur approved these changes Aug 15, 2023

View reviewed changes

pdmullen approved these changes Aug 15, 2023

View reviewed changes

src/interface/state_descriptor.hpp Outdated Show resolved Hide resolved

jdolence added 7 commits August 15, 2023 20:41

fix bug and address comments

134f1d1

move from Copy to Initialize

4cdd379

change name of package metadata flag

7ea5de2

update changelog

0480519

update docs

0657d93

update all the copyrights

fe1a6af

gah! formatting!

42e678e

jdolence changed the title ~~Draft: Provide more flexible ways to add subset MeshBlockData objects to DataCollections~~ Provide more flexible ways to add subset MeshBlockData objects to DataCollections Aug 16, 2023

bprather mentioned this pull request Aug 16, 2023

WIP: DataCollection/StateDescriptor niceties #923

Open

8 tasks

add a comment as requested

6807b61

jdolence merged commit eecd058 into develop Aug 16, 2023
45 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provide more flexible ways to add subset MeshBlockData objects to DataCollections #921

Provide more flexible ways to add subset MeshBlockData objects to DataCollections #921

jdolence commented Aug 11, 2023 •

edited

Loading

jdolence commented Aug 15, 2023

lroberts36 left a comment

Yurlungur left a comment

Yurlungur Aug 15, 2023

pdmullen left a comment

jdolence commented Aug 16, 2023

bprather commented Aug 16, 2023

jdolence commented Aug 16, 2023

Provide more flexible ways to add subset MeshBlockData objects to DataCollections #921

Provide more flexible ways to add subset MeshBlockData objects to DataCollections #921

Conversation

jdolence commented Aug 11, 2023 • edited Loading

PR Summary

PR Checklist

jdolence commented Aug 15, 2023

lroberts36 left a comment

Choose a reason for hiding this comment

Yurlungur left a comment

Choose a reason for hiding this comment

Yurlungur Aug 15, 2023

Choose a reason for hiding this comment

pdmullen left a comment

Choose a reason for hiding this comment

jdolence commented Aug 16, 2023

bprather commented Aug 16, 2023

jdolence commented Aug 16, 2023

jdolence commented Aug 11, 2023 •

edited

Loading