Adaptive Hole Map #12

ackirby88 · 2023-01-19T21:37:31Z

Initial implementation of the adaptive hole map algorithm.
To enable/disable this functionality, set line 64 in codetypes.h

#define USE_ADAPTIVE_HOLEMAP 1 // [0] original hole map, [1] adaptive hole map

I've done preliminary testing on the tioga test case: unstructured tetrahedral sphere and Cartesian background mesh.

Test Case Input (case/grid/input):

 $inputs
 nlayers=40,
 wallSpacing=1e-5,
 outerSpacing=0.1,
 outerB=0.05,
 xlo(1)=-5.,
 xlo(2)=-5.,
 xlo(3)=-5.,
 xhi(1)=5.,
 xhi(2)=5.,
 xhi(3)=5.,
 dx(1)=0.1,
 dx(2)=0.1,
 dx(3)=0.1,
 $end

Original Hole Map (./run.sh 12):

 Generating           24  partitions
 Finished generating grid ..
 Writing tecplot compatible files..
 All done
 # tioga test on           12  processes
 # tioga test : finished reading grids
 connectivity time=  0.71787999999999996     
           2          14
           0           0
           0           0
           1           7
           0           0
           0           0
           2          14
           0           0
           0           0
           3          21
           0           0
           0           0
           5          35
           0           0
           0           0
           0           0
           0           0
           2          14
           0           0
           0           0
           0           0
           0           0
        1202       10818
        1282       11538
 data update time=   7.0409999999998529E-003
-- Interpolation error statistics --
ProcId    Error
   0     0.6619866E-17
   3     0.5463998E-17
   4     0.6485183E-17
   5     0.4214754E-17
   6     0.6526902E-17
   7     0.3255305E-16
   8     0.7567119E-17
   9     0.8691998E-17
  10     0.3855176E-17
  11     0.6595807E-17
   1     0.9389718E-17
   2     0.7459847E-17
myid/meshtag/norphan/nnodes/nobc=4 2 1150 85000 0
myid/meshtag/norphan/nnodes/nobc=7 2 1138 85000 0
Tioga exited successfully

Adaptive Hole Map (./run.sh 12):

 Generating           24  partitions
 Finished generating grid ..
 Writing tecplot compatible files..
 All done
 # tioga test on           12  processes
 # tioga test : finished reading grids
  [tioga::performConnectivity::getAdaptiveHoleMap] Mesh Body 0: Levels 8, total octants 34353, total leafs 30059
 connectivity time=   9.9077000000000026E-002
           1           7
           0           0
           0           0
           0           0
           0           0
           2          14
           0           0
           0           0
           3          21
           0           0
           0           0
           5          35
           0           0
           0           0
           2          14
           0           0
           0           0
           2          14
           0           0
           0           0
           0           0
        1202       10818
        1282       11538
           0           0
 data update time=   4.9900000000000500E-003
-- Interpolation error statistics --
ProcId    Error
   0     0.6619866E-17
   2     0.7459847E-17
   4     0.6485183E-17
   6     0.6526902E-17
   7     0.3255305E-16
   9     0.8691998E-17
   1     0.9389718E-17
   3     0.5463998E-17
   5     0.4214754E-17
   8     0.7567119E-17
  10     0.3855176E-17
  11     0.6595807E-17
myid/meshtag/norphan/nnodes/nobc=7 2 1055 85000 0
myid/meshtag/norphan/nnodes/nobc=4 2 1058 85000 0
Tioga exited successfully

… case with multiblock looks coorrect. Need to extensively test.

michaeljbrazell · 2023-01-19T23:08:42Z

nice, way faster!

…with an outer/overset bc face and wall bc face -- this prevents the adaptive hole map from refining to the max number of levels. (2) Fix compiler warnings about the intstring character array buffer size.

…memory cleanup. Note: suspect small memory leak in exchangeBoxes().

jrood-nrel · 2023-01-24T20:52:38Z

Can there be something other than a define to turn this on? Also, can we verify this has been run through ASAN or valgrind to see that there are no leaks?

…compiler define variable). Also added memory cleanup for tioga shutdown -- checked with valgrind (iblank_cell is lost but only because it is handled by nalu-wind).

ackirby88 · 2023-01-25T19:57:32Z

Hi Jon,
I added a new tioga interface function to set the hole map algorithm:

void tioga_setholemapalg_(int *alg); // [0] original hole map, [1] adaptive hole map

This function replaces the compile-time define variable.

Second, I added some memory cleanup for tioga instance deletion.
There were no issues previously for repeated hole map constructions, but only when tioga_delete was called.
I verifed no memory leaks using Valgrind (single rank and 2 mpi ranks -- output files attached below).

valgrid.adaptHoleMap.1rank.txt
valgrid.adaptHoleMap.2ranks.txt

…into adaptive-hole-map

jrood-nrel · 2023-02-07T23:23:55Z

src/bookKeeping.C

+    verbose=0;
+    if(iblank[i] < 0){
+      temp=donorList[i];
+      while(temp!=NULL){


Does this need to exist?

It's a hold-over from the original function lines 320-330, so I wouldn't change the functionality.
But it does look potentially buggy with the while construct. Let me track down the history of the donorList.

I'm wondering if there's a bug in the original function. My guess is there should be
'''temp=temp->next;'''
Inside the while loop. Thoughts ?

it doesn't make any sense without some kind of increment inside the while

Does indeed seem suspect ... Do you see a reason why it hasn't broken anything yet?

I think it worked before because Jay already sorted the temp donor resolutions (finest donor first) inside insertInList and already established the iblank value to be a fringe (-1). Otherwise, it would have a better resolution than the donor and not entered that if-statement check.

…te in MeshBlock::processDonors. (2) Replace raw c-pointers for adaptive processDonors.

…into adaptive-hole-map

ashesh2512 · 2023-04-02T21:24:44Z

@ackirby88 Where is the .svn folder coming from?

ackirby88 · 2023-04-02T21:27:25Z

Where is the .svn folder coming from?

@ashesh2512 It was from the Jay's original TIOGA repository.

ackirby88 · 2023-05-02T22:36:43Z

@ashesh2512 I don't see your commits here...

ashesh2512 · 2023-02-09T20:33:07Z

src/bookKeeping.C

+    verbose=0;
+    if(iblank[i] < 0){
+      temp=donorList[i];
+      while(temp!=NULL){


Does indeed seem suspect ... Do you see a reason why it hasn't broken anything yet?

driver/CMakeLists.txt

src/codetypes.h

src/MeshBlock.h

ashesh2512 · 2023-04-29T06:07:24Z

src/MeshBlock.h

+    int nrank;
+    int masterID;   /* master rank for distributing mesh block data */
+    MPI_Comm comm;  /* communicator containing all complement ranks + master */
+    MPI_Request req;


is req used anywhere?

I don't think so -- I originally had asynchronous communications.
I'll check to see if I use it for the abutting mesh implementation (next week); if not, I'll remove it.

src/tiogaInterface.C

src/MeshBlock.C

ashesh2512 · 2023-05-03T00:26:53Z

@ashesh2512 I don't see your commits here...

@ackirby88 Turns out I never submitted the comments. You should be able to see them now.

…ap specific functions (comms), (2) macro replacement, (3) increase max level to all full 32-bit usage, (4) add back poisson_mms.

ackirby88 · 2023-05-04T21:39:32Z

@ashesh2512 I committed your recommended changes. Please give the built-in test case a try along with your other tests.

src/tioga.h

…ction using mallinfo.

ackirby88 · 2023-07-31T05:55:36Z

@jrood-nrel @ashesh2512 I've merged in the latest exawind changes with the std::min/max fixes. I've also addressed your suggestions, Jon, about the malloc.h.
This may finally be a good time from NREL's side to complete this pull request and merge in these changes so we stop diverging.

ashesh2512 · 2023-08-01T16:58:13Z

Merged through #14

ackirby88 added 4 commits December 20, 2022 20:02

Fixing header of meshblock output of tecplot files.

f0f6cb4

Fix cmakelists for driver and gridgen mpi and tioga library linking.

acddd77

WIP: check in the adaptive hole map source code.

a8cac09

Implemented adaptive hole map and initial tests for tioga test sphere…

6bbcd53

… case with multiblock looks coorrect. Need to extensively test.

ackirby88 added 3 commits January 20, 2023 17:21

(1) Add duplicate node check when an element has two adjoining faces …

b483e1a

…with an outer/overset bc face and wall bc face -- this prevents the adaptive hole map from refining to the max number of levels. (2) Fix compiler warnings about the intstring character array buffer size.

Replaced octant pointers with std::vectors. Verified correctness and …

b21a38f

…memory cleanup. Note: suspect small memory leak in exchangeBoxes().

Reenable poisson exe by fixing the fortran MPI compling issue.

4482db7

ashesh2512 requested review from ashesh2512, michaeljbrazell, jrood-nrel and psakievich January 24, 2023 20:26

Add tioga interface function to set hole map algorithm (instead of a …

3347f26

…compiler define variable). Also added memory cleanup for tioga shutdown -- checked with valgrind (iblank_cell is lost but only because it is handled by nalu-wind).

ackirby88 and others added 2 commits January 26, 2023 10:31

Merge branch 'adaptive-hole-map' of https://github.com/ackirby88/tioga …

a9924b5

…into adaptive-hole-map

Setting the hole map algorithm to the original for merging.

21047e4

jrood-nrel reviewed Feb 7, 2023

View reviewed changes

ackirby88 added 3 commits February 10, 2023 13:10

Revome old svn files.

8a77bed

(1) Fix while loop bug which didn't update the temp donorList candida…

f515c02

…te in MeshBlock::processDonors. (2) Replace raw c-pointers for adaptive processDonors.

Merge branch 'adaptive-hole-map' of https://github.com/ackirby88/tioga …

80d76a1

…into adaptive-hole-map

ashesh2512 reviewed May 3, 2023

View reviewed changes

Address comments from Ashesh: (1) conditionals around adaptive hole m…

177e016

…ap specific functions (comms), (2) macro replacement, (3) increase max level to all full 32-bit usage, (4) add back poisson_mms.

jrood-nrel reviewed Jun 27, 2023

View reviewed changes

src/tioga.h Outdated Show resolved Hide resolved

Remove malloc.h as it's nonstandard; commented out allocated data fun…

7f6cc1c

…ction using mallinfo.

ackirby88 added 2 commits July 30, 2023 23:48

Merging the std:min max from exawind.

0d93be7

Merge branch 'Exawind-exawind' into adaptive-hole-map

373cdbc

ashesh2512 closed this Aug 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adaptive Hole Map #12

Adaptive Hole Map #12

ackirby88 commented Jan 19, 2023 •

edited

Loading

michaeljbrazell commented Jan 19, 2023

jrood-nrel commented Jan 24, 2023

ackirby88 commented Jan 25, 2023

jrood-nrel Feb 7, 2023

ackirby88 Feb 8, 2023 •

edited

Loading

ackirby88 Feb 9, 2023

psakievich Feb 9, 2023

ashesh2512 Feb 9, 2023

ackirby88 May 3, 2023

ashesh2512 commented Apr 2, 2023

ackirby88 commented Apr 2, 2023 •

edited

Loading

ackirby88 commented May 2, 2023

ashesh2512 Feb 9, 2023

ashesh2512 Apr 29, 2023

ackirby88 May 3, 2023

ashesh2512 commented May 3, 2023

ackirby88 commented May 4, 2023

ackirby88 commented Jul 31, 2023

ashesh2512 commented Aug 1, 2023

Adaptive Hole Map #12

Adaptive Hole Map #12

Conversation

ackirby88 commented Jan 19, 2023 • edited Loading

michaeljbrazell commented Jan 19, 2023

jrood-nrel commented Jan 24, 2023

ackirby88 commented Jan 25, 2023

Choose a reason for hiding this comment

ackirby88 Feb 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashesh2512 commented Apr 2, 2023

ackirby88 commented Apr 2, 2023 • edited Loading

ackirby88 commented May 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashesh2512 commented May 3, 2023

ackirby88 commented May 4, 2023

ackirby88 commented Jul 31, 2023

ashesh2512 commented Aug 1, 2023

ackirby88 commented Jan 19, 2023 •

edited

Loading

ackirby88 Feb 8, 2023 •

edited

Loading

ackirby88 commented Apr 2, 2023 •

edited

Loading