Star particle IDs getting shuffled around in some outputs

[anchwr]

In certain groups of outputs from FOGGIE simulations (typically 2-8 snapshots in a row), stars that form will be assigned a temporary particle_index value, which will change to a different particle_index for the rest of the simulation after this group of snapshots. This issue makes it nearly impossible to find the exact formation location, velocity, etc. of a star particle that forms during one of the impacted steps. So far, the best workaround I've found is to grab any data I need in the first snapshot following the impacted group (e.g., if DD0733 & DD0734 have this bug, I will wait until DD0735 to grab the location etc. of a star particle that formed during DD0733).

The hallmark of the bug is a sudden apparent jump in the distances that young stars travel from one timestep to another, since any star formed during the affected steps will have its particle_index shuffled to another star formed during the same period in the first post-bug step. In the plot below, I'm showing distance traveled by young stars from one timestep to the next in 5 FOGGIE simulations. 4 instances of this bug are apparent as spikes in distance: two in Hurricane and one each in Blizzard and Maelstrom.

At @bwoshea 's suggestion, I also checked that this held true if I parsed the Enzo hdf5 files directly, rather than reading the star particle data in via yt. The plot below shows stars that formed in the ~50 Myr period leading up to the buggy timesteps in Blizzard and the distance that they travel between the final buggy step (DD0734) and the first post-bug step (DD0735). The sudden jump in distance that appears occurs just before the first buggy step (DD0733). This suggests this is an Enzo bug, not a yt bug.

Other potentially relevant things I've found:
-The particle_index shuffling only happens with stars that form during the buggy steps. The particle_index values of previously formed star particles and dark matter particles do not change during or after the buggy steps.
-The bug doesn't appear to be related to #194 , where the total number of DM particles in the simulation appears to change from step to step (presumably due to some kind of issue in writing out DM particles in ghost cells). Instances of the particle_index shuffling bug don't correlate with large fluctuations in the number of DM particles in the simulation.