Bayesian Optimization Example with WarpX Outdated

[n01r]

The Bayesian Optimization Example with WarpX seems to be outdated and is failing.

I tried it interactively on one Perlmutter node.

First, the boundary section in the WarpX input script has to be changed.
Old

# boundary
boundary.field_hi = none damped
boundary.field_lo = none damped
boundary.particle_hi = absorbing absorbing
boundary.particle_lo = absorbing absorbing

New

# boundary
boundary.field_hi = none damped
boundary.field_lo = none damped
boundary.particle_hi = none absorbing
boundary.particle_lo = none absorbing

This lets you successfully run WarpX. Each run takes 100 seconds.
Next, the analysis script throws warnings but the exploration finishes.
However, it reports Failed to evaluate trial <no. of trial> for every trial.
Is that expected?

(warpx-gpu) mgarten@nid200281:/pscratch/sd/m/mgarten/superfacilities_IFE/2024/011_try_optimas_multi_stage_warpx_example_vanilla> python run_example.py 
[INFO 07-30 17:30:05] optimas.generators.base: Generated trial 0 with parameters {'adjust_factor': 0.9121249943971634, 'lens_start': 0.3347596833705902}
[INFO 07-30 17:30:05] optimas.generators.base: Generated trial 1 with parameters {'adjust_factor': 0.8609110658988357, 'lens_start': 0.3259775773640722}
[INFO 07-30 17:30:05] optimas.generators.base: Generated trial 2 with parameters {'adjust_factor': 1.0485926766879858, 'lens_start': 0.340912622612901}
[INFO 07-30 17:30:05] optimas.generators.base: Generated trial 3 with parameters {'adjust_factor': 0.9835961840115488, 'lens_start': 0.33697925267834217}
/pscratch/sd/m/mgarten/superfacilities_IFE/2024/011_try_optimas_multi_stage_warpx_example_vanilla/analysis_script.py:15: RuntimeWarning: invalid value encountered in sqrt
  return np.sqrt(x2 * u2 - xu**2)
[INFO 07-30 17:31:51] optimas.generators.base: Failed to evaluate trial 1.
[INFO 07-30 17:31:51] optimas.generators.base: Generated trial 4 with parameters {'adjust_factor': 0.8713434485718607, 'lens_start': 0.34657188675180073}
/pscratch/sd/m/mgarten/superfacilities_IFE/2024/011_try_optimas_multi_stage_warpx_example_vanilla/analysis_script.py:15: RuntimeWarning: invalid value encountered in sqrt
  return np.sqrt(x2 * u2 - xu**2)
[INFO 07-30 17:31:53] optimas.generators.base: Failed to evaluate trial 2.
[INFO 07-30 17:31:53] optimas.generators.base: Generated trial 5 with parameters {'adjust_factor': 0.7878917461261153, 'lens_start': 0.33506758825853467}
[INFO 07-30 17:31:54] optimas.generators.base: Completed trial 3 with objective(s) {'f': (-8.115592020438127, None)} and analyzed parameter(s) {'energy_std': (7.995579263512482e-14, None), 'energy_avg': (16.331455626496947, None), 'charge': (-4.0000000000000004e-20, None), 'emittance': (2.5121479338940403e-15, None)}
/global/cfs/cdirs/m4546/mgarten/sw/perlmutter/gpu/venvs/warpx-gpu/lib/python3.11/site-packages/ax/core/data.py:286: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  return cls(df=pd.concat(dfs, axis=0, sort=True))
/global/cfs/cdirs/m4546/mgarten/sw/perlmutter/gpu/venvs/warpx-gpu/lib/python3.11/site-packages/botorch/models/utils/assorted.py:194: UserWarning: std(): degrees of freedom is <= 0. Correction should be strictly less than the reduction factor (input numel divided by output numel). (Triggered internally at ../aten/src/ATen/native/ReduceOps.cpp:1807.)
  Ymean, Ystd = torch.mean(Y, dim=-2), torch.std(Y, dim=-2)
/pscratch/sd/m/mgarten/superfacilities_IFE/2024/011_try_optimas_multi_stage_warpx_example_vanilla/analysis_script.py:15: RuntimeWarning: invalid value encountered in sqrt
  return np.sqrt(x2 * u2 - xu**2)
[INFO 07-30 17:34:17] optimas.generators.base: Generated trial 6 with parameters {'adjust_factor': 1.05, 'lens_start': 0.32}
[INFO 07-30 17:34:17] optimas.generators.base: Failed to evaluate trial 0.
[INFO 07-30 17:34:17] optimas.generators.base: Completed trial 4 with objective(s) {'f': (-8.11559202039898, None)} and analyzed parameter(s) {'energy_std': (5.453444427775179e-13, None), 'energy_avg': (16.329845970195954, None), 'charge': (-4.0000000000000004e-20, None), 'emittance': (1.4210854715202004e-14, None)}
[INFO 07-30 17:34:17] optimas.generators.base: Saved model to file after 5 evaluated trials.
[INFO 07-30 17:34:17] optimas.generators.base: Completed trial 5 with objective(s) {'f': (-8.115592020444431, None)} and analyzed parameter(s) {'energy_std': (2.7889368091584967e-13, None), 'energy_avg': (16.329498528861265, None), 'charge': (-4.0000000000000004e-20, None), 'emittance': (6.280369834735101e-16, None)}

I am attaching the libE_stats.txt and the ensemble.log.
libE_stats.txt
ensemble.log

The former contains Task Failed during run.

@RTSandberg, you wrote this example and @RemiLehe, you merged it, right? Do either of you remember what this looked like?

[n01r]

@RemiLehe, I moved the simulations here:
/global/cfs/cdirs/m3239/mgarten/superfacilities/2024/011_try_optimas_multi_stage_warpx_example_vanilla/

[n01r]

Thanks, @RemiLehe! This fixes the problem. :)