Embedded boundary edge lengths and Field arrays have different number of guard cells

[clarkse]

Allocation of the embedded boundary arrays uses ng_FieldSolver from the guard cell manager in WarpX, whereas the fields use the ng_alloc_EB value in teh guard cell manager. These values can be different, and when using the edge_lengths array in a grown field tilebox a segmentation fault occurs.

E.g. this code snippet yields a seg fault.

    amrex::MultiFab* mfx = field[0];
    amrex::MultiFab* mfy = field[1];
    amrex::MultiFab* mfz = field[2];

    const amrex::IntVect x_nodal_flag = mfx->ixType().toIntVect();
    const amrex::IntVect y_nodal_flag = mfy->ixType().toIntVect();
    const amrex::IntVect z_nodal_flag = mfz->ixType().toIntVect();

    // avoid implicit lambda capture
    auto x_external = expression[0];
    auto y_external = expression[1];
    auto z_external = expression[2];

    for ( MFIter mfi(*mfx, TilingIfNotGPU()); mfi.isValid(); ++mfi)
    {
        const amrex::Box& tbx = mfi.tilebox( x_nodal_flag, mfx->nGrowVect() );
        const amrex::Box& tby = mfi.tilebox( y_nodal_flag, mfy->nGrowVect() );
        const amrex::Box& tbz = mfi.tilebox( z_nodal_flag, mfz->nGrowVect() );

        auto const& mfxfab = mfx->array(mfi);
        auto const& mfyfab = mfy->array(mfi);
        auto const& mfzfab = mfz->array(mfi);

        amrex::Array4<amrex::Real> lx, ly, lz;
        if (EB::enabled()) {
            lx = warpx.m_fields.get(FieldType::edge_lengths, Direction{0}, lev)->array(mfi);
            ly = warpx.m_fields.get(FieldType::edge_lengths, Direction{1}, lev)->array(mfi);
            lz = warpx.m_fields.get(FieldType::edge_lengths, Direction{2}, lev)->array(mfi);
        }

        amrex::ParallelFor (tbx, tby, tbz,
            [=] AMREX_GPU_DEVICE (int i, int j, int k) {
                // skip if node is covered by an embedded boundary
                if (EB::enabled() && lx(i, j, k) <= 0) { return; }
            }
        );
    }

This was compiled and run in RZ. I am not sure if it does the same in 3D.