slow performace on 3D flow past cylinder

[vanchisen]

Hi, this morning I have tested neko simulation of flow past a cylinder at Re=80,000,
The mesh consists of 131,856 elements, 7rd SEM is used.
The simulation runs quite slowly, takes about 300 seconds each step on 7 RTX4090 GPUs, so I wonder if I have made any mistakes in setting up the runing parameters ?

The mesh file is available from following link,
https://drive.google.com/file/d/1wox2-WCoKu_bvVUj8CSX-8PyhQtckwkn/view?usp=sharing

And the parameter file *.case is on below,

{
  "version": 1.0,
  "case":
  {
  "mesh_file": "cyl.nmsh",
  "output_checkpoints": false,
  "output_at_end": true,
  "end_time": 200.0,
  "timestep": 2e-4,

  "numerics": {
    "time_order": 3,
    "polynomial_order": 7,
    "dealias": true
  },

  "fluid": {
    "scheme": "pnpn",
    "Re": 80000,

    "initial_condition": {
      "type": "uniform",
      "value": [0.0, 0.0, 0.0]
    },
      "inflow_condition": {
            "type": "uniform",
            "value": [1.0, 0.0, 0.0]
        },

    "velocity_solver": {
      "type": "cg",
      "preconditioner": "jacobi",
      "projection_space_size": 0,
      "absolute_tolerance": 1e-7,
      "max_iterations": 800
    },

    "pressure_solver": {
      "type": "gmres",
      "preconditioner": "hsmg",
      "projection_space_size": 8,
      "absolute_tolerance": 1e-5,
      "max_iterations": 800
    },

        "boundary_types": ["v", "o", "w", "", "", "", ""],
        "output_control": "nsamples",
        "output_value": 400
  }
  }
}

[timofeymukha]

Your initial condition is quite poor. You are essentially accelerating the entire flow from 0 velocity. It would be better to put the same value as at the inlet.

How are the iteration counts for the linear solvers looking in the log?

[vanchisen]

Thank you, below is the output of one of the typical time step,

t = 0.1000000E-02 [ 0.00% ]

Time-step: 6
CFL: 0.2139215E+02 dt: 0.2000000E-03

   ------------Fluid-------------  
   Projection Pressure
   Proj. vec.:   Orig. residual:
             0     0.2727242-320
   Pressure
   Iterations:   Start residual:     Final residual:
           800     0.1282938E+03       0.1248382E+03
   X-Velocity
   Iterations:   Start residual:     Final residual:
            11     0.2615724E+03       0.5065713E-07
   Y-Velocity
   Iterations:   Start residual:     Final residual:
             9     0.3702106E+01       0.7560267E-07
   Z-Velocity
   Iterations:   Start residual:     Final residual:
             9     0.7943654E+00       0.8505412E-07
   Elapsed time (s):  0.2076705E+04 Step time:  0.3456647E+03

[timofeymukha]

Yeah, the pressure sovler is not convering at all and maxing out the iteration number. I looked at the mesh, looks pretty cool. Is it standard to use periodic conditions on the upper and lower boundaries for this case?

[vanchisen]

Yes, periodic BC is imposed on spanwise(z) and cross-flow(y) direction.
Gmsh script for this mesh is on following link,
https://drive.google.com/file/d/1Sm_k1eqH3oS5d9N8IB9DzIBUujlreSjL/view?usp=sharing

[timofeymukha]

I could get your case running with the following config:

{
  "version": 1.0,
  "case":
  {
  "mesh_file": "cyl.nmsh",
  "output_checkpoints": true,
  "checkpoint_control": "tsteps",
  "checkpoint_value": "100",
  "output_at_end": true,
  "output_boundary": true,
  "end_time": 200.0,
  "timestep": 2e-5,

  "numerics": {
    "time_order": 2,
    "polynomial_order": 3,
    "dealias": true
  },

  "fluid": {
    "scheme": "pnpn",
    "Re": 80000,

    "initial_condition": {
      "type": "uniform",
      "value": [1.0, 0.0, 0.0]
    },
      "inflow_condition": {
            "type": "uniform",
            "value": [1.0, 0.0, 0.0]
        },

    "velocity_solver": {
      "type": "cg",
      "preconditioner": "jacobi",
      "projection_space_size": 0,
      "absolute_tolerance": 1e-7,
      "max_iterations": 800
    },

    "pressure_solver": {
      "type": "gmres",
      "preconditioner": "hsmg",
      "projection_space_size": 0,
      "absolute_tolerance": 1e-5,
      "max_iterations": 100
    },

    "boundary_types": ["v", "o", "w", "", "", "", ""],
    "output_control": "tsteps",
    "output_value": 100
  }
  }
}

Note:

• No projections
• Reduce poly order to test faster
• Reduced time order for better stability
• Reduced time-step size.

After a few hundred time-steps the solver starts converging. But I see some weird stuff at the outlet.

Maybe using o+dong at the outlet can help stabilize this.

[vanchisen]

Thank you for the suggestions, Timofey.

[timofeymukha]

The dong doesn't seem to help, which is unfortunate. @MartinKarp , any ideas what this could be?

[timofeymukha]

But here we don't have a boundary between a Neumann and Dirichlet bc, is there still an issue then?

[MartinKarp]

The same issue would arise when it is an internal boundary touching the Dirichlet. I have started looking at the fix, and updating the branch. Would it be possible to generate a smaller mesh for us to ease the troubleshooting @vanchisen? To replicate the issue, the important part is that there are some elements in the mesh where only the corner is touching the boundary at the inflow and outflow.

[vanchisen]

Sure Martin, Thank you.
Do you mean the slow performance is related this BC issue ?
A smaller mesh can be obtained from following link,
https://drive.google.com/file/d/1yUCYovFxhm-Vdgjp41x2Ree5FDwimCRZ/view?usp=sharing

[timofeymukha]

@vanchisen If Martin's hypothesis is correct, the issue here is not "performance", but rather that we incorrectly apply boundary conditions for particular types of mesh topologies. It seems your case will be very good to test this. Given the severity of the issue, I would say fixing this will be top priority for us.

[vanchisen]

Thank you. @timofeymukha
I have tested the same big mesh but without periodic BC, wall clock for each time iteration is same as that with periodic BC.