Coupler error; Fortran runtime error; atm%pelist

[Parag0206]

Hi,

I am trying to run MOM6+COBALTv2 coupled on a CentOS system. While running a case, I am getting the following error:

At line 1466 of file /home/mom6/CEFI-regional-MOM6/builds/../src/coupler/full//coupler_main.F90
Fortran runtime error: Index '1' of dimension 1 of array 'atm%pelist' above upper bound of 0

The coupler fails I guess.The &coupler_nml in input,nml looks like this

 &atmos_model_nml
        layout= 1,1
/

 &coupler_nml
        months = 0
        days   = 182
        current_date = 2013,4,1,0,0,0
        hours = 0
        minutes = 0
        seconds = 0
        calendar = 'NOLEAP'
        dt_cpld  = 3600
        dt_atmos = 3600
        do_atmos = .false.
        do_land = .false.
        do_ice = .true.
        do_ocean = .true.
        atmos_npes = 0,
        ocean_npes = 0,
        concurrent = .false. 
        use_lag_fluxes=.false.
        atmos_nthreads = 1
        ocean_nthreads = 1
        do_chksum = .false.
        do_endpoint_chksum = .false.
/

 &diag_manager_nml
        max_files = 600 
        flush_nc_files=.false.
        max_axes = 600
        max_num_axis_sets = 600
        max_input_fields = 2000 
        max_output_fields = 5000 
        mix_snapshot_average_fields=.false.
/

 &fms_io_nml
        fms_netcdf_restart=.true. 
        threading_read='multi'
        max_files_r = 800
        max_files_w = 800
        checksum_required=.false.
/

 &fms_nml
        clock_grain='ROUTINE' 
        clock_flags='NONE'
        domains_stack_size = 8000000
        stack_size =0
/

Can anyone help me with this?

[yichengt900]

@Parag0206 , thanks for raising this issue. Your coupler_nml looks ok to me. In order to investigate the issue further, would you mind to share your model configurations (e.g., grid files, model setup files) so I can attempt to replicate the error on my end? Additionally, could you provide details on how you compile your model and which source codes (GitHub repo, tag, etc.) you are using? This information will help me determine the root cause. Thanks.

[Parag0206]

Hi @yichengt900, Thanks for the reply!

1.) My model parameter files are here : https://github.com/Parag0206/BOB_0.073_2013. The grid and forcing files i am not sure how to share. (The total directory is around 14gb.)

2.) As listed in https://cefi-regional-mom6.readthedocs.io/en/latest/BuildMOM6.html#set-up-the-environment-and-build-the-executables, inside the "builds" folder I made a directory of my machine, where I had two files("triton-gnu.mk" and "triton-gnu.env"). The ".env" file is empty, the other I have attached.
triton-gnu.txt

3. Further details:

• CentOS 7
• GCC 9.5.0
• NETCDF-C 4.9.2
• NETCDF-FORTRAN 4.6.1
• OPENMPI 5.0.3

If you want anything else, please let me know and again thank you for your reply

[yichengt900]

@Parag0206 Thanks. One quick question: are you building your model with -t repro when using the build script linux-build.bash? If so, try removing -fbounds-check on line 31 in your triton-gnu.txt and re-building the model to see if it solves your problem.

[Parag0206]

Hi @yichengt900, Thanks.
The fix you suggested, of removing "-fbounds-check" in .mk file, does lead to elimination of that error.

But, I am now seeing a new error:

Error termination. Backtrace:
Operating system error: Cannot allocate memory
Allocation would exceed memory limit

Can you help me resolve this?

[yichengt900]

Hi @Parag0206, hmm, that's interesting. It's hard for me to debug without actually running your experiment. Could you share the grid files, at least? I can then try to reproduce your issue on my end.

[Parag0206]

Hi @yichengt900, I have shared the grid files and parameter files here:
https://github.com/Parag0206/BOB_0.073_2013

Thanks for your help. Do let me know if you require any other files

Edit: Hi, I have made a minimal(5 day) version of the model. Here is the link (https://indianinstituteofscience-my.sharepoint.com/:u:/g/personal/paragshende_iisc_ac_in/EWeXRJydoxdDq2nZhOoKl3EBaMlJ181ePKQuEMokcALVNg?e=V4YG1R)

[yichengt900]

Hi @Parag0206,

I apologize for the delayed response. I finally had the opportunity to review your configuration. It appears there are several issues, with the primary concern being your OBC netCDF files. For instance, the V11 variable for your south boundary is incorrectly shaped. The UV open boundary file should have a structure similar to the following:

netcdf uv_002 {
dimensions:
	time = UNLIMITED ; // (365 currently)
	nz_segment_002 = 35 ;
	ny_segment_002 = 1 ;
	nx_segment_002 = 155 ;
variables:
	double u_segment_002(time, nz_segment_002, ny_segment_002, nx_segment_002) ;
		u_segment_002:_FillValue = 1.e+20 ;
	double v_segment_002(time, nz_segment_002, ny_segment_002, nx_segment_002) ;
		v_segment_002:_FillValue = 1.e+20 ;
	int nz_segment_002(nz_segment_002) ;
	double time(time) ;
		time:_FillValue = 1.e+20 ;
		time:units = "hours since 1990-01-01 00:00:00.000000" ;
		time:calendar = "gregorian" ;
	int nx_segment_002(nx_segment_002) ;
	double dz_u_segment_002(time, nz_segment_002, ny_segment_002, nx_segment_002) ;
		dz_u_segment_002:_FillValue = 1.e+20 ;
	double dz_v_segment_002(time, nz_segment_002, ny_segment_002, nx_segment_002) ;
		dz_v_segment_002:_FillValue = 1.e+20 ;
	double lon_segment_002(nx_segment_002) ;
		lon_segment_002:_FillValue = 1.e+20 ;
	double lat_segment_002(nx_segment_002) ;
		lat_segment_002:_FillValue = 1.e+20 ;

I recommend reviewing our example OBC generation script to better understand the correct format for OBC files. Although there are a few minor issues, I think the OBC file problem probably should be the major one. Additionally, I suggest starting with a physics-only run to ensure the physical configuration is correct before proceeding to an ocean-biogeochemical coupled run.