Bad pe and aprun configurations for PET test case on Titan

[philipwjones]

On Titan:
./create_test PET_Ln9.T62_oQU240.GMPAS-NYF.titan_pgi
creates both a 2-threaded case and a single-threaded case.

In the single-thread case directory, env_mach_pes.xml is correct for a 64-task MPI run with 1 thread/task, but the aprun command generated is:
aprun -S 4 -n 64 -N 8 -d 1
which has a bad -S flag (should be -S8 for a 4-node run)
causing the single-thread case to fail.

In the 2-threaded case, the env_mach_pes.xml contains bad NTASKS values. It should be creating a 4-node case (64-core) with 32 MPI tasks and 2 threads per task, but env_mach_pes.xml has 64 set for NTASKS in all components. This can be manually changed and reset, but the defaults are wrong for this case.

[jgfouca]

@philipwjones , I was unable to duplicate this error. I added a unit test to aprun:

    >>> ntasks = [64, 64, 64, 64, 64, 64, 64, 64, 1]                                                         
    >>> nthreads = [1, 1, 1, 1, 1, 1, 1, 1, 1]                                                              
    >>> rootpes = [0, 0, 0, 0, 0, 0, 0, 0, 0]                                                                 
    >>> pstrids = [1, 1, 1, 1, 1, 1, 1, 1, 1]                                                    
    >>> _get_aprun_cmd_for_case_impl(ntasks, nthreads, rootpes, pstrids, max_tasks_per_node, pes_per_node, pio_numtasks, pio_async_interface, compiler, machine, run_exe) 
    ('aprun -S 8 -cc numa_node -n 64 -N 16 -d 1 acme.exe ', 4)  

I also tried to run PET_Ln9.T62_oQU240.GMPAS-NYF.titan_pgi manually on titan and I never made it to the single-thread case because the threaded case failed.

[philipwjones]

@jgfouca Hmmm... mine definitely makes it through the first threaded test and then fails with the error below and a clearly wrong aprun. (Note that the error message itself doesn't show up in the log message - reported in a different issue - but I did manage to track it down to a bad resource request)

The env_mach_pes.xml looks correct in that directory. Is there a chance the second case is inadvertently carrying over part of the environment from the threaded case (the -S 4 was appropriate for the 2-thread run)?

Let me know if there is something I can do to help debug...we're trying to get some threading mods into MPAS and the tests complete fine on Edison but would like to verify on Titan since that's where some thread bugs showed up before.

Exception during run:
ERROR: Command: 'aprun -S 4 -n 64 -N 8 -d 1 /ccs/home/pwjones/acme_scratch/csc190/PET_Ln9.T62_oQU240.GMPAS-NYF.titan_pgi.20170412_092206_pkv2pg/bld/acme.exe >> acme.log.$LID 2>&1 ' failed with error '' from dir '/lustre/atlas/scratch/pwjones/csc190/PET_Ln9.T62_oQU240.GMPAS-NYF.titan_pgi.20170412_092206_pkv2pg.single_thread/run'

[jgfouca]

@philipwjones yes, given what you've said there's a good chance something is carrying-over from the threaded case into the serial case. I'll continue to look at it.

[jgfouca]

Fixed in ESMCI with this: ESMCI/cime#1414