Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

slightly improve v3 run_script template #6534

Merged
merged 5 commits into from
Sep 5, 2024
Merged

slightly improve v3 run_script template #6534

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b435240
small mods to v3 script
mahf708 Jul 31, 2024
33b4eaa
Use $PSCRATCH instead
mahf708 Jul 31, 2024
7581ae0
fix link to best practices, save code in home
mahf708 Aug 5, 2024
fd620a5
gen custom_pelayout, better case loc, and default proj
mahf708 Aug 21, 2024
f84cad1
improve the comment regarding PROJECT
mahf708 Aug 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 59 additions & 11 deletions run_e3sm.template.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,7 @@
# E3SM Coupled Model Group run_e3sm script template.
#
# Bash coding style inspired by:
# http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming

# TO DO:
# - custom pelayout
# https://web.archive.org/web/20200620202413/http://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming

main() {

Expand All @@ -17,8 +14,9 @@ main() {

# Machine and project
readonly MACHINE=pm-cpu
# BEFORE RUNNING: CHANGE this to your project
readonly PROJECT="e3sm"
# NOTE: The command below will return your default project on SLURM-based systems.
# If you are not using SLURM or need a different project, remove the command and set it directly
readonly PROJECT="$(sacctmgr show user $USER format=DefaultAccount | tail -n1 | tr -d ' ')"
mahf708 marked this conversation as resolved.
Show resolved Hide resolved

# Simulation
readonly COMPSET="WCYCL1850"
Expand Down Expand Up @@ -46,8 +44,8 @@ readonly GET_REFCASE=TRUE
#readonly RUN_REFDATE="" # same as MODEL_START_DATE for 'branch', can be different for 'hybrid'

# Set paths
readonly CASE_ROOT="${PSCRATCH}/E3SMv3/${CASE_NAME}"
readonly CODE_ROOT="${HOME}/E3SMv3/code/${CHECKOUT}"
readonly CASE_ROOT="/pscratch/sd/r/${USER}/e3sm-scratch/${CASE_NAME}"

# Sub-directories
readonly CASE_BUILD_DIR=${CASE_ROOT}/build
Expand All @@ -56,6 +54,7 @@ readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive
# Define type of run
# short tests: 'XS_2x5_ndays', 'XS_1x10_ndays', 'S_1x10_ndays',
# 'M_1x10_ndays', 'M2_1x10_ndays', 'M80_1x10_ndays', 'L_1x10_ndays'
# * can replace XS, M, etc. with custom-XY with XY being the node count
# or 'production' for full simulation
readonly run='XS_2x5_ndays'
if [ "${run}" != "production" ]; then
Expand Down Expand Up @@ -118,6 +117,9 @@ fetch_code
# Create case
create_newcase

# Custom PE layout
custom_pelayout

# Setup
case_setup

Expand Down Expand Up @@ -322,6 +324,14 @@ create_newcase() {

echo $'\n----- Starting create_newcase -----\n'


if [[ ${PELAYOUT} == custom-* ]];
then
layout="M" # temporary placeholder for create_newcase
else
layout=${PELAYOUT}
fi

if [[ -z "$CASE_GROUP" ]]; then
${CODE_ROOT}/cime/scripts/create_newcase \
--case ${CASE_NAME} \
Expand All @@ -333,7 +343,7 @@ create_newcase() {
--machine ${MACHINE} \
--project ${PROJECT} \
--walltime ${WALLTIME} \
--pecount ${PELAYOUT}
--pecount ${layout}
else
${CODE_ROOT}/cime/scripts/create_newcase \
--case ${CASE_NAME} \
Expand All @@ -346,7 +356,7 @@ create_newcase() {
--machine ${MACHINE} \
--project ${PROJECT} \
--walltime ${WALLTIME} \
--pecount ${PELAYOUT}
--pecount ${layout}
fi


Expand Down Expand Up @@ -398,6 +408,43 @@ case_setup() {
popd
}

#-----------------------------------------------------
custom_pelayout() {

if [[ ${PELAYOUT} == custom-* ]];
then
echo $'\n CUSTOMIZE PROCESSOR CONFIGURATION:'

# Number of cores per node (machine specific)
if [ "${MACHINE}" == "pm-cpu" ]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mahf708 : could you add more E3SM machines (compy, chrysalis, anvil) here to make it more portable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good! Will address soon.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added compy, chrys and anvil. Note that @rljacob indicated in the past that this script was aimed primarily at pm-cpu users, hence I didn't add them originally. I can further customize the CASE_ROOT to suit all machines ($PSCRATCH on pm-cpu, /lcrc/group/e3sm/$USER on chrys and anvil, and /compyfs/$USER on compy if desired). I am in favor of portability (but I didn't want to make the logic complex preemptively) so let me know if you'd like that extended as well

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes because pm-cpu is our only officially supported machine.

ncore=128
elif [ "${MACHINE}" == "chrysalis" ]; then
ncore=64
elif [ "${MACHINE}" == "compy" ]; then
ncore=40
elif [ "${MACHINE}" == "anvil" ]; then
ncore=36
else
echo 'ERROR: MACHINE = '${MACHINE}' is not supported for custom PE layout.'
exit 400
fi

# Extract number of nodes
tmp=($(echo ${PELAYOUT} | tr "-" " "))
nnodes=${tmp[1]}

# Customize
pushd ${CASE_SCRIPTS_DIR}
./xmlchange NTASKS=$(( $nnodes * $ncore ))
./xmlchange NTHRDS=1
./xmlchange MAX_MPITASKS_PER_NODE=$ncore
./xmlchange MAX_TASKS_PER_NODE=$ncore
popd

fi

}

#-----------------------------------------------------
case_build() {

Expand Down Expand Up @@ -534,9 +581,10 @@ copy_script() {

local script_provenance_dir=${CASE_SCRIPTS_DIR}/run_script_provenance
mkdir -p ${script_provenance_dir}
local this_script_name=`basename $0`
local this_script_name=$( basename -- "$0"; )
local this_script_dir=$( dirname -- "$0"; )
local script_provenance_name=${this_script_name}.`date +%Y%m%d-%H%M%S`
cp -vp ${this_script_name} ${script_provenance_dir}/${script_provenance_name}
cp -vp "${this_script_dir}/${this_script_name}" ${script_provenance_dir}/${script_provenance_name}
Comment on lines -537 to +587
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that these edits were made because this specific function before won't work if the script is submitted non-interactively. With the fix, it works fine. I don't know what the problem is exactly....

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't understand why, but I'll trust you on that.


}

Expand Down