E3SM set up on Perlmutter: Adding SSH key to the ssh-agent

[abhishekh-srivastava]

Hello. I am trying to set up E3SM on Perlmutter. I am stuck on two points:

(1) Adding your SSH key to the ssh-agent (https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent): The page suggests the following command to add ssh private key to the ssh-agent:
ssh-add --apple-use-keychain ~/.ssh/id_ed25519

However, the --apple-use-keychain is specific to the apple systems. What is the equivalent command for Perlmutter? if I don't use a keychain, I have to enter the passphrase later at every step while running "run_e3sm.template.sh" .

(2) When I run the "run_e3sm.template.sh", I get the following error:

ERROR: Command: '/global/u1/a/asrivas/E3SMv3/code/latest/components/elm/bld/build-namelist -infile /pscratch/sd/a/asrivas/e3sm-scratch/your_casename/tests/XS_2x5_ndays/case_scripts/Buildconf/elmconf/namelist -csmdata /global/cfs/cdirs/e3sm/inputdata -inputdata /pscratch/sd/a/asrivas/e3sm-scratch/your_casename/tests/XS_2x5_ndays/case_scripts/Buildconf/elm.input_data_list -ignore_ic_year -namelist " &elm_inparm start_ymd=00010101 /" -use_case 1850_CMIP6_control -res r05 -clm_start_type default -envxml_dir /pscratch/sd/a/asrivas/e3sm-scratch/your_casename/tests/XS_2x5_ndays/case_scripts -l_ncpl 48 -r_ncpl 8 -lnd_frac /global/cfs/cdirs/e3sm/inputdata/share/domains/domain.lnd.r05_IcoswISC30E3r5.231121.nc -glc_nec 0 -co2_ppmv 284.317 -co2_type diagnostic -ncpl_base_period day -config /pscratch/sd/a/asrivas/e3sm-scratch/your_casename/tests/XS_2x5_ndays/case_scripts/Buildconf/elmconf/config_cache.xml -bgc bgc -nutrient cnp -nutrient_comp_pathway rd -soil_decomp ctc -methane -solar_rad_scheme top -mask IcoswISC30E3r5' failed with error 'Can't locate XML/LibXML.pm in @inc (you may need to install the XML::LibXML module) (@inc entries checked: /global/u1/a/asrivas/E3SMv3/code/latest/components/elm/bld /global/u1/a/asrivas/E3SMv3/code/latest/components/elm/bld /global/u1/a/asrivas/E3SMv3/code/latest/cime/utils/perl5lib /global/u1/a/asrivas/E3SMv3/code/latest/cime/utils/perl5lib/Config/ /global/u1/a/asrivas/E3SMv3/code/latest/components/elm/bld /global/cfs/cdirs/e3sm/perl/lib/perl5-only-switch/x86_64-linux-thread-multi /global/cfs/cdirs/e3sm/perl/lib/perl5-only-switch //////////////////////////////////////////////////////global/common/software/spackecp/perlmutter/e4s-23.08/94543/spack/opt/spack/linux-sles15-zen3/gcc-12.3.0/perl-5.38.0-pz3nhjkjcpfx2cbbquomtfa6c3bxc3xj/lib/perl5 //////////////////////////////////////////////////////global/common/software/spackecp/perlmutter/e4s-23.08/94543/spack/opt/spack/linux-sles15-zen3/gcc-12.3.0/perl-5.38.0-pz3nhjkjcpfx2cbbquomtfa6c3bxc3xj/lib/site_perl/5.38.0/x86_64-linux-thread-multi //////////////////////////////////////////////////////global/common/software/spackecp/perlmutter/e4s-23.08/94543/spack/opt/spack/linux-sles15-zen3/gcc-12.3.0/perl-5.38.0-pz3nhjkjcpfx2cbbquomtfa6c3bxc3xj/lib/site_perl/5.38.0 //////////////////////////////////////////////////////global/common/software/spackecp/perlmutter/e4s-23.08/94543/spack/opt/spack/linux-sles15-zen3/gcc-12.3.0/perl-5.38.0-pz3nhjkjcpfx2cbbquomtfa6c3bxc3xj/lib/5.38.0/x86_64-linux-thread-multi //////////////////////////////////////////////////////global/common/software/spackecp/perlmutter/e4s-23.08/94543/spack/opt/spack/linux-sles15-zen3/gcc-12.3.0/perl-5.38.0-pz3nhjkjcpfx2cbbquomtfa6c3bxc3xj/lib/5.38.0) at /global/u1/a/asrivas/E3SMv3/code/latest/cime/utils/perl5lib/Config/SetupTools.pm line 5.
BEGIN failed--compilation aborted at /global/u1/a/asrivas/E3SMv3/code/latest/cime/utils/perl5lib/Config/SetupTools.pm line 5.
Compilation failed in require at /global/u1/a/asrivas/E3SMv3/code/latest/components/elm/bld/ELMBuildNamelist.pm line 459.' from dir '/pscratch/sd/a/asrivas/e3sm-scratch/your_casename/tests/XS_2x5_ndays/case_scripts/Buildconf/elmconf'

Please suggest how to fix these problems. Any step-by-step suggestion is highly welcome.

[rljacob]

For the first one, just remove the --apple-use-keychain option. For the second one, @ndkeen don't we have that perl lib installed somewhere?

[ndkeen]

Yes. @abhishekh-srivastava asked same question as NERSC ticket and I answered.

It's not clear, but it could be that e3sm_unified is being used here. Which we know will hit this error.

[abhishekh-srivastava]

Hi Robert and Noel, Following Noel, I am using a different terminal to run the code. The path where I run the code is ***@***.***:/global/u1/a/asrivas/E3SM> As far as I understand, the environment appears on the far left side of the above path. But there is no (e3sm_unified) before asrivas@, so I assume that e3sm_unified is not used here. As Robert suggested, I am not using the --apple-use-keychain option. The “config” file looks like as follows: ***@***.***:/global/u1/a/asrivas/.ssh> ls config id_ed25519 id_ed25519.pub known_hosts ***@***.***:/global/u1/a/asrivas/.ssh> cat config Host github.com AddKeysToAgent yes IdentityFile ~/.ssh/id_ed25519 ————— When I run the code, I have to enter the passphrase multiple times, as shown below: Cloning into '/global/u1/a/asrivas/E3SMv3/code/latest/cime'... Enter passphrase for key '/global/homes/a/asrivas/.ssh/id_ed25519': Cloning into '/global/u1/a/asrivas/E3SMv3/code/latest/components/eam/src/physics/cosp2/external'... Enter passphrase for key '/global/homes/a/asrivas/.ssh/id_ed25519': Cloning into '/global/u1/a/asrivas/E3SMv3/code/latest/components/eam/src/physics/crm/pam/external'... Enter passphrase for key '/global/homes/a/asrivas/.ssh/id_ed25519': Cloning into '/global/u1/a/asrivas/E3SMv3/code/latest/components/eam/src/physics/rrtmgp/external'... Enter passphrase for key '/global/homes/a/asrivas/.ssh/id_ed25519': Cloning into '/global/u1/a/asrivas/E3SMv3/code/latest/components/elm/src/external_models/fates'... Enter passphrase for key '/global/homes/a/asrivas/.ssh/id_ed25519': Cloning into '/global/u1/a/asrivas/E3SMv3/code/latest/components/elm/src/external_models/mpp'... Enter passphrase for key '/global/homes/a/asrivas/.ssh/id_ed25519': ***@***.***:/global/u1/a/asrivas/E3SM>

…

_____ Is there any way I can save the passphrase somewhere that can be automatically used, and I do not have to type the passphrase manually?

-Abhishekh ———————————————————————————————— Abhishekh Kumar Srivastava, Ph.D. Assistant Project Scientist Department of Land, Air and Water Resources University of California, Davis Office: 223 Hoagland Hall, 1 Shields Avenue Davis, CA 95616-8627 ----------------------------------------------------------------

On Aug 20, 2024, at 11:09 AM, noel ***@***.***> wrote: Yes. @abhishekh-srivastava <https://github.com/abhishekh-srivastava> asked same question as NERSC ticket and I answered. It's not clear, but it could be that e3sm_unified is being used here. Which we know will hit this error. — Reply to this email directly, view it on GitHub <#6557 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A5QP2IRVVM5ISQDDZWNGL4DZSOA7FAVCNFSM6AAAAABM2DMQ26VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMZZHA4TQNQ>. You are receiving this because you were mentioned.

[rljacob]

After you start the agent, type "ssh-add" to add your key to the agent. Then you won't have to keep typing the passphrase.

[mt5555]

Best practice: you should not have any keys on perlmutter. You only need keys on your local laptop/workstation, and "forward" them when you log into perlmutter:

https://docs.github.com/en/authentication/connecting-to-github-with-ssh/using-ssh-agent-forwarding

[abhishekh-srivastava]

Hi Robert, OK, so I followed your instructions and then also added "eval "$(ssh-agent—s)” to my .bashrc file. This solves the problem of entering the passphrase multiple times. I made some progress with this. But now, I get the following error: Submitting job script sbatch --time 02:00:00 -q regular --account e3sm .case.run --resubmit ERROR: Command: 'sbatch --time 02:00:00 -q regular --account e3sm .case.run --resubmit' failed with error 'sbatch: error: No available node hour balance information for user asrivas, account e3sm. Cannot proceed. sbatch: error: Batch job submission failed: Unspecified error' from dir '/pscratch/sd/a/asrivas/e3sm-scratch/your_casename/tests/XS_2x5_ndays/case_scripts’ Please suggest how to fix this.

…

-Abhishekh ---------------------------------------------------------------- Abhishekh Kumar Srivastava, Ph.D. Assistant Project Scientist Department of Land, Air and Water Resources University of California, Davis Office: 223 Hoagland Hall, 1 Shields Avenue Davis, CA 95616-8627 ----------------------------------------------------------------

On Aug 20, 2024, at 11:52 AM, Robert Jacob ***@***.***> wrote: After you start the agent, type "ssh-add" to add your key to the agent. Then you won't have to keep typing the passphrase. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[rljacob]

That error is because you're trying to submit to the e3sm project. You have to change that to your NERSC project. In your CASEDIR, do "./xmlchange PROJECT=

[ndkeen]

I'm thinking you are running a launch script. Inside that script, there is probably a line that sets the project.
For example:

readonly PROJECT="e3sm"

You would just want to change that to your project at NERSC (where hours are charged)

Note this is very common first error for users outside of e3sm -- it would be great if we could find a way to set this a little smarter in our outward facing scripts. Some logic to first check if user in the e3sm project on a given machine, and if not, just use the default project for the user.

[mahf708]

I will fix this in my PR #6534, see: fd620a5

- readonly PROJECT="e3sm"
+ readonly PROJECT="$(sacctmgr show user $USER format=DefaultAccount | tail -n1 | tr -d ' ')"