Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Computer / Code configuration need support for templating #4680

Open
ltalirz opened this issue Jan 26, 2021 · 0 comments
Open

Computer / Code configuration need support for templating #4680

ltalirz opened this issue Jan 26, 2021 · 0 comments

Comments

@ltalirz
Copy link
Member

ltalirz commented Jan 26, 2021

Is your feature request related to a problem? Please describe

While it is already very useful to be able to store AiiDA computer and code configurations in yaml files like these, it its current form it is difficult to share them on the aiida code registry in a way that makes them easily reusable.

The issue is that certain parameters of the configuration (e.g. which slurm "account" to use or perhaps which scheduler queue) are typically set in the "prepend text" of the computer but will differ from user to user.

This led @yakutovicha to add a new computer just for a specific project, which leads to duplication of information and is not reusable by others.

Describe the solution you'd like

Which parameters vary can depend on the computer in question, so I don't think this can be generalized at the level of the AiiDA-internal computer representation.

However, one could consider introducing a templating language like jinja2 inside the configuration files (jinja2 inside yaml is something done by ansible as well, i.e. this combination should not pose issues).

This also allows to set default values where it makes sense (e.g. for the label, which is good to standardize but which some users may want to change):

label: "{{ label | default='daint-mc' }}"
hostname: "daint.cscs.ch"
description: Piz Daint supercomputer at CSCS Lugano, Switzerland, multicore partition.
transport: ssh
scheduler: "slurm"
work_dir: "/scratch/snx3000/{username}/aiida_run/"
shebang: "#!/bin/bash"
mpirun_command: "srun -n {tot_num_mpiprocs}"
mpiprocs_per_machine: 36
prepend_text: |
    ### computer prepend_text start ###
    #SBATCH --partition={{ slurm_partition | default('normal') }}
    #SBATCH --account={{ slurm_account }}
    #SBATCH --constraint=mc
    #SBATCH --cpus-per-task=1
    export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
    source $MODULESHOME/init/bash
    module load daint-mc
    ulimit -s unlimited
    ### computer prepend_text end ###
append_text: " "

The "interactive version" of this is straightforward to implement - simply pass all the config values through a jinja2 parser and prompt the user for missing values.

The "non-interactive version" is a bit less obvious to me. One way would be to follow the approach of the jinja2-cli and add a cli option --config-vars that takes another yaml with just

slurm_partition: normal
slurm_account: abc

(not super elegant, perhaps there are better ways...)

P.S. jinja2 doesn't clash with the existing templating mechanism used by AiiDA (e.g. {username}).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants