Rynner usage

Welcome to the Rynner wiki!

A brief reminder that this project is still raw and may contain bugs.

Script example

Whereas the original idea was to have a proper GUI the safest method to make it work is to use a separate Python script for submission. Below, we provide an example of such a script and explain which parameters are of importance.

REMINDER: for this code to work you have to set up ssh-agent first so that you don't get asked for a password/passphrase while connecting to a remote machine.

Imported modules

The MWE requires only two modules to run: parsl and rynner itself.

from parsl.channels import SSHChannel
from parsl.providers.slurm.slurm import SlurmProvider
from rynner.rynner import Rynner

Login & domain

Next step is to provide your username that you use on the remote machine and the address of the machine itself. At the moment this is a general approach to avoid any potential issues with pre-saved user credentials. The functionality allowing to have your personal settings stored permanently on a local machine is still under development. The process of entering your data can be pretty repetitive but this was the most optimal solution at the moment.

remote_username = input("Enter your username: ")
domain = input("Enter the domain: ")

connect_ = input(f'Connect as {remote_username} at {domain}? [Yy/Yes/No/Nn]:')

This code also checks if everything is entered correctly and allows you to stop execution in the case of any typos.

Server connection

If you need to amend the credentials just enter n, N, no or No. Otherwise, the code creates a connection provider that connects to your remote machine via ssh connection and allows to interact with job submissions using SLURM.

if connect_.lower() not in ('y', 'yes', ''):
    print('No connection established, exiting.')
    exit(1)
else:
    provider = SlurmProvider(
        partition='compute',
        account='scw1000',
        exclusive=False,
        walltime="00:01:00"
        channel=SSHChannel(
            hostname=domain,
            username=remote_username,
            script_dir='/home/' + remote_username + '/rynner_test_submission_scripts'
        )
    )

Here, we discuss the most important initial parameters that have to be used for you to get a stable script. There is a change it would work without them but it's highly likely not true and you won't be able to trace the error.

• partition - a section of the HPC that is used for computations, the name compute is widely used.
• account - typically, a user has a project or a few assigned to them, this is the code of the project. If you don't have any than just don't pass this parameter to SlurmProvider, if you have one than you can do both (pass and don't), if you have two or more than you have to explicitly specify which one you'd like to use. Otherwise, SLURM won't be able to submit the job.
• exclusive - defines whether you allocate the whole node for you job or not, True by default. It takes ages to get higher in the queue, use only when you really need it, don't waste computational resources by reserving extra for yourself, let other people use it too.
• walltime - walltime of the job, the default value is longer making the job hanging in the queue longer.
• channel - the way to connect to the remote machine. Both remote_username and domain have been provided earlier, script_dir is the directory on the remote machine where all SLURM submission scripts are stored. The official documentation states:

Full path to a script dir where generated scripts could be sent to.

NOTE: avoid using /tmp as the directory, since you need to change the permission of it and you can't as a regular user; also, try not to use any sub-directory of /tmp as it leads to inconsistent behaviour of submissions. Typical file paths here for an HPC should look like /home/remote_username or /scratch/remote_username + some additional directory specifically for scripts.

Running Rynner

Init

rynner = Rynner(provider,
                remote_path='/home/' + remote_username,
                local_path='/tmp/rynner_test')

run = rynner.create_run(script='cat Makefile > tmps',  # code to execute
                        uploads=['Makefile'],  # paths to files to copy to the remote machine
                        downloads=['tmps'],  # paths? to files to copy from the remote machine
                        remote_namespace='rynner_test'
                        # namespace to store data on the cluster
                        )

Upload files

rynner.upload(run)
print('upload')

Submit jobs

print(run)
rynner.submit(run)

print('submit')
print(run)

Update job status

runs = [run]
rynner.update(runs)
print('runs')
print(runs)

Download results

rynner.download(run)