Clean up interface for instruments

.github/workflows/simple-project.yml

@@ -22,7 +22,7 @@ jobs:
           sudo apt-get update -y && apt-get install -y apt-utils && apt-get upgrade -y
           sudo apt-get install -y python3 python3-dev python3-pip
           sudo apt-get install -y python-is-python3
-          python3 setup.py develop --user
+          python3 setup.py develop --user --with-instruments
           export PATH=$PATH:$HOME/.local/bin
     - name: Run Tests
       run: |

.gitignore

@@ -3,5 +3,6 @@ __pycache__
 *.rst
 .sphinx/build/*
 .sphinx/source/media/*
+jobrunner/options.py
 !.sphinx/source/index.rst
 !./README.rst

MANIFEST.in

@@ -1,4 +1,6 @@
 include README.rst
 include NOTICE
 include LICENSE
+include bin/*
+include requirements/*
 include media/*

README.rst

@@ -41,6 +41,15 @@ using,
    pip install --upgrade PyJobrunner
    pip uninstall PyJobRunner
 
+The following installation option can be used to allow for using
+customization specific to instruments.
+
+.. code::
+
+   pip install PyJobruner --user --install-option="--with-instruments"
+
+This allow for the use of the ``instrument:`` field in the Jobfile
+
 There maybe situations where users may want to install Jobrunner in
 development mode $\\textemdash$ to design new features, debug, or
 customize options/commands to their needs. This can be easily
@@ -49,7 +58,7 @@ directory and executing,
 
 .. code::
 
-   ./setup develop
+   ./setup develop --with-instruments
 
 Development mode enables testing of features/updates directly from the
 source code and is an effective method for debugging. Note that the
@@ -107,9 +116,10 @@ using Flash-X.
 
 Application of Jobrunner can be understood better with an example design
 of a computational experiment. Consider an experiment named `Project`
-representative of a publicly available dataset (https://github.com/Lab-Notebooks/Outflow-Forcing-BubbleML) for
-the work presented in (https://arxiv.org/pdf/2306.10174.pdf). The directory tree has the following
-structure,
+representative of a publicly available dataset
+(https://github.com/Lab-Notebooks/Outflow-Forcing-BubbleML) for the work
+presented in (https://arxiv.org/pdf/2306.10174.pdf). The directory tree
+has the following structure,
 
 .. code:: console
 
@@ -147,7 +157,7 @@ which is the instrument used to perform the experiments.
    #
    # Load Message Passing Interface (MPI) and
    # Hierarchical Data Format (HDF5) libraries
-   module load openmpi 
+   module load openmpi
    module load hdf5
 
 There are situations where requirements for Flash-X are not available as
@@ -167,9 +177,10 @@ these,
 
 Here the script ``setupAMReX.sh`` provides commands to get the source
 code for AMReX(https://github.com/AMReX-Codes/amrex) and build it for
-desired version and configuration. The script ``setupFlashX.sh`` sets 
-the version for Flash-X to perform the experiments. The ``Jobfile`` 
-indicates the use of these files by assigning them to specific Jobrunner commands,
+desired version and configuration. The script ``setupFlashX.sh`` sets
+the version for Flash-X to perform the experiments. The ``Jobfile``
+indicates the use of these files by assigning them to specific Jobrunner
+commands,
 
 .. code:: yaml
 
@@ -214,6 +225,8 @@ The ``Jobfile`` at this node assigns the use of ``environment.sh``,
 
    # file: Project/Jobfile
 
+   instrument: flashx
+
    # Scripts to include during jobrunner setup and submit commands
    job:
      setup:
@@ -295,8 +308,8 @@ conditions. Each configuration contains its respective ``Jobfile``,
 Scientific instruments like Flash-X require input during execution which
 is supplied in the form of parfiles with a ``.par`` extension. The
 parfiles along a directory tree are combined to create a single input
-file when submitting the job. For example, invocation of 
-``jobrunner submit simulation/PoolBoiling/earth_gravity`` combines
+file when submitting the job. For example, invocation of ``jobrunner
+submit simulation/PoolBoiling/earth_gravity`` combines
 ``pool_boiling.par`` and ``earth_gravity.par`` that is used to run the
 target executable ``flashx`` using the combination of ``environment.sh``
 and ``flashRun.sh``.
@@ -356,12 +369,12 @@ archiving or cleaning by extending the ``Jobfile`` for each study,
 Setup
 =====
 
-``jobrunner setup <JobWorkDir>`` creates a ``job.setup`` file 
-using ``job.setup`` scripts defined in Jobfiles along the
-directory tree. Jobrunner executes each script serially by changing the
-working directory to the location of the script. A special environment
-variable ``JobWorkDir`` provides the value of ``<JobWorkDir>`` supplied
-during invocation of the command.
+``jobrunner setup <JobWorkDir>`` creates a ``job.setup`` file using
+``job.setup`` scripts defined in Jobfiles along the directory tree.
+Jobrunner executes each script serially by changing the working
+directory to the location of the script. A special environment variable
+``JobWorkDir`` provides the value of ``<JobWorkDir>`` supplied during
+invocation of the command.
 
 .. code:: console
 
@@ -376,10 +389,10 @@ during invocation of the command.
 Submit
 ======
 
-``jobrunner submit <JobWorkDir>`` creates a ``job.submit`` file 
-using ``job.submit`` scripts and ``schedular.options``
-values defined in Jobfiles along the directory tree.
-``schedular.command`` is used to dispatch the resulting script.
+``jobrunner submit <JobWorkDir>`` creates a ``job.submit`` file using
+``job.submit`` scripts and ``schedular.options`` values defined in
+Jobfiles along the directory tree. ``schedular.command`` is used to
+dispatch the resulting script.
 
 .. code:: console
 
@@ -405,7 +418,8 @@ values defined in Jobfiles along the directory tree.
            ]
 
 Along with the ``job.submit`` script, ``job.input`` and ``job.target``
-files are also created in ``<JobWorkDir>`` using values defined in Jobfiles.
+files are also created in ``<JobWorkDir>`` using values defined in
+Jobfiles.
 
 Archive
 =======
@@ -436,17 +450,19 @@ Functionality of Jobrunner is best understood through example projects
 which can be found in following repositories:
 
 -  `akashdhruv/Multiphase-Simulations
-   <https://github.com/akashdhruv/Multiphase-Simulations>`_: A
-   lab notebook to manage development of Flash-X
+   <https://github.com/akashdhruv/Multiphase-Simulations>`_: A lab
+   notebook to manage development of Flash-X
 
 -  `Lab-Notebooks/Outflow-Forcing-BubbleML
-   <https://github.com/Lab-Notebooks/Outflow-Forcing-BubbleML>`_: Reproducibility
-   capsule for research papers (https://arxiv.org/pdf/2306.10174.pdf) (https://arxiv.org/pdf/2307.14623.pdf)
+   <https://github.com/Lab-Notebooks/Outflow-Forcing-BubbleML>`_:
+   Reproducibility capsule for research papers
+   (https://arxiv.org/pdf/2306.10174.pdf)
+   (https://arxiv.org/pdf/2307.14623.pdf)
 
 -  `Lab-Notebooks/Flow-Boiling-3DL
-   <https://github.com/Lab-Notebooks/Flow-Boiling-3D>`_: Execution environment for
-   running three-dimensional flow boiling simulations on high performance computing
-   systems.
+   <https://github.com/Lab-Notebooks/Flow-Boiling-3D>`_: Execution
+   environment for running three-dimensional flow boiling simulations on
+   high performance computing systems.
 
 **********
  Citation

bin/cmd.py

@@ -0,0 +1,91 @@
+"""Custom commands for BoxKit setup."""
+import os
+import sys
+import subprocess
+from setuptools.command.install import install
+from setuptools.command.develop import develop
+
+# custom command
+class CustomCmd:
+    """Custom command."""
+
+    user_options = [
+        ("with-instruments", None, "Install additional modules for instruments"),
+    ]
+
+    def initialize_options(self):
+        """
+        Initialize options
+        """
+        self.with_instruments = None  # pylint: disable=attribute-defined-outside-init
+
+    def finalize_options(self):
+        """
+        Finalize options
+        """
+        for option in [
+            "with_instruments",
+        ]:
+            if getattr(self, option) not in [None, 1]:
+                raise ValueError(f"{option} is a flag")
+
+    def run(self, user):
+        """
+        Run command
+        """
+        if user:
+            with_user = "--user"
+        else:
+            with_user = ""
+
+        if self.with_instruments:
+            subprocess.run(
+                f"{sys.executable} -m pip install -r requirements/instruments.txt {with_user}",
+                shell=True,
+                check=True,
+                executable="/bin/bash",
+            )
+
+        with open("jobrunner/options.py", "w", encoding="ascii") as optfile:
+
+            optfile.write(f"INSTRUMENTS={self.with_instruments}\n")
+
+
+# replaces the default build command for setup.py
+class InstallCmd(install, CustomCmd):
+    """Custom build command."""
+
+    user_options = install.user_options + CustomCmd.user_options
+
+    def initialize_options(self):
+        install.initialize_options(self)
+        CustomCmd.initialize_options(self)
+
+    def finalize_options(self):
+        install.finalize_options(self)
+        CustomCmd.finalize_options(self)
+
+    def run(self):
+
+        CustomCmd.run(self, self.user)
+        install.run(self)
+
+
+# replaces custom develop command for setup.py
+class DevelopCmd(develop, CustomCmd):
+    """Custom develop command."""
+
+    user_options = develop.user_options + CustomCmd.user_options
+
+    def initialize_options(self):
+        develop.initialize_options(self)
+        CustomCmd.initialize_options(self)
+
+    def finalize_options(self):
+        develop.finalize_options(self)
+        CustomCmd.finalize_options(self)
+
+    def run(self):
+
+        develop.run(self)
+        CustomCmd.run(self, self.user)

jobrunner/__init__.py

@@ -1,6 +1,10 @@
 """Initialization method for PyJobrunner"""
 
+from . import options
+
+if options.INSTRUMENTS == 1:
+    from . import instruments
+
 from . import lib
-from . import resources
 from . import api
 from . import cli

jobrunner/__meta__.py

@@ -1,7 +1,7 @@
 """Metadata for jobrunner"""
 
 __pkgname__ = "PyJobRunner"
-__version__ = "2023.11"
+__version__ = "2023.11.01"
 __authors__ = "Akash Dhruv"
 __license__ = "Apache Software License"
 __copyright__ = "Copyright (c) Akash Dhruv 2023. All Rights Reserved."

jobrunner/lib/_filetools.py

@@ -7,7 +7,10 @@
 
 # local imports
 from jobrunner import lib
-from jobrunner import resources
+from jobrunner import options
+
+if options.INSTRUMENTS == 1:
+    from jobrunner import instruments
 
 
 def CreateSetupFile(config):
@@ -77,7 +80,6 @@ def CreateInputFile(config):
                     job_toml[group].update(node_toml[group])
                 else:
                     job_toml[group] = node_toml[group]
-
         # start writing the job.input file
         with open(config.job.workdir + os.sep + "job.input", "w") as inputfile:
 
@@ -106,8 +108,8 @@ def CreateInputFile(config):
             #        else:
             #            inputfile.write(f'{" "*2}{variable} = {value}\n')
 
-        if config.instrument == "flashx":
-            resources.flashx.CreateParfile(config.job.workdir)
+        if config.instrument == "flashx" and options.INSTRUMENTS == 1:
+            instruments.flashx.CreateParfile(config.job.workdir)
 
 
 def CreateTargetFile(config):

jobrunner/lib/_parsetools.py

@@ -7,6 +7,8 @@
 import toml
 import yaml
 
+from jobrunner import options
+
 
 class __YamlLoader(yaml.SafeLoader):
     """
@@ -96,7 +98,13 @@ def ParseJobConfig(basedir, workdir):
                     )
 
                 # set values if instrument not already set
-                config[key] = work_dict[key]
+                if options.INSTRUMENTS == 1:
+                    config[key] = work_dict[key]
+
+                else:
+                    raise NotImplementedError(
+                        "[jobrunner] Not configured with instruments. Please reinstall with releveant options"
+                    )
 
                 continue
 

requirements/core.txt

@@ -0,0 +1,4 @@
+click
+toml
+pyyaml
+alive-progress==3.1.4

requirements/instruments.txt

@@ -0,0 +1,3 @@
+numpy
+h5py
+scipy==1.8.0