Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs/add howto #144

Merged
merged 11 commits into from
Aug 10, 2022
1 change: 1 addition & 0 deletions doc/source/user_guide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -47,4 +47,5 @@ the RegionProcessor and validated using DataStructureDefinition.
user_guide/variable
user_guide/region
user_guide/model-mapping
user_guide/howto
phackstock marked this conversation as resolved.
Show resolved Hide resolved
user_guide/local-usage
20 changes: 10 additions & 10 deletions doc/source/user_guide/local-usage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ Local usage of a project
.. attention:: This page is intended for users who are familiar with Python,
`git <https://git-scm.com>`_ and a service like `GitHub <https://github.com>`_.

You can use the **nomenclature** package locally (on your machine) for validation
and region-aggregration. This can be helpful as part of processing your model results,
or to ensure that a submission to an IIASA Scenario Explorer instance will succeed.
You can use the **nomenclature** package locally (on your machine) for validation and
region-aggregration. This can be helpful as part of processing your model results, or to
ensure that a submission to an IIASA Scenario Explorer instance will succeed.

The following **requirements** must be satisfied to use a :class:´nomenclature`-based
The following **requirements** must be satisfied to use a :class:`nomenclature`-based
project locally:

1. Install the **nomenclature** package (see the :ref:`installation` instructions)
Expand All @@ -28,13 +28,13 @@ project locally:
Validation against codelists
----------------------------

The easiest use case is to validate that a data file or an
:class:`IamDataFrame <pyam.IamDataFrame>` is compatible with the codelists
(lists of variables and regions) of a project's :class:`DataStructureDefinition`.
The easiest use case is to validate that a data file or an :class:`IamDataFrame
<pyam.IamDataFrame>` is compatible with the codelists (lists of variables and regions)
of a project's :class:`DataStructureDefinition`.

If there are inconsistencies with the codelists, the method
:meth:`validate <DataStructureDefinition.validate>` will raise an error.
If the scenario data is consistent, the method returns *True*.
If there are inconsistencies with the codelists, the method :meth:`validate
<DataStructureDefinition.validate>` will raise an error. If the scenario data is
consistent, the method returns *True*.

.. code-block:: python

Expand Down
85 changes: 85 additions & 0 deletions doc/source/user_guide/model-registration.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
.. _model-registration:

.. currentmodule:: nomenclature

Model registration
==================

This guide presents a quick instruction for "registering a model" in a project workflow.

It is still recommended to read the detailed instructions under :ref:`model_mapping` and
:ref:`region` in particular.

Region processing
-----------------

"Registering a model" for a **nomenclature**-based project workflow requires two
specifications:

* a model mapping to perform region aggregation from *native_regions* to
*common_regions* and renaming of model native regions (optional)
* a list of region names as they should appear in the processed scenario data

Model mapping
^^^^^^^^^^^^^

For this guide we will consider a model mapping as an example:

.. code-block:: yaml

model: model_a
native_regions:
- region_1: model_a|Region 1
- region_2: model_a|Region 2
- region_3: model_a|Region 3
common_regions:
- World:
- region_1
- region_2
- region_3
- Common region 1:
- region_1
- region_2

``model_a`` reports three regions natively, ``region_1``, ``region_2`` and ``region_3``.
These three are to be renamed to the ``{model}|{region}`` pattern as is common place in
many projects. For the ``common_regions``, there are ``World`` and ``Common region 1``.

Region definitions
^^^^^^^^^^^^^^^^^^

In order to constitute a valid "model registration", regions ``model_a|Region 1``,
``model_a|Region 2``, ``model_a|Region 3``, ``World`` and ``Common region 1`` **must**
be part of the region definitions.

``region_1``, ``region_2`` and ``region_3`` are **not required** since they refer to the
input names of ``model_a``'s regions and will be renamed in the processing.

In most cases, the common regions (in the above example ``World`` and ``Common region
1``) will already be defined in a file called ``definitions/region/regions.yaml``.

The model native regions, however, most likely need to be added. For this, a new yaml
file should be created, usually in ``definitions/region/model_native_regions/``. The
file does not need to have any special name but it is recommended to use the model name.

In order to complete the model registration for the example above, we would therefore
add a file called ``model_a.yaml`` to ``definitions/region/model_native_regions/`` with
the following content:

.. code-block:: yaml

model_a:
- model_a|Region 1
- model_a|Region 2
- model_a|Region 3

This combination of a model mapping and the definition of all regions that are part of
the processing output constitutes a complete model registration.

Continuous Integration
----------------------

In most cases, a model registration is submitted as a pull request to a project repository hosted on GitHub. As part
of this, :func:`assert_valid_structure` (details can be found here: :ref:`cli`) is run
automatically to ensure that the model registration is valid. Any regions that are, for
example mentioned in a mapping but not defined will raise an error.
15 changes: 13 additions & 2 deletions nomenclature/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,24 @@
@cli.command("validate-yaml")
@click.argument("path", type=click.Path(exists=True, path_type=Path))
def cli_valid_yaml(path: Path):
"""Assert that all yaml files in `path` can be parsed without errors"""
"""Assert that all yaml files in `path` are syntactically valid."""
assert_valid_yaml(path)


@cli.command("validate-project")
@click.argument("path", type=click.Path(exists=True, path_type=Path))
def cli_valid_project(path: Path):
"""Assert that `path` is a valid project nomenclature"""
"""Assert that `path` is a valid project nomenclature

This test includes three steps:

1. Test that all yaml files in `definitions/` and `mappings/` can be correctly read
as yaml files. This is a formal check for yaml syntax only.
2. Test that all files in `definitions/` can be correctly parsed as a
:class:`DataStructureDefinition` object comprised of individual codelists.
3. Test that all model mappings in `mappings/` can be correctly parsed as a
:class:`RegionProcessor` object. This includes a check that all regions mentioned
in a model mapping are defined in the region codelist.
"""
assert_valid_yaml(path)
assert_valid_structure(path)