Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a pyam.iiasa.platforms() function #829

Merged
1 change: 1 addition & 0 deletions RELEASE_NOTES.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# Next release

- [#829](https://github.com/IAMconsortium/pyam/pull/829) Add a `pyam.iiasa.platforms()` function for a list of available platforms
- [#826](https://github.com/IAMconsortium/pyam/pull/826) Add `read_ixmp4()` function and extend integration test
- [#825](https://github.com/IAMconsortium/pyam/pull/825) Add support for Python 3.12
- [#824](https://github.com/IAMconsortium/pyam/pull/824) Update ixmp4 requirement to >=0.7.1
Expand Down
9 changes: 6 additions & 3 deletions docs/api/iiasa.rst
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,13 @@ You will be prompted to enter your password.
*Scenario Apps* instances
-------------------------

Coming soon...
The *Scenario Apps* use the |ixmp4| package as a database backend.
You can list all available ixmp4 platforms hosted by IIASA using the following:

*Scenario Explorer* instances
-----------------------------
.. autofunctions:: platforms

*Scenario Explorer* instances (legacy service)
----------------------------------------------

The *Scenario Explorer* infrastructure developed by the Scenario Services and Scientific
Software team was developed and used for projects from 2018 until 2023.
Expand Down
1 change: 1 addition & 0 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -330,6 +330,7 @@
"pandas_datareader": ("https://pandas-datareader.readthedocs.io/en/stable", None),
"unfccc_di_api": ("https://unfccc-di-api.readthedocs.io/en/stable", None),
"nomenclature": ("https://nomenclature-iamc.readthedocs.io/en/stable", None),
"ixmp4": ("https://docs.ece.iiasa.ac.at/projects/ixmp4/en/stable", None),
}

# Set up the plotting gallery with plotly scraper
Expand Down
93 changes: 69 additions & 24 deletions docs/tutorials/iiasa.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -4,23 +4,40 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Read directly from IIASA data resources\n",
"# Query data from the IIASA database infrastructure\n",
"\n",
"The IIASA *Energy, Climate, and Environment* Program hosts a suite of **Scenario Explorer** instances and related infrastructure to support analysis of integrated-assessment pathways in IPCC reports and model comparison projects. \n",
"High-profile use cases include the [IAMC 1.5°C Scenario Explorer hosted by IIASA](https://data.ece.iiasa.ac.at/iamc-1.5c-explorer) supporting the *IPCC Special Report on Global Warming of 1.5°C* (SR15) and the Horizon 2020 project [CD-LINKS](https://data.ece.iiasa.ac.at/cd-links).\n",
"The IIASA *Energy, Climate, and Environment* Program hosts a suite of **Scenario Explorer** instances and related database infrastructure to support analysis of integrated-assessment pathways in IPCC reports and model comparison projects. \n",
"High-profile use cases include the [AR6 Scenario Explorer hosted by IIASA](https://data.ece.iiasa.ac.at/ar6) supporting the *IPCC' Sixth Assessment Report* (AR6) and the Horizon 2020 project [ENGAGE](https://data.ece.iiasa.ac.at/engage).\n",
"\n",
"IIASA's [modeling platform infrastructure](http://software.ene.iiasa.ac.at/ixmp-server) and the Scenario Explorer UI is not only a great resource on its own, but it also allows the underlying datasets to be directly queried.\n",
"**pyam** takes advantage of this ability to allow you to easily pull data and work with it in your Python data processing and analysis workflow."
"IIASA's [modeling platform infrastructure](http://docs.ece.iiasa.ac.at/) and the scenario apps are not only a great resource on its own, but it also allows the underlying datasets to be queried directly via a Rest API.\n",
"The **pyam** package takes advantage of this ability to allow you to easily pull data and work with it in your Python data processing and analysis workflow."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Connecting to a data resource (aka the database API of a Scenario Explorer instance)\n",
"## Access and permission management for project-internal databases\n",
"\n",
"Accessing a data resource is done via a **Connection** object.\n",
"By default, your can connect to all public Scenario Explorer instances. "
"By default, your can connect to all public scenario database instances.\n",
"\n",
"If you have credentials to connect to a private, project-internal database instance, \n",
"you can store this information by running the following command in a console:\n",
"\n",
"```\n",
"ixmp4 login <username>\n",
"\n",
"```\n",
"\n",
"You will be prompted to enter your password.\n",
"\n",
"<div class=\"alert alert-warning\">\n",
"\n",
"Your username and password will be saved locally in plain-text for future use!\n",
"\n",
"</div>\n",
"\n",
"When connecting to a database, **pyam** will automatically search for the configuration in a known location."
]
},
{
Expand All @@ -29,41 +46,69 @@
"metadata": {},
"outputs": [],
"source": [
"import pyam\n",
"\n",
"conn = pyam.iiasa.Connection()\n",
"conn.valid_connections"
"import pyam"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you have credentials to connect to a non-public or restricted Scenario Explorer instance,\n",
"you can store this information by running the following command in a console:\n",
"## Connecting to ixmp4 database instances hosted by IIASA\n",
"\n",
"```\n",
"ixmp4 login <username>\n",
"\n",
"```\n",
"\n",
"You will be prompted to enter your password.\n",
"The *Scenario Apps* use the **ixmp4** package as a database backend.\n",
"\n",
"<div class=\"alert alert-warning\">\n",
"\n",
"Your username and password will be saved locally in plain-text for future use!\n",
"The *Scenario Services* team is currently migrating database instances from the legacy *Scenario Explorer* infrastructure \n",
"to the new *Scenario Apps* services based on the **ixmp4** package.\n",
"\n",
"</div>\n",
"\n",
"You can list all available ixmp4 platforms hosted by IIASA using the following function."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pyam.iiasa.platforms()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Connecting to a (legacy) Scenario Explorer database\n",
"\n",
"\n",
"When initializing a new **Connection** instance, **pyam** will automatically search for the configuration in a known location."
"Accessing the database connected to a **Scenario Explorer** is done via a **Connection** object."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"conn = pyam.iiasa.Connection()\n",
"conn.valid_connections"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Reading data from a database hosted by IIASA"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The **pyam** package can read both from **ixmp4** databases (connected to a *Scenario App*) \n",
"and the (legacy) **Scenario Explorer** infrastructure with the same function.\n",
"\n",
"In this example, we will be retrieving data from the *IAMC 1.5°C Scenario Explorer hosted by IIASA*\n",
"([link](https://data.ece.iiasa.ac.at/iamc-1.5c-explorer)),\n",
"which provides the quantitative scenario ensemble underpinning\n",
Expand Down Expand Up @@ -316,7 +361,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.0"
"version": "3.11.0"
}
},
"nbformat": 4,
Expand Down
33 changes: 31 additions & 2 deletions pyam/iiasa.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,14 @@
import pandas as pd
import requests
import yaml
from ixmp4.cli import utils
from ixmp4.conf import settings
from ixmp4.conf.auth import ManagerAuth
from requests.auth import AuthBase

from pyam.core import IamDataFrame
from pyam.logging import deprecation_warning
from pyam.str import is_str
from pyam.str import is_str, shorten
from pyam.utils import (
IAMC_IDX,
META_IDX,
Expand All @@ -35,13 +36,37 @@
You are connected to the {} scenario explorer hosted by IIASA.
If you use this data in any published format, please cite the
data as provided in the explorer guidelines: {}
""".replace("\n", "")
""".replace(
"\n", ""
)
IXMP4_LOGIN = "Please run `ixmp4 login <username>` in a console"

# path to local configuration settings
DEFAULT_IIASA_CREDS = Path("~").expanduser() / ".local" / "pyam" / "iiasa.yaml"


def platforms() -> None:
"""Print a list of available ixmp4 platforms hosted by IIASA

See Also
--------
ixmp4.conf.settings.manager.list_platforms
"""

_platforms = ixmp4.conf.settings.manager.list_platforms()
utils.echo("IIASA platform".ljust(20) + "Access".ljust(10) + "Notice\n")
phackstock marked this conversation as resolved.
Show resolved Hide resolved

for p in _platforms:
utils.important(shorten(p.name, 20), nl=False)
utils.echo(str(p.accessibility.value.lower()).ljust(10), nl=False)

if p.notice is not None:
utils.echo(shorten(p.notice, 55), nl=False)
utils.echo()

utils.info("\n" + str(len(_platforms)) + " total \n")


def set_config(user, password, file=None):
raise DeprecationWarning(f"This method is deprecated. {IXMP4_LOGIN}.")

Expand Down Expand Up @@ -197,6 +222,10 @@ def _connection_map(self):
@lru_cache()
def valid_connections(self):
"""Return available resources (database API connections)"""
logger.warning(
"IIASA is migrating to a database infrastructure using the ixmp4 package."
phackstock marked this conversation as resolved.
Show resolved Hide resolved
"Use `pyam.iiasa.platforms()` to list available ixmp4 databases."
)
return list(self._connection_map.keys())

def connect(self, name):
Expand Down
10 changes: 10 additions & 0 deletions pyam/str.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,18 +65,21 @@ def _count_pipes(val):
# test = lambda x: level == x if x is not None else False
def test(x):
return level == x if x is not None else False

elif level[-1] == "-":
level = int(level[:-1])

# test = lambda x: level >= x if x is not None else False
def test(x):
return level >= x if x is not None else False

elif level[-1] == "+":
level = int(level[:-1])

# test = lambda x: level <= x if x is not None else False
def test(x):
return level <= x if x is not None else False

else:
raise ValueError("Unknown level type: `{}`".format(level))

Expand Down Expand Up @@ -141,3 +144,10 @@ def escape_regexp(s):
def is_str(x):
"""Returns True if x is a string"""
return isinstance(x, six.string_types)


def shorten(value: str, length: int):
"""Shorten a string to a given length adding `...`"""
if len(value) > length - 4:
value = value[: length - 4] + "..."
return value.ljust(length)
9 changes: 9 additions & 0 deletions tests/test_iiasa.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,15 @@
)


def test_platforms(capsys):
# test that the function does not raise an error
iiasa.platforms()
assert (
"public-test public This is a public ixmp4 test instance"
in capsys.readouterr().out
)


def test_unknown_conn():
# connecting to an unknown API raises an error
match = "You do not have access to instance 'foo' or it does not exist."
Expand Down
Loading