-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Backends entrypoints #4577
Merged
Merged
Backends entrypoints #4577
Changes from 16 commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
e005db9
Define _get_backends_cls function inside apiv2.py to read engines fro…
0ae8bbd
Read open_backends_dataset_* from entrypoints.
df4f3de
Add backend entrypoints in setup.cfg
ea08b03
Pass apiv2.py isort and black formatting tests.
131a2b6
add dependencies
aurghs e2bdeaa
add backend entrypoints and check on conflicts
aurghs 2f99e9c
black
aurghs 9c14a02
removed global variable EMGINES
aurghs d184947
black isort
aurghs a37d549
add detect_engines in __all__ init.py
aurghs 1c49d73
removed entrypoints in py36-bare-minimum.yml and py36-min-all-deps.yml
aurghs 94451c6
add entrypoints in IGNORE_DEPS
aurghs 5dd4714
Plugins test (#20)
aurghs 9a14597
fix typo
aurghs 57e5e21
style
aurghs a302c87
style
aurghs fafc8de
Merge remote-tracking branch 'origin/master' into backends-entrypoints
alexamici f0821c2
Code style
alexamici f73f062
Code style
alexamici 78b1866
fix: updated plugins.ENGINES with plugins.list_engines()
aurghs ad023c4
fix
aurghs 14bf314
One more correctness fix of the latest merge from master
alexamici File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,31 +1,78 @@ | ||
import inspect | ||
import typing as T | ||
|
||
from . import cfgrib_, h5netcdf_, zarr | ||
|
||
ENGINES: T.Dict[str, T.Dict[str, T.Any]] = { | ||
"h5netcdf": { | ||
"open_dataset": h5netcdf_.open_backend_dataset_h5necdf, | ||
}, | ||
"zarr": { | ||
"open_dataset": zarr.open_backend_dataset_zarr, | ||
}, | ||
"cfgrib": { | ||
"open_dataset": cfgrib_.open_backend_dataset_cfgrib, | ||
}, | ||
} | ||
|
||
|
||
for engine in ENGINES.values(): | ||
if "signature" not in engine: | ||
parameters = inspect.signature(engine["open_dataset"]).parameters | ||
for name, param in parameters.items(): | ||
if param.kind in ( | ||
inspect.Parameter.VAR_KEYWORD, | ||
inspect.Parameter.VAR_POSITIONAL, | ||
): | ||
raise TypeError( | ||
f'All the parameters in {engine["open_dataset"]!r} signature should be explicit. ' | ||
"*args and **kwargs is not supported" | ||
) | ||
engine["signature"] = set(parameters) | ||
import itertools | ||
import warnings | ||
from functools import lru_cache | ||
|
||
import pkg_resources | ||
|
||
|
||
class BackendEntrypoint: | ||
__slots__ = ("open_dataset", "open_dataset_parameters") | ||
|
||
def __init__(self, open_dataset, open_dataset_parameters=None): | ||
self.open_dataset = open_dataset | ||
self.open_dataset_parameters = open_dataset_parameters | ||
|
||
jhamman marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
def remove_duplicates(backend_entrypoints): | ||
|
||
# sort and group entrypoints by name | ||
backend_entrypoints = sorted(backend_entrypoints, key=lambda ep: ep.name) | ||
backend_entrypoints_grouped = itertools.groupby( | ||
backend_entrypoints, key=lambda ep: ep.name | ||
) | ||
# check if there are multiple entrypoints for the same name | ||
unique_backend_entrypoints = [] | ||
for name, matches in backend_entrypoints_grouped: | ||
matches = list(matches) | ||
unique_backend_entrypoints.append(matches[0]) | ||
matches_len = len(matches) | ||
if matches_len > 1: | ||
selected_module_name = matches[0].module_name | ||
all_module_names = [e.module_name for e in matches] | ||
warnings.warn( | ||
f"\nFound {matches_len} entrypoints for the engine name {name}:" | ||
f"\n {all_module_names}.\n It will be used: {selected_module_name}.", | ||
RuntimeWarning, | ||
) | ||
return unique_backend_entrypoints | ||
keewis marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
|
||
def detect_parameters(open_dataset): | ||
signature = inspect.signature(open_dataset) | ||
parameters = signature.parameters | ||
for name, param in parameters.items(): | ||
if param.kind in ( | ||
inspect.Parameter.VAR_KEYWORD, | ||
inspect.Parameter.VAR_POSITIONAL, | ||
): | ||
raise TypeError( | ||
f"All the parameters in {open_dataset!r} signature should be explicit. " | ||
"*args and **kwargs is not supported" | ||
) | ||
return tuple(parameters) | ||
|
||
|
||
def create_engines_dict(backend_entrypoints): | ||
engines = {} | ||
for backend_ep in backend_entrypoints: | ||
name = backend_ep.name | ||
backend = backend_ep.load() | ||
engines[name] = backend | ||
return engines | ||
|
||
|
||
def set_missing_parameters(engines): | ||
for name, backend in engines.items(): | ||
if backend.open_dataset_parameters is None: | ||
open_dataset = backend.open_dataset | ||
backend.open_dataset_parameters = detect_parameters(open_dataset) | ||
jhamman marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
|
||
@lru_cache(maxsize=1) | ||
def list_engines(): | ||
entrypoints = pkg_resources.iter_entry_points("xarray.backends") | ||
backend_entrypoints = remove_duplicates(entrypoints) | ||
engines = create_engines_dict(backend_entrypoints) | ||
set_missing_parameters(engines) | ||
return engines |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,96 @@ | ||
from unittest import mock | ||
|
||
import pkg_resources | ||
import pytest | ||
|
||
from xarray.backends import plugins | ||
|
||
|
||
def dummy_open_dataset_args(filename_or_obj, *args): | ||
pass | ||
|
||
|
||
def dummy_open_dataset_kwargs(filename_or_obj, **kwargs): | ||
pass | ||
|
||
|
||
def dummy_open_dataset(filename_or_obj, *, decoder): | ||
pass | ||
|
||
|
||
@pytest.fixture | ||
def dummy_duplicated_entrypoints(): | ||
specs = [ | ||
"engine1 = xarray.tests.test_plugins:backend_1", | ||
"engine1 = xarray.tests.test_plugins:backend_2", | ||
"engine2 = xarray.tests.test_plugins:backend_1", | ||
"engine2 = xarray.tests.test_plugins:backend_2", | ||
] | ||
eps = [ | ||
pkg_resources.EntryPoint.parse(spec) | ||
for spec in specs | ||
] | ||
return eps | ||
|
||
|
||
def test_remove_duplicates(dummy_duplicated_entrypoints): | ||
entrypoints = plugins.remove_duplicates(dummy_duplicated_entrypoints) | ||
assert len(entrypoints) == 2 | ||
|
||
|
||
def test_remove_duplicates_warnings(dummy_duplicated_entrypoints): | ||
|
||
with pytest.warns(RuntimeWarning) as record: | ||
_ = plugins.remove_duplicates(dummy_duplicated_entrypoints) | ||
|
||
assert len(record) == 2 | ||
message0 = str(record[0].message) | ||
message1 = str(record[1].message) | ||
assert "entrypoints" in message0 | ||
assert "entrypoints" in message1 | ||
|
||
|
||
@mock.patch("pkg_resources.EntryPoint.load", mock.MagicMock(return_value=None)) | ||
def test_create_engines_dict(): | ||
specs = [ | ||
"engine1 = xarray.tests.test_plugins:backend_1", | ||
"engine2 = xarray.tests.test_plugins:backend_2", | ||
] | ||
entrypoints = [pkg_resources.EntryPoint.parse(spec) for spec in specs] | ||
engines = plugins.create_engines_dict(entrypoints) | ||
assert len(engines) == 2 | ||
assert engines.keys() == set(("engine1", "engine2")) | ||
|
||
|
||
def test_set_missing_parameters(): | ||
backend_1 = plugins.BackendEntrypoint(dummy_open_dataset) | ||
backend_2 = plugins.BackendEntrypoint(dummy_open_dataset, ("filename_or_obj",)) | ||
engines = {"engine_1": backend_1, "engine_2": backend_2} | ||
plugins.set_missing_parameters(engines) | ||
|
||
assert len(engines) == 2 | ||
engine_1 = engines["engine_1"] | ||
assert engine_1.open_dataset_parameters == ("filename_or_obj", "decoder") | ||
engine_2 = engines["engine_2"] | ||
assert engine_2.open_dataset_parameters == ("filename_or_obj",) | ||
|
||
|
||
def test_set_missing_parameters_raise_error(): | ||
|
||
backend = plugins.BackendEntrypoint(dummy_open_dataset_args) | ||
with pytest.raises(TypeError): | ||
plugins.set_missing_parameters({"engine": backend}) | ||
|
||
backend = plugins.BackendEntrypoint( | ||
dummy_open_dataset_args, ("filename_or_obj", "decoder") | ||
) | ||
plugins.set_missing_parameters({"engine": backend}) | ||
|
||
backend = plugins.BackendEntrypoint(dummy_open_dataset_kwargs) | ||
with pytest.raises(TypeError): | ||
plugins.set_missing_parameters({"engine": backend}) | ||
|
||
backend = plugins.BackendEntrypoint( | ||
dummy_open_dataset_kwargs, ("filename_or_obj", "decoder") | ||
) | ||
plugins.set_missing_parameters({"engine": backend}) |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for this sort of filtering and transformation I would usually rely on comprehensions:
but there might be a better (more readable) way to express the conditional transformation ofEdit: I added the local function, but I'm still searching for a better namedecoder
. Maybe use a local function with a descriptive name?