XPublish has only one route when used with new plugin system

[mpiannucci]

So Im trying to hookup with the new plugin system but Im not getting any of the routers I expect by default. I setup my rest like this:

rest = xpublish.Rest(
    app_kws={
        "title": 'XREDS',
        "description": 'XArray Environmental Data Services exposes environmental model data in common data formats for digestion in applications and notebooks',
        "openapi_url": "/xreds.json",
    },
    cache_kws={
        "available_bytes": 1e9
    },
    datasets=None,
    plugins={
        'xreds_datasets': DatasetProvider(),
        'xpublish_wms': CfWmsPlugin(),
    },
)

The xpublish_wms plugin should expose /wms, the dataset provider is lazy loading my datasets.

When I load up /docs though I only get this:

I was pretty sure the old routers are gone and plugins should do everything now. Are the default plugins not loaded anymore? Am I missing a step?

When I log rest.plugins I get:

{'xreds_datasets': DatasetProvider(name='xreds_datasets', dataset_mapping={'gfswave_global': {'path': 's3://nextgen-dmac/kerchunk/gfswave_global_kerchunk.json'}}), 'xpublish_wms': CfWmsPlugin(name='cf_wms', dataset_router_prefix='/wms', dataset_router_tags=['wms'])}
``

[mpiannucci]

Okay the key here is that if plugins are speecified, the default ones are not loaded.... so you either have to specify all of them or specify none, then after register any other plugins.

However, with this working I get this list of plugins:

{'cf_wms': CfWmsPlugin(name='cf_wms', dataset_router_prefix='/wms', dataset_router_tags=['wms']), 'dataset_info': DatasetInfoPlugin(name='dataset_info', dataset_router_prefix='', dataset_router_tags=[]), 'module_version': ModuleVersionPlugin(name='module_version', app_router_prefix='', app_router_tags=[]), 'plugin_info': PluginInfoPlugin(name='plugin_info', app_router_prefix='', app_router_tags=[]), 'zarr': ZarrPlugin(name='zarr', dataset_router_prefix='', dataset_router_tags=['zarr']), 'opendap': OpenDapPlugin(name='opendap', dataset_router_tags=['opendap'], dataset_router_prefix='/opendap')}

So the WMS plugin in loading but it doesnt show on the docs/ page and I'm not sure why

[abkfenris]

When you specify plugins in xpublish.Rest() it disables automatic loading of the default plugins.

How plugins are loaded

If you want to add to the default plugins, you can use either xpublish.Rest.register_plugin() to add them one by one, or you can load the default plugins and then add yours to the dict before initializing xpublish.Rest.

rest = xpublish.Rest(
    app_kws={
        "title": 'XREDS',
        "description": 'XArray Environmental Data Services exposes environmental model data in common data formats for digestion in applications and notebooks',
        "openapi_url": "/xreds.json",
    },
    cache_kws={
        "available_bytes": 1e9
    },
    datasets=None,
)
rest.register_plugin(DatasetProvider(), 'xreds_datasets')
rest.register_plugin(CfWmsPlugin(), 'xpublish_wms')

Or

from xpublish.plugins.manage import load_default_plugins

rest = xpublish.Rest(
    app_kws={
        "title": 'XREDS',
        "description": 'XArray Environmental Data Services exposes environmental model data in common data formats for digestion in applications and notebooks',
        "openapi_url": "/xreds.json",
    },
    cache_kws={
        "available_bytes": 1e9
    },
    datasets=None,
    plugins={
        **load_default_plugins(),
        'xreds_datasets': DatasetProvider(),
        'xpublish_wms': CfWmsPlugin(),
    },
)

[abkfenris]

At some point I'd like to explore abstract base classes or mixins to help provide typing to the various types of plugins. Maybe even include a 'did it return a router?' test in the a cookiecutter or template for new plugins.

[mpiannucci]

Yeah that sounds good. Okay hopefully last question, is this dependency injection allowed? It is causing a crash: AttributeError: 'Depends' object has no attribute 'get':

...
    @hookimpl
    def get_dataset(self, dataset_id: str, cache: cachey.Cache = Depends(get_cache)) -> xr.Dataset:
        cache_key = f"dataset-{dataset_id}"

        ds = cache.get(cache_key)
        if not ds:
...

[mpiannucci]

Ok im guessing access to the cache isnt there for dataset providers?

[abkfenris]

The cache currently isn't available for dataset provider methods.

I think there is probably a bigger discussion about the overall caching strategy, but it's easier to add a new argument to methods with the plugin system, than it is to later take them away, so I left those changes for later.

I don't currently have a strong vision or preference as to how caching should work. Maybe we'll really need 'cache provider' plugins or similar so folks can wire in shared memory, Redis, Memcache, or who knows what other caching routes. (Similarly we may want 'dask provider' plugins).

The plugin system is also a little disconnected from the path that FastAPI can do dependency injection on, so Depends doesn't work to add a cache. For now, maybe it would make sense to manage a cache in your own dataset provider plugin.

[mpiannucci]

Sounds good, I will just manage that on my own then.

As for dask, I think we really need to talk about that because I think thats going to be the most important missing piece for my work, but i also am not a dask expert so im not sure what the workflow would even be for that. But my vision is that all pluygins should be able to use the same dask provider