Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interfaces and services for JWK management #10628

Merged
merged 29 commits into from
Feb 7, 2022

Conversation

woodruffw
Copy link
Member

@woodruffw woodruffw commented Jan 20, 2022

This adds the initial set of interfaces and services (and a small warehouse oidc CLI) for JSON Web Key (JWK) management.

Some design notes:

  • JWKs come in sets, and are keyed by their OIDC provider. For now, GitHub is the only provider Warehouse knows about.
  • The current implementation stashes the JWKs in Redis. I'm not 100% sure if this is the best place for them, so please let me know if somewhere else is better.
  • The current implementation adds a periodic (daily) job for refreshing the JWKs from each service. A user standing up a fresh instance of Warehouse can also manually fetch the JWKs by running warehouse oidc update-oidc-jwks.
  • I'm not sure whether or not it also makes sense to cache the OIDC configuration (and periodically re-fetch it). The only thing I can currently think of that we might want from it is the issuer key, but we should probably keep that hardcoded anyways.

Needs:

  • More documentation
  • Unit tests

Closes #10620.

@woodruffw woodruffw requested review from di and ewdurbin January 20, 2022 21:55
warehouse/tasks.py Outdated Show resolved Hide resolved
These should be very fast, and thus don't need a separate queue.
It makes more sense to have the OIDC_PROVIDERS map be a map of
names to issuer FQDNs, so that we don't have to store that information
elsewhere.
warehouse/oidc/services.py Outdated Show resolved Hide resolved
warehouse/cli/oidc.py Outdated Show resolved Hide resolved
@woodruffw
Copy link
Member Author

Updated the warehouse oidc CLI now that we do FOFU:

Usage: python -m warehouse oidc [OPTIONS] COMMAND [ARGS]...

  Manage the Warehouse OIDC components.

Options:
  -h, --help  Show this message and exit.

Commands:
  get-key         Return the JWK for the given provider's...
  prime-provider  Update Warehouse's JWK sets for the given...

@woodruffw woodruffw marked this pull request as ready for review January 25, 2022 17:10
@woodruffw
Copy link
Member Author

I haven't started on tests yet, but this should be good for another review pass.

@woodruffw woodruffw changed the title Draft: Interfaces and services for JWK management Interfaces and services for JWK management Jan 25, 2022
Copy link
Member

@di di left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good. We need to make sure we're emitting meaningful metrics at the right places. We probably want to use sentry_sdk.capture_message for the errors instead of logging them (would be nice to make this just output to the logger in development).

requirements/main.txt Outdated Show resolved Hide resolved
warehouse/oidc/services.py Outdated Show resolved Hide resolved
warehouse/oidc/services.py Outdated Show resolved Hide resolved
warehouse/oidc/services.py Outdated Show resolved Hide resolved
warehouse/oidc/services.py Outdated Show resolved Hide resolved
warehouse/oidc/services.py Outdated Show resolved Hide resolved
warehouse/oidc/services.py Show resolved Hide resolved
warehouse/oidc/services.py Outdated Show resolved Hide resolved
warehouse/utils/oidc.py Outdated Show resolved Hide resolved
woodruffw and others added 3 commits January 26, 2022 16:51
Co-authored-by: Dustin Ingram <di@users.noreply.github.com>
Co-authored-by: Dustin Ingram <di@users.noreply.github.com>
Adds a service factory for creating per-provider services.
Comment on lines 142 to 159
class JWKServiceFactory:
def __init__(self, provider, service_class=JWKService):
self.provider = provider
self.service_class = service_class

def __call__(self, context, request):
cache_url = request.registry.settings["oidc.jwk_cache_url"]

return self.service_class(self.provider, cache_url)

def __eq__(self, other):
if not isinstance(other, JWKServiceFactory):
return NotImplemented

return (self.provider, self.service_class) == (
other.provider,
other.service_class,
)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @di: Let me know if this looks reasonable for the factory.

(I cargo-culted the __eq__ implementation from TokenServiceFactory -- I assume it's necessary to help find_service disambiguate the factory-produced services?)

warehouse/oidc/services.py Outdated Show resolved Hide resolved
warehouse/oidc/services.py Outdated Show resolved Hide resolved
warehouse/oidc/services.py Outdated Show resolved Hide resolved
Copy link
Member

@di di left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some slight refactoring suggestions and one overall thought: Should we make this an OIDCProviderService instead? I don't see a need for having separate JWK service and OIDC service (with an interface like .verify(token)) for the same provider.

warehouse/oidc/interfaces.py Outdated Show resolved Hide resolved
warehouse/utils/oidc.py Outdated Show resolved Hide resolved
warehouse/utils/oidc.py Outdated Show resolved Hide resolved
warehouse/oidc/services.py Outdated Show resolved Hide resolved
warehouse/utils/oidc.py Outdated Show resolved Hide resolved
warehouse/oidc/services.py Outdated Show resolved Hide resolved
@woodruffw
Copy link
Member Author

Should we make this an OIDCProviderService instead?

Yep, that makes sense to me! I'll make that change.

warehouse/oidc/services.py Outdated Show resolved Hide resolved
warehouse/oidc/__init__.py Outdated Show resolved Hide resolved
@woodruffw
Copy link
Member Author

Dependencies failure in the CI looks unrelated to these changes:

# install ipython if enabled
/opt/hostedtoolcache/Python/3.8.9/x64/bin/pip-compile --upgrade --allow-unsafe -o /tmp/tmp.skKABnHln0/deploy.txt requirements/deploy.in > /dev/null
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.9/x64/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/opt/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/opt/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/piptools/scripts/compile.py", line 342, in cli
    repository = PyPIRepository(pip_args, cache_dir=cache_dir)
  File "/opt/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/piptools/repositories/pypi.py", line 106, in __init__
    self._setup_logging()
  File "/opt/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/piptools/repositories/pypi.py", line 455, in _setup_logging
    assert isinstance(handler, logging.StreamHandler)
AssertionError
make[1]: *** [Makefile:121: deps] Error 1
make[1]: Leaving directory '/home/runner/work/warehouse/warehouse'
make: *** [Makefile:139: github-actions-deps] Error 2
Error: Process completed with exit code 2.

@di di merged commit 3d239c0 into pypi:main Feb 7, 2022
@di di deleted the tob-jwk-management branch February 7, 2022 18:01
domdfcoding pushed a commit to domdfcoding/warehouse that referenced this pull request Jun 7, 2022
* python-version: bump to 3.8.9

* ci, Dockerfile: bump Python versions

* requirements, warehouse: dependencies, skeleton for JWKs

* warehouse/oidc: format

* config, oidc, utils: fill in more groundwork

* warehouse: add a basic `warehouse oidc` CLI, redis caching

* tasks: remove the separate OIDC queue

These should be very fast, and thus don't need a separate queue.

* warehouse: decompose OIDC urls a bit

It makes more sense to have the OIDC_PROVIDERS map be a map of
names to issuer FQDNs, so that we don't have to store that information
elsewhere.

* warehouse/utils: docs

* warehouse: refactor JWKs to fetch on first use

* tests/unit: fix config test

* Update requirements/main.txt

Co-authored-by: Dustin Ingram <di@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Dustin Ingram <di@users.noreply.github.com>

* warehouse: refactor JWKService

Adds a service factory for creating per-provider services.

* oidc/services: appease flake8

* warehouse: add metrics to JWKService, rewrite CLI

* warehouse/cli: remove oidc subcommand

This is no longer useful.

* warehouse: rename JWKService to OIDCProviderService, refactor

* warehouse/oidc: fix init

* warehouse: remove oidc.utils, refactor

* warehouse/oidc: re-add provider attribute

* tests: unit tests for warehouse.oidc.services

Fixes small bugs in the process.

Co-authored-by: Dustin Ingram <di@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Limetime management for OIDC JWKS
3 participants