Security Notification Systems for Python Packages

[philippeowagner]

Having the following scenario: We are open sourcing apps from time to time and maintaining them as well ;-). Our most popular Django app is django-hijack. Days back we had a little security vulnerability in hijack, nothing really problematic but it was an issue. We had no real plan how to communicate to all the users that runs the app it in production.... However, it's not just us – others have this problem too!

Fixing a security vulnerability is one thing, but communicating it is something different. Honestly we had no real tools for that, something like a mailing list for example. It feels wrong using mailing lists for this use-case anyway. How to organise them? One for all our apps, one for the GitHub organisation, one for each app/project we open source?

A great thing would be to automatically subscribe to the apps/packages I do "pip install", "subscribe" or download from the CheeseShop. In case an app has a serious security vulnerability I would be notified by my channel of choice linked to my PyPI account. Without talking about the technical details how this could be done, do you Donald et al. think this is a legit use-case that could be added to this platform and adds value for others?

I look forward for your feedback.

[nlhkabu]

@dstufft I have been talking to @philippeowagner about this by email. I personally think it is a great idea, so long as users are provided with enough control to manage their notifications, both when they install a package and later via Warehouse. 👍

[sephii]

I would be very interested by this feature. Also at the moment there's no way to tell whether a version of a package you're using has any known security vulnerability or not. The only information you can currently get is "am I running the latest version of this package?" and not "am I running a secure version of this package?".

This would also help security checks made by tools such as requires.io, which as far as I know rely on manual checking at the moment.

[sigmavirus24]

Along similar lines, perhaps some metadata along the lines of This release fixes CVE-201Y-ABCD. And the ability to search by CVE-ID?

[ewdurbin]

Could a feature be introduced into pip which blocks pip from installing a release marked as vulnerable on an opt-in basis?

Similar to the way --allow-external was introduced.

Explicitly declaring that vulnerable packages are installed would give a method for application developers to be quickly alerted... assuming sane CI environments.

Anecdotally... I have learned a bunch about the consistent progress made with setuptools and pip by adding python -m pip install -U setuptools pip to my Travis builds and waiting for a red build :)

[ewdurbin]

To clarify, I feel that the external hosting issue is a weaker security concern that fits in the same realm as "known bad" releases.

They are a kind of META-meta-data that are perfect candidates for mutable state

[apollo13]

Yes, we just discussed this at "django under the hood" and it would be great to have something in that area!

[brainwane]

Thanks for your note and sorry for the slow response!

The folks working on Warehouse have gotten funding to concentrate on improving and deploying Warehouse, and have kicked off work towards our development roadmap -- the most urgent task is to improve Warehouse to the point where we can redirect pypi.python.org to pypi.org so the site is more sustainable and reliable. Since this feature isn't something that the legacy site has, I've moved it to a future milestone.

Thanks and sorry again for the wait.

[eirnym]

@brainwane I think task to redirect old PyPi and use a new one is done and many people would prefer to have a security and vulnerability management as it was done in npm, FreeBSD pkgng or other systems.

This would save a world from problems to keep tracking many projects out from security problems.

[brainwane]

From December 2017 till the end of April 2018, PyPI had funding to get the new site up and running and perform the switchover. We did finish that task. The grant ran out and we have, as far as I know, no one paid to work on PyPI; volunteers are maintaining and improving the software and infrastructure sides of things, but we need dedicated funding to add complex features. The Packaging Working Group is seeking donations and applying for further grants to fund more design work, more and faster development and code review, and requisite project management. One of the grants we've applied for is specifically to fund security work. Sorry for the wait.

[brainwane]

Good news! The Packaging Working Group, seeking donations and further grants to fund more work, got some new funding from the Open Technology Fund, and an auditable event log -- tracked in #5863 -- is part of the current grant-funded project.

To do security notifications on PyPI packages the right way, we should wait till we have #5863 implemented, so we can draw on the event logging and use it to trigger this kind of notification.

[eirnym]

This is the excellent news, @brainwane! I'm looking forward for the implementation!

[brainwane]

@SantiagoTorres Per our conversation earlier this week, would you like to share a thought about the possibility of using in-toto for this?

[brainwane]

Now unblocked! See #5863 for related issues and requests for notifications of specific kinds of events, and #6339 for events currently being logged, and #5714 for related conversation.

[MVrachev]

I would be very interested by this feature. Also at the moment there's no way to tell whether a version of a package you're using has any known security vulnerability or not. The only information you can currently get is "am I running the latest version of this package?" and not "am I running a secure version of this package?".

There are a multiple difficulties and questions when regarding the question "is this package secure".
Questions like:

• How do you verify that a package is secure?
• If you decide to use the amount and severity of known CVE-s as measurement how would you gather that information?
• How many CVE-s or what severity should they have to consider a package as vulnerable?
• Do you imagine every package to have that information?
  If that's the case what will be the overhead of Warehouse doing that?

There is the other possibility as suggested in the begging of this issue - let the maintainers have a flag to mark their packages as vulnerable.
But then there is the other question:
If a package is not marked as vulnerable from its maintainers does it mean it's safe?

This approach could be misleading for the users of the packages if there isn't clear information that a package could be marked as vulnerably only from the maintainers and if it's not marked it doesn't mean it's safe.