RFC for linking packages to their source and build

[feelepxyz]

RFC for linking public npm packages to the source code repository and build it originated from.

View rendered version

[bnb]

One question for clarification: This RFCs repo has historically been mostly focused on the CLI. Is this RFC for the implementation of this functionality in the CLI or for implementation in the Registry?

[ljharb]

given that build output should never be committed, and builds/publishes are typically done on local machines, how can this link be verified?

[MylesBorins]

This RFC will only work with packages published from trusted build infrastructure. It will not support local publish.

[ljharb]

What are the criteria for "trusted"? What are the associated costs in standing up such infrastructure, and what portions of the community would be excluded by those costs?

[darcyclarke]

@ljharb I believe the premise of this entire RFC is predicated on builds/publishes happening in the open/cloud (as the motivation aims at getting closer to being able to meet SLSA Level 3 standards - ref. https://github.com/npm/rfcs/blob/link-packages-to-source-and-build/accepted/0000-link-packages-to-source-and-build.md#non-falsifiable-provenance-using-a-trusted-builder & https://slsa.dev/spec/v0.1/levels#detailed-explanation)

Notably, "private/inner-source" is considered a "non-goal" (ref. https://github.com/npm/rfcs/blob/link-packages-to-source-and-build/accepted/0000-link-packages-to-source-and-build.md#non-goals) so any local builds/publishes would be out of scope afaik.

[MylesBorins]

This is all covered in the RFC

https://github.com/npm/rfcs/pull/626/files#diff-95858aa7674c6ecb2578ad8c2b7ac848ac80dd36eedbbc92ce8a3968daf10771R239-R249

[ljharb]

If the manual approval allows me to input an actual second factor token, then that would be great! The things you linked don't seem to permit that yet tho.

I look forward to the RFC discussion.

[MylesBorins]

@ljharb I'm surfacing that request to the actions team.

In general I see staging / approvals as out of scope for npm as a whole, but up for challenging those assumptions.

[laurentsimon]

I would like to point out that one of the major use cases of SLSA provenance is to mitigate the threat of compromised (Npm) registry credentials. That's because the provenance provides a link to the source repo. So if an attacker were to compromise a registry credential and push a malicious package, the provenance would change from "source=github.com/genuine/source" to "source=github.com/attacker/source" or even "". The registry or another system that monitors it would immediately detect the compromise. In essence, this means that the system fails "safe" so long as:

1. you have 2FA on your registry account, protecting privileged settings, etc
2. you use an automation token to push packages.

An attacker who gets hold of an automation token can push packages but these would be immediately detected because of the change of source. In a nutshell, you don't need staged builds if you upload provenance from a trusted builder. (I'm assuming that merely pushing packages in an attempt to DoS a user's registry account is not a valuable attack).

Let me know if there are some nuances I'm missing.

(Note: you could improve on automation token using OIDC to further harden the system and improve user experience.)

[feelepxyz]

An attacker who gets hold of an automation token can push packages but these would be immediately detected because of the change of source. In a nutshell, you don't need staged builds if you upload provenance from a trusted builder.

💯 Yes great summary! We can also make it difficult to change the provenance, e.g. locking it to a particular source repo once set, or preventing publishing new versions that don't include it if it's previously been set. Changing this could be gated behind 2FA on your registry account, limited to admins for an org.

[fkautz]

If you disaggregate the system pushing the package to the repo from the system building/signing the artifact, you will be able to protect against the upload token being compromised because you can validate the signed artifacts with published public keys.

Furthermore, sigstore (I believe mentioned in this RFC, I'm still reading through it) can provide some ability to detect a compromised signing key. Will comment more on this later after reading more.

[MylesBorins]

@bnb while the repository has historically been focused on the CLI, and this RFC touches a broader surface area, we felt this work was too important to not work through it with the broader community community.

[bnb]

@MylesBorins should we expect discussion time set aside in the npm CLI RFC meeting, a separate synchronous meeting, or no synchronous meeting to discuss?

[darcyclarke]

@bnb added the "Agenda" label to the RFC. We'll ensure there's some dedicated time to discuss this in the Open RFC call Wednesday (will likely stay on the Agenda for a bit given the proposal's length, density & impact)

[jchestershopify]

Very exciting.

Some suggested improvements.

[jchestershopify]

Is there a potential dependency confusion attack here? Ie. can I get a private package to publish to rekor if I upload a package of that name to the public registry?

[feelepxyz]

Oh yeah you mean a case where you have some internal corp dep pulled from a private registry and then someone finds this name and creates a public version? Not sure we can do much to mitigate the creation of these. Do you see an added risk being able to create these?

An issue could be getting the wrong provenance attestations during verification if we just used the package name and version. We can mitigate this by querying for the attestations using a shamsum of the installed package's tarball.

[bnb]

I think what @jchestershopify means is if I have package x internally and someone publishes x, the RFC says that npm will interrogate the registry for x and get the internal package signed accidentally because x exists on the public registry. This is especially heinous in this case, since the ledger is immutable.

[feelepxyz]

When you say internal package x you mean on some other registry right? I think in this case the signature would be for the public package x and associated via the shasum of the tarball, which won't match the internal package x. Maybe I'm missing something though?

[feelepxyz]

We're also planning on creating a "release attestation" on the npm registry that gets uploaded to the public Rekor ledger. This is a signed statement that the npm registry accepted and authorized the publish for public package x on top of the "build attestation" that gets created during publish in the CI/CD system, which links the source and build to the package.

[ljharb]

unpkg.com makes it much easier - it seems like maybe addressing this (making npm packages browseable on the web safely) would be a good first step?

I'm reasonably certain that the owner of unpkg.com would be happy to transfer the IP to npm if it meant it'd be an official solution (at the least, it's worth asking)

[MylesBorins]

We already have a beta implementation of exploring code in npm and are funding improvements to it for live code audit on npmjs.com in parallel to this work.

[naugtur]

cc @feross

[ljharb]

emails are already exposed for all publishers to npm - this seems like it should be a non-goal.

[MylesBorins]

This is about the data that ends up in the public ledger, separate from the registry

[jchestershopify]

Users may be using different email addresses for their OIDC provider vs the one they use for npm.

[ljharb]

ah ok, makes sense, thanks

[feelepxyz]

emails are already exposed for all publishers to npm

edit: missed above discussion when posting this.

Yes this is true. We're not proposing changing what goes in the packument. The thing we've focused on in this RFC is the signed statement about the repo and build that will go on a public immutable signature ledger (Rekor). We didn't want a solution where this signed statement would include any kind of identifying information about maintainers.

[fkautz]

I have a friend who is an expert and advocate on this exact topic in favor of maintaining developer privacy. Would it make sense to invite them to review the proposal and provide feedback?

[feelepxyz]

@fkautz yes please 👍 sounds like they might have some great context that we're also not aware of yet.

[ljharb]

3 9's seems pretty paltry; can we aim higher?

[jchestershopify]

Bear in mind this SLO affects publication only; it won't break builds if sigstore is offline. I expect most publishing authors will be fine with waiting a little bit.

[haydentherapper]

+1, this SLO should be for signing which must be online (for fetching an identity certificate and publishing to the log).

Note that the verification mechanism should be designed with offline verification in mind. For example, querying the log for checking for inclusion would be an online action, but this can be designed to be offline with a persisted proof of inclusion.

[feelepxyz]

Yes this is only during publish when talking to Sigstore infrastructure. Sigstore is run as free-to-use public good infrastructure hosted on GCP which has SLAs from 99.9% to 99.99% on used services AFAICT.

There's a lot to figure out committing to a SLOs on a shared service like this, with people on-call from different organisations. My hope would be that we can eventually get to four 9s uptime once this process is more mature.

[fkautz]

I reached out to the sigstore team, and they pointed me to this as their current official statement, as of June 29, 2022: https://blog.sigstore.dev/an-update-on-general-availability-5c5563d4e400

They are targeting four months from June 29th (interpreting as end of October) to ramp up operations, tooling, and publishing SLOs.

[feelepxyz]

Yep I believe the current target for the Sigstore GA is 99.9% uptime for the hosted versions of Fulcio and Rekor.

[ljharb]

other than recording "an npm specifier" and "a github repo and SHA" together, how are these linked? published npm packages already include repo and SHA information in the packument, so how would sigstore's linking be different?

[MylesBorins]

This information would come as an OIDC claim from the publishing party and thus verifiable... e.g. the claim of a specific repo would come from the GitHub action itself, not meta data published to the registry from a personal machine

[darcyclarke]

Just want to be mindful/note that there's nothing in this RFC which mentions that source, CI or build envs will be or will become immutable; which puts their audibility into question (ex. I can build & publish my package with a trusted builder but then delete that action/workflow/repo - killing the link/trace)

[bnb]

Does this mean you have to use GitHub as your source repo?

[MylesBorins]

No

[feelepxyz]

We're proposing ways to make this information harder to falsify. Currently you can say the repo is anything and it's not validated, so an attacker can easily forge this information.

A part of the solution is using information from the OIDC id token that you can get from a supporting CI/CD system, this includes information about the repo and commit. The whole thing is then signed and can be verified against the CI/CD's public key for example.

[feelepxyz]

nothing in this RFC which mentions that source, CI or build envs will be or will become immutable

Yes this is true. Worth noting though that the entry on Rekor and the stored copy on npm will show an audit trail of where it was originally built from. This might still be useful for post-mortem analysis. Having a broken link could also end being a strong signal to the community that a package is no longer maintained and should not be used.

Does this mean you have to use GitHub as your source repo?

No, we'll support any provider supported by Sigstore's Fulcio service. Today this only includes GitHub Actions but it's vendor neutral and any provider that meets the following requirements will be supported:

• OIDC ID tokens identifying the current workflow/run/build.
• ID token with a custom audience (needs to be set to sigstore for Fulcio to accept it).
• Claims about the code repository, commit, build and actor

[ianlewis]

Just want to be mindful/note that there's nothing in this RFC which mentions that source, CI or build envs will be or will become immutable; which puts their audibility into question (ex. I can build & publish my package with a trusted builder but then delete that action/workflow/repo - killing the link/trace)

I'm not sure if npm will want to do this but whether links are still valid is something that can be checked during package/artifact verification.

[feelepxyz]

I'm not sure if npm will want to do this but whether links are still valid is something that can be checked during package/artifact verification.

Yeah maybe this could be optional check that can be performed during verification. We probably don't want to do this by default as it could add significant time to the verification if we're checking 1000s+ links for a typical install.

Another idea would be to send the link though a proxy service on npm, and if this service detects a 404 it updates some state on the package that shows a big red warning next to the link next time you view the package on npmjs.com for example. This might cause more headaches than it's worth though, if requests intermittently fail or if the source repo is down and suddenly a bunch of links are broken and need to be fixed when the service is back up.

[ljharb]

tbh this seems like a potential dealbreaker

[MylesBorins]

Any of these risks are a potential deal breaker 😇

Our intent and hope is that these will not prove to be a problem but we want to call out that these are indeed risks that could result in this specific approach not working out.

[bnb]

I don't disagree, but IMO this isn't because there wouldn't be but more because there's been near zero exposure.

[ljharb]

this should be considered a big concern for npm as well

[fkautz]

Would npm consider being an OIDC Provider for users who do not want to rely on a third party provider? It already has unique user names, manages user identities, and has 2FA support.

[haydentherapper]

It is an option, that was brought up in #626 (comment).

While it does provide a good user experience, the concern is that it doesn't isolate the identity provider from the package repository, so a compromise of a user account for one would compromise both.

[bnb]

Do you expect that this data will be able to be reflected in third-party registry caches like Verdaccio, jFrog Artifactory, and Sonatype Nexus, or is it being designed in a way that will be proprietary to the npm website? Additionally, will the API be somewhat reproducible for those third-party registries for non-public modules?

[bnb]

Ah, this was answered in the bullets below.

[feelepxyz]

is it being designed in a way that will be proprietary to the npm website? Additionally, will the API be somewhat reproducible for those third-party registries for non-public modules?

We don't want it to be proprietary to the official npm registry, which is partly why we're excited about adopting Sigstore. We're planning to document what is required by third-party npm registries to support this functionality and make it as easy as possible to adopt.

[fkautz]

There are several of us in both the CNCF and OpenSSF who are willing to help spec this out or provide guidance. Please let us know if you would like assistance.

[bnb]

It would be nice to be able to have the existing npm audit signatures functionality and the new functionality as independent features. Not saying that this shouldn't be added to npm audit signatures but more that we should also ensure that end-users can run the two sets of functionality independently.

[ljharb]

Indeed; the entire point of npm audit foo when we discussed npm audit signatures was that every foo would be independently configurable and invokable.

[feelepxyz]

existing npm audit signatures functionality and the new functionality as independent features

Thoughts on verifying all available signatures (registry and build) when running npm audit signatures but allowing you to filter the type with an argument like: npm audit signatures --type=registry|build (naming tbd)?

[MylesBorins]

There is going to be broader shared behavior between the various signature verification commands such as "what do you do if packages are not yet in cache" that lead me to feeling a single command makes far more sense

[bnb]

FWIW I'm not at all against a single command but I have seen so many situations where people want just one task and don't care about other tasks in a bundled command.

It feels... very un-npm-y to me not to be able to opt-in to choosing more limited but precise sets of work that do the exact thing I want completed, especially when the functionality is fundamentally not related (checking what you received is what's on npm vs. what this RFC proposes).

[bnb]

Also to answer @feelepxyz: that's a totally fine solution to me!

[ljharb]

npm audit is intended to be the single command - every subpart of it is intended to be granularly configurable and disableable by users, so that nobody is forced to do a check they don't want.

[fkautz]

If the consensus is to split it up into two separate functions, I would recommend splitting something like this:

npm audit signatures # runs everything related to signatures
npm audit signatures compare # only computes the hashes and compares
npm audit signatures integrity # validates the signature and ensures it is present and trusted in sigstore

This way, npm audit remains a single command, and npm audit signatures does not result in a user misunderstanding what it does.

[bnb]

It might be good to clarify at some point in this RFC if "source repository" must be GitHub or not. I'm 100% sure this will be a question that comes up, so it'd be good to be clear about that.

[feelepxyz]

There's no requirement on the source repo being a GitHub repo. The requirement is on the CI/CD system. So for example when CircleCI supports the Fulcio service (which verifies OIDC id tokens from the cicd system and mints short-lived signing certificates) you would be able to use any source repo supported by Circle.

Will clarify this 👍

[fkautz]

Make sure to consider that the source git repo may change, e.g. tags are mutable. branches may be deleted and recreated with new content. Malicious content could be uploaded (by the dev, or through a compromised user credential), a build submitted, then the branch deleted and recreated in an attempt to hide the payload.

(May already be covered, still reading through the document).

[feelepxyz]

Malicious content could be uploaded (by the dev, or through a compromised user credential), a build submitted, then the branch deleted and recreated in an attempt to hide the payload.

TBH we don't yet have great answers around these issues. We could try and mitigate against some things on the npm side, e.g. by optionally verifying the repo still exists during verification but verifying the correct code still exists is probably not feasible.

It would be great if we could somehow create immutable checkpoints in the source code repo/CI system. One naive idea could be to package up a shallow clone of the git repo as a build artifact 🤔

[bnb]

Almost all of these seem to be effectively external, non-maintainer benefits. Are there any more maintainer-focused goals? Right now, I see very little incentive for maintainers to be excited for this outside of ones like me who work at companies and get paid to do this kind of maintenance.

[feelepxyz]

This is good feedback 👍 part of the story is not placing any kind of maintenance overhead on maintainers, for example having to manage keys. Another part in my mind is making publishing from CI/CD safer and easier, which is not covered in this RFC but we're planning as a follow up.

We want to allow authorising the publish using the same OIDC id token from the CI/CD system. So instead of using a long-lived access token to authorize the publish, you would log into your npm dashboard and set up which source repo is allowed to publish what package. The publish step then exchanges the OIDC id token for a short lived access token after the npm registry has checked the source repo from the id token matches the allowed one.

Once you've set up publishing in this way, you would not need to use long-lived access tokens and the risk of stolen credentials basically goes away if you also use 2FA on your npm account. The setup for this would be exactly the same as what we're proposing in this RFC so you essentially get it for free. I imagine adoption would go up a lot once we have both features in place and all major CI/CD providers are fully supported.

[MarkLodato]

+1 to more clearly articulating the benefits to maintainers. You may also wish to separate goals (e.g. improve security) from requirements (e.g. zero overhead).

To me, the main benefit for maintainers is:

• Maintainers gain increased protection against malicious upload by compromised credentials or (other) rogue maintainers.

[bnb]

Are there any potential alternatives that have been identified if this is the case?

[feelepxyz]

Not concrete alternatives but the rough shape would probably look similar to the architecture in this RFC with the difference that npm/github would be running the services instead. Either fully fledged or a paired down version.

It would be sad if we got to this as it would probably mean that we either end up with some proprietary signature service just for npm/github or potentially multiple competing public good instances.

[haydentherapper]

If this RFC is only about provenance generated through CI, this is good.

Do you want to support other identities signing provenance or artifacts such as email or SPIFFE IDs?

[kommendorkapten]

For now, it's only focusing on CI systems integrated with Fulcio. I would say that it's harder to define policies around email and SPIFFE IDs. One central concept is to offer a set of "trusted builders" across all supported CI systems. As references to those should be part of the certificate (i.e they are validated by Fulcio) we have a way forward to identify/validate those packages built by a trusted builder.

But this does not of course rule out future extensions.

[feelepxyz]

Do you want to support other identities signing provenance or artifacts such as email or SPIFFE IDs?

Possibly yes eventually! We haven't figured out how to support private package and enterprise uses-cases here. I can imagine that we will want to support several different signing identities for these use-cases.

Less sure about email identities as it really only tells us who published the package and not where and how the package was built. It also introduces the issue of linking that identity to an authorized maintainer. Open to exploring ideas around it though.

[fkautz]

I'd like to help here if possible. I have experience with SPIFFE, SBOM generation, and in-toto and am happy to assist. I can also ask colleagues involved with projects like in-toto to help weigh in on the design.

The trick here I think will be to sign a top level document that embeds the claims. Perhaps the developer key can be used to sign the commit, then the commit can be embedded a document that is signed by the SPIFFE SVID, then submitted to Sigstore.

[haydentherapper]

+1, this is a critical point - How do you create a verification policy that maps an identity to a package in such a way that a) you can update it in the case of compromise or a new maintainer takes over, and b) you can differentiate between a valid update and an update that occurs because an maintainer's account on an IdP is compromised? Avoiding this is ideal.

[haydentherapper]

Agreed, no way to check that a user has MFA enabled on one of those accounts for example, as this isn't included in identity tokens.

[bnb]

IMO this should borderline be a requirement. I'm amazed that this isn't supported. If my publishing email is j@gmail.com but I publish the password publicly, this means anyone could sign in to that email and publish as me which... fundamentally gets around the entire point of this.

[haydentherapper]

A concern that has been brought up in the past is including the MFA status in identity tokens makes it easier to find targets. It would probably be simpler for all identity providers to require MFA.

[bnb]

It would probably be simpler for all identity providers to require MFA.

No opposition from me :)

[haydentherapper]

(From a Fulcio maintainer) This would be great!

[bnb]

I would also posit that "What's the benefit for maintainers" is an extremely important question. This current set is assuming that maintainers care and want to do signing, which isn't guaranteed.

If there's not a good benefit for maintainers, this will create a bad situation where maintainers are pressured by consumers to take on this additional workload with no tangible benefit. Resentment and hostility easily follow in situations like that (see: typescript definitions for package maintainers who don't want to do typescript).

[jchestershopify]

There will inevitably be maintainers who object to signing for a variety of reasons. While I think it is reasonable to listen, a software repository also answers to the needs of end users.

For a maintainer a compromised package is perhaps embarrassing and distressing. For end users it is potentially catastrophic. And there are far more end users than maintainers. I don't see any other way to do the utilitarian calculus than to find that the needs of the end users for enhanced security are paramount.

edit: to be clear I am a passer by, not an npm person.

[ljharb]

In any ecosystem where you have two cohorts of users, producers and consumers, it's simply not viable to do utilitarian calculus, because neither group can exist without the other. Both groups' needs must be weighed together.

eg, mandatory 2FA for maintainers is annoying and inconvenient, but the burden on maintainers is relatively small, and the benefits for maintainers and end users is exceedingly high, so it works out to enforce it.

If the burden is too large, or the benefits aren't bulletproof enough or high enough, it wouldn't likely be worth the tradeoffs.

[jchestershopify]

Both groups' needs must be weighed together.

That is in fact the calculus. Two variables: cost to maintainers, cost to end users, integrated under a curve with respect to the spectrum of potential outcomes. The area under the curve is dominated by the impact on users because there are so many and because they are more negatively affected than maintainers by a compromise.

If the burden is too large, or the benefits aren't bulletproof enough or high enough, it wouldn't likely be worth the tradeoffs.

The burden here is quite minimal. One click for the common case.

[ljharb]

I think it needs to be minimal for the uncommon cases too. Prolific maintainers can be eccentric, myself included, and doing both unpaid technical and emotional work does not tend to predispose humans to being charitable.

[mlieberman85]

@feelepxyz a CI/CD machine is just as attackable as my local machine, though - perhaps more so, since it's a known target that's exposed to the open web. Why can't my local machine - something that has proven to be more trustworthy so far than all CI providers I'm aware of, largely due to nobody trying to attack it I'm sure - be able to sign it?

I would really argue that this isn't true, unless you're running something without any security controls. I would assume the CI/CD systems here would be follow standards, be pen tested, etc. Compare that with a local workstation that could be downloading arbitrary software, running arbitrary software in addition to the build you're running. That build could be attacked from a pretty unbounded number of vectors. Compare that with a CI/CD system that has adequate network, identity, etc. controls, there would be minimized attack surface.

For me it isn't that I don't trust your personal build machine, its that we can't enforce any kind of provenance claims as to what repo the package was built from. An attacker that's publishing a malicious version of some package from their local machine could easily state that the package is from the original legitimate repo, without it actually reflecting the changes in the published package.

This doesn't fix the problem right? A CI/CD system that has been compromised can falsify some elements of the provenance just as well as a laptop.

[ljharb]

This doesn't fix the problem right? A CI/CD system that has been compromised can falsify some elements of the provenance just as well as a laptop.

This is my point, I think - that there's nothing inherent about a CI/CD system that isn't also true about a personal laptop, and vice versa - the only difference is likelihoods due to competence, target awareness/attractiveness, which software is being run, etc.

[mlieberman85]

That's true. However, I think generally CI/CD is easier to secure compared to your laptop. CI/CD system can be audited, and configured through code, to ensure that for example that builds happen hermetically, only download pre-approved tools with a provenance trail. Your laptop unless it's managed by some other system doesn't haven't similar properties. Also in the case of packages that have multiple developers and maintainers it makes it more clear that packages are built through one system that has been approved by the project.

In the case of let's say GHA, you need to trust that GHA is doing the right things from a workflow perspective, OIDC, etc. GHA is taking on a lot of risk to their reputation, but most likely have resources to do so. If I build an npm package from my local laptop it's mostly on me. I could inadvertently have a rootkit on my machine, using a compromised OS kernel, have downloaded malicious software, etc. It is harder to trust anonymous actor's laptop compared to a potentially audited/threat modeled CI/CD system or service.

[ljharb]

Perhaps - but there are few personal laptop environments more objectively trustable than mine based on track record :-) without a doubt the majority will want to use this system, and probably should! However, if it's made a requirement - explicitly, or implicitly via creating metadata that effectively punishes an author for not complying - that seems like it would be harming maintainers for a very dubious benefit.

[jchestershopify]

Trust has economies of scale. Asking me to trust every random laptop means that, to achieve a similar level of confidence to a highly defended central provider, I would need to interview a bunch of people who, as has been pointed out elsewhere in this discussion, might object to the added burden.

Put another way: I don't know you. I know Github. Given my current priors, I trust them more than you, and it's costly for both you and I to update those priors.

[bnb]

As someone unfamiliar with this ledger: Are there any concerns around exponential growth of this ledger and the insertion of new entries slowing down publishing? Other.... ledgers that I'm aware of begin to hit speed issues that would be concerning at the volume of all published packages.

Additionally, when you say The Cosign functionality will be embedded directly in npm CLI does this mean that the CLI will reach out to the Fulcio/Rekor APIs, or that additional binaries would be included?

[haydentherapper]

The ledger is sharded - Every year a new tree is created, so this should alleviate concerns around performance. Additionally, while not currently implemented, we'd like to have log mirrors to distribute the load, but this will require the community operating mirrors, which comes with performance expectations and storage costs.

The Cosign functionality will be embedded directly in npm CLI

I assume this is referring to performing the same behavior as Cosign, using the sigstore-js library to call out to Rekor/Fulcio.

[bnb]

Every year a new tree is created, so this should alleviate concerns around performance.

I'd be really curious to know what the throughput is. npm has... a lot of publishes, and I get concerned about even a yearly reset if this will also include other ecosystems that are also growing + new ecosystems that haven't yet been invented.

What happens when the equivalent of all packages in 2022 are published in one week? 😅

[feelepxyz]

It will likely take a very long time before all daily package publishes are signing their builds. Assuming the uptick will be gradual over time.

The database that backs the transparency logs of both Rekor and Fulcio has very high throughput. It's the same database that powers the certificate transparency log for all SSL certs being issued and performs over 2,000 writes per second.

[bnb]

It will likely take a very long time before all daily package publishes are signing their builds. Assuming the uptick will be gradual over time.

IIRC npm's growth over time hasn't necessarily been gradual 😅

The database that backs the transparency logs of both Rekor and Fulcio has very high throughput. It's the same database that powers the certificate transparency log for all SSL certs being issued and performs over 2,000 writes per second.

IMO it would be really nice to do some of the math before this RFC is ratified to see how much runway we have until we hit 2,000 writes per second across all the ecosystems that have begun considering this. It'd be good to know that this is/isn't a problem and how far out we need to start thinking about being concerned :)

[jchestershopify]

I agree that some envelope arithmetic would be ideal. My current estimate is that it'll be no big deal: 2k RPS is 2.88M per day. From searching a bit I see counts of ~2M packages in npm. I doubt that every package is signed every day, so there's some headroom.

The 2k probably isn't a hard upper limit either. Let's Encrypt reported early last year on their current DB servers -- 2x EPYC giving 64 cores, with 2TB RAM. It's possible to buy 4 socket EPYC systems with up to 4TB of RAM.

[bnb]

I know this is a problem introduced by proxy of using sigstore, but I'm uncomfortable with the likely future where maintainers who want to publish a package are relatively forced by their users to identify with a limited set of platforms that have extremely valid and well-documented criticisms around privacy (Google/Microsoft tracking, GitHub backing up on cookies policy), politics (GitHub ICE, Microsoft DoD/JEDI deals, Google's Dragonfly, to name a few), and other ethical issues (treatment of women/Black/trans/other marginalized employees).

I don't fully believe that opting out of this feature for those reasons is a particularly equitable solution, nor one that will be an actual choice in the future.

[jchestershopify]

This is mentioned as the Vendorization problem. RubyGems was the first to encounter this discussion and a number of alternatives were floated both within the RubyGems discussion and in discussion with peers in other ecosystems (including npm).

1. Grow the list of endorsed IdPs. The current list isn't meant to be final, it's just what was convenient to bootstrap. There are many candidate IdPs who are non-commercial.
2. Operate an IdP specific to npm. This solves the vendorization problem (you've already decided to trust npm) but it almost completely cancels a critical advantage of independent IdPs, which is that to sign and upload an attacker will need to compromise two accounts.
3. A "neutral" IdP, independent of npm, probably shared among multiple ecosystem software repositories, operated by some trusted third party (at a first guess, the OpenSSF).
4. An email-based flow for verification. This has mostly been rejected as "OIDC over SMTP" -- a real possibility of protocol design flaws make this highly risky.

[bnb]

I'm fine with #1 and maybe #3 (I fully understand why #2 and #4 are less than ideal). I'd really just want to see that list expanded somewhat significantly before this launches.

[feelepxyz]

We're sidestepping some of the problems with email based identities in this proposal by just supporting workflow identities from supported CI/CD systems. That said, we might wan to support email based identities in future if support improves.

[nicorikken]

I certainly hope vendorization will be addressed from the beginning by offering alternative solutions. As other package managers run into the same issue, I see an opportunity for collaboration. If vendorization isn't addressed, you can expect a vocal group to resent this proposal, causing confusion and doubt and ultimately leading to failure of the whole thing. Improving supply chain security is too important to fail, so it should be done right to get everybody on board.

[jchestershopify]

For reference, some further discussion of the Neutral IdP idea took place in Fulcio issue 444.

[feelepxyz]

I certainly hope vendorization will be addressed from the beginning by offering alternative solutions.

💯 One of our internal goals of publishing this RFC is to help start these conversions with CI providers. I'm also excited about Sigstore as a solution to software signing for this reason. It's a vendor neutral entity that can help create alignment.

I'm not closed or opposed to extending npm package signing to support some kind of "author/maintainer" IdP in future that could allow maintainers to publish from their laptops. For the first iteration I think we'll start with only supporting build or workflow identities from CI/CD providers.

[bnb]

If this is the goal, you are going to get questions about 2FA and automated publishing. While strides have been made, are there further efforts to continue to make publishing from CI a reasonable experience given that this will effectively force all maintainers of large-scale packages to publish from CI?

[feelepxyz]

Copying answer from another thread.

are there further efforts to continue to make publishing from CI a reasonable experience given that this will effectively force all maintainers of large-scale packages to publish from CI?

Yes! We're considering a proposal to authorise the publish using the same OIDC id token from the CI/CD system (still early days so haven't committed to this work yet). So instead of using a long-lived access token to authorize the publish, you would log into your npm dashboard and set up which source repo is allowed to publish what package. The publish step then exchanges the OIDC id token for a short lived access token after the npm registry has checked the source repo from the id token matches the allowed one.

Once you've set up publishing in this way, you would not need to use long-lived access tokens and the risk of stolen credentials basically goes away if you also use 2FA on your npm account. The setup for this would be exactly the same as what we're proposing in this RFC so you essentially get source and build linking for free if you set up OIDC publishing. I imagine adoption would pick up a lot once we have both features in place and all major CI/CD providers are fully supported.

I think we can also ease the migration by providing reusable workflows for publishing npm packages on CI/CD. On actions you could start by using a manual workflow trigger to publish.

[MylesBorins]

While longer term we are hoping to implement something on top of OIDC we are currently working on Granular access tokens as a successor to automation tokens. These tokens will:

• Be able to be scoped to a specific package or scope
• Scoped to read or read/write permissions
• Has an expiry date

As mentioned above longer term we are looking at OIDC to remove the needs for tokens and a streamlined GitHub experience to make setting up everything a breeze. All of this is intended to be build on open technologies so that other CI/CD environments can benefit from similar implementation and that we can explore scaling these approaches to other package registries.

None of the above is part of the signing work outlined in this RFC though.

What I can say is that the remainder of this year has a big focus on "package security" for publishing in the same way we've spent the last 9 months extremely focused on "account security"

[bnb]

If this means that GitHub Actions are the only way to publish verified packages, I'm a hard and vocal -1 on this until that's not the case. There are extremely valid reasons to not use GitHub Actions that haven't been addressed in the years it's been out, and there are major packages in the ecosystem that do not use GItHub Actions for those reasons.

[bnb]

(from the following context that does not seem to be the case for npm to ship this, but I am going to leave this here to ensure that perspective is logged)

[feelepxyz]

100%, GitHub Actions is the only provider that fully supports the requirements from Fulcio today but I'd like to see other CI/CD providers like GitLab and Circle be supported before this functionality launches for npm.

[everett1992]

Would it be feasable to allow any OICD provider to include signatures with published npm packages and let the npm cli configure which providers it trusts? npmjs.com can choose to only display verified badges for Github Actions and other trusted providers but other communities could choose to trust providers of their choice.

[feelepxyz]

Would it be feasable to allow any OICD provider to include signatures with published npm packages and let the npm cli configure which providers it trusts?

The current proposal relies on the OIDC Identity Providers trusted by Fulcio, which is currently defined here: https://github.com/sigstore/fulcio/tree/main/federation

On top if this I could also see npm providing some way for you to write your own policies of which of these providers to trust. The hope is that the list of supported and trusted providers in Fulcio grows significantly over time as its vendor neutral and open source.

One idea to make it easier to support more providers in Fulcio would be to come up with a spec for ID token claims, so any provider that supports some officially supported set of claims could be supported more easily in Fulcio.

[ljharb]

can you elaborate here on why a local machine that i can configure is different than a CI machine that i can configure?

[mlieberman85]

It's a state space problem. A CI/CD system that is hosted immutably and can be verified/audited will be simpler to reason about compared with a laptop that usually has lots of arbitrary stuff installed.

[feelepxyz]

Supporting signing from both CI/CD and local machines increases the scope of the initial work quite a bit so we've decided to just start with signing from CI/CD as the identity solution (OIDC id tokens issued by the CI provider) does not extend to local machines.

There are ways we can support signing from local machines in future, e.g. using proof of email linked to a maintainer but this gives different guarantees than signing from CI/CD.

Related to this, my hope is that all public packages are eventually published from open, auditable and automated systems with all the other benefits this brings like ephemeral environments and the ability to prove what went into a build. Maybe I'm deluding myself on the possibility of this though? 🤔

I also think the experience and selection of automated build systems could be vastly improved to make publishing from one as easy as toggling a button or two on whatever source control system you might use that just works with the package setup/registry and versioning scheme you use.

[ljharb]

@feelepxyz the possibility is only there if the community freely decides to make that move - and it’s not a free decision unless the existing mechanisms are equally privileged.

[bnb]

Related to this, my hope is that all public packages are eventually published from open, auditable and automated systems with all the other benefits this brings like ephemeral environments and the ability to prove what went into a build. Maybe I'm deluding myself on the possibility of this though? 🤔

I think there are two reasons why this is challenging:

• exploration - whether from someone just learning JavaScript or from someone just exploring a new solution to a problem they have, this adds barriers and weight.
• tooling - currently, npm simply does not provide the necessary tools that make such automated publishing safe and reliable for anyone operating past a certain scale or who requires a certain level of trust.

I agree that things could be dramatically easier. Unfortunately that does require substantial investments, and it is seemingly often very challenging for certain teams to actually secure the necessary resources to be able to accomplish the outcomes that are required to achieve that ease.

[bnb]

Any updates on the progress of this?

[feelepxyz]

Any updates on the progress of this?

@bnb yes! We're actively working on this and aiming to open up a beta in the new year. I'm planning to merge this RFC soon as we've settled on the approach for now.

We just wrote about Sigstore reaching General Availability (GA) and are working on a JavaScript client for Sigstore that will be used to generate this signed source/provenance information.

There have been several concerns and questions raised about the accessibility of this feature and how we could roll this capability out to more developers, e.g. if you can't use CI to publish your package, or how we could support already published/legacy packages.

We're starting out by supporting signing newly published public packages from trusted CI systems (the goal being non-forgeable/non-falsifiable information about the source and build). The barrier of entry will initially be higher than we want it to be. The tooling won't be as good as we want it to be but our aim is to invest in this over time and get to the point where the secure way is the path of least resistance. This will take a time but we're committed to investing in this area.

[ljharb]

O.o did we discuss this again in an RFC call such that it's appropriate to merge this?

[ljharb]

@feelepxyz why is a non-npm team member merging an npm RFC prior to having full discussion about it? That's not a good look for Microsoft (or Github) as an acquirer.

This is a very low value security property that could potentially cause a lot of harm to the ecosystem by de-facto punishing perfectly secure packages without this provenance info. More to the point, if perfectly secure packages don't choose to ship this provenance info (like my 10+% of npm packages likely will never do) then why would the ecosystem adopt it as a signal?

[trevrosen]

@ljharb thanks for your contributions to the Node community. This RFC was open for comment for ~3 months before we merged it, and we signaled our intention to merge in a comment almost two weeks ago. This issue is the proper forum for discussion on the RFC, and that discussion has dwindled to nil over the last two months.

This is a very low value security property that could potentially cause a lot of harm to the ecosystem by de-facto punishing perfectly secure packages without this provenance info. More to the point, if perfectly secure packages don't choose to ship this provenance info (like my 10+% of npm packages likely will never do) then why would the ecosystem adopt it as a signal?

This is not a constructive comment.

We will emphasize once again that the new functionality described in this RFC is optional, and you're free to use it or disregard it as you choose.

[ljharb]

@trevrosen The only people who should be merging things on an npm repo is the npm team, not "any github employee", and npm RFCs have never before been merged a) based on a time limit, b) without extensive discussion in an RFC call, or c) by someone who isn't on the npm team. The RFC has been commented on a number of times, and many of the comments are not in favor. What's the point of an RFC if you're going to merge it before these concerns are addressed in an RFC call?

Which part of my comment is not constructive? Provenance simply isn't of high value for an ecosystem that doesn't ship binaries, and incomplete (in terms of a dep graph) provenance info is of equally low value. If you disagree, I'd love to hear your reasons.

Functionality being optional does not mean there can't be ecosystem effects. If a "security" property exists, and it gains enough adoption, anyone not using it will be effectively punished for not having it - which makes it non-optional.