Support package signing and verification

[ryankurte]

It should be pretty easy to have a basic flow so CI created images can be signed, and verified on pull. Not sure what algorithm(s) / tools we should use? Accessibility is key, both IRL and in CI, and it needs to all work well cross-platform.

• Add public key (and key type?) to metadata (let's b64 it or something)
• Search for a signature file corresponding to each package (.sig or appropriate to the format)
• Generate checksum / validate signature against public key on pull

[badboy]

signify could be one way to support that.
I have a Rust implementation of it. Though it's not been touched since 2018 it might be not too much work to update it & expose parts of it as a library so that binstall could use it.

The Cargo.toml would then have the public key listed and .sig` file or similar contains the signature to verify.

[NobodyXu]

Sigstore has reached v1.0.0 and it provides a rust crate sigstore-rs for verifying the signature.

[sunshowers]

For nextest-rs/nextest#369, I had a look at https://docs.rs/sigstore/latest/sigstore/ and seems like it should be possible to:

1. create a signature bundle at release time using cosign sign-blob: https://docs.sigstore.dev/cosign/signing_with_blobs, including in GitHub Actions
2. upload the signature bundle to GitHub Releases
3. verify the bundle as in https://github.com/sigstore/sigstore-rs/blob/main/examples/cosign/verify-bundle/main.rs

Now, this just verifies that the bundle uploaded also matches the artifact uploaded. I think this would be a really useful initial step.

To make this better:

• We also need to verify the identity of the signer (which should probably be stored in Cargo.toml? This way crates.io becomes the root of trust). It's not totally clear to me how to do that from the current sigstore API. Route away from experimental to 1.0 sigstore/sigstore-rs#274 (comment) mentions this, I believe:

  I think the verify_blob* methods could use a variant that supports Fulcio based certificates + verification constraints.

• We also need to ensure that the artifact and bundle haven't changed since the initial release, in case a malicious entity takes over control of the GitHub repo and starts clobbering old release artifacts. This might need to be done using timestamps, possibly with https://docs.rs/sigstore/0.7/sigstore/cosign/bundle/struct.Bundle.html#structfield.signed_entry_timestamp. It's possible using OCI to store artifacts rather than GitHub Releases might also help.

I'm not a security expert and I'm almost certainly missing something. It would also be good to maybe open an issue with sigstore-rs people discussing this.

[sunshowers]

More thoughts.

1. I believe that GitHub Releases is not a reliable place to store artifacts that can never be changed in the future. Is this correct?
2. Timestamps aren't great as a way to verify authenticity, ideally we'd use hashes. But assuming 1 is true, there's no reliable place to store hashes on GitHub Releases.
3. Does uploading either the artifact or the signature bundle to OCI solve the "malicious user takes over GitHub Actions and changes old artifacts" threat model? I don't know if OCI artifacts on ghcr or other registries can be changed. It's worth verifying this.
4. It would also be really cool to have the binstall verification algorithm be its own non-copyleft-licensed crate, so non-binstall users can use the same algorithm.

[ryankurte]

I believe that GitHub Releases is not a reliable place to store artifacts that can never be changed in the future. Is this correct?
Does uploading either the artifact or the signature bundle to OCI solve the "malicious user takes over GitHub Actions and changes old artifacts" threat model?

hmm, you can delete and re-upload artifacts, but so long as the signature is valid is it important that these are immutable? i think the risks one is attempting to mitigate are worth some though here.

• for folks worried about an exact binary they may need to pin a signature or a hash for a given platform and version
• most users will fetch new version / binaries anyways so long as they're correctly signed so is a new version meaningfully different from an updated binary with the same version? (eg. compromised actions / keys could be used to publish a patch release which any non-exact version filter would update to anyway)

slightly aside one of the things i've been thinking about a bit is how annoyingly repetitive setting up gh actions CI is for rust tools... maybe it'd be worth us investigating putting together a workflow template with variables for the usual stuff (rust version, platforms, platform packages, cross) that could include packaging / signing / publishing with whatever mechanisms we do use.

[sunshowers]

slightly aside one of the things i've been thinking about a bit is how annoyingly repetitive setting up gh actions CI is for rust tools

A couple of existing efforts:

• https://github.com/taiki-e/upload-rust-binary-action which ideally would follow whatever pattern binstall supports (I know @NobodyXu works on both the projects). I use this for nextest, it's great: https://github.com/nextest-rs/nextest/blob/1cd93240e022f28968bffbbec4e6ab7a9a585dee/.github/workflows/release.yml#L120-L313
• https://github.com/axodotdev/cargo-dist

[sunshowers]

hmm, you can delete and re-upload artifacts, but so long as the signature is valid is it important that these are immutable? i think the risks one is attempting to mitigate are worth some though here.

for folks worried about an exact binary they may need to pin a signature or a hash for a given platform and version

Yes, security-conscious users would like to pin the version and platform and have it be guaranteed to always resolve to the same artifacts. (modulo cargo-binstall itself being compromised, but they'll likely want to pin that to an exact version too)

most users will fetch new version / binaries anyways so long as they're correctly signed so is a new version meaningfully different from an updated binary with the same version? (eg. compromised actions / keys could be used to publish a patch release which any non-exact version filter would update to anyway)

Yes. I think publishing a new version that is bad is materially different from a malicious actor surreptitiously updating an old binary. (crates.io has the same philosophy, right?)

[sunshowers]

Ah sorry, NobodyXu doesn't work on upload-rust-binary-action. But it's part of the same general family of actions as https://github.com/taiki-e/install-action which they do work on, haha :)

[NobodyXu]

I believe that GitHub Releases is not a reliable place to store artifacts that can never be changed in the future. Is this correct?

@sunshowers We could put the public key/checksum inside Cargo.toml since it is actually immutable and you can count on it.

If the registry is hacked, then no matter how secure your GitHub release is, it won't matter since the attacker can change Cargo.toml to point to whatever release they like.

Also, we are working on checksum support for registry #1183 , it will provide guarantees on security (presumably because crates.io index and the crates.io storage can be provided by two different sets of servers).

[sunshowers]

We could put the public key/checksum inside Cargo.toml since it is actually immutable and you can count on it.

Interesting idea -- how would you do that? I guess I imagined modifying Cargo.toml as part of the release process was generally off limits.

[NobodyXu]

We could put the public key/checksum inside Cargo.toml since it is actually immutable and you can count on it.

Interesting idea -- how would you do that? I guess I imagined modifying Cargo.toml as part of the release process was generally off limits.

You can put a public key inside package.binstall under Cargo.toml, then use that private key to sign your packages released.

Once the Cargo.toml is uploaded to crates.io, it stays there and is immutable.

[NobodyXu]

You could still setup automatic upload to crates.io on GHA using the recent crates.io scoped token (which we are already using) and it can be revoked if your GitHub get hacked somehow.

[sunshowers]

Gotcha! So, hmm, I think just storing the public key doesn't quite solve the threat model that I outlined, because: assume that the private key is stored as an environment secret on GHA. A malicious actor can:

• gain access to the repository
• read the private key from GHA
• compromise old binaries
• re-sign them with the private key

In other words, we don't just need the public key to be stored in immutable storage, we also need some sort of identifier for the binary. This could be just as simple as a hash, or a certificate, or something that just makes sure that once a binary is published it never changes.

Also, users will have to manage their own keys, which is something experts can likely do but new users could have trouble with. (In this case, to perform a release via GHA, you'd have to store the private key as a secret, then carefully destroy the key material on the local machine.)

While asking people to use keys is definitely one way to go about it and does solve some problems, ideally users would also be able to perform signing via OpenID Connect.

[NobodyXu]

read the private key from GHA

If you store that in the secret, then even the admin cannot read it, it can only be accessed inside GitHub Action.
Although they can try to reveal it by changing the GitHub Action.

In other words, we don't just need the public key to be stored in immutable storage, we also need some sort of identifier for the binary. This could be just as simple as a hash, or a certificate, or something that just makes sure that once a binary is published it never changes.

Hmmm, perhaps we can store a root certificate in Cargo.toml, then derive a public key based on hash of the crates tarball uploaded to crates.io?

(Of course, if you can generate a new hash/public-key and modify Cargo.toml in each release, it will definitely guarantee security.)

While asking people to use keys is definitely one way to go about it and does solve some problems, ideally users would also be able to perform signing via OpenID Connect.

Thanks, I will read it later.

[NobodyXu]

I think the hash of the crate tarball can definitely be used when verifying the pre-built binaries, after all, the binary is built from the crate tarball.

[NobodyXu]

While asking people to use keys is definitely one way to go about it and does solve some problems, ideally users would also be able to perform signing via OpenID Connect.

I skim through it, honestly I don't think it will solve the "someone replace the release artifacts" given that it still requires the developer to provide an identity token either in CI if automated, or locally.

Perhaps I misunderstood and it does have mechanism to solve it, but IMHO the most secure way is still to provide a checksum inside Cargo.toml and update it on every release, given that crates.io is immutable and always trusted.

Also cc @taiki-e since I would also love to hear feedback from you.

[passcod]

We can do away with user key management via the same process sigstore works. That is, in CI:

1. Create a key pair
2. Optionally add a certificate to sigstore that attests identity via OIDC
3. Sigstore-sign all artifacts with that keypair
4. Add the public key to Cargo.toml
5. Throw away the keypair
6. Publish the crate

As crate publishes are immutable, new artifacts can't be uploaded for that version, and as the keypair only exists in the context of that one CI job, it can't be stolen in the future.

Verification is either:

1. SHA256-hash a given artifact
2. Look it up in sigstore
3. List out all crate-metadata attestations attached to the artifact
4. Verify one of these matches the public key from crate metadata

or:

1. Download associated sigstore certificate/signature distributed alongside artifacts
2. Verify it was signed by the public key from crate metadata
3. Verify it matches the artifact

[passcod]

Alright, how about this:

[metadata.binstall.signing]
algorithm = "minisign"
pubkey = "RWT+oj++Y0app3N4K+PLSYTKhtXimltIHxhoFgyWjxR/ZElCG0lDBDl5"
file = "{ url }.minisig"

We add support for this optional section to the binstall metadata. algorithm and pubkey are mandatory, file is optional and defaults to { url }.sig (where url is the url we're downloading, and we include all the other fields too just in case you want to do something freaky).

algorithm can initially only be "minisign" because that's pretty popular, self-contained, doesn't have a thousand options, has good rust support, and keys are small. Later we can add GPG and Cosign and whatever else.

pubkey is the string representation of the public key in whatever format is native to the algorithm, in this case base64. Later we can add things like:

pubkey = { file = "signing.pub" }
pubkey = { url = "https://someserver.online/signing.pub" }

to support loading from elsewhere maybe.

Then binstall would, if the section is there:

1. Check the file template for syntax (to fail early if it's broken)
2. Do package resolution as normal, but don't download the file yet
3. Render that file template
4. Download that file as a signature. I think we can just keep it in memory, for minisign it will be less than 400 bytes and for GPG it should be at most single digit kilobytes.
5. Download the actual file, with the checksum thing @NobodyXu added recently enabled and configured for the algorithm
6. Verify the signature
7. Proceed with installation.

We should add:

• a --only-signed flag to refuse to install non-signed packages
• a --skip-signatures flag to disable this entire thing (e.g. if a packager messes up their publish and someone really wants to download things anyway)

That leaves the modalities of signing to the packager. As we've discussed, there's two main approaches:

1. Persistent keypair, so the section above would be more or less static and the packager is responsible for protecting their key. This approach supports --git too.
2. Ephemeral keypair, so a new key is generated before publish and its pubkey is added to the Cargo.toml just in time before publish, then the private key deleted when done with. This approach doesn't support installing via --git unless the publish process commits the Cargo.toml with the signature.

That will at least introduce package signing to the ecosystem; with this initial approach implemented we'll then be able to get more feedback from both packagers, users, and other community members. Because we'll namespace under metadata.binstall, we won't step on any feet, and then hopefully in the long term a common solution will emerge (with cargo-dist, crates.io, or whomever).

How does that sound?

[NobodyXu]

It sounds great and aligns with how I would like this to be done.

[passcod]

I've started work on a PR to do this!

[passcod]

Alright y'all, we've shipped the first version of binstall with signing support!

Find more info in the release notes and documentation.

[NobodyXu]

cc @sunshowers @ryankurte @somehowchris @taiki-e

We will also open to new signing algorithms and PRs for improving our signing mechanism!

[passcod]

Tracking/discussion issues for additional algorithm:

• Signing: support GPG #1424
• Signing: support sigstore #1425
• Signing: support SSH #1426