The sigstore-the-local-way tutorial will teach you how to build and configure a local sigstore stack (cosign, rekor, fulcio) and use it to sign and verify container signatures. It can be completed in about 15 minutes.
This tutorial is heavily based on Sigstore the Hard Way, though simplified for local use without skipping any of the sigstore-specific steps. For example, this tutorial omits actions relating to provisioning services via the Google Cloud Platform, DNS updates, HAProxy, or Certbot. This new tutorial has also been modified for cross-platform use and was developed using OpenBSD and fish.
This tutorial explores three levels of signing & verification that sigstore makes available, adding new dependencies each time:
- Signing a container against a local OCI registry
- Signing a container against a local OCI registry and the Rekor transparency log
- Keyless signing a container against a local OCI registry and the Rekor transparency log
This tutorial involves launching several foreground services, so using a terminal multiplexer such as tmux or screen is highly recommended.
As a bonus, this repository includes a launch script to relaunch the sigstore stack using tmux any time after the completion of the tutorial.
If you run into an error, be sure to check the Appendix for a list of errors you might bump into.
Installing the full-stack requires the Go programming language, a SQL database, and a handful of security tools:
- Arch Linux:
sudo pacman -S mariadb git softhsm opensc go
- Debian|Ubuntu:
sudo apt-get install -y mariadb-server git softhsm2 opensc
- Fedora:
sudo dnf install mariadb-server git go softhsm opensc
- FreeBSD:
doas pkg install mariadb105-server git softhsm2 opensc
- Gentoo:
sudo emerge mariadb git go softhsm opensc
- macOS:
brew install mariadb go softhsm opensc
- OpenBSD:
doas pkg_add mariadb-server git go softhsm2 opensc
- NetBSD:
doas pkgin install mariadb-server git go softhsm2 opensc
Verify that the Go version in your path is v1.16 or higher:
go version
If your Go version is too old, uninstall it and install the latest from https://go.dev/dl/
We'll run through the steps in the above rectangles, starting from left (ko) to right (cosign verify).
sigstore can sign against any container registry. In the interest of keeping things local, we'll run a local registry. You may want to run this in tmux or screen as it is one of the many foreground services you will start:
go install github.com/google/go-containerregistry/cmd/registry@latest
$HOME/go/bin/registry
If it's successful, the registry command will quietly hang until an incoming request arrives. Start a new terminal window or tab and let's move on!
For this demo, we will build the rekor-cli tool into an unsigned image and push it locally.
Check out the rekor codebase, and use ko
to build and push an unsigned image to our local registry:
mkdir -p $HOME/sigstore-local/src
cd $HOME/sigstore-local/src
git clone https://github.com/sigstore/rekor.git
go install github.com/google/ko@latest
cd $HOME/sigstore-local/src/rekor/cmd
KO_DOCKER_REPO=localhost:1338/demo $HOME/go/bin/ko publish ./rekor-cli
Here is what successful output looks like:
2022/02/03 15:38:35 Published localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9@sha256:184a7313e59492c366e505acdb91eeeca0abdbc40281c0dd4933aab161179760
localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9@sha256:184a7313e59492c366e505acdb91eeeca0abdbc40281c0dd4933aab161179760
Install the latest release:
go install github.com/sigstore/cosign/cmd/cosign@latest
The most basic usage of cosign uses a local key pair. You can use any password you like, even an empty one:
cd $HOME/sigstore-local
$HOME/go/bin/cosign generate-key-pair
Sign the container we just published:
$HOME/go/bin/cosign sign --key cosign.key localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9:latest
And validate the signature:
$HOME/go/bin/cosign verify --key cosign.pub localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9 || echo OHNO
The successful verification output looks verbose and scary, but everything checks out:
Verification for localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9:latest --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- The signatures were verified against the specified public key
- Any certificates were verified against the Fulcio roots.
[{"critical":{"identity":{"docker-reference":"localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9"},"image":{"docker-manifest-digest":"sha256:3a46c2e44bfe8ea0231af6ab2f7adebd0bab4a892929b307c0b48d6958863a4d"},"type":"cosign container image signature"},"optional":null}]
Congratulations! You have signed your first container using sigstore. It is also possible to sign binaries or other blobs of text using cosign
, see Working with other artifacts.
The way we've signed a container so far only relies on a single mutable source of truth: the container registry. With Rekor, we will introduce a second immutable source of truth to the system.
While Sigstore can use multiple database backends, this tutorial uses MariaDB. Once you've installed the prerequisites, run the following to start the database up locally in a locked-down fashion:
- Arch:
sudo mariadb-install-db --user=mysql --basedir=/usr --datadir=/var/lib/mysql; sudo systemctl start mariadb && sudo mysql_secure_installation
- Debian|Ubuntu:
sudo mysql_secure_installation
- Fedora:
sudo systemctl start mariadb && sudo mysql_secure_installation
- FreeBSD:
sudo sudo service mysql-server start && sudo mysql_secure_installation
- macOS:
brew services start mariadb && sudo mysql_secure_installation
- OpenBSD:
doas mysql_install_db && doas rcctl start mysqld && doas mysql_secure_installation
During the secure script, I recommend skipping the password change, but answering "YES" to everything else. Once complete, create the database tables which Trillian will need:
cd $HOME/sigstore-local/src/rekor/scripts
sudo sh -x createdb.sh
Install Trillian, which provides a tamper-proof append-only log based on Merkle Trees via a gRPC API. The data will be stored in the MariaDB database we created.
go install github.com/google/trillian/cmd/trillian_log_server@latest
go install github.com/google/trillian/cmd/trillian_log_signer@latest
go install github.com/google/trillian/cmd/createtree@latest
Start the log_server, which will provide a Trillian API to both Rekor and the Certificate Transparency frontends (also referred to as personalities):
$HOME/go/bin/trillian_log_server -http_endpoint=localhost:8090 -rpc_endpoint=localhost:8091 --logtostderr
Start the log signer, which periodically checks the database and sequences the data into a merkle tree:
$HOME/go/bin/trillian_log_signer --logtostderr --force_master --http_endpoint=localhost:8190 -rpc_endpoint=localhost:8191
The Trillian system is multi-tenant and can support multiple independent Merkle trees. Run this command to send a gRPC request to create a tree and save the log_id for future use:
$HOME/go/bin/createtree --admin_server localhost:8091 | tee $HOME/sigstore-local/trillian.log_id
The Rekor project provides a restful API based server for validation and a transparency log for storage. Install it from source:
cd $HOME/sigstore-local/src/rekor
go install ./cmd/rekor-cli ./cmd/rekor-server
Start rekor:
$HOME/go/bin/rekor-server serve --trillian_log_server.port=8091 --enable_retrieve_api=false
Upload a test artifact to rekor:
cd $HOME/sigstore-local/src/rekor
$HOME/go/bin/rekor-cli upload --artifact tests/test_file.txt --public-key tests/test_public_key.key --signature tests/test_file.sig \
--rekor_server http://localhost:3000
If it works, the following will be output:
Created entry at index 0, available at: http://127.0.0.1:3000/api/v1/log/entries/d2f305428d7c222d7b77f56453dd4b6e6851752ecacc78e5992779c8f9b61dd9
You can inspect the resulting record with:
curl -s http://127.0.0.1:3000/api/v1/log/entries/d2f305428d7c222d7b77f56453dd4b6e6851752ecacc78e5992779c8f9b61dd9
With Rekor setup, we can now sign and upload the signature for our image:
COSIGN_EXPERIMENTAL=1 $HOME/go/bin/cosign sign --key $HOME/sigstore-local/cosign.key \
--rekor-url=http://localhost:3000 \
localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9
Verify the container against the OCI attestation and the Rekor record:
COSIGN_EXPERIMENTAL=1 $HOME/go/bin/cosign verify --key $HOME/sigstore-local/cosign.pub \
--rekor-url=http://localhost:3000 \
localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9
With this invocation, cosign will check the OCI metadata and rekor. Rekor in-turn, will use Trillian and the MariaDB database we setup earlier to verify the certificate. Success looks like this:
Verification for localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9:latest --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- The claims were present in the transparency log
- The signatures were integrated into the transparency log when the certificate was valid
- The signatures were verified against the specified public key
- Any certificates were verified against the Fulcio roots.
[{"critical":{"identity":{"docker-reference":"localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9"},"image":{"docker-manifest-digest":"sha256:35b25714b56211d548b97a858a1485b254228fe9889607246e96ed03ed77017d"},"type":"cosign container image signature"},"optional":{"Bundle":{"SignedEntryTimestamp":"MEUCIGhbOHcduQOWrsL8CaAHeSB1pQXintGyo2OlEs7yflWcAiEA2Wk/WeT5GOpYkpV2bZzaZBEt925W00VOAE/aHi7yoIY=","Payload":{"body":"eyJhcGlWZXJzaW9uIjoiMC4wLjEiLCJraW5kIjoiaGFzaGVkcmVrb3JkIiwic3BlYyI6eyJkYXRhIjp7Imhhc2giOnsiYWxnb3JpdGhtIjoic2hhMjU2IiwidmFsdWUiOiI4Yzg1YjNhMjQ5Y2I1MjNlYTNiYjRiM2RiN2RmMTc0Zjc0ZjI0NGJiNmJmN2QyNjI3ZjJjNTZlNmYzZjliZmQzIn19LCJzaWduYXR1cmUiOnsiY29udGVudCI6Ik1FWUNJUUM4NXZrMEoxQ0dCdFVGMEtBVXpCOHRCWG10TzkreFNQS2NldG4wYm52eGVnSWhBT0lnWG9Xa3FoR2FiWm8xRFFUem5GaTFKRU5vL0VvSDg5bGh0OWthZWNpOCIsInB1YmxpY0tleSI6eyJjb250ZW50IjoiTFMwdExTMUNSVWRKVGlCUVZVSk1TVU1nUzBWWkxTMHRMUzBLVFVacmQwVjNXVWhMYjFwSmVtb3dRMEZSV1VsTGIxcEplbW93UkVGUlkwUlJaMEZGY0d0WWFITTNSWGxoY1V0V1VsbFdMMkZ3Y0RsVE4ybGtNRTFxZVFwaU5sTXZiMnhIWkhoeWJuSnVaakZ3VlU5eFVFbFRVVzlWYVZseE1WTjRURUpvVEVWaFp6aHJTSFV2WTA1dlpUQllXR2g0VURGdGRHcDNQVDBLTFMwdExTMUZUa1FnVUZWQ1RFbERJRXRGV1MwdExTMHRDZz09In19fX0=","integratedTime":1643917737,"logIndex":1,"logID":"4d2e4729bc008d76b4962364d19fe3f7a7b7bd58627bbafa0c19d9eac9797291"}}}}]
Go take an ice cream break if you got this far. You earned it!
Fulcio is a service for dynamically generating code-signing certificates based on authenticated OpenID Connect identities. This allows for key-less signing from authenticated service accounts or humans.
For this demo, we will use Fulcio to dynamically create code signing certificates based on GitHub identities.
cd $HOME/sigstore-local/src
git clone https://github.com/sigstore/fulcio.git
cd fulcio
go install .
SoftHSM implements a cryptographic store accessible through a PKCS #11 interface. You can use it to explore PKCS #11 without having a Hardware Security Module. For this demo, we will configure sigstore to reference tokens in $HOME/sigstore-local/tokens:
mkdir -p $HOME/sigstore-local/tokens
printf "\
directories.tokendir = $HOME/sigstore-local/tokens
log.level = DEBUG
" > $HOME/sigstore-local/softhsm2.conf
export SOFTHSM2_CONF=$HOME/sigstore-local/softhsm2.conf
Create your first HSM token:
softhsm2-util --init-token --slot 0 --label fulcio
For this tutorial, set both the SO (Security Officer) and User PIN to 2324
.
Configure OpenSC:
- (FreeBSD|OpenBSD|NetBSD):
export PKCS11_MOD=/usr/local/lib/softhsm/libsofthsm2.so
- (Arch|Debian|Ubuntu):
export PKCS11_MOD=/usr/lib/softhsm/libsofthsm2.so
- Fedora:
export PKCS11_MOD=/usr/lib64/libsofthsm2.so
- macOS:
export PKCS11_MOD=$(brew --prefix softhsm)/lib/softhsm/libsofthsm2.so
Create a configuration file for the pkcs11 crypto library using the user PIN code you specified in the previous step:
mkdir -p $HOME/sigstore-local/config
echo "{ \"Path\": \"$PKCS11_MOD\", \"TokenLabel\": \"fulcio\", \"Pin\": \"2324\" }" > $HOME/sigstore-local/config/crypto11.conf
Save your key into the HSM:
SOFTHSM2_CONF=$HOME/sigstore-local/softhsm2.conf pkcs11-tool \
--module=$PKCS11_MOD \
--login \
--login-type user \
--keypairgen \
--id 1 \
--label PKCS11CA \
--key-type EC:secp384r1
Create a CA root certificate:
cd $HOME/sigstore-local
SOFTHSM2_CONF=$HOME/sigstore-local/softhsm2.conf $HOME/go/bin/fulcio createca \
--org=acme \
--country=USA \
--locality=Anytown \
--province=AnyPlace \
--postal-code=ABCDEF \
--street-address="123 Main St" \
--hsm-caroot-id 1 \
--out ca-root.pem
The ct_server
is an RFC6962-compliant certificate transparency log that stores only the code-signing certificates issued by Fulcio.
go install github.com/google/certificate-transparency-go/trillian/ctfe/ct_server@latest
Next we need to setup a private key. I'll use 2324 again, but you can use anything 4-characters or longer:
export PASS=<password entered>
cd $HOME/sigstore-local
openssl ecparam -genkey -name prime256v1 -noout -out ct_unenc.key
openssl ec -aes256 -in ct_unenc.key -out ct_private.pem -passout pass:$PASS
openssl ec -aes256 -in ct_unenc.key -out ct_public.pem -pubout -passout pass:$PASS
rm ct_unenc.key
Look up the Trillian log ID we previously created, and set the LOG_ID variable to the resulting value:
cat $HOME/sigstore-local/trillian.log_id
export LOG_ID=<value of trillian.log_id>
Then populate the Certificate Transparency configuration file. It will fill in the password and log id:
printf "\
config {
log_id: $LOG_ID
prefix: \"sigstore\"
roots_pem_file: \"$HOME/sigstore-local/ca-root.pem\"
private_key: {
[type.googleapis.com/keyspb.PEMKeyFile] {
path: \"$HOME/sigstore-local/ct_private.pem\"
password: \"$PASS\"
}
}
}
" | tee $HOME/sigstore-local/ct.cfg
Start the certificate transparency server:
$HOME/go/bin/ct_server -logtostderr -log_config $HOME/sigstore-local/ct.cfg -log_rpc_server localhost:8091 -http_endpoint 0.0.0.0:6105
If successful, the output will look like this:
I0128 11:42:16.401794 65425 main.go:121] **** CT HTTP Server Starting ****
...
I0128 11:42:16.511090 65425 instance.go:85] Start internal get-sth operations on sigstore (8494167753837750461)
Dex is a federated OpenID Connect Provider, which connects OpenID identities from multiple providers to drive authentication for other apps. Dex will serve as your OIDC issuer. Unfortunately, Dex doesn't support go install
, so you need to build it manually:
cd $HOME/sigstore-local/src
git clone https://github.com/dexidp/dex.git
cd dex
gmake build || make build
cp bin/dex $HOME/go/bin
For this demonstration, we'll use GitHub to authenticate requests. Visit GitHub: Register a new OAuth Application, and fill in the form accordingly:
- Application Name:
My Local Sigstore Adventure
- Homepage URL, use
http://localhost/
- Authorization callback URL:
http://localhost:5556/callback
When you click Register Application, it will output a client ID. Save it to your environment:
export GITHUB_CLIENT_ID=<your id>
Click the Generate a new client secret
button, and copy the long alphanumeric string it emits into your environment:
export GITHUB_CLIENT_SECRET=<your client secret>
Create a Dex configuration that answers local OAuth requests, delegating the authentication to GitHub:
printf "\
issuer: http://localhost:5556
storage:
type: sqlite3
config:
file: ./dex.db
web:
http: 127.0.0.1:5556
frontend:
issuer: sigstore
theme: light
logger:
level: "debug"
format: "json"
oauth2:
responseTypes: [ "code" ]
skipApprovalScreen: false
alwaysShowLoginScreen: true
staticClients:
- id: sigstore
public: true
name: 'sigstore'
connectors:
- type: github
id: github-sigstore-test
name: GitHub
config:
clientID: $GITHUB_CLIENT_ID
clientSecret: $GITHUB_CLIENT_SECRET
redirectURI: http://localhost:5556/callback
" | tee $HOME/sigstore-local/dex-config.yaml
Start dex:
$HOME/go/bin/dex serve $HOME/sigstore-local/dex-config.yaml
Populate the Fulcio configuration file:
printf '
{
"OIDCIssuers": {
"http://localhost:5556": {
"IssuerURL": "http://localhost:5556",
"ClientID": "sigstore",
"Type": "email"
}
}
}
' > $HOME/sigstore-local/config/fulcio.json
Start Fulcio:
cd $HOME/sigstore-local
SOFTHSM2_CONF=$HOME/sigstore-local/softhsm2.conf $HOME/go/bin/fulcio serve \
--config-path=config/fulcio.json --ca=pkcs11ca --hsm-caroot-id=1 --ct-log-url=http://localhost:6105/sigstore \
--host=127.0.0.1 --port=5000 --metrics-port=2113
If it is working, you will see a message similar to:
2022-01-27T16:35:11.359-0800 INFO app/serve.go:173 127.0.0.1:5000
Now let's try some experimental cosign features: Integration with the Rekor transparency log and keyless signatures using the Fulcio CA. Fulcio will instead rely on a combination of certificates stored in SoftHSM and the OIDC tokens provided by Dex and Github:
NOTE: If you are running cosign on a non-local machine, wait 2 minutes for the Enter verification code
prompt, and then forward the Dex webserver port to your local workstation using ssh -L 5556:127.0.0.1:5556 <dex server>
. Then visit the URL and enter the resulting verification code into the terminal.
Sign the container with our local certificate:
SIGSTORE_CT_LOG_PUBLIC_KEY_FILE=$HOME/sigstore-local/ct_public.pem \
COSIGN_EXPERIMENTAL=1 $HOME/go/bin/cosign sign \
--oidc-issuer=http://localhost:5556 \
--fulcio-url=http://127.0.0.1:5000 \
--rekor-url=http://localhost:3000 \
localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9
Successful output will look like this:
**Warning** Using a non-standard public key for verifying SCT: /home/t/sigstore-local/ct_public.pem
Successfully verified SCT...
tlog entry created with index: 17
Pushing signature to: localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9
NOTE: If you get a NAME_UNKNOWN: Unknown name
error, re-run the ko publish
command in step 1.2.
Verify the certificate:
SIGSTORE_ROOT_FILE=$HOME/sigstore-local/ca-root.pem COSIGN_EXPERIMENTAL=1 \
$HOME/go/bin/cosign verify --rekor-url=http://localhost:3000 \
localhost:1338/demo/rekor-cli-e3df3bc7cfcbe584a2639931193267e9
Congratulations! You made it!
If you have rebooted and wish to bring the local sigstore stack up again, you can do so if you have checked out this repository and have tmux installed:
sh launch-sigstore.sh
After killing any daemons started:
sudo mysql -u root -e "DROP DATABASE IF EXISTS test;"
rm -Rf $HOME/sigstore-local
cosign:
NAME_UNKNOWN: Unknown name
: The package name you are trying to sign does not appear in the registry. Re-run theko publish
step and confirm you are signing the correct package name.
ct_server:
Failed to read config: failed to parse LogConfigSet from "/Users/t/sigstore-local/ct.cfg" as text protobuf (proto: (line 3:3): invalid value for int64 type: prefix)
: Confirm that thelog_id
field is set properly in$HOME/sigstore-local/ct.cfg
. It's probably empty!failed to load private key: pemfile: empty password for file
: Check the password field in$HOME/sigstore-local/ct.cfg
. It's probably empty!x509: decryption password incorrect
: The password in$HOME/sigstore-local/ct.cfg
doesn't match what's on disk. Re-run step 3.4.Server exited: listen tcp 0.0.0.0:6105: bind: address already in use
: There is already a ct_server process running. Find it usinglsof -i TCP:6105
, then kill it.Failed to retrieve STH for sigstore (30): rpc error: code = NotFound desc = tree ### not found
: Thelog_id
field in$HOME/sigstore-local/ct.cfg
does not match what is in the MySQL database. Re-run thecreatetree
command and place the resulting ID number inct.cfg
.
fulcio:
could not open config file: config/crypto11.conf
: Check that$HOME/sigstore-local
is your current working directory. If so, you missed part of 3.3.no key pair was found matching label "PKCS11CA"
: SoftHSM does not know about your CA certificate. Re-run the pkcs11 command in step 3.3.getting signer: getting key from Fulcio: retrieving cert:
: TBDlisten tcp :2112: bind: address already in use
: Another process is already using the default metrics publishing port. Try again with--metrics-port=2113
(or any other available port).
github:
404 Page Not Found
: The GitHub clientID or clientSecret field in$HOME/sigstore-local/dex-config.yaml
is incorrect. Run through step 3.5 again.
openssl:
unable to write private key | CRYPTO_internal:result too small
: You pressed enter at a prompt that requires a password containing at least 4 characters.
pkcs11-tool:
No slots.
: You may have forgotten to set a value for $PKCS11_MOD.Failed to load pkcs11 module
: The path specified in--module
is incorrect. Check the output offile $PKCS11_MOD
rekor-cli:
Post "http://localhost:3000/api/v1/log/entries": dial tcp [::1]:3000: connect: connection refused
: Rekor isn't running.createLogEntry default &{Code:500 Message:Unexpected result from transparency log}
: Check that you only have onerekor-server
process running. If so, try restarting it.
- Official Sigstore Documentation: Everything you ever wanted to know about Sigstore
- sigstore-scaffolding: Setup sigstore using Kubernetes
- Sigstore the Hard Way: Setup sigstore using Google Compute Engine
- Bring your own TUF: Setup your own TUF root of trust