notation verify command error messages

[yizha1]

This is not an exhausted list. The purpose is to improve the error messages for better UX.

Trust Policy

Example of a correct trustpolicy.json file

{
    "version": "1.0",
    "trustPolicies": [
        {
            "name": "wabbit-networks-images",
            "registryScopes": [ "localhost:5000/net-monitor" ],
            "signatureVerification": {
                "level" : "strict"
            },
            "trustStores": [ "ca:wabbit-networks.io" ],
            "trustedIdentities": [
                "x509.subject:CN=wabbit-networks.io,O=Notary,L=Seattle,ST=WA,C=US"
            ]
        }
    ]
}

Error cases

• Error#1: trustpolicy.json file non-existent/wrong name/wrong path

$ notation verify $IMAGE
Error: open /home/<user>/.config/notation/trustpolicy.json: no such file or directory

• Error#2: Invalid value for trust policy property registryScopes

# "registryScopes": [ "localhost:5000" ]

$ notation verify $IMAGE
Error: registry scope "localhost:5000" is not valid, make sure it is the fully qualified registry URL without the scheme/protocol. e.g domain.com/my/repository

• Error#3: The artifact to be verified is not stored in the repositories listed in trust policy property registryScopes

# "registryScopes": [ "localhost:5000/acme-rockets" ]

$ notation verify $IMAGE
Signature verification failed for all the signatures associated with digest: sha256:1d04c2f932abdcbb2f4024afa9a715636aa914354e470fa5a0629a556e371416
Error: artifact "localhost:5000/net-monitor@sha256:1d04c2f932abdcbb2f4024afa9a715636aa914354e470fa5a0629a556e371416" has no applicable trust policy

• Error#4: Invalid signatureVerification level

# "signatureVerification": {
#                "level" : "somelevel"
#            }
 
$ notation verify $IMAGE
Error: trust policy statement "wabbit-networks-images" uses invalid signatureVerification value "somelevel"

• Error#5: invalid type of trust store configured

#  "trustStores": [ "dca:wabbit-networks.io" ],

$ notation verify $IMAGE
Error: trust policy statement "wabbit-networks-images" uses an unsupported trust store type "dca" in trust store value "dca:wabbit-networks.io"

• Error#6: invalid name of trust store or non-existent trust store

# "trustStores": [ "ca:wrongname" ]

$ notation verify $IMAGE
Signature verification failed for all the signatures associated with digest: sha256:1d04c2f932abdcbb2f4024afa9a715636aa914354e470fa5a0629a556e371416
Error: signature verification failed

• Error#7: missing x509.subject: in trustedIdentities

#  "trustedIdentities": [
#                "CN=wabbit-networks.io,O=Notary,L=Seattle,ST=WA,C=US"
#            ]

$ notation verify $IMAGE
Error: trust policy statement "wabbit-networks-images" has trusted identity "CN=wabbit-networks.io,O=Notary,L=Seattle,ST=WA,C=US" missing separator

• Error#8: wrong separator in trustedIdentities, for example: wrongseparator

#  "trustedIdentities": [
#                "wrongseparator: CN=wabbit-networks.io,O=Notary,L=Seattle,ST=WA,C=US"
#            ]

$ notation verify $IMAGE
Signature verification failed for all the signatures associated with digest: sha256:1d04c2f932abdcbb2f4024afa9a715636aa914354e470fa5a0629a556e371416
Error: signature verification failed

• Error#9: wrong subject info

#  "trustedIdentities": [
#                "x509.subject: wrong-subject-info"
#            ]

$ notation verify $IMAGE
Signature verification failed for all the signatures associated with digest: sha256:1d04c2f932abdcbb2f4024afa9a715636aa914354e470fa5a0629a556e371416
Error: signature verification failed

Trust store

Error cases

• Error#10: The certificate is invalid

$ notation cert list
/home/<user>/.config/notation/truststore/x509/ca/wabbit-networks.io/invalid-certificate.pem
$ notation verify $IMAGE
Signature verification failed for all the signatures associated with digest: sha256:1d04c2f932abdcbb2f4024afa9a715636aa914354e470fa5a0629a556e371416
Error: signature verification failed

[yizha1]

Need to improve error message for Error#6, 8, 9, 10. There are errors during verify process that I cannot verify them all, need to consider how to capture and show the necessary errors. Otherwise, the troubleshooting cost will be too high. Let's check what we can do in rc.1 timeframe. We can improve it release by release.

@toddysm @vaninrao10 @priteshbandi ^^

[vaninrao10]

Yi are we planning to address all the 10 scenarios as part of rc.1 or just few of it ?

[JeyJeyGao]

After added debug log:

• Error#6: invalid name of trust store or non-existent trust store

# "trustStores": [ "ca:wrongname" ]

$ notation verify $IMAGE
ERRO[0002] error while loading the trust store, "/home/jj/.config/notation/truststore/x509/ca/wabbit-networks.ioc" does not exist 
Signature verification failed for all the signatures associated with digest: sha256:cd5eef6b6a6750c9850a8d7b1a5435f35f1a1808d66c74e265f6b7ec290bea47
Error: signature verification failed

• Error#8: wrong separator in trustedIdentities, for example: wrongseparator

#  "trustedIdentities": [
#                "wrongseparator: CN=wabbit-networks.io,O=Notary,L=Seattle,ST=WA,C=US"
#            ]

$ notation verify $IMAGE
ERRO[0000] no x509 trusted identities are configured in the trust policy "wabbit-networks-images" 
Signature verification failed for all the signatures associated with digest: sha256:cd5eef6b6a6750c9850a8d7b1a5435f35f1a1808d66c74e265f6b7ec290bea47
Error: signature verification failed

• Error#9: wrong subject info

#  "trustedIdentities": [
#                "x509.subject: wrong-subject-info"
#            ]

$ notation verify $IMAGE
Error: distinguished name (DN) " wrong-subject-info" is not valid, it must contain 'C', 'ST', and 'O' RDN attributes at a minimum, and follow RFC 4514 standard

• Error#10: The certificate is invalid

$ notation cert list
/home/<user>/.config/notation/truststore/x509/ca/wabbit-networks.io/invalid-certificate.pem
$ notation verify $IMAGE
ERRO[0002] error while loading the trust store, error while reading certificates from "/home/jj/.config/notation/truststore/x509/ca/wabbit-networks.io/wabbit-networks.io.crt": x509: malformed certificate 
Signature verification failed for all the signatures associated with digest: sha256:cd5eef6b6a6750c9850a8d7b1a5435f35f1a1808d66c74e265f6b7ec290bea47
Error: signature verification failed

[yizha1]

It looks good. I have additional questions but not to your updates.
The following two output messages seem duplicated.

Signature verification failed for all the signatures associated with digest: sha256:cd5eef6b6a6750c9850a8d7b1a5435f35f1a1808d66c74e265f6b7ec290bea47
Error: signature verification failed