Delegate to targets/releases per extended MVP #80

trishankatdatadog · 2020-06-09T20:56:58Z

Signed-off-by: Trishank Karthik Kuppusamy trishank.kuppusamy@datadoghq.com

Signed-off-by: Trishank Karthik Kuppusamy <trishank.kuppusamy@datadoghq.com>

trishankatdatadog · 2020-06-17T21:03:25Z

Paging @radu-matei to take a look and help 🙂

radu-matei · 2020-06-29T19:17:05Z

We have two failing tests for now, which I am currently investigating:

FAIL	github.com/cnabio/signy/pkg/docker	13.762s
--- FAIL: TestVerify (0.00s)
    os_test.go:15: 
        	Error Trace:	os_test.go:15
        	Error:      	Received unexpected error:
        	            	invalid metadata found: invalid scheme for key '': should be 'rsassa-pss-sha256', got: ''
        	Test:       	TestVerify
--- FAIL: TestValidate (0.00s)
    os_test.go:25: 
        	Error Trace:	os_test.go:25
        	Error:      	Received unexpected error:
        	            	cannot load layout from file file ../../testdata/intoto/root.layout: open ../../testdata/intoto/root.layout: no such file or directory
        	Test:       	TestValidate
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xdaa04a]

I assume this is happening because in-toto tools are not installed by default in GitHub Actions workers.

trishankatdatadog · 2020-06-29T19:20:26Z

We have two failing tests for now, which I am currently investigating:

FAIL	github.com/cnabio/signy/pkg/docker	13.762s
--- FAIL: TestVerify (0.00s)
    os_test.go:15: 
        	Error Trace:	os_test.go:15
        	Error:      	Received unexpected error:
        	            	invalid metadata found: invalid scheme for key '': should be 'rsassa-pss-sha256', got: ''
        	Test:       	TestVerify
--- FAIL: TestValidate (0.00s)
    os_test.go:25: 
        	Error Trace:	os_test.go:25
        	Error:      	Received unexpected error:
        	            	cannot load layout from file file ../../testdata/intoto/root.layout: open ../../testdata/intoto/root.layout: no such file or directory
        	Test:       	TestValidate
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xdaa04a]

I assume this is happening because in-toto tools are not installed by default in GitHub Actions workers.

I think the second error is because the file was renamed and moved to the minimal root layout example

radu-matei · 2020-06-29T19:37:24Z

Ah, you're right.
Changing the directory to the correct one:

--- FAIL: TestVerify (0.00s)
    os_test.go:15: 
        	Error Trace:	os_test.go:15
        	Error:      	Received unexpected error:
        	            	failed verification: No signature found for key '556caebdc0877eed53d419b60eddb1e57fa773e4e31d70698b588f3e9cc48b35'
        	Test:       	TestVerify

This commit updates the minimal in-toto root layout, removes an unused signature, as well as updates the directory structure, putting the links at the same level as the layout and public key. (the directory structure should be updated in the future, but for now, getting the tests to pass should suffice.) Finally, it updates `docker_test.go` and `os_test.go` accordingly. Signed-off-by: Radu M <root@radu.sh>

radu-matei · 2020-06-29T23:04:30Z

Thanks for the help, @adityasaky and @trishankatdatadog!
Now both verification tests are passing.

cmd/sign.go

pkg/intoto/metadata.go

radu-matei · 2020-06-30T05:15:33Z

pkg/intoto/metadata.go

+}
+
+// WriteMetadataFiles writes the content of a metadata object into files in a directory
+// TODO


Is this supposed to be used?

Yes, this is one of the features I need help with: collecting the root layout, its pubkeys, all the links, and writing them to this directory for verification...

Let me know if I can help with this. If nothing else, I can listen and help work through what we want this function to do.

radu-matei · 2020-06-30T05:23:11Z

pkg/tuf/common.go

@@ -11,14 +11,15 @@ import (
 	"path/filepath"


Re - the comment at the top of this file.
The comment was relevant when the file contained transport-related code, which was indeed adapted from the Notary repo.

That is not the case anymore, so we should update the comments accordingly.

It might be true elsewhere, so how to best resolve this overall?

pkg/tuf/common_test.go

pkg/tuf/keys.go

pkg/tuf/sign.go

radu-matei · 2020-06-30T05:39:45Z

pkg/tuf/verify.go

 )

+// GetTargetAndSHA returns the target with roles and the SHA256 of the target file


nit for the future: when moving bits of code around, it's pretty difficult to understand whether this was just moved, or something else also changed.

Yeah, sorry about that, I felt guilty about it, too...

Signed-off-by: Trishank Karthik Kuppusamy <trishank.kuppusamy@datadoghq.com>

radu-matei · 2020-07-29T06:01:42Z

pkg/intoto/metadata.go

 	}
-
-	return canonicaljson.RawMessage(raw), nil
+	return bytes, nil


So this function returns ioutil.ReadFile, right?
It might be easier to just use it directly.

💯 yes, please go for it, sorry missed it after refactoring it

Agreed, Radu's simplification seems like a good change.

carolynvs

Reviewing just the first 1/4 of the files. I'm still working my way through this as I learn, sorry!

carolynvs · 2021-02-02T16:11:50Z

pkg/docker/docker.go

+	if err != nil {
+		return "", err
+	}
+	repo, err := registry.ParseRepositoryInfo(r)


I don't understand what this second function call does for you? It seems that you can return r.Name directly and the test still passes.

carolynvs · 2021-02-02T16:19:49Z

pkg/intoto/metadata.go

-	Key    []byte            `json:"key"`
-	Layout []byte            `json:"layout"`
-	Links  map[string][]byte `json:"links"`
+	Data       []byte   `json:"data"`


This struct is a bit confusing to understand what it actually represents. Seems like it sometimes populates Data with the contents of either a layout, link or key but you can't tell which is stored based on the struct fields. Then in some cases either public key or link is populated as well.

I am wondering if it really makes sense to reuse in all of these cases? I am brand new to this so I'm really just trying to understand, and am not saying that it needs to be changed (other than the documentation perhaps to clarify).

carolynvs · 2021-02-02T16:23:18Z

pkg/intoto/metadata.go

 	if err != nil {
-		return err
+		return nil, fmt.Errorf("cannot encode custom metadata into canonical json: %v", err)


suggestion: this error message will get bubbled up into CNAB tooling eventually, so it would help to say "in-toto custom metadata" so that it's more clear where things went wrong. Otherwise custom metadata means something completely different in cnab (the custom section in the bundle.json).

carolynvs · 2021-02-02T16:27:38Z

pkg/intoto/metadata.go

+	for _, fileinfo := range fileinfos {
+		filename := fileinfo.Name()
+		if !strings.Contains(filename, ".link") {
+			return nil, fmt.Errorf("%s does not have a .link suffix", filename)


Do we have to assume that every file in the directory must be a link? I could see someone wanting to have another file, such as a readme that may provide additional documentation or info about the files in the directory. Does it make sense to instead skip files that aren't links?

carolynvs · 2021-02-02T16:29:31Z

pkg/intoto/metadata.go

+}
+
+// WriteMetadataFiles writes the content of a metadata object into files in a directory
+// TODO


Let me know if I can help with this. If nothing else, I can listen and help work through what we want this function to do.

carolynvs · 2021-02-02T16:44:18Z

pkg/tuf/delegations.go

+	// the public keys used to verify the delegatee
+	publicKeys := []data.PublicKey{publicKey}
+	// the target paths entrusted to the delegatee
+	paths := make([]string, 2)


This creates an array with 2 indexes allocated, and then you append two more, resulting in an array of length 4. I assume you meant this instead

Suggested change

paths := make([]string, 2)

paths := make([]string, 0, 2)

carolynvs · 2021-02-02T16:44:42Z

pkg/tuf/delegations.go

+
+// Delegate all paths ("*") to targets/releases.
+// https://github.com/theupdateframework/notary/blob/f255ae779066dc28ae4aee196061e58bb38a2b49/cmd/notary/delegations.go
+func delegateToReleases(repo client.Repository, publicKey data.PublicKey) error {


Does this function have a unit test?

carolynvs · 2021-02-02T16:47:36Z

pkg/tuf/keys.go

+	switch len(keyList) {
+	case 0:
+		log.Debugf("No %s key available, need to make one", releasesRoleName)
+		return cryptoService.Create(releasesRoleName, r.GetGUN(), data.ECDSAKey)


Is this creating a new key if one wasn't found? Is it persisted somewhere? Some more comments about what is happening here, and if it should ever be okay to happen outside of tests would be really helpful.

general suggestion: When I call functions that I don't control, I like to wrap the error using github.com/pkg/errors.Wrap(err, "message with more context") so that when the error is bubbled up, you can tell where the code failed. Usually error messages from stdlib and other libraries don't have enough info to tell what the task was that failed, what file was being used, etc. For example, you could get an error like "invalid repository path" or something like that and have no way to trace it back to this line, to signy vs the calling CNAB tool, etc.

carolynvs · 2021-02-02T16:54:32Z

pkg/tuf/keys.go

 	case 1:
 		log.Debug("Nothing to do, only one targets key available")
+		return nil
 	case 2:
 		// First, we publish current changes to repository in order to list roles.
 		// FIXME: Find a find better way to list roles w/o publishing changes first.


You may want to create an issue to track this, so it's not lost. Seems like publishing first is pretty unideal. Have you asked Notary if they would be willing to accept a PR to make this easier?

carolynvs · 2021-02-02T16:59:22Z

pkg/tuf/keys.go

-		err = r.RotateKey(data.CanonicalTargetsRole, false, []string{thatTargetsKeyID})
+		log.Debugf("That targets keyID: %s", thatKeyID)
+		log.Debugf("Before rotating targets key from %s to %s", thisKeyID, thatKeyID)
+		err = r.RotateKey(data.CanonicalTargetsRole, false, []string{thatKeyID})


So this case statement and line here are assuming that the current key (thiskeyid) was created automagially and really thatKeyId is the one to reuse? It's not clear if there are other situations where you can have two target keys or if it's only caused by notary's automagic.

Also if this function is unit or integration testable, this seems complicated enough to warrant it.

trishankatdatadog · 2021-02-24T17:28:41Z

Thanks for all your great comments, @carolynvs! So sorry I haven't gotten around to this yet, but been swamped with life and work. I can't promise I'll get to it this week, but please keep prodding me every two weeks as you see fit...

trishankatdatadog mentioned this pull request Jun 9, 2020

Review CNAB-Sec cnabio/cnab-spec#369

Closed

5 tasks

trishankatdatadog force-pushed the trishankatdatadog/delegate-targets branch from 04028e5 to ed58458 Compare June 10, 2020 03:09

delegate all paths to targets/releases

2313823

Signed-off-by: Trishank Karthik Kuppusamy <trishank.kuppusamy@datadoghq.com>

trishankatdatadog force-pushed the trishankatdatadog/delegate-targets branch from ed58458 to 2313823 Compare June 10, 2020 03:13

trishankatdatadog added 2 commits June 12, 2020 17:13

sign and verify bundles as delegated targets

9dd82d0

Signed-off-by: Trishank Karthik Kuppusamy <trishank.kuppusamy@datadoghq.com>

signing in-toto metadata ala MVP

5616526

Signed-off-by: Trishank Karthik Kuppusamy <trishank.kuppusamy@datadoghq.com>

trishankatdatadog requested a review from radu-matei June 17, 2020 21:03

adityasaky mentioned this pull request Jun 29, 2020

Require fully conformant metadata when loading in-toto/in-toto#186

Open