Document what happens to a VCH when content trust is enabled on a project

[stuclem]

In #1651, @hickeng mentioned that when content trust is enabled on a project, some modification is made to the VCHs in that project.

We do not currently document this, and can find no information about it beyond what @hickeng said in #1651.

@reasonerjt can you please provide some info about what content trust does to a VCH? Thanks!

[stuclem]

See also vmware/vic#6778, which is the initial issue for implementing content trust in VCHs.

[stuclem]

See also the manual testing instructions, Test 2-06 Trusted Content.

[stuclem]

Setup:

1. Deploy VCH with --registry-ca so that it can connect to VIC registry.
2. In Management Portal, add the VCH to a project (not necessarily the default project).
3. Enable content trust on that project.

How this affects the VCH:

• VCH is automatically put into whitelist mode, and only the VIC Registry instance from the project is included in the whitelist.
• VCH can now only pull signed images from that registry.

Questions:

• What happens if whitelist mode is already enabled on the VCH before it is added to the project, and the VCH's whitelist includes other registries? Is the VCH's whitelist overridden?
• What happens if a project includes more than one registry, for example other VIC Registry/Harbor instances? Are all of the project's registries added to the VCH's whitelist?
• What happens if you add a VCH to two projects (currently possible and supported), one of which has content trust enabled and one that doesn't?
• Does it make a difference if you enable CT on the project before or after you add a VCH?

[stuclem]

I did a bit of testing and came up with the following answers to my questions above:

• What happens if whitelist mode is already enabled on the VCH before it is added to the project, and the VCH's whitelist includes other registries? Is the VCH's whitelist overridden?
  • Yes, the VCH's whitelist is overridden. If you subsequently disable CT on the project, the previous whitelist configuration is restored.
• What happens if a project includes more than one registry, for example other VIC Registry/Harbor instances? Are all of the project's registries added to the VCH's whitelist?
  • No, only the default registry instance for the project is added to the VCH's whitelist.
• What happens if you add a VCH to two projects (currently possible and supported), one of which has content trust enabled and one that doesn't?
  • Only the settings from the first project that you register the VCH with are applied. For example:
    • If you add a VCH to project_1 and enable CT, the whitelist is updated.
    • If you also add that VCH to project_2 and enable/disable CT, the whitelist config remains the same as before.
    • If you disable CT on project_1 and then enable CT on project_2, the whitelist config remains as it was when you disabled CT on project_1. Namely, the whitelist on the VCH is either disabled or reverts to the whitelist config with which the VCH was deployed.
    • If you remove the VCH from project_1 and then enable/disable CT on project_2, the settings that you make in project_2 are applied.
• Does it make a difference if you enable CT on the project before or after you add a VCH?
  • No, the VCH's whitelist is updated regardless of whether or not it was already in the project when CT is enabled.

We already document that adding the same VCH to projects in different instances of Management Portal is not supported. We should probably note that nothing prevents you from doing this, though.

@hickeng and @lazarin does the above reflect what you would expect?

[martin-borisov]

Stuart, thanks for looking into this in detail. Honestly, we don't have such a scenario in our tests and we have not really validated it, but I will ask @isyarov to go over your steps and try to reproduce them.

[stuclem]

This also applies to 1.3.

[stuclem]

This will require updates in the following sections:

• Whitelist Registry Mode in Configure Registry Access, in the vSphere Admin doc.
• Configure Project Settings in the Cloud admin doc.
• Using vSphere Integrated Containers Registry with Content Trust in Configure the Docker Client for Use with vSphere Integrated Containers in the developer doc.

[isyarov]

Hi Stuart,
Once a VCH is deployed, it's whitelist preferences cannot be modified. When a VCH is deployed in whitelist mode, this restricts the VCH to be able to pull images only from those registries, which are whitelisted. This list is passed as a parameter when deploying the VCH and it cannot be modified after that.
Content trust has nothing to do with the VCH whitelist. CT is enforced by harbor and if enabled, harbor will decline all pull requests for images that are not signed.
And admiral whitelisting is a third thing. It is enforced only when provisioning from admiral. If you provision an image which is whitelisted in the VCH, but not in admiral it would fail from the admin portal, but will succeed from the CLI.

1. So for your first question: No, the VCH whitelist is not overridden, just the registry declines the pull request if CT is turned on.
2. For the second question: No, the VCH whitelist is specified when deploying it and no modifications are made/can be made afterwards
3. The third question: Here the situation is a bit confusing. I don't think that what you are describing relates to the fact that the VCH is added to two projects.
   Looking through your steps i would guess that you are trying to provision in project_2 an image that is associated with project_1 and that's where the content trust restriction kicks in.
   I played around a bit too and came to the conclusion that CT is enforced not per project, bur per images, associated with that project. Let me try to be more specific with an example.

• We have two projects: project_1 and project_2
• In project_1 we have an image called image_1 and respectively in project_2 - image_2
• We add the same VCH to both projects(unrelated, to my opinion)
• In project_1 we enable content trust and try to provision image_1 from project_1 -> this fails because of CT enforcement
• In project_2 we provision image_1 -> this fails because of CT enforcement (on the image associated with project_1)
• In project_2 we provision the image image_2 -> this succeeds because content trust is disabled in project_2

4. The fourth question: refer to the answers of the first two questions

I hope that i was able to answer your questions and hope that the info is not misleading/incorrect. Maybe @hickeng can correct me if that's the case.

[stuclem]

Hi @isyarov, apologies for not replying sooner, but I only just noticed your post. Maybe I am misunderstanding something, but it really looks to me like enabling/disabling CT on a project does modify the VCHs that are in that project. Here are my test steps:

1. Deploy a VCH, vch-CT-test, with whitelist mode turned off, and passing it the certificate for my Registry instance. I use the vSphere Client UI, but here's the generated vic-machine command:

vic-machine-windows create --name vch-CT-test --compute-resource /VSAN-DC/host/VSAN-Cluster --image-store Datastore4 --base-image-size 8GB --volume-store Datastore4:default --bridge-network vic-bridge --bridge-network-range 172.16.0.0/12 --public-network vic-public --tls-cname vch-CT-test --certificate-key-size 2048 --no-tlsverify --user Administrator@VSPHERE.LOCAL --thumbprint 37:B3:E6:5D:EB:AE:6D:E4:25:30:88:39:E2:C3:63:58:C8:8D:5D:1C --target sc-rdops-vm02-dhcp-52-37.eng.vmware.com/VSAN-DC --registry-ca ca.crt --ops-user vic-ops@vsphere.local --ops-grant-perms

2. Run docker info on vch-CT-test. You see that whitelist mode is not enabled:

3. Pull an image from Docker Hub with no problem.
4. Create a project, ct-test-project, in Management Portal and add the VCH to that project.
5. Enable content trust on ct-test-project.
6. Run docker info on vch-CT-test. Now whitelist mode is enabled, and my Registry is listed:

7. Attempts to pull an image from Docker Hub fail:

8. Deploy another VCH, vch-CT-test-2, this time with whitelist mode enabled and with two Registry instances in the whitelist:

vic-machine-windows create --name vch-CT-test-2 --compute-resource /VSAN-DC/host/VSAN-Cluster --image-store Datastore4 --base-image-size 8GB --volume-store Datastore4:default --bridge-network vic-bridge --bridge-network-range 172.16.0.0/12 --public-network vic-public --tls-cname vch-CT-test-2 --certificate-key-size 2048 --no-tlsverify --user Administrator@VSPHERE.LOCAL --thumbprint 37:B3:E6:5D:EB:AE:6D:E4:25:30:88:39:E2:C3:63:58:C8:8D:5D:1C --target sc-rdops-vm02-dhcp-52-37.eng.vmware.com/VSAN-DC --whitelist-registry 10.160.54.109 --whitelist-registry 10.153.199.171 --registry-ca ca4.crt --registry-ca ca5.crt --ops-user vic-ops@vsphere.local --ops-grant-perms

9. Run docker info on vch-CT-test-2. Whitelist mode is enabled and two registries are listed:

10. Add vch-CT-test-2 to ct-test-project (that has CT enabled).
11. Run docker info on vch-CT-test-2. Whitelist mode is enabled, but now only the Registry instance that includes the project is listed:

This is based on the tests described in https://github.com/vmware/vic-product/blob/master/tests/manual-test-cases/Group2-OVA-Features/2-06-Trusted-Content.md, which test the implementation of vmware/vic#6778. So, this is all expected behaviour. But, it does mean that adding a VCH to project with CT enabled does modify the VCH. Which brings me back to the question that I raised in vmware-archive/admiral#264, about whether we should recommend against adding a VCH to more than one project.

[stuclem]

I forgot to add the last step. If you disable CT on the project and run docker info on the VCHs again, the configurations with which they were deployed are restored:

[stuclem]

Some further steps, continuing the example above from where it left off. To recap, we have:

• A VCH named vch-CT-test, that was deployed with whitelist mode disabled.
• A project named ct-test-project, that includes vch-CT-test, and currently has content trust turned off.

Let's try adding vch-CT-test to another project, while it's still in ct-test-project.

1. Create a new project, ct-test-project-2.
2. Add vch-CT-test to ct-test-project-2. It is still also in ct-test-project.
3. Enable content trust on ct-test-project-2 but leave CT disabled on ct-test-project.
4. In ct-test-project-2, attempt to provision a container from Docker Hub. It fails because vch-CT-test is in whitelist mode. This is normal, because CT is enabled on ct-test-project-2:

5. Now attempt to provision a container from Docker Hub in ct-test-project, which has CT disabled. It also fails because whitelist mode is enabled.

This is not normal, because ct-test-project is not enforcing content trust and the VCH was deployed without a whitelist. So, users of ct-test-project should expect to be able to deploy images from Docker Hub. However, because CT is enabled on ct-test-project-2, the VCH would appear to have been put into whitelist mode for all projects.

This is why I am asking in vmware-archive/admiral#264 whether we should discourage using the same VCH in more than one project.

[isyarov]

Hi Stuart,
I apologize for the apparently incorrect information. It was based on https://vmware.github.io/vic-product/assets/files/html/1.4/vic_vsphere_admin/vch_registry.html#whitelist-registry where it is clearly stated that:

 You cannot modify a VCH whitelist after the initial deployment of the VCH.

I was able to reproduce what you've described and it seems that you're right and in fact, yes the VCH whitelist settings seem to change. I do not know how/when/what changes this settings. This too should probably be documented as it seems that the above statement is not true. And i can't get what is the connection between content trust and whitelisting. I thought that CT is to do with signed images push/pull enforcement whereas whitelisting is to do with registry enforcement, apparently that's not the case.
Again apologies for the incorrect info, it seems someone else has to shed some light on the matter.

[stuclem]

Thanks @isyarov. @hickeng @zjs @reasonerjt can you please confirm that the behaviour as described above is correct?

And to come back to my question from vmware-archive/admiral#264, should we recommend against using a VCH in more than one project?