Add carvel test to e2e

[absoludity]

Signed-off-by: Michael Nelson minelson@vmware.com

Description of the change

See #4093

Benefits

We can have CI checking that as a minimum a simple carvel install works.

Possible drawbacks

Longer CI? We should parallelize the helm and carvel tests.

Applicable issues

• fixes Run our e2e tests when using the Carvel plugin #4093

Additional information

Creating as draft until CI passes.
Currently failing on CI with:

Running 1 test using 1 worker

     [chromium] › tests/carvel/04-default-deployment.spec.js:8:1 › Deploys package with default values in main cluster
Logging in Kubeapps via OIDC in host: http://kubeapps-ci.kubeapps/
Logged into Kubeapps!
Creating release "test-04-release-42289"

Too long with no output (exceeded 10m0s): context deadline

even though locally it gets past this point.

Feel free to review. I'll try to get it passing tomorrow.

[castelblanque]

Too long with no output (exceeded 10m0s): context deadline exceeded

This is CircleCI killing the process as it has >10m without any output, as the message suggests.

Deployments timeout is set to >> Deployments timeout: 86 mins, >10 of course. So I would say that it is stuck waiting for the deployment to be ready at:

  await page.waitForSelector("css=.application-status-pie-chart-number >> text=1", {
    timeout: utils.getDeploymentTimeout(),
  });
  await page.waitForSelector("css=.application-status-pie-chart-title >> text=Ready", {
    timeout: utils.getDeploymentTimeout(),
  });

Could it be related to #4447 ? The pie chart not being refreshed.

A different issue would be to investigate why the deployment timeout is calculated as 86 mins 😄

[castelblanque]

Just for the record, if we ever want to increase the no-output timeout (I don't think it is worth for this particular issue), it can be done in CircleCI with no_output_timeout param.
See example here.

[absoludity]

Too long with no output (exceeded 10m0s): context deadline exceeded

This is CircleCI killing the process as it has >10m without any output, as the message suggests.

Yep, I realise, just unsure why it happens like that on CI but locally I see more output.

Deployments timeout is set to >> Deployments timeout: 86 mins, >10 of course. So I would say that it is stuck waiting for the deployment to be ready at:

  await page.waitForSelector("css=.application-status-pie-chart-number >> text=1", {
    timeout: utils.getDeploymentTimeout(),
  });
  await page.waitForSelector("css=.application-status-pie-chart-title >> text=Ready", {
    timeout: utils.getDeploymentTimeout(),
  });

Could it be related to #4447 ? The pie chart not being refreshed.

Again, locally I see the pie chart for this one (why I chose it - the issue with #4447 was that the resource refs only included CRDs, not deployments, in the packages I was trying, so we never watched for the resources, hence no pie chart).

A different issue would be to investigate why the deployment timeout is calculated as 86 mins smile

Heh, fun.

[absoludity]

Again, locally I see the pie chart for this one (why I chose it - the issue with #4447 was that the resource refs only included CRDs, not deployments, in the packages I was trying, so we never watched for the resources, hence no pie chart).

The issue was that the check was looking for the pie chart showing 1 pod ready. The carvel package I'm testing with produces 3 pods. It passed locally because one pod came up (and the page updated) before the others came up (and it updated again, but the test had already succeeded).

Let's see if it passes here now.

[absoludity]

sigh - still something different in the CI env than from locally where it passes :/

[absoludity]

Woo - increasing the no output timeout enabled a failure with an artifact, showing that the test is entering the release name, but then never moving on to select the service account:

Still keen to know why this can pass locally but not on CI, but checking service accounts etc.

[absoludity]

Hah - so locally I was running with a service account login, which meant that the namespace was defaulting to the kubeapps namespace (since the service account token was from there), where the service account kubeapps-operator was available. When running on CI, it ends up in the default namespace, where that service account is not present.

[absoludity]

Carvel test is now passing, just trying to ensure that we don't re-enable packaging.helm when upgrading to support operators, since it causes issues re-installing postgres with the password.

[absoludity]

Woo - and it's ready to land :)

[antgamdia]

Awesome!! It's great we finally have Carvel e2e support, thanks!

[antgamdia]

Mmm, perhaps we should pass this var also from the circleci config, like we already do for the OLM_VERSION ?

[absoludity]

Woops, yes. Forgot to update it. Done.

[castelblanque]

Carvel test is now passing

Cool!! Great to see it passes OK in CI.