Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Push Factory Image" failure checking cypress/factory:4.0.1 #1093

Closed
MikeMcC399 opened this issue Jun 4, 2024 · 9 comments
Closed

"Push Factory Image" failure checking cypress/factory:4.0.1 #1093

MikeMcC399 opened this issue Jun 4, 2024 · 9 comments
Assignees
@MikeMcC399
Labels
bug

Comments

@MikeMcC399
Copy link
Collaborator

Issue

The CircleCI workflow job "Push Factory Image" is failing to get the status of cypress/factory:4.0.1 from Docker Hub cypress/factory.

The error is

message: 'unsupported schema version 2 in cypress/factory:4.0.1 manifest',

Examining the manifest of cypress/factory:4.0.1 with the following command shows that it differs in format from the manifest for cypress/factory:4.0.0 (and from cypress/base, cypress/browsers and cypress/included)

docker manifest inspect cypress/factory:4.0.1

shows

   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",

instead of

   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",

Logs

https://app.circleci.com/pipelines/github/cypress-io/cypress-docker-images/1959/workflows/3c12d584-8378-4f1d-8722-bcc504b21e2d/jobs/59620 from PR

shows

npm WARN exec The following package was not found and will be installed: docker-image-not-found@1.1.1
npm WARN deprecated inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
npm WARN deprecated rimraf@2.4.5: Rimraf versions prior to v4 are no longer supported
npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated glob@6.0.4: Glob versions prior to v9 are no longer supported
npm WARN deprecated debug@4.1.1: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
npm WARN deprecated tough-cookie@2.0.0: ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130
got an error fetching info about Docker image docker.io/cypress/factory:4.0.1
got an error other than image not found
RestError [InvalidContentError]: unsupported schema version 2 in cypress/factory:4.0.1 manifest
    at _afterCall (/home/circleci/.npm/_npx/299ed24e1d89a4d2/node_modules/docker-registry-client/lib/registry-client-v2.js:1469:24)
    at IncomingMessage.finish (/home/circleci/.npm/_npx/299ed24e1d89a4d2/node_modules/docker-registry-client/lib/docker-json-client.js:171:13)
    at Object.onceWrapper (node:events:632:28)
    at IncomingMessage.emit (node:events:530:35)
    at endReadableNT (node:internal/streams/readable:1696:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  jse_shortmsg: 'unsupported schema version 2 in cypress/factory:4.0.1 manifest',
  jse_info: {},
  message: 'unsupported schema version 2 in cypress/factory:4.0.1 manifest',
  [stack]: 'InvalidContentError: unsupported schema version 2 in cypress/factory:4.0.1 manifest\n' +
    '    at _afterCall (/home/circleci/.npm/_npx/299ed24e1d89a4d2/node_modules/docker-registry-client/lib/registry-client-v2.js:1469:24)\n' +
    '    at IncomingMessage.finish (/home/circleci/.npm/_npx/299ed24e1d89a4d2/node_modules/docker-registry-client/lib/docker-json-client.js:171:13)\n' +
    '    at Object.onceWrapper (node:events:632:28)\n' +
    '    at IncomingMessage.emit (node:events:530:35)\n' +
    '    at endReadableNT (node:internal/streams/readable:1696:12)\n' +
    '    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)',
  body: {
    code: 'InvalidContent',
    message: 'unsupported schema version 2 in cypress/factory:4.0.1 manifest'
  },
  restCode: 'InvalidContent',
  cause: <ref *1> [Function: we_cause] {
    [length]: 1,
    [name]: 'we_cause',
    [arguments]: null,
    [caller]: null,
    [prototype]: { [constructor]: [Circular *1] }
  },
  name: 'InvalidContentError',
  statusCode: 400
}
exiting with code 1
Docker hub has image cypress/factory:4.0.1 or not responding
We should stop in this case

Analysis

Manifest

cypress/factory:4.0.1

$ docker manifest inspect cypress/factory:4.0.1
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 865,
         "digest": "sha256:bbf8a2e6fda64adb6c87e9793493240667e9e3a43874e71563f5963076942b85",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 865,
         "digest": "sha256:7cfe1c11128c9e4a119b7ad3a6bb4ce13a705cc8f00436f4174a964e102fe4fd",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:936a49c206ffcdfeb0faed320876c164b671f862cd97e815a8cfa9ba3c61eff6",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:fdb5a0baf2cb8b69d44a4d435b0ff6fdd1b6877de919cf0a74bc4aae0c93a4e1",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      }
   ]
}

cypress/factory:4.0.0

$ docker manifest inspect cypress/factory:4.0.0
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 899,
         "digest": "sha256:7a9498698dc9ceacb1a3e93cc602436f6e135c734a701681ef0fe7304e2cc601",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 899,
         "digest": "sha256:abbafb58e45003f37c1e2a0466c36af8707917a7b682eed9546ed928b2ceeec3",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]
}

Log Push Factory Image for 4.0.1

https://app.circleci.com/pipelines/github/cypress-io/cypress-docker-images/1944/workflows/d32dcb15-f148-4cf7-a4d1-3b1f48ab6141/jobs/59206

Check for existence

npm WARN exec The following package was not found and will be installed: docker-image-not-found@1.1.1
npm WARN deprecated inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
npm WARN deprecated rimraf@2.4.5: Rimraf versions prior to v4 are no longer supported
npm WARN deprecated glob@6.0.4: Glob versions prior to v9 are no longer supported
npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated debug@4.1.1: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
npm WARN deprecated tough-cookie@2.0.0: ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130
got an error fetching info about Docker image docker.io/cypress/factory:4.0.1
Got definite NotFoundError
exiting with code 0
Docker hub says image cypress/factory:4.0.1 does not exist

building Docker image for target factory

#14 exporting to image
#14 exporting manifest sha256:265f8134ce492f9219dd8c71ff531f33fa38e2e4254873e981ef94bba7427103 0.0s done
#14 exporting config sha256:a8eb011e2c5ca149e4f573a8b902f10596a378dbcebc94eb67ea1a95a21911fa 0.0s done
#14 exporting manifest sha256:206ac86fb72641f54d01c8068bafa8e216169cd4ce433aa408de646176e885eb 0.0s done
#14 exporting config sha256:de63602eb3ffcd58ce5dbde4fa501245b01748e3e17b91a78406557327465997 0.0s done
#14 exporting manifest list sha256:d287e6a9570956dbd6f70efb871bf86db5aebb94bb331dcf47ddb65d5fba205b 0.0s done
#14 pushing layers
#14 pushing layers 5.4s done
#14 pushing manifest for docker.io/cypress/factory:latest@sha256:d287e6a9570956dbd6f70efb871bf86db5aebb94bb331dcf47ddb65d5fba205b
#14 pushing manifest for docker.io/cypress/factory:latest@sha256:d287e6a9570956dbd6f70efb871bf86db5aebb94bb331dcf47ddb65d5fba205b 0.5s done
#14 pushing layers 0.2s done
#14 pushing manifest for docker.io/cypress/factory:4.0.1@sha256:d287e6a9570956dbd6f70efb871bf86db5aebb94bb331dcf47ddb65d5fba205b
#14 pushing manifest for docker.io/cypress/factory:4.0.1@sha256:d287e6a9570956dbd6f70efb871bf86db5aebb94bb331dcf47ddb65d5fba205b 0.3s done
#14 DONE 25.3s
@MikeMcC399 MikeMcC399 added the bug label Jun 4, 2024
@MikeMcC399
Copy link
Collaborator Author

@jennifer-shehane

I don't understand why this issue has occurred and it is in an area where I have no experience! I can't tell if was a CircleCI glitch or something else.

If it is not addressed it will prevent any new cypress/factory images from being published.

Depending on the root cause it may also affect the automatic publication of cypress/base, cypress/browsers and cypress/included images.

@MikeMcC399
Copy link
Collaborator Author

MikeMcC399 commented Jun 4, 2024 •
edited
Loading

There is a possibility that the image cypress/factory:4.0.1 is corrupted through 3 parallel-running PRs which each checked if there was an image cypress/factory:4.0.1 present, found there was not, and then each built one and published it:

The 2nd of the PRs #1079 also effectively bumped the Docker client version from Docker 20.10.18 to 26.0.2. This might have all been too much in one go!

Perhaps cypress/factory:4.0.1 should be manually deleted so that it can be recreated?

@MikeMcC399 MikeMcC399 self-assigned this Jun 4, 2024
@MikeMcC399
Copy link
Collaborator Author

It seems there were some breaking changes in https://github.com/docker/buildx

@MikeMcC399 MikeMcC399 mentioned this issue Jun 4, 2024
Merged
@MikeMcC399
Copy link
Collaborator Author

After this is merged it will require a manual delete of the incompatible cypress/factory:4.0.1 image

@jennifer-shehane
Copy link
Member

K, let me know if that works and if the deletion is required.

@MikeMcC399
Copy link
Collaborator Author

@jennifer-shehane

Sorry that a revert was necessary, but at least only one image was caught up in this. I only noticed by accident when I was looking into another failure, as the workflow itself does not return an error. It's only visible when you examine the logs manually.

Status

https://app.circleci.com/pipelines/github/cypress-io/cypress-docker-images/1967/workflows/63050ae3-636c-46a2-a570-c5a14187d2e1/jobs/59867/parallel-runs/0/steps/0-103?invite=true#step-103-1209_95

RestError [InvalidContentError]: unsupported schema version 2 in cypress/factory:4.0.1 manifest

So, yes please, deletion of cypress/factory:4.0.1 on Docker Hub is needed. I don't think it checks on AWS, so I'm guessing that it will just overwrite there.

@jennifer-shehane
Copy link
Member

I deleted the 4.0.1 tag in cypress/factory. Hopefully that worked.

@MikeMcC399
Copy link
Collaborator Author

@jennifer-shehane

I deleted the 4.0.1 tag in cypress/factory. Hopefully that worked.

Thank you! That should be good, and the next merge to master will test that out.

In the meantime I checked by hand and docker-image-not-found is happy again! 😃

$ docker-image-not-found --repo cypress/factory:4.0.1
got an error fetching info about Docker image docker.io/cypress/factory:4.0.1
Got definite NotFoundError
exiting with code 0

It does however still fail when no tag is defined, which defaults to latest, and latest is still linked to the problematic manifest. I expect that will get overwritten at the next release. We'll see!

@MikeMcC399
Copy link
Collaborator Author

MikeMcC399 commented Jun 4, 2024 •
edited
Loading

Closing as resolved. I will be monitoring what happens next!

Edit:

@MikeMcC399 MikeMcC399 mentioned this issue Jun 4, 2024
Closed
This was referenced Jul 3, 2024
Closed
Closed
Merged
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MikeMcC399 MikeMcC399

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jennifer-shehane @MikeMcC399