Harbor cosign signature manifest is not saved as artifact accessory after replication

[karaguo]

We notice a flaky issue happening at our harbor instances that:

After a replication job, the destination registry UI, one artifact shows no cosign signature, while at the source registry it has the signature.

To check the artifact item in db, I got no accessory associated with the artifact by

curl -skL https://admin:<omitted>@<omitted>/api/v2.0/projects/library/repositories/<omitted>/artifacts/<omitted>?with_accessory=true&with_signature=true

However, crane manifest shows that the signature manifest is in harbor storage.

crane manifest <host>/<artifact>:sha256-<digest>.sig  | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 233,
    "digest": "sha256:<omitted>"
  },
  "layers": [
    {
      "mediaType": "application/vnd.dev.cosign.simplesigning.v1+json",
      "size": 276,
      "digest": "sha256:<omitted>",
      "annotations": {
        "dev.cosignproject.cosign/signature": "<omitted>,
        "dev.sigstore.cosign/certificate": "-----BEGIN CERTIFICATE-----<omitted>-----END CERTIFICATE-----\n",
        "dev.sigstore.cosign/chain": ""
      }
    }
  ],
  "annotations": {...}
}

Any thought on why the flake happens?

[MinerYang]

Which version of these 2 harbor instance?

[karaguo]

@MinerYang The source harbor is v2.6.2, and destination harbor is 2.8.4

[MinerYang]

Could you share the screenshot of your replication rule and the result of this curl -skL https://admin:<omitted>@<omitted>/api/v2.0/projects/library/repositories/<omitted>/artifacts/<omitted>?with_accessory=true&with_signature=true ?

[karaguo]

The response of curl -skL https://admin:<omitted>@<omitted>/api/v2.0/projects/<omitted>/repositories/<omitted>/artifacts/<omitted>?with_accessory=true&with_signature=true

root@<omitted>:~# curl -skL https://admin:<omitted>@<omitted>/api/v2.0/projects/<omitted>/repositories/<omitted>/artifacts/<omitted>?with_accessory=true&with_signature=true
[1] 1392264
root@<omitted>:~# {"accessories":null,"addition_links":{"build_history":{"absolute":false,"href":"/api/v2.0/projects/<omitted>/repositories/<omitted>/artifacts/sha256:<omitted>/additions/build_history"}},"annotations":{<omitted>},"digest":"sha256:<omitted>","extra_attrs":{"architecture":"amd64","author":"","config":{"Cmd":["nginx","-g","daemon off;"],"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Labels":{<omitted>},"StopSignal":"SIGQUIT","User":"nginx","Volumes":{"/run":{},"/var/cache/nginx":{},"/var/log/nginx":{}},"WorkingDir":"/"},"created":"2023-12-08T19:51:53.73133776Z","os":"linux"},"icon":"sha256:<omitted>","id":1938,"labels":null,"manifest_media_type":"application/vnd.docker.distribution.manifest.v2+json","media_type":"application/vnd.docker.container.image.v1+json","project_id":21,"pull_time":"0001-01-01T00:00:00.000Z","push_time":"2024-01-08T03:03:45.789Z","references":null,"repository_id":71,"size":49740265,"tags":[{"artifact_id":1938,"id":1931,"immutable":false,"name":"v2.8.4-gke.4","pull_time":"0001-01-01T00:00:00.000Z","push_time":"2024-01-08T03:03:45.932Z","repository_id":71,"signed":false}],"type":"IMAGE"}
^C
[1]+  Done                    curl -skL https://admin:kKm29DWBvHOUNpQB@10.200.0.8:10443/api/v2.0/projects/gpc-system-container-images/repositories/private-cloud-staging%2Fgoharbor%2Fharbor-portal/artifacts/v2.8.4-gke.4?with_accessory=true

And the replication rule looks like

[Correction]: an incorrect replication rule was attached, and the rule is use matching url instead of labels.

[MinerYang]

Has the cosign signature artifact itself been pushed to the destination harbor? Could you share the screenshot of this repository and check the existence of signature manifest in the harbor storage directory instead of using crane?

docker exec -it <registry-pod> /bin/bash
cd /storage/docker/registry/v2/blobs/sha256/<prefix-of-signature-digest>
cat < signature-digest.> data

[karaguo]

Sure in the storage, we can see the sig manifest

harbor [ /var/lib/registry/docker/registry/v2/blobs/sha256/67 ]$ cat <omitted>/data 
{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","config":{"mediaType":"application/vnd.docker.container.image.v1+json","size":233,"digest":"sha256:<omitted>"},"layers":[{"mediaType":"application/vnd.dev.cosign.simplesigning.v1+json","size":304,"digest":"sha256:<omitted>","annotations":{"dev.cosignproject.cosign/signature":"<omitted>=","dev.sigstore.cosign/certificate":"-----BEGIN CERTIFICATE-----<omitted>-----END CERTIFICATE-----\n","dev.sigstore.cosign/chain":""}}],"annotations":{<omitted>, "<omitted>.tag":"sha256-32b16e80<omitted>.sig"}}

From dest harbor, the artifact show no sig

From src harbor, the harbor has sig

At this incident, we have both src and dst with v2.8.4

[MinerYang]

Could you kindly share the actual source resource filter name, repository name,signature tag and attach the source harbor audit log of the specific replication task via the UI? And get the signature artifact itself via below cmd

 curl -kv https://admin:<pwd>@<hostname>/api/v2.0/projects/<project-name>/repositories/<repository-name>/artifacts/<.sig tag of signature>

[karaguo]

I found a log that confirms my assumption that the sig manifest is pushed before its artifact manifest, and when the artifact is pushed, the artifact doesn't include the sig as accessory in db.

(^The sig is create 15 mins before its own artifact)

And the sig is pushed (10:16) somehow 15 mins early then its corresponding repo's replication execution task (10:29). The whole replication "end_time":"2024-01-10T22:31:24.000Z", "start_time":"2024-01-10T22:15:03.655Z"

It looks a replication issue

[karaguo]

@MinerYang the replication task log:

2024-01-10T22:29:58Z [INFO] [/controller/replication/transfer/image/transfer.go:139]: client for source registry [type: harbor, URL: http://core:8080, insecure: true] created
2024-01-10T22:29:58Z [INFO] [/controller/replication/transfer/image/transfer.go:149]: client for destination registry [type: harbor, URL: https://10.200.0.8:10443, insecure: true] created
2024-01-10T22:29:58Z [INFO] [/controller/replication/transfer/image/transfer.go:182]: copying ***-system-container-images/***********-staging/ntp-relay-server:[1.12.0-****.****,sha256-387f4799ed3506850dd14bb9f9859109d4fbe3ae11e21524e4eccc6501d40151.sig](source registry) to ***-system-container-images/***********-staging/ntp-relay-server:[1.12.0-****.****,sha256-387f4799ed3506850dd14bb9f9859109d4fbe3ae11e21524e4eccc6501d40151.sig](destination registry)...
2024-01-10T22:29:58Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying ***-system-container-images/***********-staging/ntp-relay-server:1.12.0-****.****(source registry) to ***-system-container-images/***********-staging/ntp-relay-server:1.12.0-****.****(destination registry)...
2024-01-10T22:29:58Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact ***-system-container-images/***********-staging/ntp-relay-server:1.12.0-****.**** ...
2024-01-10T22:29:58Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact ***-system-container-images/***********-staging/ntp-relay-server:1.12.0-****.**** pulled
2024-01-10T22:29:58Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:05b3439058a026ee336d57ee40ccec31c0eba8c6fbe86c2882c1ac4b379471d9(the 1th running)...
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:339]: the blob sha256:05b3439058a026ee336d57ee40ccec31c0eba8c6fbe86c2882c1ac4b379471d9 already exists on the destination registry, skip
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:05b3439058a026ee336d57ee40ccec31c0eba8c6fbe86c2882c1ac4b379471d9 completed
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:23d56352c17f01d8f39b7f68f883288f64c500ff02707f7d9337fee771c259d0(the 1th running)...
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:339]: the blob sha256:23d56352c17f01d8f39b7f68f883288f64c500ff02707f7d9337fee771c259d0 already exists on the destination registry, skip
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:23d56352c17f01d8f39b7f68f883288f64c500ff02707f7d9337fee771c259d0 completed
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:d53a0184841d3691683c0aff4ec52aa402fad5b8272410ca834a3ad45edf1065(the 1th running)...
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:339]: the blob sha256:d53a0184841d3691683c0aff4ec52aa402fad5b8272410ca834a3ad45edf1065 already exists on the destination registry, skip
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:d53a0184841d3691683c0aff4ec52aa402fad5b8272410ca834a3ad45edf1065 completed
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:b0a2b3e20bbf8e5ebdc54eaab7e35a164701bcb4805e9a56b423420e44d588ca(the 1th running)...
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:339]: the blob sha256:b0a2b3e20bbf8e5ebdc54eaab7e35a164701bcb4805e9a56b423420e44d588ca already exists on the destination registry, skip
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:b0a2b3e20bbf8e5ebdc54eaab7e35a164701bcb4805e9a56b423420e44d588ca completed
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:11e179caa15e6e116eca2cdfb3b913be26d45b29d5ed95a49bcc9114b74a94ef(the 1th running)...
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:339]: the blob sha256:11e179caa15e6e116eca2cdfb3b913be26d45b29d5ed95a49bcc9114b74a94ef already exists on the destination registry, skip
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:11e179caa15e6e116eca2cdfb3b913be26d45b29d5ed95a49bcc9114b74a94ef completed
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:7fd511c2b8b5b5f10ef2db7f905fe6aba408726a4ab9ee6425bf2b0eb5fa2d22(the 1th running)...
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:339]: the blob sha256:7fd511c2b8b5b5f10ef2db7f905fe6aba408726a4ab9ee6425bf2b0eb5fa2d22 already exists on the destination registry, skip
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:7fd511c2b8b5b5f10ef2db7f905fe6aba408726a4ab9ee6425bf2b0eb5fa2d22 completed
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:8a0e4b9a840f6cc678a4795a63a84a6ebd3eb40ac4c6c552e91dd5aff96227ab(the 1th running)...
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:339]: the blob sha256:8a0e4b9a840f6cc678a4795a63a84a6ebd3eb40ac4c6c552e91dd5aff96227ab already exists on the destination registry, skip
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:8a0e4b9a840f6cc678a4795a63a84a6ebd3eb40ac4c6c552e91dd5aff96227ab completed
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:3c2cba919283a210665e480bcbf943eaaf4ed87a83f02e81bb286b8bdead0e75(the 1th running)...
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:339]: the blob sha256:3c2cba919283a210665e480bcbf943eaaf4ed87a83f02e81bb286b8bdead0e75 already exists on the destination registry, skip
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:3c2cba919283a210665e480bcbf943eaaf4ed87a83f02e81bb286b8bdead0e75 completed
2024-01-10T22:29:59Z [INFO] [/controller/replication/transfer/image/transfer.go:496]: pushing the manifest of artifact ***-system-container-images/***********-staging/ntp-relay-server:1.12.0-****.**** ...
2024-01-10T22:30:00Z [INFO] [/controller/replication/transfer/image/transfer.go:508]: the manifest of artifact ***-system-container-images/***********-staging/ntp-relay-server:1.12.0-****.**** pushed
2024-01-10T22:30:00Z [INFO] [/controller/replication/transfer/image/transfer.go:253]: copy ***-system-container-images/***********-staging/ntp-relay-server:1.12.0-****.****(source registry) to ***-system-container-images/***********-staging/ntp-relay-server:1.12.0-****.****(destination registry) completed
2024-01-10T22:30:00Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying ***-system-container-images/***********-staging/ntp-relay-server:sha256-387f4799ed3506850dd14bb9f9859109d4fbe3ae11e21524e4eccc6501d40151.sig(source registry) to ***-system-container-images/***********-staging/ntp-relay-server:sha256-387f4799ed3506850dd14bb9f9859109d4fbe3ae11e21524e4eccc6501d40151.sig(destination registry)...
2024-01-10T22:30:00Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact ***-system-container-images/***********-staging/ntp-relay-server:sha256-387f4799ed3506850dd14bb9f9859109d4fbe3ae11e21524e4eccc6501d40151.sig ...
2024-01-10T22:30:00Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact ***-system-container-images/***********-staging/ntp-relay-server:sha256-387f4799ed3506850dd14bb9f9859109d4fbe3ae11e21524e4eccc6501d40151.sig pulled
2024-01-10T22:30:00Z [INFO] [/controller/replication/transfer/image/transfer.go:226]: the artifact ***-system-container-images/***********-staging/ntp-relay-server:sha256-387f4799ed3506850dd14bb9f9859109d4fbe3ae11e21524e4eccc6501d40151.sig already exists on the destination registry, skip
2024-01-10T22:30:00Z [INFO] [/controller/replication/transfer/image/transfer.go:204]: copy ***-system-container-images/***********-staging/ntp-relay-server:[1.12.0-****.****,sha256-387f4799ed3506850dd14bb9f9859109d4fbe3ae11e21524e4eccc6501d40151.sig](source registry) to ***-system-container-images/***********-staging/ntp-relay-server:[1.12.0-****.****,sha256-387f4799ed3506850dd14bb9f9859109d4fbe3ae11e21524e4eccc6501d40151.sig](destination registry) completed

The sig artifact from src harbor

curl -vk https://admin:***@10.128.4.128/api/v2.0/projects/****-system-container-images/repositories/*************-staging%252Fntp-relay-server/artifacts/sha256-387f4799ed3506850dd14bb9f9859109d4fbe3ae11e21524e4eccc6501d40151.sig
*   Trying 10.128.4.128:443...
* TCP_NODELAY set
* Connected to 10.128.4.128 (10.128.4.128) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=CA; L=MountainView; O=*******; OU=*******; CN=10.128.4.128
*  start date: Jan 10 20:01:50 2024 GMT
*  expire date: Jan  7 20:01:50 2034 GMT
*  issuer: C=US; ST=CA; L=MountainView; O=*******; OU=*******; CN=10.128.4.128
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Server auth using Basic with user 'admin'
> GET /api/v2.0/projects/****-system-container-images/repositories/*************-staging%252Fntp-relay-server/artifacts/sha256-387f4799ed3506850dd14bb9f9859109d4fbe3ae11e21524e4eccc6501d40151.sig HTTP/1.1
> Host: 10.128.4.128
> Authorization: Basic ***=
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx
< Date: Thu, 11 Jan 2024 05:02:20 GMT
< Content-Type: application/json
< Content-Length: 1379
< Connection: keep-alive
< Set-Cookie: sid=e2c8f4f4aa41b0efbd5c2de42984a3b5; Path=/; Secure; HttpOnly
< X-Request-Id: 336dcdcd-2255-479b-8d1a-e7da3e09b415
< Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
< X-Frame-Options: DENY
< Content-Security-Policy: frame-ancestors 'none'
< 
{"accessories":null,"addition_links":{"build_history":{"absolute":false,"href":"/api/v2.0/projects/****-system-container-images/repositories/*************-staging%252Fntp-relay-server/artifacts/sha256:785314dbfd144ea21f619cf663739f78db2c07844bdc26c67a48a9b54371ebb1/additions/build_history"}},"annotations":{"org.*******.****.harbor.project":"****-system-container-images","org.*******.****.harbor.repo":"*************-staging/ntp-relay-server","org.*******.****.harbor.tag":"sha256-387f4799ed3506850dd14bb9f9859109d4fbe3ae11e21524e4eccc6501d40151.sig"},"digest":"sha256:785314dbfd144ea21f619cf663739f78db2c07844bdc26c67a48a9b54371ebb1","extra_attrs":{"architecture":"","author":"","config":{},"created":"0001-01-01T00:00:00Z","os":""},"icon":"sha256:0048162a053eef4d4ce3fe7518615bef084403614f8bca43b40ae2e762e11e06","id":5477,"labels":null,"manifest_media_type":"application/vnd.oci.image.manifest.v1+json","media_type":"application/vnd.docker.container.image.v1+json","project_id":215,"pull_time":"0001-01-01T00:00:00.000Z","push_time":"2024-01-10T20:40:30.301Z","references":null,"repository_id":949,"size":3247,"tags":[{"artifact_id":5477,"id":1963,"immutable":false,"name":"sha256-387f4799ed3506850dd14bb9f9859109d4fbe3ae11e21524e4eccc6501d40151.sig","pull_time":"0001-01-01T00:00:00.000Z","push_time":"2024-01-10T20:40:30.312Z","repository_id":949,"signed":false}],"type":"IMAGE"}
* Connection #0 to host 10.128.4.128 left intact

The sig artifact from dst

curl -vk https://admin:***@10.200.0.8:10443/api/v2.0/projects/***-system-container-images/repositories/*********-staging%252Fntp-relay-server/artifacts/sha256-387f4799ed3506850dd14bb9f9859109d4fbe3ae11e21524e4eccc6501d40151.sig
*   Trying 10.200.0.8:10443...
* TCP_NODELAY set
* Connected to 10.200.0.8 (10.200.0.8) port 10443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Jan 10 21:32:54 2024 GMT
*  expire date: Jan  9 21:32:54 2025 GMT
*  issuer: CN=harbor-cert
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Server auth using Basic with user 'admin'
* Using Stream ID: 1 (easy handle 0x555d0940d8a0)
> GET /api/v2.0/projects/***-system-container-images/repositories/*********-staging%252Fntp-relay-server/artifacts/sha256-387f4799ed3506850dd14bb9f9859109d4fbe3ae11e21524e4eccc6501d40151.sig HTTP/2
> Host: 10.200.0.8:10443
> authorization: Basic *****==
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 200 
< content-type: application/json
< set-cookie: sid=03a5f52fe9a532023bb2a93b45705b1f; Path=/; HttpOnly; Secure
< x-request-id: f1fb99ae-b128-41e0-89fd-5bf82cf26b88
< date: Thu, 11 Jan 2024 05:07:30 GMT
< content-length: 1370
< x-envoy-upstream-service-time: 7
< 
{"accessories":null,"addition_links":{"build_history":{"absolute":false,"href":"/api/v2.0/projects/***-system-container-images/repositories/*********-staging%252Fntp-relay-server/artifacts/sha256:785314dbfd144ea21f619cf663739f78db2c07844bdc26c67a48a9b54371ebb1/additions/build_history"}},"annotations":{"org.*********.***.harbor.project":"***-system-container-images","org.*********.***.harbor.repo":"*********-staging/ntp-relay-server","org.*********.***.harbor.tag":"sha256-387f4799ed3506850dd14bb9f9859109d4fbe3ae11e21524e4eccc6501d40151.sig"},"digest":"sha256:785314dbfd144ea21f619cf663739f78db2c07844bdc26c67a48a9b54371ebb1","extra_attrs":{"architecture":"","author":"","config":{},"created":"0001-01-01T00:00:00Z","os":""},"icon":"sha256:0048162a053eef4d4ce3fe7518615bef084403614f8bca43b40ae2e762e11e06","id":20,"labels":null,"manifest_media_type":"application/vnd.oci.image.manifest.v1+json","media_type":"application/vnd.docker.container.image.v1+json","project_id":11,"pull_time":"0001-01-01T00:00:00.000Z","push_time":"2024-01-10T22:16:17.022Z","references":null,"repository_id":17,"size":3247,"tags":[{"artifact_id":20,"id":20,"immutable":false,"name":"sha256-387f4799ed3506850dd14bb9f9859109d4fbe3ae11e21524e4eccc6501d40151.sig","pull_time":"0001-01-01T00:00:00.000Z","push_time":"2024-01-10T22:16:17.161Z","repository_id":17,"signed":false}],"type":"IMAGE"}
* Connection #0 to host 10.200.0.8 left intact

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[MinerYang]

Hi @karaguo ,

We harbor support push accessory and subject artifact in either order #19906
Please upgrade to v2.9.2+ and have a try

[MinerYang]

WorkAround:
https://github.com/goharbor/harbor/wiki/Harbor-FAQs#cosign-signatures-accessory