OPA does not save bundle files to disk

[Sergey-Kizimov]

OPA does not actually save the bundle files on disk, even though if in config persist: true and it reports that bundles has been saved on disk.
So, if OPA is restarted and the OCI repository is not unavailable at that moment, the startup will fail.

I've tested this on official OPA docker image in k8s and locally

opa version:

✗ opa version

Version: 0.67.1
Build Commit: 
Build Timestamp: 
Build Hostname: 
Go Version: go1.22.5
Platform: darwin/amd64
WebAssembly: unavailable

command:
opa run --server --config-file config.yaml --addr=0.0.0.0:8443 --skip-version-check --log-level=debug

config.yaml

---
services:
  opa-registry:
    url: https://<<path to opa registry>>
    type: oci
    credentials:
      bearer:
        scheme: "Bearer"
        token: <<token>>

bundles:
  tags:
    service: opa-registry
    resource: <<OCI registry tag>>
    persist: true
    polling:
      min_delay_seconds: 3
      max_delay_seconds: 20

logs:

✗ opa run --server --config-file config.yaml --addr=0.0.0.0:8443 --skip-version-check --log-level=debug
Flag --skip-version-check has been deprecated, "skip-version-check" is deprecated. Use "disable-telemetry" instead
{"addrs":["0.0.0.0:8443"],"diagnostic-addrs":[],"level":"info","msg":"Initializing server.","time":"2024-08-16T14:41:42-07:00"}
{"level":"debug","msg":"maxprocs: Leaving GOMAXPROCS=12: CPU quota undefined","time":"2024-08-16T14:41:42-07:00"}
{"level":"info","msg":"Starting bundle loader.","name":"tags","plugin":"bundle","time":"2024-08-16T14:41:42-07:00"}
{"level":"debug","msg":"OCI - Download starting.","time":"2024-08-16T14:41:42-07:00"}
{"level":"debug","msg":"OCIDownloader: using auth plugin: *rest.bearerAuthPlugin","time":"2024-08-16T14:41:42-07:00"}
{"host":"test-registry.io","level":"debug","msg":"resolving","time":"2024-08-16T14:41:42-07:00"}
{"host":"test-registry.io","level":"debug","msg":"do request","request.header.accept":"application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*","request.header.user-agent":"containerd/1.7.20+unknown","request.method":"HEAD","time":"2024-08-16T14:41:42-07:00","url":"https://test-registry.io/v2/test-metadata/metadata-store/manifests/latest"}
{"level":"debug","msg":"Server initialized.","time":"2024-08-16T14:41:42-07:00"}
{"host":"test-registry.io","level":"debug","msg":"fetch response received","response.header.accept-ranges":"bytes","response.header.cache-control":"no-store","response.header.connection":"keep-alive","response.header.content-disposition":"attachment; filename=\"manifest.json\"","response.header.content-length":"615","response.header.content-type":"application/vnd.oci.image.manifest.v1+json","response.header.date":"Fri, 16 Aug 2024 21:41:42 GMT","response.header.docker-content-digest":"sha256:dd5e961b0989b6aaaf4098ccc8b05f9ca0a2a04d7c328ce164021aa4c16ccd77","response.header.docker-distribution-api-version":"registry/2.0","response.header.etag":"96067ecc4bbd063d86a5d217655cc7daf614bad5","response.header.last-modified":"Tue, 25 Jun 2024 10:08:37 GMT","response.header.strict-transport-security":"max-age=31536000; includeSubDomains","response.header.x-artifactory-docker-registry":"opa-registry","response.header.x-artifactory-filename":"manifest.json","response.header.x-artifactory-id":"89cc505d3980364bd8c80e2fa3562fd49562b230","response.header.x-artifactory-node-id":"test-artifactory-primary-1","response.header.x-checksum-md5":"2b1ff85b736d445430d1c618fa112c00","response.header.x-checksum-sha1":"96067ecc4bbd063d86a5d217655cc7daf614bad5","response.header.x-checksum-sha256":"dd5e961b0989b6aaaf4098ccc8b05f9ca0a2a04d7c328ce164021aa4c16ccd77","response.header.x-jfrog-version":"Artifactory/7.93.3 79303900","response.header.x-request-id":"fc4e7dc515438b46a6862ed8d58e75c1","response.status":"200 ","time":"2024-08-16T14:41:42-07:00","url":"https://test-registry.io/v2/test-metadata/metadata-store/manifests/latest"}
{"desc.digest":"sha256:dd5e961b0989b6aaaf4098ccc8b05f9ca0a2a04d7c328ce164021aa4c16ccd77","host":"test-registry.io","level":"debug","msg":"resolved","time":"2024-08-16T14:41:42-07:00"}
{"level":"debug","msg":"Bundle activation in progress (). Opening storage transaction.","name":"tags","plugin":"bundle","time":"2024-08-16T14:41:42-07:00"}
{"level":"debug","msg":"Opened storage transaction (4).","name":"tags","plugin":"bundle","time":"2024-08-16T14:41:42-07:00"}
{"level":"debug","msg":"Closing storage transaction (4).","name":"tags","plugin":"bundle","time":"2024-08-16T14:41:42-07:00"}
{"level":"debug","msg":"Persisting bundle to disk in progress.","name":"tags","plugin":"bundle","time":"2024-08-16T14:41:42-07:00"}
{"level":"debug","msg":"Bundle persisted to disk successfully at path .opa/bundles/tags.","name":"tags","plugin":"bundle","time":"2024-08-16T14:41:42-07:00"}
{"level":"info","msg":"Bundle loaded and activated successfully. Etag updated to aa18f1686eacc8e661db6770db2e1783cf3fe74131fe1e82834ae7e37303094a.","name":"tags","plugin":"bundle","time":"2024-08-16T14:41:42-07:00"}
{"level":"debug","msg":"OCI - Waiting 11.685026324s before next download/retry.","time":"2024-08-16T14:41:42-07:00"}
{"level":"debug","msg":"OCI - Download starting.","time":"2024-08-16T14:41:54-07:00"}
{"level":"debug","msg":"OCIDownloader: using auth plugin: *rest.bearerAuthPlugin","time":"2024-08-16T14:41:54-07:00"}
.....

✗ ls -al .opa/bundles/tags/bundle.tar.gz
-rw-------  1 user  staff  0 Aug 16 14:41 .opa/bundles/tags/bundle.tar.gz

Once started, OPA log error with code EOF and starts downloading it from the remote OCI repository.

✗ opa run --server --config-file config.yaml --addr=0.0.0.0:8443 --skip-version-check --log-level=debug
Flag --skip-version-check has been deprecated, "skip-version-check" is deprecated. Use "disable-telemetry" instead
{"addrs":["0.0.0.0:8443"],"diagnostic-addrs":[],"level":"info","msg":"Initializing server.","time":"2024-08-16T14:52:12-07:00"}
{"level":"debug","msg":"maxprocs: Leaving GOMAXPROCS=12: CPU quota undefined","time":"2024-08-16T14:52:12-07:00"}
{"level":"error","msg":"Failed to load bundle from disk: bundle read failed: archive read failed: EOF","name":"tags","plugin":"bundle","time":"2024-08-16T14:52:12-07:00"}
{"level":"info","msg":"Starting bundle loader.","name":"tags","plugin":"bundle","time":"2024-08-16T14:52:12-07:00"}
{"level":"debug","msg":"OCI - Download starting.","time":"2024-08-16T14:52:12-07:00"}
{"level":"debug","msg":"OCIDownloader: using auth plugin: *rest.bearerAuthPlugin","time":"2024-08-16T14:52:12-07:00"}
{"host":"ttest-registry.io","level":"debug","msg":"resolving","time":"2024-08-16T14:52:12-07:00"}

[ashutosh-narkar]

There maybe a bug in the handling of OCI bundles. Looking at this code, we probably need a io.TeeReader if OPA is configured to persist the bundle. Feel free to submit a fix if you'd like.

[srenatus]

Are you using a tag other than "latest"? I could imagine that making a difference...

[Sergey-Kizimov]

I tried using the latest and specific tags, the behavior is the same, there is no difference

[Sergey-Kizimov]

I've create PR for fixing this issue #6945