ERROR: rebase working image: manifest.json had unexpected number of entries: 2

[altfatterz]

Summary

pack rebase is failing with:

ERROR: rebase working image: manifest.json had unexpected number of entries: 2

Reproduction

Steps

1. Checkout an example repo and create an image:

$ git clone https://github.com/altfatterz/buildpacks-demo.git
$ cd buildpacks-demo
$ pack build buildpacks-demo --builder paketobuildpacks/builder:base --run-image paketobuildpacks/run:1.0.10-base-cnb

2. Verify image is created:

$ docker images

REPOSITORY                 TAG               IMAGE ID       CREATED        SIZE
paketobuildpacks/run       1.0.10-base-cnb   564f9efbf27d   11 days ago    90.7MB
paketobuildpacks/builder   base              03aa716c9552   41 years ago   558MB
buildpacks-demo            latest            c312718ad92b   41 years ago   263MB

3. Rebase the buildpacks-demo with a newer base image

pack rebase buildpacks-demo --run-image paketobuildpacks/run:1.0.11-base-cnb

Current behavior

It fails with error:

1.0.11-base-cnb: Pulling from paketobuildpacks/run
Digest: sha256:f393fa2927a2619a10fc09bb109f822d20df909c10fed4ce3c36fad313ea18e3
Status: Image is up to date for paketobuildpacks/run:1.0.11-base-cnb
Rebasing buildpacks-demo on run image paketobuildpacks/run:1.0.11-base-cnb
ERROR: rebase working image: manifest.json had unexpected number of entries: 2

Expected behavior

The new image is not produced.

Environment

pack report

Pack:
  Version:  0.15.1+git-79adc30.build-1660
  OS/Arch:  darwin/amd64

Default Lifecycle Version:  0.9.3

Supported Platform APIs:  0.3, 0.4

Config:
(no config file found at /Users/altfatterz/.pack/config.toml)

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.4.2-docker)
  scan: Docker Scan (Docker Inc., v0.5.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 5
 Server Version: 20.10.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.121-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.778GiB
 Name: docker-desktop
 ID: NZMK:XXXX:MEFA:GTFK:6U6F:7V3U:B5AM:CQMT:EDGR:UWD5:EGQ5:JO5Z
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: gateway.docker.internal:3128
 HTTPS Proxy: gateway.docker.internal:3129
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

[dfreilich]

Hey @altfatterz ,

Thanks for the really clear error report! I tried to replicate it, but I wasn't successfully able to do so:

$  pack rebase buildpacks-demo --run-image paketobuildpacks/run:1.0.11-base-cnb
1.0.11-base-cnb: Pulling from paketobuildpacks/run
Digest: sha256:f393fa2927a2619a10fc09bb109f822d20df909c10fed4ce3c36fad313ea18e3
Status: Downloaded newer image for paketobuildpacks/run:1.0.11-base-cnb
Rebasing buildpacks-demo on run image paketobuildpacks/run:1.0.11-base-cnb
*** Images (e52aedb7431f):
      buildpacks-demo
Rebased Image: e52aedb7431f0973dc1d89c6ac73e4921fc9d1a9bc042bc4ecbe6041bf32693e
Successfully rebased image buildpacks-demo

My docker info:

$  docker info
Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0)
  buildx: Build with BuildKit (Docker Inc., v0.4.2-tp-docker)
  scan: Docker Scan (Docker Inc., v0.3.4)

Server:
 Containers: 29
  Running: 3
  Paused: 0
  Stopped: 26
 Images: 41
 Server Version: 19.03.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.39-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 1.94GiB
 Name: docker-desktop
 ID: DPWL:PDY4:EAG6:ZDAJ:VAON:XFI7:6SWP:5MT5:II3B:W2WH:YSCT:VAJL
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 61
  Goroutines: 69
  System Time: 2021-01-04T11:49:32.2434211Z
  EventsListeners: 3
 HTTP Proxy: gateway.docker.internal:3128
 HTTPS Proxy: gateway.docker.internal:3129
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  dfreilich.local:5000
  localhost:5000
  127.0.0.0/8
 Live Restore Enabled: false

[ekcasey]

@altfatterz The next release of pack should contain the fix for this issue buildpacks/imgutil#63. A an explanation of the issue can be found here buildpacks/imgutil#81

While intended to resolve #80 this also resolves #63 as a side effect because we now save an image to disk by ID, avoiding situations where manifest.json might contain multiple entries. This previously happened because we were passing a tag to docker save without expanding an implicit :latest tag. While docker inspect will treat my/image as my/image:latest docker save will return all images with repository my/image.

You should be able to work around this in the meantime by adding an explicit :latest tag. E.g

pack rebase buildpacks-demo:latest --run-image paketobuildpacks/run:1.0.11-base-cnb

[altfatterz]

@ekcasey great and thanks for the workaround.

[dwillist]

Closing as I can no longer reproduce with pack 0.17.0 @altfatterz please reopen if this issue persists.