Eclipse Docker Tooling support

[mlegenovic]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Docker compatibility endpoint /images/json does not comply with the Docker API specification. Eclipse Docker plugin calls /images/json?all=true and expects this structure: https://github.com/dmandalidis/docker-client/blob/master/src/main/java/org/mandas/docker/client/messages/Image.java. Podman returns a JSON like this:

$ curl http://localhost:8080/images/json?all=true | jq                            
[
  {
    "Id": "50541622f4f179450f4acec7d16964499525932a81263eafb91e699671f58ee4",
    "RepoTags": [
      "docker.io/library/redis:6"
    ],
    "Created": "2020-07-22T02:15:29.603086742Z",
    "Size": 107545246,
    "Names": [
      "docker.io/library/redis:6"
    ],
    "Digest": "sha256:e495ab323824662760af192548758d93365ca2e796eb0ef54b7bf3a2bd775df5",
    "Digests": [
      "sha256:e495ab323824662760af192548758d93365ca2e796eb0ef54b7bf3a2bd775df5",
      "sha256:7d1ebef417593273b2f7b137f615393b21723d93568868c7fe5aad0401469152"
    ],
    "History": [
      "50541622f4f179450f4acec7d16964499525932a81263eafb91e699671f58ee4",
      "<missing>",
      "<missing>",
      "<missing>",
      "<missing>",
      "<missing>",
      "<missing>",
      "<missing>",
      "<missing>",
      "<missing>",
      "<missing>",
      "<missing>",
      "<missing>",
      "<missing>",
      "<missing>",
      "<missing>"
    ]
  }
]

Deserialisation fails with following error message:

An internal error occurred during: "Refreshing Images...".
Entity input stream has already been closed.

Steps to reproduce the issue:

1. Call /images/json?all=true

2. Compare result with expected response schema described here: https://docs.docker.com/engine/api/v1.40/#operation/ImageList

3. Podman does not return all required fields

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      2.0.6
API Version:  1
Go Version:   go1.14.6
Built:        Tue Sep  1 21:26:51 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.1
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-2.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 81d18b6c3ffc266abdef7ca94c1450e669a6a388'
  cpus: 4
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: f32vm
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.11-200.fc32.x86_64
  linkmode: dynamic
  memFree: 7107022848
  memTotal: 16783425536
  ociRuntime:
    name: crun
    package: crun-0.14.1-5.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14.1
      commit: 598ea5e192ca12d4f6378217d3ab1415efeddefa
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 0
  swapTotal: 0
  uptime: 2h 31m 0.15s (Approximately 0.08 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/mile/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/mile/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/mile/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 1598988411
  BuiltTime: Tue Sep  1 21:26:51 2020
  GitCommit: ""
  GoVersion: go1.14.6
  OsArch: linux/amd64
  Version: 2.0.6

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.0.6-1.fc32.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Eclipse 2020-09 (4.17) with Docker Tooling Plugin

[mheon]

Can you possibly find the type of struct that Eclipse is trying to unmarshal into, and what version of the Docker API Eclipse was written to support?

[mlegenovic]

Link to java struct is available in description, but here again: https://github.com/dmandalidis/docker-client/blob/master/src/main/java/org/mandas/docker/client/messages/Image.java

Version compatibility is described here: https://github.com/dmandalidis/docker-client#version-compatibility.

It looks that they support latest version, that Podman also supports.

[mheon]

Issues I notice: Digests in our JSON output should be RepoDigests. VirtualSize, Labels, and ParentId are missing entirely (presumably set to omitempty).

[mlegenovic]

presumably set to omitempty

Yes, this was also comment from jwhonce in #7327

To save on bandwidth when encoding JSON we have routinely omitted zero-value fields. If this is causing decoding issues, we can revisit this design decision.

[rhatdan]

I am not sure how much bandwidth we are saving by omitting these fields. If tools have merged broken behaviour from the Docker API into their tools, then we will just have to deal with it. Bug for Bug compatibility.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@jwhonce @mheon where do we stand on this. Seems like just removing ommitempty for docker API would fix the problem?

[mheon]

Did we ever fix the RepoDigests issue?

I would check if that resolves things first, JSON should be tolerant of missing entries.

[mlegenovic]

To solve first problem in Eclipse RepoDigests is not needed, but in Docker API specified as required. This patch solves first problem:

diff --git a/pkg/domain/entities/images.go b/pkg/domain/entities/images.go
index 101542a98..d8f123625 100644
--- a/pkg/domain/entities/images.go
+++ b/pkg/domain/entities/images.go
@@ -52,12 +52,12 @@ func (i *Image) Id() string { // nolint
 
 type ImageSummary struct {
        ID          string            `json:"Id"`
-       ParentId    string            `json:",omitempty"` // nolint
+       ParentId    string                                // nolint
        RepoTags    []string          `json:",omitempty"`
-       Created     int64             `json:",omitempty"`
-       Size        int64             `json:",omitempty"`
+       Created     int64
+       Size        int64
        SharedSize  int               `json:",omitempty"`
-       VirtualSize int64             `json:",omitempty"`
+       VirtualSize int64
        Labels      map[string]string `json:",omitempty"`
        Containers  int               `json:",omitempty"`
        ReadOnly    bool              `json:",omitempty"`

Second problem is solved with this patch (RegistryConfig is specified as nullable in API spec, but for some reason docker-client library used in Eclipse Docker plugin requires it):

diff --git a/pkg/api/handlers/compat/info.go b/pkg/api/handlers/compat/info.go
index 2bb165522..4b3a390f1 100644
--- a/pkg/api/handlers/compat/info.go
+++ b/pkg/api/handlers/compat/info.go
@@ -17,6 +17,7 @@ import (
        "github.com/containers/podman/v2/pkg/api/handlers/utils"
        "github.com/containers/podman/v2/pkg/rootless"
        docker "github.com/docker/docker/api/types"
+       "github.com/docker/docker/api/types/registry"
        "github.com/docker/docker/api/types/swarm"
        "github.com/google/uuid"
        "github.com/pkg/errors"
@@ -103,7 +104,7 @@ func GetInfo(w http.ResponseWriter, r *http.Request) {
                PidsLimit:          sysInfo.PidsLimit,
                Plugins:            docker.PluginsInfo{},
                ProductLicense:     "Apache-2.0",
-               RegistryConfig:     nil,
+               RegistryConfig:     new(registry.ServiceConfig),
                RuncCommit:         docker.Commit{},
                Runtimes:           getRuntimes(configInfo),
                SecurityOptions:    getSecOpts(sysInfo),

Next problem is missing Args attribute in containers endpoint, other two attributes (Path and RestartCount) are not needed in plugin, but assigned values are wrong (podman inspect shows correct values, i got them from there):

diff --git a/pkg/api/handlers/compat/containers.go b/pkg/api/handlers/compat/containers.go
index 00be8e845..61ec9e7a6 100644
--- a/pkg/api/handlers/compat/containers.go
+++ b/pkg/api/handlers/compat/containers.go
@@ -318,8 +318,8 @@ func LibpodToContainerJSON(l *libpod.Container, sz bool) (*types.ContainerJSON,
        cb := types.ContainerJSONBase{
                ID:              l.ID(),
                Created:         l.CreatedTime().Format(time.RFC3339Nano),
-               Path:            "",
-               Args:            nil,
+               Path:            inspect.Path,
+               Args:            inspect.Args,
                State:           &state,
                Image:           imageName,
                ResolvConfPath:  inspect.ResolvConfPath,
@@ -328,7 +328,7 @@ func LibpodToContainerJSON(l *libpod.Container, sz bool) (*types.ContainerJSON,
                LogPath:         l.LogPath(),
                Node:            nil,
                Name:            fmt.Sprintf("/%s", l.Name()),
-               RestartCount:    0,
+               RestartCount:    int(inspect.RestartCount),
                Driver:          inspect.Driver,
                Platform:        "linux",
                MountLabel:      inspect.MountLabel,

Next patch is not needed in Eclipse plugin, but it fixes wrong value in VirtualSize attribute (i got it also from podman inspect code):

diff --git a/pkg/api/handlers/types.go b/pkg/api/handlers/types.go
index 6bb5f5101..b95ba6fc0 100644
--- a/pkg/api/handlers/types.go
+++ b/pkg/api/handlers/types.go
@@ -215,7 +215,7 @@ func ImageToImageSummary(l *libpodImage.Image) (*entities.ImageSummary, error) {
                Created:      l.Created().Unix(),
                Size:         int64(*size),
                SharedSize:   0,
-               VirtualSize:  l.VirtualSize,
+               VirtualSize:  int64(*size),
                Labels:       labels,
                Containers:   containerCount,
                ReadOnly:     l.IsReadOnly(),

Next problem is images/search endpoint. Here how Docker API spec looks like: https://docs.docker.com/engine/api/v1.40/#operation/ImageSearch
and here is what i get from podman (it looks like docker compatibility endpoint is not implemented):

curl --unix-socket /run/user/1000/podman/podman.sock http://localhost/images/search?term=redis | jq

{
    "Index": "fedoraproject.org",
    "Name": "registry.fedoraproject.org/f31/redis",
    "Description": "",
    "Stars": 0,
    "Official": "",
    "Automated": "",
    "Tag": ""
}

The last problem I found before I stopped looking for others is wrong result when i try to create a container from GUI. Container is created, but it can not be started. A look with podman inspect shows that some attributes are wrong (compared with inspect-output when i create container with podman create or podman run).

[rhatdan]

@mlegenovic Could you open these patches as a PR?

[mlegenovic]

BTW, IntelliJ docker plugin has the same problem, also there a container instance is created but cannot be started. I think relevant part is this:

...
          "Path": "redis-server",
          "Args": [
               "redis-server"
...

but i'm not 100% sure. Here is how it looks like when i start the container with podman create:

...
          "Path": "docker-entrypoint.sh",
          "Args": [
               "redis-server"
          ],
...

There are also some other differences, but i'm not sure if they are relevant. It is very simple to reproduce the problem.

[mlegenovic]

@rhatdan Of course, i can do it.

[mheon]

This comes back to our discussions around testcontainers - the Docker API never specifies that these fields are required, but it seems like Java implementations of JSON parsing are unable to handle omitting empty values? I don't really like doing this just for one language's quirks, but it seems like we may not have a choice.

[mlegenovic]

@mheon This is from Docker API spec:

Size integer Required

so, it is specified as required. I agree that these fields are not really required for a tool to work, but on the other side Size should not be empty because podman inspect shows a size, so it is a bug in that endpoint implementation.

[mlegenovic]

Sorry, my mistake, i meant VirtualSize, Size is present and has correct value.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mlegenovic Has the situation improved on this in the main branch?

[mlegenovic]

@rhatdan Most things are already working, there is only a problem with the end-point to pull images. I just had no time to find out what exactly is the problem. What I saw is, docker sends progress information during the download, and podman not. Maybe this is the only problem. I need some time to analyze it.
Will close the issue as soon as everything is working.

[rhatdan]

Great, I think there was some work on podman-remote build to send progress information, might be related.

[mlegenovic]

@rhatdan I have tested podman-remote, and it has the same behavior. I have found one endpoint which works with stream-writer: /libpod/images/pull (pkg/api/handlers/libpod/images_pull.go, function ImagesPull). JSON structure is not the same, but principle is correct. I though I can use it to fix behavior of /images/create endpoint (monkey see, monkey do), but I underestimated it. My Go skills are not good enough.

Just to demonstrate the difference:

Podman:

$ curl -X POST 'http://localhost:8080/images/create?fromImage=docker.io%2Flibrary%2Fmysql&tag=latest'
{"status":"pulling image () from docker.io/library/mysql:latest","error":"","progress":"","progressDetail":{},"id":"c8562eaf9d81c779cbfc318d6e01b8e6f86907f1d41233268a2ed83b2f34e748"}

Docker:

$ curl --unix-socket /run/docker.sock -X POST 'http://d/images/create?fromImage=docker.io%2Flibrary%2Fmysql&tag=latest'
{"status":"Pulling from library/mysql","id":"latest"}
{"status":"Already exists","progressDetail":{},"id":"a076a628af6f"}
{"status":"Pulling fs layer","progressDetail":{},"id":"f6c208f3f991"}
{"status":"Pulling fs layer","progressDetail":{},"id":"88a9455a9165"}
{"status":"Pulling fs layer","progressDetail":{},"id":"406c9b8427c6"}
{"status":"Pulling fs layer","progressDetail":{},"id":"7c88599c0b25"}
{"status":"Pulling fs layer","progressDetail":{},"id":"25b5c6debdaf"}
{"status":"Pulling fs layer","progressDetail":{},"id":"43a5816f1617"}
{"status":"Pulling fs layer","progressDetail":{},"id":"1a8c919e89bf"}
{"status":"Pulling fs layer","progressDetail":{},"id":"9f3cf4bd1a07"}
{"status":"Pulling fs layer","progressDetail":{},"id":"80539cea118d"}
{"status":"Pulling fs layer","progressDetail":{},"id":"201b3cad54ce"}
{"status":"Pulling fs layer","progressDetail":{},"id":"944ba37e1c06"}
{"status":"Waiting","progressDetail":{},"id":"1a8c919e89bf"}
{"status":"Waiting","progressDetail":{},"id":"9f3cf4bd1a07"}
{"status":"Waiting","progressDetail":{},"id":"7c88599c0b25"}
{"status":"Waiting","progressDetail":{},"id":"80539cea118d"}
{"status":"Waiting","progressDetail":{},"id":"201b3cad54ce"}
{"status":"Waiting","progressDetail":{},"id":"43a5816f1617"}
{"status":"Waiting","progressDetail":{},"id":"25b5c6debdaf"}
{"status":"Waiting","progressDetail":{},"id":"944ba37e1c06"}
{"status":"Downloading","progressDetail":{"current":424,"total":1741},"progress":"[============\u003e                                      ]     424B/1.741kB","id":"f6c208f3f991"}
{"status":"Download complete","progressDetail":{},"id":"f6c208f3f991"}
{"status":"Extracting","progressDetail":{"current":1741,"total":1741},"progress":"[==================================================\u003e]  1.741kB/1.741kB","id":"f6c208f3f991"}
{"status":"Downloading","progressDetail":{"current":15480,"total":1419257},"progress":"[\u003e                                                  ]  15.48kB/1.419MB","id":"406c9b8427c6"}
{"status":"Downloading","progressDetail":{"current":42703,"total":4178395},"progress":"[\u003e                                                  ]   42.7kB/4.178MB","id":"88a9455a9165"}
{"status":"Extracting","progressDetail":{"current":1741,"total":1741},"progress":"[==================================================\u003e]  1.741kB/1.741kB","id":"f6c208f3f991"}
{"status":"Pull complete","progressDetail":{},"id":"f6c208f3f991"}
{"status":"Downloading","progressDetail":{"current":298240,"total":1419257},"progress":"[==========\u003e                                        ]  298.2kB/1.419MB","id":"406c9b8427c6"}
{"status":"Downloading","progressDetail":{"current":85664,"total":4178395},"progress":"[=\u003e                                                 ]  85.66kB/4.178MB","id":"88a9455a9165"}
{"status":"Downloading","progressDetail":{"current":658688,"total":1419257},"progress":"[=======================\u003e                           ]  658.7kB/1.419MB","id":"406c9b8427c6"}
{"status":"Downloading","progressDetail":{"current":232837,"total":4178395},"progress":"[==\u003e                                                ]  232.8kB/4.178MB","id":"88a9455a9165"}
{"status":"Downloading","progressDetail":{"current":826624,"total":1419257},"progress":"[=============================\u003e                     ]  826.6kB/1.419MB","id":"406c9b8427c6"}
{"status":"Downloading","progressDetail":{"current":376064,"total":4178395},"progress":"[====\u003e                                              ]  376.1kB/4.178MB","id":"88a9455a9165"}
{"status":"Downloading","progressDetail":{"current":1023232,"total":1419257},"progress":"[====================================\u003e              ]  1.023MB/1.419MB","id":"406c9b8427c6"}
{"status":"Verifying Checksum","progressDetail":{},"id":"406c9b8427c6"}
{"status":"Download complete","progressDetail":{},"id":"406c9b8427c6"}
{"status":"Downloading","progressDetail":{"current":965888,"total":4178395},"progress":"[===========\u003e                                       ]  965.9kB/4.178MB","id":"88a9455a9165"}
{"status":"Downloading","progressDetail":{"current":1407457,"total":4178395},"progress":"[================\u003e                                  ]  1.407MB/4.178MB","id":"88a9455a9165"}
{"status":"Downloading","progressDetail":{"current":1849825,"total":4178395},"progress":"[======================\u003e                            ]   1.85MB/4.178MB","id":"88a9455a9165"}
{"status":"Downloading","progressDetail":{"current":115,"total":115},"progress":"[==================================================\u003e]     115B/115B","id":"7c88599c0b25"}
{"status":"Download complete","progressDetail":{},"id":"7c88599c0b25"}
...

[rhatdan]

@jwhonce PTAL

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mlegenovic Where do we stand on this at this point?

[mlegenovic]

@rhatdan I have improved my go skills in the meantime, and could implement create?fromImage endpoint to work like /libpod/images/pull (with stream-writer). Unfortunately, it's still not good enough. Problem is that feedback comes per layer. If you pull an image with a large layer, only one json is created per layer, and you wait again very long without any feedback. Java docker client has a timeout, and if it does not receive any feedback for more then 20 seconds (or something like this), it interrupts the download and pull fails.

I have analyzed with debugger if podman even offers the possibility to provide such feedback. And I have found something (a channel), but due to lack of time not yet tried. Result will for sure not be identical to that what docker provides, but this is also not needed. I will try it when I have time, and if it works I will create a PR. Otherwise i will close this issue because a larger refactoring will be needed to offer similar functionality (not only in podman, but also in https://github.com/containers/image).

I have seen some people have the same problem because they use products that use the same java docker client library. If I can solve it, multiple issues can be closed.

[mlegenovic]

@rhatdan It works, I was successful. Will create PR in the next few days.

[rhatdan]

Awesome.

[mlegenovic]

Last known problem is fixed, so issue can be closed.

[rhatdan]

Awesome, now if we could get Eclipse Podman tooling, to use enhanced podman features...

[mlegenovic]

Eclipse uses 3rd party library to integrate docker. Only GUI is implemented in Eclipse, everything else is 3rd party. I do not know if podman has such a library written in java (I know there is one for python). If not, then I think first step would be to write such a library, or to extend existing docker-based libraries to support native podman features.

I don't know if there are issues in the eclipse project to support podman, but I have seen that Jetbrains user (Intellij) are asking for proper podman integration (including podman native features). Many people write there "podman is the future, please support podman", "kubernetes has dropped docker support, please support podman". So community interest is there.

[rhatdan]

Awesome. Last time I wrote Java, was before the millennial. :^(

[KhurramSSiddiqui]

Nice, I see that this issue has been closed and PRs merged to containers:master
I have just started exploring Podman. I faced this problem immediately on trying to create a connection from Eclipse CDT 2020-12-R to Podman 2.2.1 (Built Feb 9, 2021).
When would these changes make their way to the next build of Podman 2.2.1? (If such a build is planned/scheduled -or- 2.2.2 maybe?)
I tried looking around but couldn't find this information on my own (newbie).
Thanks.

[mheon]

The 2.2.x branch of Podman is no longer being actively developed - we're working on 3.1 now, with 3.0 staying around for a while as a long-term support release