Provide a shared library for C users

[lpgc]

/kind feature

in order to enable access to libpod APIs from languages other than golang, libpod.so should be built as a C shared library

[baude]

huh? you want us to rewrite in c?

[baude]

btw, the official response here is that you can use the RESTFul interface for this kind of stuff.

[lpgc]

huh? you want us to rewrite in c?

no build the library "-buildmode=c-shared"

[baude]

@lpgc have you tried this on our repo and it works? if so, how about making a make target and a readme on how to use it?

[lpgc]

@lpgc have you tried this on our repo and it works? if so, how about making a make target and a readme on how to use it?

I'll give it my best shot! and report back here... I need to do a little background on the full effects of this buildmode on the resulting shared object before I "ready, fire, aim" ... it may require a separate .so from the native golang one...

Rgds

• larry

[TomSweeneyRedHat]

@lpgc thanks for running with this, a c library could be very useful for some folks.

[lpgc]

@lpgc thanks for running with this, a C library could be very useful for some folks.

Happy to! - my motivation is somewhat selfish as I want to be able to call into the API from Java (via JNA or JNI) and would prefer to "avoid" doing this via REST if possible... but that is always an option to fallback on...

[jmguzik]

@lpgc @TomSweeneyRedHat as far as I know, according to the documentation, it is impossible to create shared C lib out of functions which are using complex go structures as well as interfaces.
@lpgc what you could do, if you really do not want to use provided API, you can write a library in Go, using provided binding, and then export it to shared C lib, stripping complicated types and interfaces. I think it is the quickest way, provided you do not need full functionality, just some parts.

Disclaimer: I am not the expert, I faced a similar problem myself. If you have more experience and now something more, please share.
Further reading (about limitations): https://medium.com/swlh/build-and-use-go-packages-as-c-libraries-889eb0c19838

[lpgc]

@lpgc @TomSweeneyRedHat as far as I know, according to the documentation, it is impossible to create shared C lib out of functions which are using complex go structures as well as interfaces.
@lpgc what you could do, if you really do not want to use provided API, you can write a library in Go, using provided binding, and then export it to shared C lib, stripping complicated types and interfaces. I think it is the quickest way, provided you do not need full functionality, just some parts.

Disclaimer: I am not the expert, I faced a similar problem myself. If you have more experience and now something more, please share.
Further reading (about limitations): https://medium.com/swlh/build-and-use-go-packages-as-c-libraries-889eb0c19838

@jmguzik thanks - sounds like this may be the direction to go, I guess I will decide how much of the API I need and I will take it from there!

I'll close this issue for now, and re-open if I go ahead and create a "simple" API binding that would be generally useful to others!

[jmguzik]

@lpgc issue is closed, but if you are still interested in some kind of starting point I wrote something like this:
https://github.com/jmguzik/podman-bindings-c
Repo follows example from Demos and transforms it to library. Lib is then used in main.c

[jmguzik]

@lpgc forgot to add: I tested it against podman 2.2.1

[lpgc]

@jmguzik very much appreciated, thanks for this... I was planning to do something similar! seems like a reasonable approach to create an appropriate set of "C" compatible "wrappers" to the underlying Golang structures and APIs!

Kudos my friend!

[jmguzik]

@lpgc No problem.. it seemed interesting so I invested a while to see how it works (my background is C/C++).
I had memory allocation issues (but no worries, an example is fine now). Please remember to use Valgrind and memory/address sanitizers to check everything. Sometimes memory/address sanitizer can point to a bug outside of a code (in 3rd party lib), and nothing you can do about it, but still, it is good to scan and fix your own code.
One more piece of advice, use the multi return struct instead of the return via assignment (popular in C). There were memory issues with ret via arg assignment too.

[lpgc]

@lpgc No problem.. it seemed interesting so I invested a while to see how it works (my background is C/C++).
I had memory allocation issues (but no worries, an example is fine now). Please remember to use Valgrind and memory/address sanitizers to check everything. Sometimes memory/address sanitizer can point to a bug outside of a code (in 3rd party lib), and nothing you can do about it, but still, it is good to scan and fix your own code.
One more piece of advice, use the multi return struct instead of the return via assignment (popular in C). There were memory issues with ret via arg assignment too.

good to know cheers!

[matejvasek]

@jmguzik I looked at the code and it seems to me it wraps libpod rest API, is that correct? I am not sure if that's what @lpgc want?

[lpgc]

@jmguzik I looked at the code and it seems to me it wraps libpod rest API, is that correct? I am not sure if that's what @lpgc want?

correct but in general the mechanism is the same ... to provide a C-friendly wrapper to underlying libpod API

[matejvasek]

Have you tried: java -jar openapi-generator-cli.jar generate -i swagger.yaml -g c -o myProjDir --skip-validate-spec. The -g flag is language/framework.

[matejvasek]

also creates rest api wrapper for Java if you want

[lpgc]

yeah I dont want the REST API ...

[jmguzik]

@matejvasek I understand Go bindings are not using REST API (according to instruction: https://github.com/containers/podman/blob/master/docs/tutorials/podman-go-bindings.md). This mechanism avoids using REST API, though it has some (unexplored) limitations.
Making shared C lib means you can use the native Go interface in C, C++, Java (JNI) or even Python. Probably other languages are also possible.

[lpgc]

@matejvasek I understand Go bindings are not using REST API (according to instruction: https://github.com/containers/podman/blob/master/docs/tutorials/podman-go-bindings.md). This mechanism avoids using REST API, though it has some (unexplored) limitations.
Making shared C lib means you can use the native Go interface in C, C++, Java (JNI) or even Python. Probably other languages are also possible.

+1!

[matejvasek]

I think I see where are you heading in future.

Right now
https://github.com/jmguzik/podman-bindings-c/blob/main/libpodc.go
calls images.Pull()
from "github.com/containers/podman/v2/pkg/bindings/images"
which internally do:
response, err := conn.DoRequest(nil, http.MethodPost, "/images/pull", params, header)

[jmguzik]

@matejvasek To be honest I do not quite get your response.
Some calls from bindings are delegated to http, but bindings use unix socket connection to running podman service, like in example:
bindings.NewConnection(context.Background(), "unix://run/podman/podman.sock")

[matejvasek]

My point is that "github.com/containers/podman/v2/pkg/bindings/*" is wrapper lib calling rest API. You need running service serving that API.

[matejvasek]

I believe that @lpgc don't want to depend on rest api provided by some service, he probably need to wrap some other package than "github.com/containers/podman/v2/pkg/bindings/*", so it works standalone without any service.

[jmguzik]

@matejvasek To be honest I am from community so I do not know internals of bindings so well.
In the blog post it was written:

In the release of Podman 2.0, we removed the experimental tag from its recently introduced RESTful service. While it might be interesting to interact with a RESTFul server using curl, using a set of Go based bindings is probably a more direct route to a production ready application. Let’s take a look at how easily that can be accomplished.

It cleary suggests/implies bindings are more native. If it is not the case, maybe other functions are needed to be wrapped. I followed blog post and translated main.go to libpodc.go believing I am interacting with native interface. Another thing is Pull (and inner function) just may use http interface to download image, I do not know.
Maybe @TomSweeneyRedHat who was involved in the discussion some time ago and probably is more experienced can comment on this.
Running service is of course podman service, connection to that service is using unix socket and what podman is doing under the hood, I can't exactly tell.

[jmguzik]

According to article, podman service is a must be to get bindings working. So whatever @lpgc needs, he has to take this into consideration.

[jmguzik]

One more comment: I think we want to eliminate this Podman REST <-> OurApp and replace it with this: Podman Unix Socket <-> OurApp, not get rid of REST completely.

[matejvasek]

It cleary suggests/implies bindings are more native.

Totally more native if you use Golang.
Still I don't see how it's useful from C.

You can use swagger generated C: http://a.tmp.ninja/Sm3YxYj6N0l1.tar

I think we want to eliminate this Podman REST <-> OurApp and replace it with this: Podman Unix Socket <-> OurApp

I don't understand, Podman Unix Socket is the rest api, isn't it?

/cc @jwhonce @mheon

[jmguzik]

@matejvasek I think you are missing one point. Golang has a feature called cgo which enables possibility to compile Go code into shared C lib. That way it acts the same as you are using Go.
I am taking advantage of this mechanism here so:

Totally more native if you use Golang.
Still I don't see how it's useful from C.

If you do not see it usefull from Golang then you do not see it useful from C. Golang functionality is embed into shared C lib:
$ go build -buildmode=c-shared -o libpodc.so libpodc.go

[jmguzik]

Podman Unix Socket is not rest api.

[lpgc]

@matejvasek I think you are missing one point. Golang has a feature called cgo which enables possibility to compile Go code into shared C lib. That way it acts the same as you are using Go.
I am taking advantage of this mechanism here so:

Totally more native if you use Golang.
Still I don't see how it's useful from C.

If you do not see it usefull from Golang then you do not see it useful from C. Golang functionality is embed into shared C lib:
$ go build -buildmode=c-shared -o libpodc.so libpodc.go

exactly @jmguzik my intent is to be able to call directly into podman (libpod) API from a language other than golang hence the need for a shared lib with C bindings ...

[matejvasek]

Podman Unix Socket is not rest api.

The socket is serving HTTP requests. API is described here: https://docs.podman.io/en/latest/_static/api.html#operation/libpodImagesPull.

[matejvasek]

You can invoke it as it was almost as it was TCP:
curl -s --unix-socket /run/user/1001/podman/podman.sock "http:/host/v1.38/images/json" -v -X GET

[jmguzik]

But still this is IPC socket which is a way better than Http. There is a difference to fight for.

[matejvasek]

@lpgc if you want it to be able to run standalone -- deamonless, then you probably will need to wrap stuff from "github.com/containers/podman/v2/pkg/domain/infra/abi".

[matejvasek]

But still this is IPC socket which is a way better than Http.

I think you can access socket even with code generated by swagger.

[jmguzik]

@lpgc if you want it to be able to run standalone -- deamonless, then you probably will need to wrap stuff from "github.com/containers/podman/v2/pkg/domain/infra/abi".

This may be tricky since functions there are too complex to use cgo mechanism.

Anyway this whole thing was just an excercise for me, as topic seemed interesting. I still think it is much more beneficial to wrap Golang bindings using shared C lib, because as a user you do not need to design interface yourself and you have native bindings at almost zero cost (cost of wrapper). IPC unix socket is quick enough and running service is not a problem, as for native go programs interacting with podman it is also designed to be this way. Anyway REST in C++ and especially C is not better solution anyway due to need of exploring external libs.
Probably there are other solutions, you can always design your own curls and send them over socket. Maybe you could use swagger to access IPC socket as you say. This would require further work and exploration and while I might be interested in exploring this for C/C++, java is beyond my scope of interest.

[matejvasek]

Anyway REST in C++ and especially C is not better solution anyway due to need of exploring external libs.

valid point

you need to install at least libcurl

[matejvasek]

This may be tricky since functions there are too complex to use cgo mechanism.

I don't know how difficult that could be.

With regard to java:
If you wrap "github.com/containers/podman/v2/pkg/bindings/*" into C shared lib, then you need to wrap that lib using JNA/JNI (maybe swig can help). And in the end you still end up with solution that needs deamon.
In such a case you may as well auto-generate Java code and avoid that wrapper of wrapper.

[jmguzik]

I don't know how difficult that could be.

You can export only basic simple types and structs. Lambdas and interfaces are no go. For this interface a (complicated?) wrapper would be needed in Go.

In such a case you may as well auto-generate Java code and avoid that wrapper of wrapper.

Topic from the beginning had C in name and I focused on C. I think from C/C++ perspective. This one should be discussed further with @lpgc as he wanted to take C lib further.

[matejvasek]

Maybe I misunderstood:

Happy to! - my motivation is somewhat selfish as I want to be able to call into the API from Java (via JNA or JNI) and would prefer to "avoid" doing this via REST if possible... but that is always an option to fallback on...

This can be understood as:
a) I don't want to use REST at all, I want to run deamonless
b) I don't care if REST is used, I just don't want to see it (do it manually)

If b) is true then wrapping "github.com/containers/podman/v2/pkg/bindings/*" might be good idea.

I initially assumed a).

and by REST I mean libpod API no matter if it runs on unix socket or tcp socket.

[rhatdan]

If you don't want to do rest API, then you would be better off looking at wrapping pkg/domain/infra/abi. This is the direct calls into libpod from Podman. No service required. Note that this interface is not stable and not locked down. Perhaps in the future we could.

[jmguzik]

@rhatdan In regards to your anwser:

Note that this interface is not stable and not locked down. Perhaps in the future we could.

How to see this comment from @TomSweeneyRedHat:

@lpgc thanks for running with this, a c library could be very useful for some folks.

To explain, the interface in the directory you provided is quite difficult to cgo it. Would you accept contributions that will make wrappers and allow to generate C shared libs without REST? These files should be in your repo to keep them up to date. If interface is not stable and has to be maintained outside of podman's realm, this would not make much sense.

From my perspective it was just an exercise to use podman without the need of usage REST in C/C++ (which is a nightmare IMO). I found a leak in podman by the way, so I think it was a good exercise. It could be taken further as topic seems interesting, but I don't know if there is a real need.

[rhatdan]

Sure we would love to work with the community on this, the difficulty is once we define the API we have to make keep it static/stable, which is what we have done with the Rest API. I am not sure if @mheon is ready to specify a stable API to libpod yet.
The question I have is do we want a stable libpod low level API, or something higher like pkg/domain/infra/api, which would be closer to a podman wrapper API on libpod.

[matejvasek]

@rhatdan here it depends what people mean by: "I don't want to use REST". See my comment above.
If they don't really care if they call REST internally --- meaning they want to be just shielded form HTTP calls,
then @jmguzik approach may make sense.

[matejvasek]

Another approach of shielding would be providing wrapper like https://github.com/containers/podman-py but in C.

[jmguzik]

@rhatdan That is why wrapper around functions should be made inside your realm. In subproject (like podman-py) or inside podman itself. Even if not stable, it could be provided as experimental and worked on.
@matejvasek In my understanding "I don't want to use REST" could mean I do not want to see REST in my program. I think it was an initial idea behind this post. I understood it that way.

[lpgc]

@rhatdan That is why wrapper around functions should be made inside your realm. In subproject (like podman-py) or inside podman itself. Even if not stable, it could be provided as experimental and worked on.
@matejvasek In my understanding "I don't want to use REST" could mean I do not want to see REST in my program. I think it was an initial idea behind this post. I understood it that way.

actually my original intent was for a C API calling directly into libpod and not a wrapper onto the REST API, which is
"language independent" anyway, and as far as language access from Java is about as "easy" as calling a "C" based API.

My actual intent was to allow a language environment to provide (some of) the same function as libpod w/o having to deploy any other component other than the shared object itself as part of the application/service deployed.

I know the difference is a subtle one and perhaps not clear as to the advantage of one over the other ... I guess another way to look at this is perhaps as a C API to the various OCI artifacts etc ... like the major verbs and entities ...