How to support plugins

[yurishkuro]

We continue to be asked if we can "support X as storage backend" (e.g. #331, #421). Provided that the authors are willing to contribute, maintain, and support such backend implementations, we still have an open question of whether we want to accept those contributions into the main jaeger repository. I could be wrong, by my initial reaction is that we should keep them in separate "contrib" style repositories, for the following reasons:

1. having half a dozen implementations is going to bloat the size of the binary, increase compile / testing time
2. having them in core repo suggests that they are officially supported, same as Cassandra/ES, but we don't have expertise in all those different storage solutions, and cannot be on the hook to support them

If we do keep them in the contrib repos, however, we need an approach to allow end users to use those implementations without having to rebuild the backend from source.

One such approach is using Go plugins (https://golang.org/pkg/plugin/), for example as done in Kanali. I think it is feasible to package plugins as individual containers and mount them into a shared volume where the main Jaeger binaries can locate and load them.

cc @pavolloffay @black-adder - any thoughts?

Update 12/28/2017

An alternative approach mentioned in the comments below is the sidecar plugin model (e.g. https://github.com/hashicorp/go-plugin) where the plugin runs as a separate process and the main binary communicates with it via RPC, e.g. gRPC streams. It's worth noting, however, that this approach is still a special case of in-process plugin model, so we need to start there and answer the questions below. For each type of plugin we can support a built-in "sidecar" variant.

Update 8/2/2018

Per #422 (comment), I think the following is a reasonable, actionable, and realistic plan. If someone wants to volunteer, ping me.

• define protobuf version of SpanWriter and SpanReader interfaces
• implement gRPC client and server, where server delegates to the respective storage.SpanReader/Writer interfaces, and client implements them.
• extend storage factory with two new types, e.g. h-plugin (h for harshicorp) and g-plugin (plain gRPC). The h-plugin should support a cli flag for the name of the plugin executable. The g-plugin should support a cli flag for grpc server host:port.
• implement in-memory storage as g-plugin using gRPC client/server defined above
• implement one of the other storage types (Cassandra or ES) as h-plugin as a template.
• TBD: how to pass configuration to the h-plugins. Because plugins are plain executables, they can use viper just like the main binaries, and the cli flag with the plugin command line might be a long string (or the user can pass params via env vars). We probably should provide a template for main, so that the actual main for a plugin is very short.
• update documentation with example of building an h-plugin.
• replace Cassandra with in-memory shared service in crossdock integration test.

Update 9/4/2018

Someone pointed out that Go's pkg/plugin now supports MacOS and Linux. This removes a significant development hurdle with using the native plugins, and makes it a viable option which is probably simpler to implement than the gRPC-based harshicorp model.

[jpkrohling]

I think Kubernetes is also planning on supporting plugins for external storage. I'm not familiar with how it works, but it would perhaps be worth checking if it would make sense for us to follow the same path.

[yurishkuro]

I think K8s plugins will be runnable containers, i.e. integration is out of process. A lot of our use cases require in-process integration for performance reasons, e.g. span sanitizers, filters.

[omeid]

I believe out of process integration (effectively middleware/brokers) would be practical for storage, of course, for things like sanitizers, filters, et al. that is a different story.

[frankgreco]

What I found developing Kanali is that plugins are amazing for allowing users to add their own custom functionality. For example, if people design plugins for Kanali, the only requirement is that they implement a certain interface. One side affect is that the .so files aren't small and so while the main program's binary remains small, the Docker image is larger.

I think I agree with @omeid that storage should be out of process. For things like these, I'd propose using gRPC to communicate with out of process integration. For in-process items, such as encrypting tag values, +1 for golang plugins.

[yurishkuro]

I've been thinking this past week about how to support plugins in Jaeger. Some considerations:

• the plugins themselves must be configurable, e.g. Cassandra storage has a bunch of parameters
  • configuring via env variables is the easiest
  • configuring via cmd line arguments (as we do now) is desired (see How to support composable command line arguments #608)
• there must be different types of plugins, e.g. storage vs. span adjusters
• there should be an easy story of deploying plugins without rebuilding docker images
  • the easiest way seems to be to point the main binary to a directory that stores plugins in sub-directories. The main binary will iterate and load all of them. Each plugin could be self-describing, e.g. by implementing one of the supported plugin interfaces like StoragePlugin, AdjusterPlugin, etc.
  • I don't have a good strategy yet how plugins can be loaded from other docker images, open to suggestions.

[frankgreco]

Addressing your last point, when kanali@2.0.0 is released, I think i'm going to do the following to load plugin. With this approach, the kanali pod does not have to be changed when new plugins are required or changed.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kanali
  namespace: default
spec:
  template:
    metadata:
      labels:
        app: kanali
    spec:
      serviceAccountName: kanali
      containers:
      - name: kanali
        image: northwesternmutual/kanali
        command: ["/kanali", "start"]
        volumeMounts:
        - name: plugins
          mountPath: /plugins
          readOnly: true
      initContainers:
      - name: kanali-plugin-apikey
        image: fbgrecojr/kanali-plugin-apikey:local
        imagePullPolicy: IfNotPresent
        command: ["cp", "/apiKey_v2.0.0-rc.1.so", "/plugins/"]
        volumeMounts:
        - name: plugins
          mountPath: /plugins
      volumes:
      - name: plugins
        emptyDir: {}

Addressing your first point, making these plugins configurable is tricky. ENV variables will work but I haven't found a solution for dynamically changing the cli interface. The problems lies in the fact that a golang plugin's init function is called when they are opened. This of course would happened after the init functions for the code doing the opening is executed.

[yurishkuro]

`command: ["cp", "/apiKey_v2.0.0-rc.1.so", "/plugins/"]

I was thinking of something very similar, thanks for posting!

Re dynamic CLI, I have a proposal in #608.

[radekg]

Food for thought - hashicorp terraform also does pluggable backends. AFAIR, they use protobuf for intra-process communication. Might be worth having a look at how they do it.

[frankgreco]

@yurishkuro Ran into a nasty issue relating to plugins with Kanali that I thought I'd share.

In theory, plugins should be able to be dynamically loaded without the need to modify Jaeger's container image. In the context of Kubernetes, the recommended approach would be to use init containers. The init container would move the compiled plugin into the location where the Jaeger binary is configured to look for them.

However, there is a nasty bug that prevents this from being the case until at least Go v1.11. Evidence of this can be tracked in the following issues:

• plugin: gopath type != vendor type golang/go#18827
• plugin: Utility of Go plug-ins diminished by vendored dependencies golang/go#20481

The root cause of the issue stems from the fact that unless a plugin is compiled using the exact same vendor/ instance as Kanali, bad things will happen. For example, global variables will be duplicated, init functions will execute twice, etc.

[frankgreco]

The current workaround that I use entails building your custom plugin within the same Dockerfile as Kanali. The goal is to take the superset of dependencies that Kanali and every plugin requires and build them all using this aggregated dependency tree. This will only work if all intersecting dependencies utilize the exact same revision.

[yurishkuro]

Thanks for pointing those out, @frankgreco. It seems like a complete show stopper to me until there's a better fix from Go. Even if the plugin is build with the same vendor as Jaeger core, there's going to be tight coupling between Jaeger version and plugin build, since the next Jaeger version might have upgrades to the deps. I actually want to give the hashicorp gRPC based plugin framework a try.

[frankgreco]

@yurishkuro I agree! I'm going to checkout their gRPC framework as well.

[zdicesare]

@yurishkuro Curious if there's been any new developments here as I am interested in working on a storage plugin. Is harshicorp gRPC still being evaluated?

[yurishkuro]

Now that the Protobuf model has been merged, the development of gRPC-based plugins can go ahead without blocking on the rest of #773. We have not evaluated harshicorp plugins system, however. I am not sure what it provides that's not achieved by plain gRPC.

[yurishkuro]

I just had a look at harshicorp/go-plugin. If I understand it correctly, even though it uses gRPC (can also use net/rpc), it spawns the plugin as a child process and uses Unix domain socket as the comms channel. This sounds good for implementing plugins for real storage. It somewhat derails another plan I had to build in-memory storage as a shared gRPC service, which would be useful for our crossdock integration tests (instead of always spawning Cassandra), but it won't work with hashicorp/go-plugin since we need a single shared storage process, not two separate child processes for collector and query. However, hashicorp/go-plugin requires us to define a gRPC service and implement client and server anyway, so maybe we can reuse those.

I will update the top of the ticket with what I think is an actionable and realistic plan.

[radekg]

Hi @yurishkuro, is it possible that your in-memory storage issue could be worked around with some atomic handling of plugin start? It is possible to define a grpc server with a custom listen address, go-plugin seems to support reattaching: https://github.com/hashicorp/go-plugin/blob/master/client.go#L117.

If plugins were responsible for their own configuration, they could be given an explicit listen address for the grpc server, if they were not able to bind, they would simply exit.

What do you think?

We at https://github.com/Klarrio use Jaeger for some of the interesting bits on our platform. We are interested in having storage plugins implemented and we would happily contribute.

[isaachier]

I don't think lock files are relevant here. If I understand correctly, @yurishkuro means that Hashicorp plugins are always spawned as a subprocess, so cannot be shared among multiple services. Collector and query need to access the same database in order for the tests to succeed.

However, I don't see any problem with implementing a storage plugin interface in plain grpc without Hashicorp plugins. That should resolve everything.

[radekg]

@isaachier one could write this from scratch but Hashicorp plugins support reattaching, the only inconvenience is that there is a plugin instance starting and stopping immediately due to not being able to bind. Why writing everything from scratch?

[olivierboucher]

Here you go #1214

I will look at the options available for passing configuration.

great, I will make sure to re-read the whole thread as well as #1050

[radekg]

@olivierboucher cool, hashicorp plugins appear to work!

[yurishkuro]

Curious alternative approach (https://github.com/mholt/caddy/wiki): adding the plugins to the source code and building a custom binary, i.e. does not involve distributing pre-built binaries for core server + plugins.

[jpkrohling]

adding the plugins to the source code and building a custom binary

A company that needs a custom plugin would then have to build Jaeger from the sources themselves?

[yurishkuro]

Yes, that's why I don't like that solution. They actually have a website where you can select the plugins in the UI and it builds the binary for you and allows to download it, which is ever more odd for automatic deployment & raises security concerns.

[jpkrohling]

It sounds cool, but I probably wouldn't want to use it as a customer, nor support it as a vendor :-)

[yurishkuro]

Indeed, hence I labeled it curious rather that useful.

[etsangsplk]

@yurishkuro It would be tremendous help if this feature be finished and merged, to let end user to plugin their own storage without pulling down the binary for other backends; and easier to add additional backends easily without tying to to this repo. Do you have a time line for this feature?

[jpkrohling]

@etsangsplk we discussed it last Friday and I think the consensus is to move forward with https://github.com/hashicorp/go-plugin, with InfluxDB being the first implementation (#272). No promises about the timeline, but we are closer to getting it done now than we were one week ago :-)

[totallyunknown]

Hello, we'd (github.com/exaring) like to provide plugin support for storage backend with Go's pkg/plugin support. Later on we'd like to build a storage plugin for DynamoDB. What is the current state of the discussion/progress here?

[jpkrohling]

The problem with Go's pkg/plugin approach is that it (currently) requires your plugins to be built against a very specific revision/tag of Jaeger. Whenever you update your Jaeger instance, you'll need to recompile your plugins, or you risk introducing some bugs that are nasty and hard to detect/diagnose.

[etsangsplk]

@jpkrohling Is there is weekly meeting group for jaegertracing backend (or a page that record the meeting notes)?

[yurishkuro]

https://www.jaegertracing.io/get-in-touch/#bi-weekly-project-meetings

[jacobmarble]

@yurishkuro @jpkrohling
I can't make the next two project meetings. Can you share the status of this issue? I'm not aware of a branch where external plugin work is happening, would love to poke around if something exists.

[yurishkuro]

@jacobmarble we usually don't use feature branches. The work is happening on some PRs linked to this issue, e.g. #1323, #1214.

[yurishkuro]

#1461 is merged 🎉 🎉 🎉 🎈 🎈 🎈

Many thanks to @olivierboucher and @chvck 👏 👏 👏

Remaining task: add documentation #1518.

[pavolloffay]

@yurishkuro can we close this one?