Consider FQ transport #1233
Comments
|
This came out of a discussion with @postwait. Depending on how many folks are interested in this, it could end up in a separate repo or in the monorepo (this repo) |
|
From @eirslett on May 26, 2016 21:58 What isn't high-performant about Kafka? |
|
@eirslett well when they last used zipkin, it only supported scribe. I don't recall a performance comparison with kafka, although less complexity was mentioned (ex no zookeeper dep) |
|
From @postwait on May 26, 2016 22:10 Not sure how a performance benchmark is relevant to the discussion? Fq can Additionally, I don't think anyone was suggesting the removal of Kafka - |
|
Summary so far: With some relative ease one could make a proof of concept which employs FQ as a custom collector. It doesn't look viable to do more than that until fq has binaries and a jar published to artifact repositories. Our build and dev story is far easier to manage when we don't need to compile c as a prerequisite to running tests. details.. Steps to create fq are like below main things is that it creates a fqd and friends, as well a jar file for the client connector. It wasn't hard to get it started though I haven't tested it, yet I think the challenge we'd have with this is we usually get dependencies from packages or otherwise. Testing would be easier to integrate if there were a current fq homebrew and apt (for our CI server) package. Otherwise we'd end up with what could be brittle pre-install routines that build the code. There's also the java side, where we typically consume published jars that include some version of whatever the connector is. The fq code has no dependencies, so it could be feasible to build locally (ie source import), but my concern would be about support issues around that. Ex we don't have experience or a means to run unit tests against it. # build and install ck
$ git clone https://github.com/concurrencykit/ck.git; cd ck/
$ ./configure && make && make install
$ cd ..
# build and install jlog
$ git clone https://github.com/omniti-labs/jlog.git; cd jlog/
$ autoconf; ./configure; make; make install
$ cd ..
# build and install fq
$ git clone https://github.com/circonus-labs/fq.git; cd fq/
$ make; make install |
|
From @postwait on May 27, 2016 1:5 We'll look at publishing the fq java client jars to the maven central repo. On Thu, May 26, 2016 at 6:58 PM, Adrian Cole notifications@github.com
|
|
Sounds good. So, I took a few stabs while in the air and probably didn't A sample using the java api in a producer consumer context would be nice, Without a sample, it is harder (at least for me) to understand which |
|
From @eirslett on May 27, 2016 8:39 I wasn't saying it's either Kafka or fq; I was just pointing out we already have I'm all for adding a transport for fq if people are willing to contribute one (adriancole sounds really motivated) but we should handle it the same way we handle zipkin-hbase, which is similar, community/ecosystem-wise. |
|
Sheesh sorry about mis-introducing this. There are many who have had problems with scribe due to its synchronous |
|
From @postwait on May 27, 2016 23:24 Backs away slowly and exits the room.
|
|
Ps to be transparent about this, I looked into this because I met circonus and we chatted :) It was interesting enough for me to want to take a look to see if I could help them move off a fork. Spikes are important. I get swamped with other things and I don't usually have the energy to try lots of things simultaneously. I opened this issue to log thoughts so the energy I spent might become helpful to circonus. Frankly, I am out of energy and already focusing on other things. That said, if we get a code snippet, I am interested enough to help sketch a proof of concept that Circonus might be able to use. So motivation on my end is that I am motivated to finish the exploration, but I wouldn't conflate that with a priority of a somewhat niche transport being higher than our more requested backlog :) |
|
Sorry about this, Theo. I'll close this issue as it has become a mess. I'm still keen to help, but I think it would be less distracted to explore this out of the mono repo, as it would be less distracting. |
|
From @postwait on June 7, 2016 14:28 @adriancole fqclient artifact is now published via maven http://search.maven.org/#artifactdetails%7Ccom.circonus%7Cfqclient%7C0.0.1%7Cjar A default program for zipkin thrift-encoded messages would be: "prefix:"zipkin.thrift."" The program we use to bridge fq onto scribe is here: https://github.com/circonus-labs/fq2scribe So a fully functional fq subscription to zipkin thrift data lies within: |
|
From @eirslett on June 7, 2016 16:35 Great work! I guess fq2scribe could be ported to c or go, for an additional performance boost? |
|
From @postwait on June 7, 2016 16:40 It performs okay today, but it quite Rube Goldberg. If performance is On Tue, Jun 7, 2016 at 12:35 PM, Eirik Sletteberg notifications@github.com
|
|
From @eirslett on June 7, 2016 17:8 Latency is not a big deal (1 second is fine end-to-end), but high throughput is critical. |
|
From @postwait on June 7, 2016 17:10 The avoiding scribe itself is critical as it is RPC. Integrating an
|
|
re-opening as the topic has been gratefully resurrected |
|
The code for an fqclient is here: https://github.com/circonus-labs/fq/tree/master/java The published jars are here: https://mvnrepository.com/artifact/com.circonus/fqclient/0.0.1 I'm sure work can be done to improve performance over what is there already, we primarily use C to communicate with Fq. |
|
Thanks, theo!
|
From @adriancole on May 26, 2016 21:27
Circonus are zipkin users who have had to work off forks due to the lack of a high-performance span transport.
Their library libmtev uses fq.
By adding an FQ transport, we can bring circonus and users of mtev back into the zipkin ecosystem, and no longer reliant on forks.
Copied from original issue: openzipkin/zipkin-java#252
The text was updated successfully, but these errors were encountered: