[WIP] Implement Consul Backend #767

ajvb · 2017-02-10T21:40:18Z

This is an initial attempt at a consul backend for Teleport.

It's heavily based off of the existing etcd backend, with the main difference of borrowing the TTL logic of the boltdb backend.

Initial Questions:

In the etcd backend, there is stopC and cancelC but they don't seem to do anything. What is there purpose? Is this something I should add to the Consul backend?
Can I get some guidance on how to test this? I have very little experience with Teleport, as this PR is really needed before my company can even consider Teleport.
~~I haven't used godeps in a while, how do I properly add github.com/hashicorp/consul/api to it?~~

Fixes #423

gravitational-jenkins · 2017-02-10T21:40:20Z

Can one of the admins verify this patch?

klizhentas · 2017-02-10T23:00:01Z

thanks for giving it a try. One question I have - have you considered using sessions to achieve TTL semantics instead of custom implementation?

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/consul-tool/0Y034z4mo_4/eJ-1x97JNgAJ

I haven't used sessions for TTL myself, may be someone from consul team can give us advice.

Regarding your questions:

In the etcd backend, there is stopC and cancelC but they don't seem to do anything. What is there purpose? Is this something I should add to the Consul backend?

stopC and cancelC are used in tests, I will take a closer look, may be they are obsolete though

Can I get some guidance on how to test this? I have very little experience with Teleport, as this PR is really needed before my company can even consider Teleport.

We have an acceptance test suite used for all backends here:

https://github.com/gravitational/teleport/blob/master/lib/backend/test/suite.go

See for example how we use it for Etcd:

https://github.com/gravitational/teleport/blob/master/lib/backend/etcdbk/etcd_test.go

I haven't used godeps in a while, how do I properly add github.com/hashicorp/consul/api to it?

To properly add it, in your $GOPATH (caution this command updates your workspace!!!)

# this is to align your dependencies with teleport's ones
cd teleport
godep restore

Then

# this will add new consul values
godep save ./lib/... ./tool/... ./integration/...

ekristen · 2017-02-11T00:30:15Z

I would definitely suggest you take a look at the libkv implementation of consul and realistically it might be best to just switch to using libkv for all backends and that way the community can contribute back to libkv as well.

Personally I've had a love hate relationship with consul sessions when dealing with them directly in the past. It is an interesting spin on how to expire key/value pairs vs the more traditional TTL on a specific key/value pair.

Ultimately you have to first create a session and then any key/value pairs you create you have to reference that session. That session then should either be maintained for the duration or closed out.

Note: DO NOT create a session for every key/value pair, this will overwhelm any consul cluster and it is not how the sessions are designed to work.

ajvb · 2017-02-11T00:40:16Z

@klizhentas

thanks for giving it a try. One question I have - have you considered using sessions to achieve TTL semantics instead of custom implementation?

I haven't used sessions for TTL myself, may be someone from consul team can give us advice.

I have not. I'm reaching out to my Hashicorp connection to see if I can get the relevant questions answered.

stopC and cancelC are used in tests, I will take a closer look, may be they are obsolete though

Ok awesome

We have an acceptance test suite used for all backends here:

Ok awesome, I may have further questions related to running it, but I'll give it a try.

To properly add it, in your $GOPATH (caution this command updates your workspace!!!)

Done, thanks.

@ekristen

I would definitely suggest you take a look at the libkv implementation of consul

Will do.

realistically it might be best to just switch to using libkv for all backends and that way the community can contribute back to libkv as well.

I'm not against this, but this is absolutely not my call and I cannot justify the needed time for it. I want to be able to try out Teleport at work, and simply having consul support is what is needed.

Personally I've had a love hate relationship with consul sessions when dealing with them directly in the past. It is an interesting spin on how to expire key/value pairs vs the more traditional TTL on a specific key/value pair.

Ultimately you have to first create a session and then any key/value pairs you create you have to reference that session. That session then should either be maintained for the duration or closed out.

Note: DO NOT create a session for every key/value pair, this will overwhelm any consul cluster and it is not how the sessions are designed to work.

Thank you so much for this information, I did not know all of this and Hashicorp's docs are lacking.

I will take a look at libkv's consul implementation, but I am uncertain at this time how I can use sessions for TTL in the way Teleport expects.

ekristen · 2017-02-11T00:42:42Z

FWIW @ajvb it might be easiest to just use libkv to do the consul backend in this PR, and once it is in, other backends can start to migrate to it. Just a thought.

ajvb · 2017-02-11T00:47:27Z

@ekristen Looking at libkv, I am actually becoming fond of that idea. Would definitely make this implementation a lot simpler. Let give that a shot.

klizhentas · 2017-02-11T00:48:27Z

@ekristen thanks for your concern about using sessions for TTL, looking at this:

https://github.com/docker/libkv/blob/master/store/consul/consul.go#L203
https://github.com/docker/libkv/blob/master/store/consul/consul.go#L133

So it seems that library does exactly what you are suggesting not to do.

I made several attempts at using libkv, but then backed off, to use it as a generic implementation backend mostly because I had different approaches to various backends.

klizhentas · 2017-02-11T00:51:48Z

@ajvb so here's the plan:

let's figure out whether the session per TTL approach is scalable first.
- If it's not scalable we will have to stick to manual workaround that you proposed in your PR
- If it is scalable and Consul team recommends using it, we will apply it to your PR.

ekristen · 2017-02-11T00:54:37Z

Caveat: it depends on how many keys you are doing and my experience was with Node.JS not golang, so there could be a difference in how things were implemented.

I will say that docker uses libkv with swarm and using consul as a backend and I have not seen problems with it.

When I suggest it not be done, it was definitely in reaction to doing thousands of writes/reads per minute which depending on session length can cause many many sessions to be created.

Here is some information on the session implementation hashicorp/consul#524

ekristen · 2017-02-11T00:55:47Z

Looking at the code you referenced, it looks like it will attempt to re-use an existing session should it be available before it tries to create a new one.

ekristen · 2017-02-11T00:56:28Z

One more thing to note, sorry for the multiple comments ...

I have this branch that I haven't opened a pull request on yet that enables using Consul ACL tokens
https://github.com/ekristen/libkv/tree/consul-acl-token

ajvb · 2017-02-11T02:27:06Z

@klizhentas Sounds good. Shot off an email to my Hashicorp AE (we have an enterprise account with them), so hopefully I can get answers by Tuesday or so.

klizhentas · 2017-02-11T02:29:29Z

awesome @ajvb I will be glad to assist you with testing/updating/reviewing PR, looking forward to getting it merged!

ajvb · 2017-02-13T22:25:27Z

@klizhentas Do we have numbers for "scope and scale of key/value or session usage"? What is an expected number of sessions to be created within an hour or a day for sessions-per-kvpair?

kontsevoy · 2017-02-13T23:09:01Z

@ajvb sessions are used when people SSH into boxes. So... the answer is number of employees x number of SSH sessions an average employee does per second ;)

The highest pressure point for the key/value subsystem is actually server pings. the auth server will call "upsert()" on every node in a cluster every 30 seconds AFAIK. So for a 1,000 node clusters expect 33 upserts per second.

ajvb · 2017-02-14T00:19:40Z

@kontsevoy Server pings and when a user ssh's into a server are the only times a new K/V pair is added to the backend?

klizhentas · 2017-02-14T03:49:23Z

@ajvb There are other keys stored in the DB (like certificate authorities and users) but the amount of writes is negligible compared to the scenarios mentioned by @kontsevoy

ajvb · 2017-02-14T19:12:59Z

@kontsevoy @klizhentas

What I told them:

As I understand it, the norm will be something like 10-40 KVPairs
created a second. The variation depends on the size of your Teleport deployment.

Response from Hashicorp Support:

Hi AJ,

For your use case in the Teleport backend, the answers to the two questions you
posed are as follows:

>>> When I attach a session with a TTL to a KVPair in Consul, when the TTL
>>> expires, is it deleted? 

If you choose the delete behavior when creating the session
(https://www.consul.io/docs/agent/http/session.html#session_create)
then the k/v is deleted when the TTL expires.

>>> Is it reasonable to create a session for each KVPair? Will this overwhelm consul
>>> or is this alright?

For this number of keys/sessions and even into the 100s, a 1:1 key/session mapping
will be alright and not overwhelm Consul.

👍 🎉 💃 I should have some time this week to swap my custom sessions implementation.

ajvb

@klizhentas

klizhentas · 2017-02-14T20:32:09Z

@ajvb then we have it all cleared, let me know once you are ready and I will review and test

ajvb · 2017-02-15T23:06:25Z

@klizhentas So I have a fun problem. If you go to https://www.consul.io/docs/agent/http/session.html#session_create and look at the TTL parameter, you will see this line:

If specified, it must be between 10s and 86400s currently.

The current test suite uses sessions with a TTL of 1 second and 2 second, which totally blows up.

If shorter than 10s TTL's is a requirement then we will need to go back to using the embedded TTL like BoltDB.

klizhentas · 2017-02-15T23:51:12Z

@ajvb 1 sec TTL is just for test, you can adjust for consul if that's a problem. We never really use anything less than 30 sec in prod anyways

klizhentas · 2017-04-25T19:47:29Z

@ajvb I'm going to close this due to the lack of activity. Feel free to reopen if you get a chance to work on it.

flyinprogrammer · 2017-09-17T20:00:17Z

so i know this was closed because of inactivity, but any chance we want to revisit getting this working?

AJ Bahnken added 5 commits February 10, 2017 11:31

initial consulbk

300ecec

Full initial consul backend

861dcca

examples/consul/ changes from copy of etcd's dir

af86d9e

Implemented TTL's based off boltdb backend

0170899

Missed an s/etcd/consul/

73990e0

Merge branch 'master' into ajvb/consul

739a693

Added new consul-related packages

27e5675

Merge branch 'master' into ajvb/consul

69d2c21

AJ Bahnken added 2 commits February 14, 2017 12:02

Implement real consul sessions for consul backend

56a2f09

Merge branch 'master' into ajvb/consul

9e39dcb

ajvb force-pushed the ajvb/consul branch from 7908077 to 9e39dcb Compare February 14, 2017 20:04

ajvb commented Feb 14, 2017

View reviewed changes

AJ Bahnken added 2 commits February 14, 2017 15:24

Fixed up basic errors in consul_test.go

76baad1

Use a newer version of consul

84f254c

Added support for 'forever'

079fbeb

klizhentas closed this Apr 25, 2017

hatched pushed a commit to hatched/teleport-merge that referenced this pull request Nov 30, 2022

Pin local shell to the specified cluster (gravitational#767)

5606eec

hatched pushed a commit that referenced this pull request Dec 20, 2022

Pin local shell to the specified cluster (#767)

c367059

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Implement Consul Backend #767

[WIP] Implement Consul Backend #767

ajvb commented Feb 10, 2017 •

edited

Loading

gravitational-jenkins commented Feb 10, 2017

klizhentas commented Feb 10, 2017 •

edited

Loading

ekristen commented Feb 11, 2017

ajvb commented Feb 11, 2017

ekristen commented Feb 11, 2017

ajvb commented Feb 11, 2017

klizhentas commented Feb 11, 2017

klizhentas commented Feb 11, 2017 •

edited

Loading

ekristen commented Feb 11, 2017

ekristen commented Feb 11, 2017

ekristen commented Feb 11, 2017

ajvb commented Feb 11, 2017

klizhentas commented Feb 11, 2017 •

edited

Loading

ajvb commented Feb 13, 2017

kontsevoy commented Feb 13, 2017

ajvb commented Feb 14, 2017

klizhentas commented Feb 14, 2017

ajvb commented Feb 14, 2017

ajvb left a comment •

edited

Loading

klizhentas commented Feb 14, 2017

ajvb commented Feb 15, 2017 •

edited

Loading

klizhentas commented Feb 15, 2017 •

edited

Loading

klizhentas commented Apr 25, 2017

flyinprogrammer commented Sep 17, 2017

[WIP] Implement Consul Backend #767

[WIP] Implement Consul Backend #767

Conversation

ajvb commented Feb 10, 2017 • edited Loading

gravitational-jenkins commented Feb 10, 2017

klizhentas commented Feb 10, 2017 • edited Loading

ekristen commented Feb 11, 2017

ajvb commented Feb 11, 2017

ekristen commented Feb 11, 2017

ajvb commented Feb 11, 2017

klizhentas commented Feb 11, 2017

klizhentas commented Feb 11, 2017 • edited Loading

ekristen commented Feb 11, 2017

ekristen commented Feb 11, 2017

ekristen commented Feb 11, 2017

ajvb commented Feb 11, 2017

klizhentas commented Feb 11, 2017 • edited Loading

ajvb commented Feb 13, 2017

kontsevoy commented Feb 13, 2017

ajvb commented Feb 14, 2017

klizhentas commented Feb 14, 2017

ajvb commented Feb 14, 2017

ajvb left a comment • edited Loading

Choose a reason for hiding this comment

klizhentas commented Feb 14, 2017

ajvb commented Feb 15, 2017 • edited Loading

klizhentas commented Feb 15, 2017 • edited Loading

klizhentas commented Apr 25, 2017

flyinprogrammer commented Sep 17, 2017

ajvb commented Feb 10, 2017 •

edited

Loading

klizhentas commented Feb 10, 2017 •

edited

Loading

klizhentas commented Feb 11, 2017 •

edited

Loading

klizhentas commented Feb 11, 2017 •

edited

Loading

ajvb left a comment •

edited

Loading

ajvb commented Feb 15, 2017 •

edited

Loading

klizhentas commented Feb 15, 2017 •

edited

Loading