ElasticSearch 5 support

[AlmogBaku]

Solves #1410

This PR finally add the support for ElasticSearch 5 (with backward compatibility for ES2)

[k8s-reviewable]

This change is 

[AlmogBaku]

/assign @AlmogBaku @andyxning @huangyuqi

[AlmogBaku]

@rikatz can you help us out with reviewing this?

[rikatz]

@AlmogBaku sure. Will review tomorrow :)

[AlmogBaku]

I added a support for Ignest pipeline.

[rikatz]

Sorry @AlmogBaku I've been pretty busy in work. Have seen that this worked now. Will try to test the code tomorrow and give you a feedback :)

[AlmogBaku]

@rikatz any update?

[rikatz]

@AlmogBaku sorry. I'll be testing this today and will return you in the end of this day!

[rikatz]

@AlmogBaku while running heapster with elasticsearch 5, it gaves me the following error:

E0510 15:30:33.101740 12777 factory.go:84] Failed to create sink: Failed to create ElasticSearch client: Failed to an ElasticSearch Client: no Elasticsearch node available

I'm using the following directive: --sink="elasticsearch:?nodes=http://ipaddress:9200&index=testEvent&ver=5"

The Elasticsearch is available and answering:

curl http://ipaddress:9200
{
  "name" : "Pz3qaoM",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "BUd5OBEGTpCIE6Xak5qglA",
  "version" : {
    "number" : "5.4.0",
    "build_hash" : "780f8c4",
    "build_date" : "2017-04-28T17:43:27.229Z",
    "build_snapshot" : false,
    "lucene_version" : "6.5.0"
  },
  "tagline" : "You Know, for Search"
}

Am I missing something?

[rikatz]

@AlmogBaku some more info:

I'm running a standalone Elasticsearch, and started it now with 'sniff=false' (as in docs). Running with ver, disabling health check, all have the same result.

The following is a dump from communication between Heapster and Elastic:

HEAD / HTTP/1.1
Host: 10.X.X.X:9200
User-Agent: Go-http-client/1.1

HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 327

GET /_nodes/http HTTP/1.1
Host: 10.X.X.X:9200
User-Agent: elastic/5.0.36 (linux-amd64)
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-encoding: gzip
transfer-encoding: chunked

[AlmogBaku]

@rikatz

1. please double check your config
2. please send me logs with verbose enabled (as described here https://github.com/kubernetes/heapster/blob/master/docs/debugging.md )
3. do you use usr/pwd?

[AlmogBaku]

@rikatz ?

[chancez]

@AlmogBaku I for what it's worth, I got this working just fine. I had to make a small addition to the PR to add an option to disable SSL verification on the http.Client since I'm using a self signed TLS cert for ES (using the elasticsearch-operator) and didn't bother with getting the certs correctly mounted in the eventer container, but otherwise it works great.

Here's the manifest I'm using: https://gist.github.com/e725dea50300bafc781382546bfe6f2b

The additional code is just this:

httpClient := &http.Client{}
if len(opts["verifySSL"]) > 0 {
	verifySSL, err := strconv.ParseBool(opts["verifySSL"][0])
	if err != nil {
		return nil, errors.New("Failed to parse URL's verifySSL value into a bool")
	}
	httpClient.Transport = &http.Transport{
		TLSClientConfig: &tls.Config{
			InsecureSkipVerify: !verifySSL,
		},
	}
}
startupFnsV2 = append(startupFnsV2, elastic2.SetHttpClient(httpClient))
startupFnsV5 = append(startupFnsV5, elastic5.SetHttpClient(httpClient))

[AlmogBaku]

@chancez I guess that's belong to a different PR..

[chancez]

@AlmogBaku Yeah, I'm waiting for this one to get merged and I'll open a PR for it. Mostly wanted to say this is working great perfectly for me.

[rikatz]

Sorry!!! I've been pretty busy at work :( will test again on monday and will let you know! Ricardo P. Katz

…

Em 20 de mai de 2017, às 14:22, Almog Baku ***@***.***> escreveu: @rikatz ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[rikatz]

@chancez Can you please quickly write how you started ES5 and Heapster?

Thanks!

[chancez]

@rikatz I'm using https://github.com/upmc-enterprises/elasticsearch-operator and the manifest I'm using is in the gist above. Just know I've made a minor modification to this code to work with my elasticsearch, as mentioned, so I can disable SSL verification.

[rikatz]

OK. I'm testing this again now and will let you know

[rikatz]

@AlmogBaku tested here in ES5 and it worked really fine :)

LGTM!

Thanks and sorry about delaying this so much :(

[rikatz]

I was just wondering here if it wouldn't be better to have a 'Generic Timestamp' (like a @timestamp) for each metric, as this breaks each Timestamp into CpuMetricsTimestamp, MemoryMetricsTimestamp and this, in kibana makes a lot difficult to create dashboards (as each metric have it's own timestamp and not a generic one).

Also, after pushing some metrics I've faced some 'Failed to push data to sink: ElasticSearch Sink' but don't know why. Heapster is pushing metrics anyway, so probably is something specific to some object inside my cluster.

[rikatz]

@AlmogBaku other 'review'. While creating a dashboard in Grafana, I was trying to aggregate metrics per namespace and per pod-name.

When using MetricsTags.pod_id it worked fine, but when using MetricsTags.pod_name I've faced the error related to fielddata

Don't know if this is fine for now, but probably we need a better field mapping in this Sink, as those searches/aggregations could be used :)

Anything I can help with?

[SupreethKadalur]

Hello @AlmogBaku , @rikatz - Please let us know if ES5 PR is merged and ready for use. I have the same problem and unable to send data from heapster v 1.3.0 to ES5

@rikatz - I totally agree that we should have @timestamp for each metrics :-) this will solve many problems I have while creating dashboards in kibana.

[DirectXMan12]

Alright, squashed into three commits (Godeps, ES5, ingest). Should be good to merge once it passes the tests.

[googlebot]

So there's good news and bad news.

👍 The good news is that everyone that needs to sign a CLA (the pull request submitter and all commit authors) have done so. Everything is all good there.

😕 The bad news is that it appears that one or more commits were authored by someone other than the pull request submitter. We need to confirm that they're okay with their commits being contributed to this project. Please have them confirm that here in the pull request.

Note to project maintainer: This is a terminal state, meaning the cla/google commit status will not change from this state. It's up to you to confirm consent of the commit author(s) and merge this pull request when appropriate.

[DirectXMan12]

/retest

[DirectXMan12]

looks like a flake unrelated to this PR

/retest

[AlmogBaku]

/retest

[DirectXMan12]

looks good, merging

[rikatz]

@AlmogBaku thanks!! :)

[DirectXMan12]

Thanks everyone for all their hard work on this :-)

[AlmogBaku]

Hallelujah!

On Fri, Jun 23, 2017 at 9:51 PM Solly Ross ***@***.***> wrote: Thanks everyone for all their hard work on this :-) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1607 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGCpoaNcw6wBGcbA2uZMuRjeWtjAbafks5sHAkegaJpZM4NFmG0> .

-- <http://www.rimoto.net/> www.rimoto.com <http://www.rimoto.net/> *Almog Baku* *CTO & Cofounder * Mobile: +972.50.2288.744 Social: * <http://www.facebook.com/AlmogBaku> <http://www.linkedin.com/in/almogbaku>*

[SupreethKadalur]

@DirectXMan12 , @AlmogBaku - I see that ES5 fix is merged and Heapster v1.4.0-beta.0 is created. I'm now using docker image "gcr.io/google_containers/heapster-amd64:v1.4.0-beta.0" but still see the same problem I had with heapster-amd64:v1.3.0 image. Am I missing something here? Please suggest. Thanks

[chancez]

@SupreethKadalur This got merged after that pre-relase, you'll have to wait for another pre-release or build it yourself.

[SupreethKadalur]

Thank you all for the support and fix. Now ES5 issue is fixed with heapster v1.4.0

[piosz]

Yes, this is included in 1.4.0 as described in https://github.com/kubernetes/heapster/releases/tag/v1.4.0