Metrics "Sidecar" (scraping container) support

[jeefy]

WIP

This is the Dashboard side for using the metrics server. Fixes #2986

• Functional Support (through the additional container)
• Finish rebasing to "v2.0.0-alpha"
• Official Repo for the scraping container (now at https://github.com/kubernetes-sigs/dashboard-metrics-scraper)
• Docker Hub dev registry for the scraping container
• Update head deploy scripts to include scraping container

[jeefy]

/hold

[jeefy]

/approve cancel

[codecov]

Codecov Report

Merging #3504 into master will increase coverage by 0.7%.
The diff coverage is 68.35%.

@@            Coverage Diff            @@
##           master    #3504     +/-   ##
=========================================
+ Coverage   46.57%   47.28%   +0.7%     
=========================================
  Files         173      181      +8     
  Lines        8066     8389    +323     
  Branches       59       71     +12     
=========================================
+ Hits         3757     3967    +210     
- Misses       4079     4178     +99     
- Partials      230      244     +14

Impacted Files	Coverage Δ
.../app/backend/integration/metric/heapster/client.go	74.38% <ø> (ø)	⬆️
src/app/backend/integration/metric/manager.go	50% <0%> (-8.7%)	⬇️
...p/backend/integration/metric/sidecar/restclient.go	0% <0%> (ø)
...c/app/backend/integration/metric/sidecar/common.go	100% <100%> (ø)
...app/backend/integration/metric/sidecar/selector.go	70.31% <70.31%> (ø)
...c/app/backend/integration/metric/sidecar/client.go	74.38% <74.38%> (ø)
...rc/app/backend/integration/metric/sidecar/model.go	77.77% <77.77%> (ø)
...p/backend/integration/metric/common/aggregation.go	89.09% <0%> (-1.82%)	⬇️
src/app/backend/handler/filter.go	45.45% <0%> (-1.39%)	⬇️
src/app/frontend/overview/helper.ts	89.47% <0%> (-0.53%)	⬇️
... and 31 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f661b8a...bbbbc84. Read the comment docs.

[jeefy]

I'm not going to block on the gcr.io repo at this point.

At this point I'm good moving this into master, tackling any emergent bugs, and getting people testing both.

To test, people will have to use aio/deploy/recommended/kubernetes-dashboard-head.yaml -- However, head will test both head of Dashboard, as well as head of metrics-scraper.

🎉

/hold cancel
/assign @maciaszczykm @floreks

[jeefy]

/hold cancel (I can't prow sometimes)

[jeefy]

@maciaszczykm @floreks I think it's time for a review.

In this PR I also added a "local" deployment YAML. For reference, here's how I just tested it.

minikube start
minikube addons list # ensure metrics-server addon is enabled
eval $(minikube docker-env)
npm run docker:build:head
kubectl apply -f aio/deploy/recommended/kubernetes-dashboard-local.yaml
kubectl proxy

And then viewing the dashboard via kubectl proxy. Let me know if there's anything else needed to dive into this.

[maciaszczykm]

/approve

Let's wait for @floreks with LGTM.

[floreks]

I wouldn't add this local deployment file to our aio/deploy directory as it should contain official deployments. From my understanding, this deployment will only work if we first build an image locally and then try to run it.

If we want to keep it then I'd move it maybe to a test-resources dir, rename it to something like metrics-test-deployment.yaml and also rename dashboard deployment inside the file so it does not collide with the official head deployment.

@jeefy @maciaszczykm WDYT?

[maciaszczykm]

@floreks SGTM.

[jeefy]

Sounds good, I'll update the PR later this morning.

[pires]

@jeefy I'm here rooting for you. What a champ you have been with this PR 🙇

[pierluigilenoci]

@ianlewis @PeWu please take a look 😘

[maciaszczykm]

LGTM, cc @floreks.

[floreks]

Let's get this in! 🍾

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: floreks, maciaszczykm

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [floreks,maciaszczykm]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jeefy]

Thanks for the support y'all!

[eloyekunle]

Thanks @jeefy

[jseguillon]

Wow @jeefy you're the boss ! Thanks !

[RainingNight]

Good!

[tdmalone]

Thanks so much for this @jeefy! Just to clarify, once a new version is cut (although I note it has been last year since this happened...), this means heapster won't be required anymore and Dashboard will be able to display metrics from metrics-server?

[jeefy]

@tdmalone Correct. Dashboard will (now) deploy two containers, the dashboard itself and the Dashboard Metrics Scraper. Since metrics-server only shows a single moment of time in the cluster, the scraper is needed to poll/store the metrics for look-up. The next thing to tackle is being able to point the Dashboard at Prometheus, but that'll be easier to do after the efforts of our GSOC students (Yay!)

Also, we should be cutting an RC at the end of this month, but you can (technically) spin up the Dashboard off the master branch using the kubernetes-dashboard-head.yaml deployment.

Just keep in mind, that's literally a build off master, not an actual release, there be dragons there, etc etc. 😄

[varuzam]

I run the kubernetes dashboard with the manifest https://gist.githubusercontent.com/varuzam/d624547b4286dd11206e1fc0a442f78d/raw/579905246348002b7b9895c60b3ae996de3f664a/kube-dashboard.yaml
The dashboard can not get metrics from the scrapper. Here is logs of the scrapper:

{"level":"info","msg":"Database updated: 2 nodes, 7 pods","time":"2019-06-18T19:05:06Z"}
{"level":"info","msg":"URL: /","time":"2019-06-18T19:05:14Z"}
{"level":"info","msg":"URL: /","time":"2019-06-18T19:05:24Z"}
{"level":"info","msg":"URL: /api/v1/api/v1/dashboard/namespaces/kube-system/pod-list/kubernetes-dashboard-86df9bb89b-l2226,metrics-server-54fddf7759-t8nnr,coredns-58b4966dd4-5gjxx,coredns-58b4966dd4-p4cn6/metrics/cpu/usage_rate","time":"2019-06-18T19:05:30Z"}

It looks like the dashboard uses wrong url with doubled '/api/v1'

[floreks]

@varuzam how did you run the scraper? In or outside of the cluster? It does work when both Dashboard and the scraper are running inside. There is an issue if I am trying to access it using remote option.

@jeefy

[varuzam]

@floreks I run the scraper in the one pod with the dashboard inside a cluster.
My deployment - https://gist.githubusercontent.com/varuzam/d624547b4286dd11206e1fc0a442f78d/raw/579905246348002b7b9895c60b3ae996de3f664a/kube-dashboard.yaml