-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Vsphere module with metrics. WIP #4028
Conversation
Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run. |
1 similar comment
Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run. |
@amandahla Thanks a lot for kicking off this contribution. Let me know when we should do a first review. Be aware that we recently switched to govendor to handle the dependencies and we try to keep the dependencies for a module in a vendor directory local to the module. |
Looks like you are off to a good start. So would Metricbeat run directly on each ESXi host or would it run on the vCenter host? |
Hello @andrewkroh, actually Metricbeat can collect from any Vmware SDK URL (ESXi/VCenter). |
Hi @ruflin . I made some modifications and I think that now it's ready for a first review. Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot for the addition. I left a few minor comments.
- Could you add a CHANGELOG entry?
- Could you add a short readme on how to test it? Similar to what I did here for the kubelete module: https://github.com/elastic/beats/tree/master/metricbeat/module/kubelet/_meta/test as vsphere is also not that straight forward to test as the other modules.
@@ -0,0 +1,10 @@ | |||
#- module: vsphere |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Be aware that we change the logic here a bit in #4049
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✅
metricbeat/include/list.go
Outdated
@@ -85,6 +85,10 @@ import ( | |||
_ "github.com/elastic/beats/metricbeat/module/system/network" | |||
_ "github.com/elastic/beats/metricbeat/module/system/process" | |||
_ "github.com/elastic/beats/metricbeat/module/system/socket" | |||
_ "github.com/elastic/beats/metricbeat/module/vsphere" | |||
_ "github.com/elastic/beats/metricbeat/module/vsphere/datastore_usage" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need the appendix usage
for the metricset names? Are there potential other metricsets for datastore
for example?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually thinking about it...No, we don't need. I removed now.
✅
@@ -0,0 +1,4 @@ | |||
== vsphere Module | |||
|
|||
This is the vsphere Module. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you add some more details here, for example link to vSphere (installation) page or docs? Also did you test with a specific version?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✅
}, | ||
"vsphere":{ | ||
"datastore_usage":{ | ||
"example": "datastore_usage" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needs update with an example event. You can run go test -tags=integration github.com/elastic/beats/metricbeat/module/vsphere/... -data
when your environment is running.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll try to use this in this case:
http://www.virtuallyghetto.com/2017/04/govcsim-neat-incubation-project-vcenter-server-esxi-api-based-simulator.html
type: long | ||
description: > | ||
Used percent of the datastore | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can add format: percent
here for Kibana.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✅
"testing" | ||
) | ||
|
||
func TestFetchEventContents(t *testing.T) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Test missing
@@ -0,0 +1,97 @@ | |||
{ | |||
"comment": "", | |||
"ignore": "test", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would expect here "ignore": "test github.com/elastic/beats"
as otherwise if you remove the libs from the vendor dir and then run govendor sync
it will also load libbeat etc. into it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✅
}, | ||
"vsphere":{ | ||
"virtualmachine_usage":{ | ||
"example": "virtualmachine_usage" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
needs update
"memory": common.MapStr{ | ||
"used": common.MapStr{ | ||
"guest": common.MapStr{ | ||
"bytes": vm.Summary.QuickStats.GuestMemoryUsage * 1024, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What are the raw values of all these * 1024
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
vm.Summary.QuickStats.GuestMemoryUsage, vm.Summary.QuickStats.HostMemoryUsage and vm.Summary.Config.MemorySizeMB are values in MB so I had to multiply all for 1024 to get bytes values.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1024*1024?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
😳
) | ||
|
||
func TestFetchEventContents(t *testing.T) { | ||
assert.EqualValues(t, 1, 1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needs update
Hi @ruflin . I've made all the changes that you asked but now I don't know why the Travis build is failing. It's something related to filebeat testsuit in OSX nothing regardless this module. :-) |
@amandahla Great. I restarted the build manually and will have a look at the PR soon. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Two requests which can also happen after merging:
- Create the
data.json
with some example events - Add Dashboards
}, | ||
"vsphere":{ | ||
"datastore":{ | ||
"example": "datastore" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be nice to have here a full example data set. This can happen in a follow up PR.
}, | ||
"vsphere":{ | ||
"host":{ | ||
"example": "host" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Example doc here would be nice.
jenkins, test it |
@amandahla Thanks a lot for the contribution. |
Thanks! :-) |
@amandahla If you have some time to open a new PR with the |
@ruflin Sure, I'm already working on it! 👍 |
🎉 🎉 🎉 🎉 |
I get a Bad request using this module. I am using the same credentials that I am using with Chef Knife Vsphere module. |
@petericebear I saw that Knife uses "knife[:vsphere_host] = "vcenter-hostname"" |
I now have this: metricsets: ["virtualmachine"] with insecure to true |
And with this setting: I get the error: |
Please try |
This was the fix :) Thank you for your swift reply |
So it worked fine now? Did you just need to put get_custom_fields: false? |
Only some bytes in the data are negative. But I think I saw a PR for that? I added the /sdk to the url, and I changed the metricsets from: to:
Running on OSX (Latest) |
Yes, this PR solved that |
Are you working on the functionality of disk usage and network usage for virtual machines? |
I found out that Performance Manager can retrieve stats by level and key like cpu.usage.average. If this a possibility for a beats module for given the keys to retrieve? That way we can have more data out of vSphere into Elasticsearch. I saw that the data changed with this module is changed every 60 seconds so the 10second interval is to low as default. If you have another solution for using the Performance Manager for making similar graphs in Kibana as in vCenter I would be really grateful. If you need more information. Please contact me, very happy to help |
@petericebear Hi, thanks for your suggestion. I tried to use this but I couldn't find a way to work (https://github.com/vmware/govmomi/blob/master/performance/manager.go). I found it too difficult since we need for now just a few things. And I'm not sure, but I think that using Performance Manager we don't collect realtime metrics, just historical. |
Hi, I installed vspher beat from elastic 6 beta. But I can't see any sample dashboard for this beat. Dose it come with any dashboard? |
Hi. This is a initial version for Vsphere module, just collecting datastore usage metrics.
We used Govmomi library and the example in:
https://github.com/vmware/govmomi/blob/master/examples/datastores/main.go
TODO: