vSphere Provider: Mapping out the Next Steps

[phinze]

Wanted to kick off a higher level discussion of what needs to be done on the vSphere provider and in what order.

• What are the important missing resources?
• Are there any enhancements that need to be made to the existing functionality?
• What do we need to do to ensure the provider works with all common versions of vSphere in the wild?

Pinging @tkak and @mkuzmin to chime in as well as anybody else with interest/knowledge in the community.

[v1gnesh]

• To handle the NIC problem, the code could be updated to drop the NIC(s) from the template (if any) and do its usual business of creating NIC(s). This can be introduced as an optional parameter so that those with no NIC(s) in their template won't be affected
• Ability to build using a CustSpec that's available in vCenter (with the option of overriding specific parameters if needed)
• Steps to format the vmdk and make it accessible within the guest OS (guest scripts). I think that the only returnable value when initiating a program within the guest is the process ID, so success/failure must be dealth with within the initiated script itself and shouldn't be tracked by the vSphere provider

[mkuzmin]

@v1gnesh NIC issue is discussed in #3559.
For other two items - create separate issues please.

[mkuzmin]

Indeed, we need to discuss many things, but at the moment the plugin is broken, and I wasn't able to launch any VMs yet. #3537 is a show-stopper issue: until it's fixed I cannot finish my testing, so cannot talk about future tasks.
@tkak, can we expect some support from you?

[tkak]

• Add flexible parameters support in acceptance test with environment value like openstack provider
• Windows template support
• Memory reservation parameter support
• Test VMware vSphere 6
• Add update feature for changing a number of CPU, memory size, network interfaces and disks

[chrislovecnm]

@tkak once @phinze gives me more info I will be running vSphere 6 testing. Also you noticed that govmomi is supporting 6.0 now ... vmware/govmomi#356

@phinze what is the minimal viable components that a provider is to support? Nics, Disk, CPU, ram, etc...

[phinze]

Okay, got some basic acceptance test docs over in #3683. We got @tkak's crasher fix in #3636 merged. There's a candidate fix for #3559 in review over at #3652.

what is the minimal viable components that a provider is to support?

Because each provider is so different, it's difficult to define much in the way of standards across providers. Generally it's up to the community using the provider to help guide the priorities, as they are the ones who understand best which features are most crucial to have.

[sbarie]

Support for Linked Clones would be a great addition!

[chrislovecnm]

So here is the tally so far:

• Add flexible parameters support in acceptance test with environment value like openstack provider
• Windows template support
• Memory reservation parameter support
• Test VMware vSphere 6
• Add update feature for changing a number of CPU, memory size, network interfaces and disks
• Support for different vmdk disk types: thin, thick, eager zeroed thick, etc
• better test support
• linked clones
• Ability to build using a CustSpec that's available in vCenter (with the option of overriding specific parameters if needed)
• Steps to format the vmdk and make it accessible within the guest OS (guest scripts). I think that the only returnable value when initiating a program within the guest is the process ID, so success/failure must be dealth with within the initiated script itself and shouldn't be tracked by the vSphere provider
• add config value to handle 'insecure' SSL connections. For example when vSphere url is using a self-signed certificate.

What about networking? For instance: vlans, ports groups, ip pools, virtual switches?

@tkak / @phinze do you want to add priorities? I am thinking we can divide and conquer a bit. Also in your opinion do we have a minimal viable product, or do we have some more work to do? I think we are really close ...

Please folks we need to vote on the P1's in this group!!

Also @tkak we appreciate you getting this code in and doing the work that you have done!! どうもありがとうございます

[chrislovecnm]

Question for the group. Who a google hangout next week be good to review this stuff? We have a bunch of different TZ's here, but I would probably recommend working it around @tkak's schedule

[phinze]

@chrislovecnm Thanks for the summary!

As I am not a vSphere user, I can't really provide any input as to which features are most important to do first, nor can I be of much help in determining what defines a "minimum viable product" here.

I can say that the SSL connection issue in #3521 is important for security and an easy win for whoever wants to pick it up.

Re: Hangout - I'm not sure a synchronous meeting is really necessary at this point. It seems to me that anything we'd video chat about could be easily handled using GitHub as we've been doing.

Rather than individual feature requests - what we could really use from a high level is for the folks with an understanding of the vSphere domain model to propose potential Terraform resources and the backing APIs that would support those resources.

I can provide an equivalent example from AWS, and perhaps a vSphere expert can chime in using this pattern.

In AWS we have:

• aws_instance - a resource representing an instance in EC2
  • Created with an EC2 RunInstances API call
  • Read with EC2 DescribeInstances
  • Updated via calls to EC2 ModifyInstanceAttribute as well as a few other specific calls.
  • Deleted by calling EC2 TerminateInstances
• aws_ebs_volume - represents a block storage volume
  • Created with a EC2 CreateVolume API call
  • Read with EC2 DescribeVolumes
  • Currently can only update its tags via {Create,Update,Delete}Tags API calls, other modifications require replacement
  • Deleted by calling EC2 DeleteVolume
• aws_ebs_volume_attachment - associates a volume with an instance
  • Created with AttachVolume
  • Read from the details of DescribeVolumes
  • Removed with DetachVolume

Currently in vSphere we have:

• vsphere_virtual_machine - represents a running VM
  • Created via DeployVM if a template is provided or CreateVM if not?
  • Deleted with PowerOff followed by Destroy

So - what else could we use in vSphere?

@tkak - As I dig through the code looking for models - I see that there are a lot of functions named create* being referenced that are really building up data structures which will end up being shoved into the CreateVM/ DeployVM API calls. It might be nice to flop those to use build instead of create to help distinguish the fact that they are not calling any API methods.

[tkak]

I'm sorry for the late response. I tried to list potential Terraform resources for VMware vSphere. But, I'm not sure how much demand is there in the resources except for vsphere_virtual_machine. Because I think the resources are used by the VMware vSphere admins, not users. @mkuzmin and @chrislovecnm, what do you think?

• vsphere_virtual_machine - represents a running VM
  • Created via DeployVM if a template is provided or CreateVM if not?
  • Deleted with PowerOff followed by Destroy
• vsphere_datacenter - represents a datacenter with https://github.com/vmware/govmomi/tree/master/govc/datacenter
  • Created with CreateDatacenter
  • Deleted with Destroy
• vsphere_datastore - represents a datacenter
  • Created? - There is no methods in govmomi now.
  • Deleted? - There is no methods in govmomi now.
• vsphere_cluster - represents a cluster compute resource with https://github.com/vmware/govmomi/tree/master/govc/cluster
  • Created with CreateCluster
  • Deleted with Destroy
• vsphere_resource_pool - represents a resource pool with https://github.com/vmware/govmomi/tree/master/govc/pool
  • Created with Create
  • Deleted with Destroy
• vsphere_virtual_switch - represents a virtual switch with https://github.com/vmware/govmomi/tree/master/govc/dvs
  • Created with CreateDVS
  • Deleted with Destroy
• vsphere_virtual_portgroup - represents a virtual port group with https://github.com/vmware/govmomi/blob/master/govc/dvs/portgroup/add.go
  • Created with AddPortgroup
  • Deleted? - There is no methods in govmomi now.

@tkak - As I dig through the code looking for models - I see that there are a lot of functions named create* being referenced that are really building up data structures which will end up being shoved into the CreateVM/ DeployVM API calls. It might be nice to flop those to use build instead of create to help distinguish the fact that they are not calling any API methods.

OK. I'll fix that. 👍

[chrislovecnm]

@tkak awesome! We also may want to look at the ability to use govmomi to exec commands on the vm's as well.

[talarczykco]

Admittedly low priority, but the ability to set annotations on the VM in vsphere_virtual_machine would be nice. I have minimal Go-fu but a strong desire to learn; feel free to contact me if I can help.

[chrislovecnm]

@mijit help is ALWAYS appreciated!!! Email is on the way

[chrislovecnm]

A separate conversation with @mijit he mentioned:

• multiple data centers
• multiple VLANS/dSwitches

@mijit can you provide more details?

[ringods]

@chrislovecnm you can always @-mention me if you need testing of new Terraform vSphere resources. I have a vSphere 6 single machine at home and wanting to get this Terraform managed.

[matt-deboer]

Any interest in adding cloud-init capability? I looked at this only briefly, and it seems that people doing this on esxi have to do this via config-drive in an ISO. The idea would be to allow a user_data string attribute on the vm, similar to the AWS provider (and others).
Is anybody already doing this, or have some other route to leverage cloud-init on vsphere (maybe deltacloud)?

[Preskton]

@tkak @mkuzmin @chrislovecnm Windows template support (#5490) is currently critical on my list. I have folks willing to poke at it to try to get basic support going to enable a workload we need to deploy. I'd like to see who, if anyone, has ideas/guidance/support for what needs to be done to get it working.

[aheeren]

@Preskton This works on my machine, looking for feedback to get it ready for prime-time:
#6087

[Preskton]

@aheeren We're cloning & pulling it in now. Will update you later today. Looks like great stuff - thanks for hopping in the discussion and letting us know what you've been up to! cc: @rossedman

[chrislovecnm]

@ringods you mind giving my branch a spin? I tweaked the WaitForIp call, hopefully fixing the timing problem with getting ip addresses. I modified @tkak's PR, because I thought we needed the call sooner.

It is on this branch. I can submit as a PR as well. https://github.com/chrislovecnm/terraform/tree/vsphere-ip-wait-issue

If any other go gurus can comment on moving the WaitForIp sooner in the function call, please pipe in.

[ringods]

@chrislovecnm maybe I can find some time coming weekend. Will let you know the outcome.

[rossedman]

@chrislovecnm @phinze @mkuzmin Hey guys, @aheeren branch is working really well. I know this doesn't add new resources but this is a pretty serious outstanding bug. Would someone mind reviewing this? We have tested it on vSphere 5.5 and vSphere 6.0.

Also, I have vSphere 6.0 at home if you need me to test some things

[Preskton]

@chrislovecnm @phinze @mkuzmin /second @rossedman's request above. Having good success with @aheeren's changes and would like to see these reviewed asap so we can then get it merged in.

cc @burzinpatel - this is the PR we have been discussing over email.

[chrislovecnm]

@rossedman might you test drive https://github.com/chrislovecnm/terraform/tree/vsphere-ip-wait-issue for me

I have not done a PR cause we may kill another PR that is in ;) I don't have a test bed and won't put in a PR that has not been smoked tested.

[vipulsabhaya]

@chrislovecnm Is there any interest in supporting NSX in Terraform?

[chrislovecnm]

@vipulsabhaya do not know. At the point we are mostly concentrating on vsphere. Does NSX have a go API?

[vipulsabhaya]

@chrislovecnm Not that I have seen, NSX has a REST api, so it may need to be added to govmomi or similar

[chrislovecnm]

@vipulsabhaya a supporting API is the first step. You many want to put in a request to the govmomi maintainers, if this support is of interest to you. If you develop you may want to work on that support. Thanks Chris

[rossedman]

@vipulsabhaya I would like to see NSX. There is definitely a need there but would love to see more vSphere resources covered first.

[dagnello]

@chrislovecnm starting to work towards adding file resource, I have created the following issue (#6366) for tracking.

[chrislovecnm]

@rossedman we are of like minds.

It may be till next week before I have a chance to create a roadmap issue. Busy busy 😄

[dkalleg]

Not sure if this is the right forum for this question.. pl point me elsewhere if needed. Regarding vm update to add a disk (I'm hoping some vmware folks can chime in here):

From vSphere I can add a disk to a running vm, but I cannot see that disk from within the vm; fdisk -l doesn't show it and nothing new under /dev/sd*. On the vm there are various ways to rescan the SCSI controller for new disks, but does anybody know a way to handle hot-attach without having to run commands on the guest OS? For now I'm find focusing on Ubuntu. I'm told in IRC that RHEL is hot-attaching disks, not sure how.

I can think of 3 potential solutions:

1. Some vm configuration parameter to enable hot-plugging that I don't know aobut
2. some govmomi api calls to do scsi rescan I don't know about
3. document for users that they need to do a remote-exec to rescan scsi when adding a new disk.

Thoughts?

[chrislovecnm]

@mixacha you seemed to have done disk stuff before. Can you comment on @dkalleg's question??

@dkalleg I think we need to assume that a remote-exec needs to be run, and probably a reboot. At the least they have to format or add the disk to lvm for instance. What @mixacha mentioned is that hot add functionality is often not supported because it is a performance hit. I recommend that we proceed assuming that hot add has been turned off.

The following is a pretty good write up about adding a disk hot: http://ryandlane.com/blog/2010/03/12/adding-new-hard-drives-live-on-an-ubuntu-guest-with-vmware-esx/

But again a lot of this depends on OS and settings at an esx level.

[dkalleg]

After some internal discussions, it sounds like leaving the scsi rescan up to the user (whether via remote-exec or other means) is the preferable option. This is a guest OS level operation, which Terraform should be agnostic to. Maybe we can add a parameter to the vm resources disk definition like hot-add which would determine whether we force a guest OS reboot on new disk update or not. Thoughts?

[chrislovecnm]

@dkalleg I would say that we should default to reboot, and have an override functionality. Not sure how the other providers handle it, but would model them. I know for instance to update sizing on a gce instance, you have to power it down first.

[dkalleg]

@chrislovecnm @phinze @tkak et al. Are there plans to move up our govmomi version? I've learned that there is a fix/feature thats come in since 0.3.0 that I'd be interested in taking advantage of. In short, with 0.3.0, we cannot detach a disk from a vm without destroying that disk. I'm sure theres other improvements/features too.

[chrislovecnm]

@dkalleg yes we need the latest for another bug 👍

[dkalleg]

@chrislovecnm Is anyone working on it? Do we have an eta?

[chrislovecnm]

@thetuxkeeper actually is having some issues with it. I cc'ed you on another issue. If you guys want to figure it out and get a PR in ... we can try to expedite it

[chrislovecnm]

@thetuxkeeper can you try to bump the version of govmomi again? I am getting an error doing it ;(

[chrislovecnm]

@thetuxkeeper @dkalleg the version bump is not trivial :( We have breaking changes ... working on it. Guys let me know if you want to cut a separate PR on this or if I need to focus on it.

Thanks

[chrislovecnm]

@thetuxkeeper @dkalleg @markpeek working on getting the version bump

[chrislovecnm]

Roadmap started ... please comment!!

Issue: #6565

[chrislovecnm]

Hey all need some reviews please, documentation improvments.

#6661

@markpeek pretty please.

[chrislovecnm]

If anyone has DRS and a few minutes to test the new DRS code, please holler!!

Code is here: #7031

Initial support for VMs. Disks, Files, and CDs are not yet supported.

[ringods]

And to make good progress on all of this: proper test automation.
@chrislovecnm can you have a look at my findings in #7973 to see if this helps in testing the terraform plugin against a real vsphere?

[mrjcleaver]

Is this initiative dead? Who from Hashicorp or vmWare is championing or sponsoring this effort to make sure it sees the light of day?

@phinze do you own it?

[mrjcleaver]

@markpeek - is VMware committed to helping progress the Terraform driver?

[ktham]

Is there a plan forward for #5843? it's causing terraform to force new resources one every VM.

We're doing an evaluation this month to see if we'd want to switch our provisioning system from our homegrown tooling written with rbvmomi and replace it with terraform. We love terraform so far! We don't usually have opportunities to evaluate new tooling and we have the opportunity/bandwidth to do so in the next month-ish before we have to scrap this exploration and move on.

Will gladly help where we can!! :-)

[mitchellh]

@mrjcleaver We're actively communicating with VMware as well as other interested companies to help push forward the vSphere work. Nobody at HashiCorp works on vSphere full time and it isn't an official HashiCorp provider at the moment (only AWS, Azure, Google at the time of writing this comment).

I'm going to close this since we're trying to go away from bigger "question" issues since they're too open ended and end up sitting open for years. Individual vsphere bugs exist and most releases still include vsphere improvements. Internally at HC, we're working hard to find a better long term solution and we'll share as soon as we can. Externally, we welcome any contributors.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.