Unknown CPI error ''Unknown'' with message ''undefined method `config_spec'' for nil:NilClass'' in ''create_stemcell'' CPI method

[chenweienn]

Describe the bug
Customer created Windows stemcell 2019.23, 2019.25, 2019.26 following this doc: https://docs.pivotal.io/application-service-windows/2-9/create-vsphere-stemcell-automatically.html
The windows ISO version is 2019.25.17763
The vSphere version is 6.5.0.30000
They hit above error when upload to the same vSphere env with bosh upload-stemcell command.

To Reproduce
Steps to reproduce the behavior:

1. Customer can reproduce it by bosh upload-stemcell
2. We tried combinatinos in lab vSphere env

• vSphere 6.5 + stemcell 2019.23; cannot reproduce the same error
• vSphere 7 + stemcell 2019.26; cannot reproduce the same error

We think the issue is on customer vSphere side that the create_import_spec does not work correctly which causes import_spec_result.import_spec being nil in this code.
Could you advice how to further troubleshoot this issue? Which particular log from vSphere should we look into? Thanks!
We could share the stemcell built by customer and vCenter log via sharepoint.

CPI Error Log
from BOSH task 240032:

D, [2020-10-02T00:42:21.663677 #9669] [task:240032] DEBUG -- DirectorJobRunner: [external-cpi] [cpi-145857] response: {"result":null,"error":{"type":"Unknown","message":"undefined method `config_spec' for nil:NilClass","ok_to_retry":false},"log":"Rescued Unknown: undefined method `config_spec' for nil:NilClass. backtrace: /var/vcap/data/packages/vsphere_cpi/6e7fcfa5d2b52470253acc9932c68ea7a07409f5/lib/cloud/vsphere/cloud.rb:214:in `block (3 levels) in create_stemcell'
/var/vcap/data/packages/vsphere_cpi/6e7fcfa5d2b52470253acc9932c68ea7a07409f5/lib/cloud/vsphere/cloud.rb:204:in `each'
/var/vcap/data/packages/vsphere_cpi/6e7fcfa5d2b52470253acc9932c68ea7a07409f5/lib/cloud/vsphere/cloud.rb:204:in `block (2 levels) in create_stemcell'
/var/vcap/data/packages/ruby-2.6.5-r0.29.0/d0bf3ee2098eae0fa7273832839bf6cdce5aa1cf/lib/ruby/2.6.0/tmpdir.rb:93:in `mktmpdir'
/var/vcap/data/packages/vsphere_cpi/6e7fcfa5d2b52470253acc9932c68ea7a07409f5/lib/cloud/vsphere/cloud.rb:173:in `block in create_stemcell'
/var/vcap/data/packages/vsphere_cpi/6e7fcfa5d2b52470253acc9932c68ea7a07409f5/vendor/bundle/ruby/2.6.0/gems/bosh_common-1.3262.24.0/lib/common/thread_formatter.rb:49:in `with_thread_name'
/var/vcap/data/packages/vsphere_cpi/6e7fcfa5d2b52470253acc9932c68ea7a07409f5/lib/cloud/vsphere/cloud.rb:168:in `create_stemcell'
/var/vcap/data/packages/ruby-2.6.5-r0.29.0/d0bf3ee2098eae0fa7273832839bf6cdce5aa1cf/lib/ruby/2.6.0/forwardable.rb:230:in `create_stemcell'
/var/vcap/data/packages/vsphere_cpi/6e7fcfa5d2b52470253acc9932c68ea7a07409f5/vendor/bundle/ruby/2.6.0/gems/bosh_cpi-2.5.0/lib/bosh/cpi/cli.rb:89:in `public_send'
/var/vcap/data/packages/vsphere_cpi/6e7fcfa5d2b52470253acc9932c68ea7a07409f5/vendor/bundle/ruby/2.6.0/gems/bosh_cpi-2.5.0/lib/bosh/cpi/cli.rb:89:in `run'
/var/vcap/packages/vsphere_cpi/bin/vsphere_cpi:48:in `<top (required)>'
/var/vcap/data/packages/ruby-2.6.5-r0.29.0/d0bf3ee2098eae0fa7273832839bf6cdce5aa1cf/lib/ruby/site_ruby/2.6.0/bundler/cli/exec.rb:63:in `load'
/var/vcap/data/packages/ruby-2.6.5-r0.29.0/d0bf3ee2098eae0fa7273832839bf6cdce5aa1cf/lib/ruby/site_ruby/2.6.0/bundler/cli/exec.rb:63:in `kernel_load'
/var/vcap/data/packages/ruby-2.6.5-r0.29.0/d0bf3ee2098eae0fa7273832839bf6cdce5aa1cf/lib/ruby/site_ruby/2.6.0/bundler/cli/exec.rb:28:in `run'
/var/vcap/data/packages/ruby-2.6.5-r0.29.0/d0bf3ee2098eae0fa7273832839bf6cdce5aa1cf/lib/ruby/site_ruby/2.6.0/bundler/cli.rb:476:in `exec'
/var/vcap/data/packages/ruby-2.6.5-r0.29.0/d0bf3ee2098eae0fa7273832839bf6cdce5aa1cf/lib/ruby/site_ruby/2.6.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/var/vcap/data/packages/ruby-2.6.5-r0.29.0/d0bf3ee2098eae0fa7273832839bf6cdce5aa1cf/lib/ruby/site_ruby/2.6.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/var/vcap/data/packages/ruby-2.6.5-r0.29.0/d0bf3ee2098eae0fa7273832839bf6cdce5aa1cf/lib/ruby/site_ruby/2.6.0/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/var/vcap/data/packages/ruby-2.6.5-r0.29.0/d0bf3ee2098eae0fa7273832839bf6cdce5aa1cf/lib/ruby/site_ruby/2.6.0/bundler/cli.rb:30:in `dispatch'
/var/vcap/data/packages/ruby-2.6.5-r0.29.0/d0bf3ee2098eae0fa7273832839bf6cdce5aa1cf/lib/ruby/site_ruby/2.6.0/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/var/vcap/data/packages/ruby-2.6.5-r0.29.0/d0bf3ee2098eae0fa7273832839bf6cdce5aa1cf/lib/ruby/site_ruby/2.6.0/bundler/cli.rb:24:in `start'
/var/vcap/data/packages/ruby-2.6.5-r0.29.0/d0bf3ee2098eae0fa7273832839bf6cdce5aa1cf/lib/ruby/gems/2.6.0/gems/bundler-2.1.2/exe/bundle:46:in `block in <top (required)>'
/var/vcap/data/packages/ruby-2.6.5-r0.29.0/d0bf3ee2098eae0fa7273832839bf6cdce5aa1cf/lib/ruby/site_ruby/2.6.0/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/var/vcap/data/packages/ruby-2.6.5-r0.29.0/d0bf3ee2098eae0fa7273832839bf6cdce5aa1cf/lib/ruby/gems/2.6.0/gems/bundler-2.1.2/exe/bundle:34:in `<top (required)>'
/var/vcap/packages/ruby-2.6.5-r0.29.0/bin/bundle:23:in `load'
/var/vcap/packages/ruby-2.6.5-r0.29.0/bin/bundle:23:in `<main>'"}, err: I, [2020-10-02T00:41:14.686728 #9778]  INFO -- [req_id cpi-145857]: Starting create_stemcell...

Expected behavior
Being able to upload the stemcell.

Release Version & Related Info (please complete the following information):

• CPI Version : vSphere CPI: 54.1.0
• BOSH Director Version: 270.11.1
• Stemcell Name & Version: have tried windows stemcell version 2019.23, 2019.25, 2019.26
• vCenter Version: vSphere version is 6.5.0.30000
• NSX(T/V) Version (If using): No use

[watarumohawk]

Some updates on this issue:

• We have determined that the VM base image's hardware version (aka compatibility) was set to ESXi 6.5, not 6.0.
• We created a new stemcell with 6.0 compatibility, and the bosh director was able to successfully upload the stemcell.

[julian-hj]

If I am reading this issue correctly, the outstanding problem is that the vSphere CPI doesn't bother to check for errors after importing a stemcell file, and as a result, the error message we produce is confusing and unhelpful.

Basically, we should be adding an error check between these 2 lines:

            import_spec_result = import_ovf(name, ovf_file, cluster.resource_pool.mob, datastore.mob)

            system_disk = import_spec_result.import_spec.config_spec.device_change.find do |change|

We know from the stack trace that import_spec_result.import_spec was nil. My guess is that import_spec_result.error probably had a useful error message that would have helped resolve the issue.

I will close and reopen this issue to create a tracker story.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[julian-hj]

We added new/better error handling for invalid OVF files in this commit which was the outstanding problem for this issue, so I will go ahead and close it.