gem_package using chef_zero installing packages into /tmp/kitchen-chef-zero making binstubs unavailable to chef

[adnichols]

We ran into this with bluepill so I'll use that as the example. Take a cookbook which depends on the 'bluepill' cookbook and starts a simple service:

Cookbook available here: https://github.com/adnichols/test-gem.git

template "#{node[:bluepill][:conf_dir]}/sleep.pill" do
  source 'sleep.pill.erb'
  owner 'root'
  group 'root'
  mode  '644'
  action :create
end

bluepill_service 'sleep' do
  action [:stop, :enable, :load, :start]
end

If I converge this cookbook in test-kitchen I run into the following error when it tries to start the service via bluepill:

         * bluepill_service[sleep] action load[2013-11-05T20:13:56+00:00] INFO: Processing bluepill_service[sleep] action load (test-gem::default line 35)

       ================================================================================
       Error executing action `load` on resource 'bluepill_service[sleep]'
       ================================================================================


       Errno::ENOENT
       -------------
       No such file or directory - /opt/chef/embedded/bin/bluepill load /etc/bluepill/sleep.pill

Indeed this is accurate - if I login to the box after being converged, the bluepill binstub isn't available in the path:

[vagrant@default-centos-64-x86-64 ~]$ which bluepill
/usr/bin/which: no bluepill in (/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/vagrant/bin)
[vagrant@default-centos-64-x86-64 ~]$ /opt/chef/embedded/bin/gem list | grep bluepill
[vagrant@default-centos-64-x86-64 ~]$

It appears that bluepill is actually being installed under the test-kitchen ruby but is not functional there:

[vagrant@default-centos-64-x86-64 ~]$ /tmp/kitchen-chef-zero/gems/bin/bluepill
/opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/dependency.rb:247:in `to_specs': Could not find bluepill (>= 0) amongst [bigdecimal-1.1.0, bundler-1.1.5, chef-11.4.4, erubis-2.7.0, highline-1.6.18, io-console-0.3, ipaddress-0.8.0, json-1.5.4, mime-types-1.23, minitest-2.5.1, mixlib-authentication-1.3.0, mixlib-cli-1.3.0, mixlib-config-1.1.2, mixlib-log-1.6.0, mixlib-shellout-1.1.0, net-ssh-2.6.7, net-ssh-gateway-1.2.0, net-ssh-multi-1.2.0, net-ssh-multi-1.1, ohai-6.16.0, rake-0.9.2.2, rdoc-3.9.4, rest-client-1.6.7, ruby-shadow-2.2.0, systemu-2.5.2, yajl-ruby-1.1.0] (Gem::LoadError)
    from /opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/dependency.rb:256:in `to_spec'
    from /opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems.rb:1231:in `gem'
    from /tmp/kitchen-chef-zero/gems/bin/bluepill:22:in `<main>'

It's unclear to me why this is happening. This was not happening before we updated to beta4, but that update also pulled in chef 11.8 and we aren't sure which is causing the problem.

We also created a cookbook which simply performs a gem_package 'bluepill' and it behaved the same way installing the bluepill gem into a location which isn't in the normal chef path.

This problem appears to be specific to using chef_zero inside test-kitchen. If we use chef_solo this works fine under both chef 11.6 & 11.8.

We have also observed that using the "require_chef_omnibus:" parameter in our .kitchen.yml doesn't appear to affect the version of chef which is used when using chef_zero. This appears to be related to the installation of knife-essentials which pulls in the latest version of chef, and that version appears to be used to converge the system vs. the omnibus installed version.

The other thing we tried, and we may be misunderstanding what this does, is to set "ruby_binpath:" that was added in #225 . This doesn't seem to change the ruby that is used during the converge, but that may not be the purpose.

We aren't able to move forward with our cookbooks which rely on bluepill and we assume this same problem will impact testing any cookbook which installs gems via gem_package. Can someone advise either a workaround or a solution to this?

[adnichols]

As a follow-up, I rolled things back to using test-kitchen v1.0.0-beta3 and the recipe worked as expected.

[jkeiser]

@schisamo there have been a few bugs around this that make me suspect the sandboxed stuff, but it's only a suspicion.

[schisamo]

This def looks like an issue caused by the sandbox. I'll take a look and see what we can do to bust the sandbox.

—
Sent from Mailbox for iPhone

On Tue, Nov 5, 2013 at 3:54 PM, John Keiser notifications@github.com
wrote:

@schisamo there have been a few bugs around this that make me suspect the sandboxed stuff, but it's only a suspicion.

Reply to this email directly or view it on GitHub:
#240 (comment)

[schisamo]

^^ @adnichols I used your repo to verify the above commit fixes things. If you don't want to wait for the next TK release you can just clone this repo and do a rake install to run off HEAD.

I also wanted to share share with you a simplified .kitchen.yml which is equivalent to what you currently have in your repo:

---
driver_plugin: vagrant
driver_config:
  require_chef_omnibus: 11.4.4
provisioner: chef_zero
platforms:
- name: centos-6.4
suites:
- name: default
  run_list:
    - recipe[bluepill]
    - recipe[test-gem]

I was able to remove quite a bit since you had many of the defaults:
https://github.com/opscode/kitchen-vagrant/blob/master/lib/kitchen/driver/vagrant.rb#L37-L45

One great thing about using the default values for box name is these boxes will be installed only once as you move between Test Kitchen-enabled cookbooks that leverage the same default values.

[sethvargo]

Fixed in 55c69d4

[adnichols]

Awesome - thanks folks for the fast turnaround. Also, definitely appreciate the simplified kitchen.yml - we auto-generate ours so it's generally consistent across cookbooks but if we can simplify it, all the better. I'll give this a whirl tomorrow and holler if we run into any problems.

[petecheslock]

@schisamo Is this related as well? https://gist.github.com/petecheslock/bb464c1ebb5ec12f49d7

Happens on a kitchen verify - but i'm not quite sure why. Can't replicate with TK beta.3 - so thinking related to the sandbox and omnibus chef (with embedded ruby)

[schisamo]

@petecheslock When was that instance last built? I made this fix in busser-serverspec recently:
test-kitchen/busser-serverspec#4

[schisamo]

@petecheslock Try to rebuild the instance OR blow away /tmp/kitchen-busser which should force busser and all plugins to be reinstalled.

[petecheslock]

@schisamo Actually spoke too soon - did another test and it TK specific - but somehow related to TK on HEAD (vs tk beta.3)

https://gist.github.com/petecheslock/5754e7f196e9c8892f4c

In the gist - you can see the test pass in beta.3 - but in beta 4 or HEAD it fails.

The test is to look at the gems install in the omnibus package (chef in this case).

but most interesting is that I can't run the "chef-client --version" command either because it can't find chef gem.

[schisamo]

@petecheslock Can you please open a separate issue and I'll try to figure out a fix.