Bundler environment not properly reset?

[tknerr]

Forking off an issue from the related discussion here:
hashicorp/vagrant#6158

So I have:

• ChefDK 0.7.0 installed
• Vagrant 1.7.4 installed (via the .msi installer; it's not present as a gem)
• the most recent bundler version installed: gem install bundler -v 1.10.6

Now whenever I bundle exec kitchen create I get an error like this:

C:\Repos\_github\bills-kitchen\target\build\repo\vagrant-workflow-tests\playground\sample-toplevel-cookbook>bundle exec kitchen create 1204
-----> Starting Kitchen (v1.4.2)
-----> Creating <default-ubuntu-1204>...
       Vagrant experienced a version conflict with some installed plugins!
       This usually happens if you recently upgraded Vagrant. As part of the
       upgrade process, some existing plugins are no longer compatible with
       this version of Vagrant. The recommended way to fix this is to remove
       your existing plugins and reinstall them one-by-one. To remove all
       plugins:

           rm -r ~/.vagrant.d/plugins.json ~/.vagrant.d/gems

       Note if you have an alternate VAGRANT_HOME environmental variable
       set, the folders above will be in that directory rather than your
       user's home directory.

       The error message is shown below:

       Bundler could not find compatible versions for gem "bundler":
         In Gemfile:
           vagrant (= 1.7.4) x86-mingw32 depends on
             bundler (<= 1.10.5, >= 1.5.2) x86-mingw32

         Current Bundler version:
           bundler (1.10.6)
       This Gemfile requires a different version of Bundler.
       Perhaps you need to update Bundler by running `gem install bundler`?
       Could not find gem 'bundler (<= 1.10.5, >= 1.5.2) x86-mingw32', which is required by gem 'vagrant (= 1.7.4) x86-mingw32', in any of the sources.

This is weird because Vagrant should never see that it's called from a bundled context, eventually with a bundler version that is not compatible. In fact Vagrant ships with it's own version of bundler (1.10.5 in that case) and should never see the outer bundler.

@sethvargo suggests here that this might have something to do with the bundler environment not being correctly cleaned / restored

[sethvargo]

@tknerr it's not the bundler environment so much as the ruby environment.

[tknerr]

Ooops, it might in fact be an upstream Vagrant issue though.

So this is my Vagrant:

C:\Repos\_github\bills-kitchen\target\build\repo\vagrant-workflow-tests\playground\sample-toplevel-cookbook>which vagrant
C:\Repos\_github\bills-kitchen\target\build\tools\vagrant\HashiCorp\Vagrant\bin\vagrant.EXE

Just to make sure, even in the bundle context it's the right one:

C:\Repos\_github\bills-kitchen\target\build\repo\vagrant-workflow-tests\playground\sample-toplevel-cookbook>bundle exec which vagrant
C:\Repos\_github\bills-kitchen\target\build\tools\vagrant\HashiCorp\Vagrant\bin\vagrant.EXE

So plain vagrant status works, but bundle exec vagrant status is already failing with that error:

C:\Repos\_github\bills-kitchen\target\build\repo\vagrant-workflow-tests\playground\sample-toplevel-cookbook>vagrant status
Current machine states:

sample-app                not created (virtualbox)

The environment has not yet been created. Run `vagrant up` to
create the environment. If a machine is not created, only the
default provider will be shown. So if a provider is not listed,
then the machine is not created for that environment.

C:\Repos\_github\bills-kitchen\target\build\repo\vagrant-workflow-tests\playground\sample-toplevel-cookbook>bundle exec vagrant status
Vagrant experienced a version conflict with some installed plugins!
This usually happens if you recently upgraded Vagrant. As part of the
upgrade process, some existing plugins are no longer compatible with
this version of Vagrant. The recommended way to fix this is to remove
your existing plugins and reinstall them one-by-one. To remove all
plugins:

    rm -r ~/.vagrant.d/plugins.json ~/.vagrant.d/gems

Note if you have an alternate VAGRANT_HOME environmental variable
set, the folders above will be in that directory rather than your
user's home directory.

The error message is shown below:

Bundler could not find compatible versions for gem "bundler":
  In Gemfile:
    vagrant (= 1.7.4) x86-mingw32 depends on
      bundler (<= 1.10.5, >= 1.5.2) x86-mingw32

  Current Bundler version:
    bundler (1.10.6)
This Gemfile requires a different version of Bundler.
Perhaps you need to update Bundler by running `gem install bundler`?
Could not find gem 'bundler (<= 1.10.5, >= 1.5.2) x86-mingw32', which is required by gem 'vagrant (= 1.7.4) x86-mingw32', in any of the sources.

@sethvargo so you would expect too that vagrant uses it's own internal bundler version, right?

[tknerr]

@sethvargo thanks for the clarification!

So here's my gem env for the specific case above:

C:\Repos\_github\bills-kitchen\target\build\repo\vagrant-workflow-tests\playground\sample-toplevel-cookbook>gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 2.4.4
  - RUBY VERSION: 2.1.6 (2015-04-13 patchlevel 336) [i386-mingw32]
  - INSTALLATION DIRECTORY: C:/Repos/_github/bills-kitchen/target/build/home/.chefdk/gem/ruby/2.1.0
  - RUBY EXECUTABLE: C:/Repos/_github/bills-kitchen/target/build/tools/chefdk/embedded/bin/ruby.exe
  - EXECUTABLE DIRECTORY: C:/Repos/_github/bills-kitchen/target/build/home/.chefdk/gem/ruby/2.1.0/bin
  - SPEC CACHE DIRECTORY: C:/Repos/_github/bills-kitchen/target/build/home/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: C:/ProgramData
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86-mingw32
  - GEM PATHS:
     - C:/Repos/_github/bills-kitchen/target/build/home/.chefdk/gem/ruby/2.1.0
     - C:/Repos/_github/bills-kitchen/target/build/tools/chefdk/embedded/lib/ruby/gems/2.1.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
     - "install" => "--user"
     - "update" => "--user"
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - C:\Repos\_github\bills-kitchen\target\build\tools\scripts
     - C:\Repos\_github\bills-kitchen\target\build\tools\docker
     - C:\Repos\_github\bills-kitchen\target\build\tools\chefdk\bin
     - C:\Repos\_github\bills-kitchen\target\build\home\.chefdk\gem\ruby\2.1.0\bin
     - C:\Repos\_github\bills-kitchen\target\build\tools\chefdk\embedded\bin
     - C:\Repos\_github\bills-kitchen\target\build\tools\consul
     - C:\Repos\_github\bills-kitchen\target\build\tools\packer
     - C:\Repos\_github\bills-kitchen\target\build\tools\terraform
     - C:\Repos\_github\bills-kitchen\target\build\tools\vagrant\HashiCorp\Vagrant\bin
     - C:\Repos\_github\bills-kitchen\target\build\tools\portablegit\cmd
     - C:\Repos\_github\bills-kitchen\target\build\tools\portablegit
     - C:\Repos\_github\bills-kitchen\target\build\tools\kdiff3
     - C:\Repos\_github\bills-kitchen\target\build\tools\cwrsync\cwRsync_5.4.1_x86_Free
     - C:\Repos\_github\bills-kitchen\target\build\tools\vagrant\HashiCorp\Vagrant\embedded\bin
     - C:\Repos\_github\bills-kitchen\target\build\tools\conemu
     - C:\Repos\_github\bills-kitchen\target\build\tools\atom\Atom\resources\cli
     - C:\Repos\_github\bills-kitchen\target\build\tools\atom\Atom\resources\app\apm\bin
     - C:\Repos\_github\bills-kitchen\target\build\tools\putty
     - C:\Repos\_github\bills-kitchen\target\build\tools\devkit\bin
     - C:\Repos\_github\bills-kitchen\target\build\tools\devkit\mingw\bin

Anything I could check for that would indicate the ruby environment is not properly set?

[tknerr]

@sethvargo could repro this in a clean environment without any ChefDK or kitchen-vagrant at all:

X:\>gem install bundler -v 1.10.6
Fetching: bundler-1.10.6.gem (100%)
Successfully installed bundler-1.10.6
Parsing documentation for bundler-1.10.6
Installing ri documentation for bundler-1.10.6
Done installing documentation for bundler after 7 seconds
1 gem installed

X:\>set PATH=C:\Repos\_github\bills-kitchen\target\build\tools\vagrant\HashiCorp\Vagrant\bin;%PATH%

X:\>echo "" > Gemfile

X:\>bundle exec vagrant status
Vagrant experienced a version conflict with some installed plugins!
This usually happens if you recently upgraded Vagrant. As part of the
upgrade process, some existing plugins are no longer compatible with
this version of Vagrant. The recommended way to fix this is to remove
your existing plugins and reinstall them one-by-one. To remove all
plugins:

    rm -r ~/.vagrant.d/plugins.json ~/.vagrant.d/gems

Note if you have an alternate VAGRANT_HOME environmental variable
set, the folders above will be in that directory rather than your
user's home directory.

The error message is shown below:

Bundler could not find compatible versions for gem "bundler":
  In Gemfile:
    vagrant (= 1.7.4) x86-mingw32 depends on
      bundler (<= 1.10.5, >= 1.5.2) x86-mingw32

  Current Bundler version:
    bundler (1.10.6)
This Gemfile requires a different version of Bundler.
Perhaps you need to update Bundler by running `gem install bundler`?
Could not find gem 'bundler (<= 1.10.5, >= 1.5.2) x86-mingw32', which is required by gem 'vagrant (= 1.7.4) x86-mingw32', in any of the sources.

@sethvargo should I open an issue in Vagrant for this, or is this expected?

My understanding was that vagrant should never see the outer bundler version, only the one it ships with.

[sethvargo]

Hi @tknerr

bundle exec vagrant is almost never expected to work. We do not support executing Vagrant as a gem. In fact, Vagrant is no longer packaged as a gem on rubygems.org (not since 0.5.x).

[tknerr]

@sethvargo yes I know, I don't have it installed as a gem :-)

Just double-checked:

X:\>gem list vagrant

*** LOCAL GEMS ***



X:\>which vagrant
C:\Repos\_github\bills-kitchen\target\build\tools\vagrant\HashiCorp\Vagrant\bin\vagrant.EXE

X:\>vagrant -v
Vagrant 1.7.4

However, my test-kitchen and kitchen-vagrant versions are managed in a Gemfile, so in the end they shell out to Vagrant from within a bundle context.

That's why I was calling it via bundle exec for a minimal repro case.

[sethvargo]

@tknerr that's my point though - they should not be retaining the context from within a bundle. They should be shelling out using the same environment as if you ran vagrant yourself from a terminal.

bundle exec is going to repro because Vagrant isn't designed to be run inside of a bundle.

[tknerr]

@sethvargo ok, got it. Thanks for clarification and your patience! :-)

[ksubrama]

I've submitted #191 to attempt to fix this so that we can bump bundler's version in chef-dk.

[mfischer-zd]

I don't think this is test kitchen's problem to solve. Vagrant's loader (when installed as an Omnibus package) should be unsetting any conflicting environment variables set by Bundler.

[sethvargo]

@mfischer-zd Vagrant actually does that as of Vagrant 1.7.4. We grab the environment that existed before Vagrant was executed and reset to that (hashicorp/vagrant-installers#57). The problem is that Test Kitchen is "execing" to Vagrant but not resetting it's environment. Vagrant has no way of knowing what the environment was before Test Kitchen was run. Does that make sense?

[mfischer-zd]

@sethvargo I see the save/restore part, but as far as I can tell, only RUBYOPT is being unset by the launcher. RUBYLIB, GEM_HOME and GEM_PATH are not modified. (Correct me if I'm mistaken.)

[sethvargo]

@mfischer-zd that is the pre-Ruby Go launcher that just copies the current env (which might include GEM_* and RUBY* envvars, especially if the caller uses a ruby version manager like RVM or chruby. When Vagrant runs, bundler will modify all those fields, but then Vagrant restores the original (VAGRANT_ORIGINAL_ENV) anytime it execs to a subprocess.

The problem occurs if Vagrant is executed from within an existing Ruby that isn't properly unsetting those variables when it shells out to Vagrant. Omnibus Chef works fine because it takes the same "restore the entire env before invocation when shelling out" approach. I think TK just needs to mirror that.

[mfischer-zd]

The problem occurs if Vagrant is executed from within an existing Ruby that isn't properly unsetting those variables when it shells out to Vagrant.

Vagrant should expect that. My contention is that any omnibus-packaged application that arranges for its own runtime environment (and that includes Vagrant) should consider its environment tainted and act accordingly so that it can load its own Gems without the caller getting in its way.

It is true that if Vagrant does this, its process descendants won't have access to the bundler/rbenv/etc environment associated with TK. I don't see this as a practical problem, though. What would break?

[b-dean]

The fix for this makes it so we can't have a pre_create_command that uses the bundle

driver:
  pre_create_command: bundle exec do_something.rb

I want to do that, I'd have to have it install the bundle again. This doesn't seem like the right fix to solve the vagrant vs bundler problems. I think it's a vagrant issue because you can clearly make it happen w/o even using test-kitchen or kitchen-vagrant. They (vagrant) should be resetting RUBYLIB. Its launcher is already doing that for RUBYOPT.