Upgrading to Bundler 1.16 causes bundler to look for Gemfile in root directory

[Gerg]

Summary

@jenspinney and @deniseyu discovered this issue when using bundler in a docker image we build containing bundler 1.16: https://github.com/cloudfoundry/capi-dockerfiles/blob/26ac90206a234ad29b2775500d471a267d8fced4/capi-ruby-units/Dockerfile#L4

The error looks like this:

[!] There was an error parsing `Gemfile`: No such file or directory @ rb_sysopen - /Gemfile. Bundler cannot continue.

Our workaround is to explicitly sets BUNDLE_GEMFILE to point to the correct Gemfile.

Details

After upgrading from Bundler 1.15 to 1.16, bundler config is reporting that the BUNDLE_GEMFILE env var is set to /Gemfile, despite no such environment variable exisiting.

Looking in /usr/local/bundle, we see a bunch of files in the /usr/local/bundle/bin directory that all follow this pattern:

#!/usr/bin/env ruby
# frozen_string_literal: true

#
# This file was generated by Bundler.
#
# The application 'clockwork' is installed as part of a gem, and
# this file is here to facilitate running it.
#

bundle_binstub = File.expand_path("../bundle", __FILE__)
load(bundle_binstub) if File.file?(bundle_binstub)

require "pathname"
ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../../../../Gemfile",
  Pathname.new(__FILE__).realpath)

require "rubygems"
require "bundler/setup"

load Gem.bin_path("clockwork", "clockwork")

That ../../../../../Gemfile filepath looks pretty suspicious, and we don't know why Bundler is generating it.

[rheaton]

I wonder if the behavior I'm seeing is related.
We upgraded to 1.16 from 1.15.3 and discovered the adding a --system --gemfile=path/to/Gemfile set the BUNDLE_GEMFILE environment variable outside of the execution of that command.

So in 1.15, the following worked:

bundle install --system --gemfile=/path/1/Gemfile && bundle install --system --gemfile=/path/2/Gemfile
cd /path/1 && bundle exec ruby -e "require 'gem_only_in_1_gemfile'"

but it fails to require gem_only_in_1_gemfile in 1.16 as bundler has set BUNDLE_GEMFILE to path/2/Gemfile in its config.

This commit looked like a likely offender, but I only poked around for a few minutes: 4337a49#diff-22d99f042aaf58eb9bf0886d01594802

It was surprising to me that a --gemfile flag persisted after the execution of the command, and more interestingly, different behavior from 1.15.3.

cc @supertopher

[deivid-rodriguez]

Hi, what's the value of bundle config bin?

[segiddins]

I wonder if #6201 has fixed this

[segiddins]

Has Bundler v1.16.1 fixed this?

[thegeorgeous]

@segiddins, #6201 has only fixed it partially.

We put Gemfile and Gemfile.lock in container root when installing gems during build. This leads to all the binstubs pointing to that Gemfile. Since we use a /app folder for development, we end up having to run BUNDLE_GEMFILE=./Gemfile bundle install once. This changes all the binstubs to the point to the appropriate Gemfile after which it is possible to continue using it as before.

[segiddins]

I don't understand what the problem is there? Are you not using the gemfile you're installing?

[lucasmazza]

We encounter this problem last week while upgrading our app to Ruby 2.4/Bundler 1.16 using the docker official images, and I think the following can expand on which scenarios this is happening:

The Docker image has a predefined BUNDLE_BIN that is available on PATH (https://github.com/docker-library/ruby/blob/ca5366d8e8afa5f6cf810a07740a50e1f4eeab36/Dockerfile-alpine.template#L99-L106), which doesn't play along when your Gemfile changes place (like placing it on / first to install the dependencies and then copying your code to /app and using it from there) or having multiple Gemfiles (like our mono repo) that might not share the same dependencies.

The first bundle install will install a /usr/local/bundle/bin/bundle binstub with the relative path of where the Gemfile first where, and then all bundle calls are using the generated binstub (that should be app specific) instead of the initial executable (that should be the "system" one).

We added a ENV BUNDLE_GEMFILE='./Gemfile' to our Dockerfile so every bundle command uses a relative Gemfile, instead of falling back to the first Gemfile path that was used.

I think it would be interesting if the bundler binstub could be more portable regarding where the Gemfile it looks for is, but I also feels that the docker image is more opinionated than users would expect, so I'm not sure on how we can make a broader fix for this.

I hope this helps to shed a light on the issue.

[agate]

Our team is using official ruby docker image. And we ran into this problem one month ago. We couldn't figure out what is the reason. But we did find one workaround. Either run source /etc/profile or gem install bundler at very beginning of our docker start script.

I also wrote a problem demo script:
https://gist.github.com/agate/513f669a86a52ed7190de6698360cd84

#!/usr/bin/env bash

function colorize_echo() { echo -e "\033[0;${1}m${@:2}\033[0m"; }
function red_echo()      { colorize_echo 31 $@; }
function green_echo()    { colorize_echo 32 $@; }
function yellow_echo()   { colorize_echo 33 $@; }
function blue_echo()     { colorize_echo 34 $@; }

CURRENT_DIR="$(pwd)"
DIR_NAME=docker-ruby-bundle-test
IMAGE_NAME=$DIR_NAME:latest

blue_echo "Create test dir"
mkdir -p $DIR_NAME

cd $DIR_NAME

blue_echo "Create dockerfile"
cat <<END > Dockerfile
FROM ruby:2.3

ADD Gemfile /tmp/bundle/
RUN cd /tmp/bundle && bundle install
END

blue_echo "Create gemfile"
cat <<END > Gemfile
# frozen_string_literal: true

source "https://rubygems.org"

git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }

gem "json", "2.0.3"
END

blue_echo "Build docker image"
docker build --tag $IMAGE_NAME .

blue_echo "Problem demo A"
docker run --rm $IMAGE_NAME bash -c '
echo hostname:
hostname
echo pwd:
pwd
echo ls:
echo `ls`
echo bundle:
bundle'
red_echo "What? How pwd:/ can run bundle? Where is the Gemfile?"

blue_echo "Problem demo B"
docker run --rm $IMAGE_NAME bash -c '
echo hostname:
hostname
echo pwd:
pwd
echo ls:
echo `ls`
cat <<END > Gemfile
# frozen_string_literal: true
source "https://rubygems.org"

git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }

gem "json", "2.0.4"
END
bundle'
red_echo "What? Why still use the json 2.0.3?"

blue_echo "Problem demo C"
docker run --rm $IMAGE_NAME bash -c '
cd /tmp/bundle
echo hostname:
hostname
echo pwd:
pwd
echo ls:
echo `ls`
cat <<END > Gemfile
# frozen_string_literal: true
source "https://rubygems.org"

git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }

gem "json", "2.0.4"
END
bundle'
green_echo "OK... Now it works."

blue_echo "Cleanup"
docker rmi $IMAGE_NAME
cd "$CURRENT_DIR"
rm -rf $DIR_NAME

The output is like this:

It looks like there is a cache problem. You can see even if I create a new Dockerfile in the running container. The bundler won't use it. It will keep using the previous one. And no matter I cd into which directory. I always can run bundle install

[agate]

Thanks @lucasmazza for the ENV BUNDLE_GEMFILE='./Gemfile' solution. But it still have one problem. So when you cd into a sub dir and you want to bundle in it. You will get:
There was an error parsing Gemfile: No such file or directory @ rb_sysopen - /Gemfile. Bundler cannot continue. or Gemfile not found error. It's because the env variable hard coded the Gemfile and bundler won't look up the dir tree to find the closest Gemfile.

[deivid-rodriguez]

Sad to see this is still causing trouble, I thought it had been fixed. There's also related discussion in #6162.

I remember workarounding this with ENV BUNDLE_BIN= so that no bundler binstubs are generated. Maybe it works for you @agate.

[agate]

Nice! Thanks @deivid-rodriguez the ENV BUNDLE_BIN= workaround solution is working for us. Thank you very much!

[deivid-rodriguez]

No problem, happy that it works! :)

[agate]

@deivid-rodriguez
I think ENV BUNDLE_BIN= will cause some other problem: If I run bundle install with this env I will get:

Installing railties 5.0.6
Errno::EISDIR: Is a directory @ rb_sysopen - /MY/APP/PATH
An error occurred while installing railties (5.0.6), and Bundler cannot continue.
Make sure that `gem install railties -v '5.0.6'` succeeds before bundling.

If I remove BUNDLE_BIN= in my env. bundle install works fine.

BTW, if I just run gem install railties -v '5.0.6'. It will install it without any problem. But I don't know why bundle will raise error.

[deivid-rodriguez]

Too bad... :( Back to square one, then...

[rheaton]

We've created a repository reproducing this issue via Docker, if anybody has any ideas of how to fix this (for now, we've pinned to an old version of bundler).

Branch master has bundler 1.16.1 with the failure:
https://github.com/rheaton/bundler_issue_6154

Branch bundler_1_15 has bundler 1.15.3 without the failure:
https://github.com/rheaton/bundler_issue_6154/tree/bundler_1_15

Thank you!

Rachel & @heycait

[indirect]

@rheaton @heycait thank you so much for this repro case! Hopefully that will help us get things figured out.

[segiddins]

Can you reproduce this without Docker?

[rheaton]

@segiddins Nope, in our repo we created a non-Docker version that works: https://github.com/rheaton/bundler_issue_6154/blob/master/install.sh

Edit: I tried it out on OSX and couldn't reproduce, so it could be a Linux-specific problem or a Docker-specific problem. Not sure which. :-/

[fgrehm]

We're seeing this problem on our docker alpine images as well. We tried BUNDLE_GEMFILE='./Gemfile' but it did not work for us.

What did work was disabling spring! 😅 we sticked a ENV DISABLE_SPRING=1 to our Dockerfile and called it a day. I haven't figured out exactly what's wrong with spring and unfortunately I don't have enough cycles to investigate it but hopefully this might help future people that ends up here.

I'd also be happy to provide more information about my env if needed!

[eanlain]

I was able to build off of @rheaton work and reproduce this issue without using Docker: https://gist.github.com/eanlain/83f97f919701c864fb68dfdcf813cf5a

I also made an attempt at fixing the issue in #6469.

[salzig]

Same Problem for us. We build a "prebuild" image using the official ruby base images with our Gemfiles as a kind of cache, so we don't have to install every gem from rubygems for every job.

# Dockerfile: this is build and published internally as prebuild image
FROM ruby:2.5.0
COPY Gemfile Gemfile.lock .
RUN bundle install && rm Gemfile Gemfile.lock

This prebuild image is then used in CI to speed up test/spec/lint/etc execution with gitlab-ci.

Gitlab-CI will clone the Repository into /builds/group/project and uses this directory as working directory.

Gitlab-CI will in the end execute our script

# .gitlab-ci.yml
image: ruby:2.5.0
before_script:
  - bundle install
  - bundle exec rake test:prepare

lint:
  script: bundle exec rake ci:lint

spec:
  script: bundle exec rake ci:spec

All our jobs are now failing, cause the Gemfile (/Gemfile) which was used for prebuild, isn't available anymore.

IMHO: bundle install shouldn't even try in this situation to look for another Gemfile, when there is a Gemfile in the current directory.

[stefansedich]

Interesting so I just ran into this same issue using the ruby:2.5 docker image, however if I used a BUNDLE_PATH other than /user/local/bundle the generated bin stub was totally different and looked like this:

#!/usr/bin/env ruby
#
# This file was generated by RubyGems.
#
# The application 'rspec-core' is installed as part of a gem, and
# this file is here to facilitate running it.
#

require 'rubygems'

version = ">= 0.a"

if ARGV.first
  str = ARGV.first
  str = str.dup.force_encoding("BINARY") if str.respond_to? :force_encoding
  if str =~ /\A_(.*)_\z/ and Gem::Version.correct?($1) then
    version = $1
    ARGV.shift
  end
end

if Gem.respond_to?(:activate_bin_path)
load Gem.activate_bin_path('rspec-core', 'rspec', version)
else
gem "rspec-core", version
load Gem.bin_path("rspec-core", "rspec", version)
end

But as soon as the path was /usr/local/bundle the generated stub was as per Gregs example above (generated by bundler and with a werid ../../ path).

Anyway I realized I had an old image using Bundler 1.16.1 so I trashed that and fetched a fresh image which has Bundler 1.16.2 and the issue has gone away, no matter what my BUNDLE_PATH is the generated stub is as per my pasted version above and everything works as expected.

[stefansedich]

OK I get it now it was https://github.com/docker-library/ruby/pull/209/files that removed BUNDLE_BIN that fixed it!

[colby-swandale]

This issue seems to have been fixed. Closing.