Install chef-dk from one gemfile #765

jkeiser · 2016-03-25T20:42:14Z

This implements several things:

Installs all gems in the chef-dk from a single Gemfile and Gemfile.lock at the top of the repository.
rake dependencies allows you to update that lock file safely.
- It updates Gemfile.lock as well as Gemfile.windows.lock.
- Gemfile.windows.lock includes the versions of gems in Gemfile.lock via a bit of code in Gemfile that reads in Gemfile.lock and adds explicit pins if you are doing a bundle lock for windows.
- It uses bin/bundle-platform to set Gem.platforms before solving, allowing us to compute an OS-agnostic lock in Gemfile.lock and a Windows-specific one in Gemfile.windows.lock.
- It updates acceptance/Gemfile.lock and omnibus/Gemfile.lock as well to pin test and build tooling.
- It runs bundle outdated at the end and yells loudly if any gems are not at their latest version. Exceptions can be made for this via the ALLOWED_OUTDATED_GEMS constant in tasks/bundle_util.rb. This is what protects us from, for example, downgrading chef because the solver noticed a conflict with a constraint from test-kitchen.
Builds native gems individually in their own software definitions named config/software/chef-dk-gem-<gemname>.rb. This allows long-running gem installs to be cached and speed up the build overall. For example, chef-dk-gem-dep-selector-libgecode will:
- Set default_version to the version of dep-selector-libgecode in the top-level Gemfile.lock
- gem install dep-selector-libgecode -v <version> --ignore-dependencies with proper build setup (--ignore-dependencies is to ensure we only get versions that are in our lockfile and not just whatever dep-selector-libgecode wants).
- config/files/chef-dk-gem/build-chef-dk-gem/gem-install-software-def.rb is what implements the above things.
Downloads and installs all remaining gems in config/software/chef-dk.rb. There are many gems installed in this step, and they are installed with bundle install --without development no_<os> --jobs=4 --retries=4 --frozen.
Goes through all gems that were installed from git or path (i.e. gem 'chef', github: 'chef/chef') and runs rake install from each one to genuinely install the gem. Then removes the git-installed gems.
bundle locks the Gemfiles in the berkshelf, test-kitchen, chef and chef-dk gems.
- Pins the lockfile to the distribution's versions of any runtime dependencies.
- Exceptions can be made to this pinning via a constant in config/files/chef-dk/build-chef-dk.rb.
- The development dependencies and bundled software are not installed in the chef-dk, just locked.
- We run bundle check with without development test changelog maintenance guard to ensure that any runtime dependencies in the bundle are in the chef-dk and weren't missed by the fact that we're no longer bundle installing their Gemfile.
- When chef verify is later run from these directories, they do a bundle install which installs any dev dependencies required for the tests onto the test instance.
/opt/chef-dk/Gemfile.lock contains a lockfile that serves as a full manifest for installed gems.

Smaller things about this patch:

Handles ruby-shadow as a gem dependency group in Gemfile--it is in the no_aix and no_windows groups.
Handles all things that were part of the chef software definition inline (there weren't many--powershell and such).

lamont-granquist · 2016-03-28T21:34:01Z

chef-dk-gemset/chef-dk-gemset.gemspec

+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'chef-dk-gemset/version'


can we make this just chef-dk/version and nuke the whole lib structure under here?

It's all gone :)

mwrock · 2016-03-28T22:22:49Z

Perhaps we don't need the gemset gem and can just include the dependencies right in chef-dk. That feels more straight forward to me.

jkeiser · 2016-03-28T23:08:41Z

@mwrock done! Made it a lot simpler, thanks :)

mwrock · 2016-03-28T23:39:27Z

👍 after appbundler merge and removing temporary branch

lamont-granquist · 2016-03-29T00:36:05Z

README.md

+
+To build the chef-dk, we use the omnibus system. Go to the [omnibus README](omnibus/README.md) to find out how to build!
+
+To update the chef-dk's dependencies, run `rake depenedencies`. This will update the `Gemfile.lock` and `Gemfile.windows.lock` and show you any outdated dependencies (some outdated dependencies are to be expected; just scan them for anything that looks particularly bad).


typeo: depenedencies

lamont-granquist · 2016-03-29T00:41:06Z

👍

1. Delete lockfiles before running bundle update 2. Use Bundler.with_clean_env to avoid loading the wrong bundler when running bundle commands

- Move version constants to the top at `version_policy.rb` - Add newlines between sections of build to make it easier to read - Rename `rake dependencies:update_conservative` to `dependencies:update[conservative]` Include version_policy.rb in gem

chef-supermarket added the Signed CLA label Mar 25, 2016

jkeiser force-pushed the jk/chef-dk-gemset branch from a341e33 to 4d47c01 Compare March 28, 2016 18:29