Policyfile: Can't use both internal and community Supermarket sources

[shruthi-venkateswaran]

As per https://github.com/chef/chef-dk/blob/master/POLICYFILE_README.md

My policyfile looks like :

name   "test"

default_source :community, "https://<internal-supermarket>.com"

run_list   "recipe[internal-cookbook]", "recipe[rabbitmq]"

But it doesn't fetch the rabbitmq(opensource) cookbook from supermarket.chef.io

It doesn't allow to specify multiple community sources either.

The error I get:

Caching Cookbooks...
Error: Failed to generate Policyfile.lock
Reason: (Solve::Errors::NoSolutionError) Unable to satisfy the following requirements:

- `rabbitmq (>= 0.0.0)` required by `user-specified dependency`

[danielsdeleo]

You'd need to have 2 default_source statements, one for each supermarket you wish to use. I double checked and github indicates the pull request is included in 0.7.0, so you should be good to go after you add another default_source statement. Going to close this for now, but if you still have trouble after fixing your policyfile, comment on this ticket and I will reopen.

cf. #450

[shruthi-venkateswaran]

@danielsdeleo Thanks for the prompt reply.
What is the recommendation if there is a conflict in the cookbook name?As of now it throws a conflict error. Can we specify which source to look at first?

[danielsdeleo]

If you only have one or two conflicts, you have to have a cookbook "name", source_options statement, but unfortunately I think these are a little unwieldy if you are using supermarkets. It would be something like:

cookbook "apt", artifactserver: "https://supermarket.chef.io:443/api/v1/cookbooks/apt/versions/2.8.2/download"

[shruthi-venkateswaran]

@danielsdeleo
The problem is using multiple sources. For eg:

name   "test"
default_source :community
default_source :community, "https://<internal-supermarket>.com"

run_list   "recipe[internal-cookbook]"

The above also fails with the conflict error. Even though I am not requiring any opensource cookbook.

Also the interesting thing to note is that conflict is for another cookbook called chef which is in the internal and opensource Supermarket. So seems like it is doing some kind of indexing at the beginning itself. This is a problem if somebody uploads a cookbook to the internal site with the same name as a cookbook in opensource supermarket. That will break it for everyone else, no one will be able to do 'chef install'.
Can we have a Berksfile/Gemfile like behaviour where it takes the order of the sources into account,
Another option would be to do some kind of namespacing. Thoughts ?

[danielsdeleo]

The conflict part is (mostly) by design. ChefDK is just feeding all the dependency data into a solver library, so there's not an easy way to know if any of the conflicting cookbooks end up influencing the eventual dependency solution. Though it could be done by searching all valid versions of all of the dependencies of the items in your run list. If there are in fact any conflicts, ChefDK has no way to tell if (for example) the "mysql" cookbook on your internal supermarket bears any resemblance to the one on the public supermarket. So generally it's recommended that you either:

1. Only keep internal cookbooks on your internal supermarket, source public cookbooks from the public supermarket, and use git sources if you need to temporarily fork a public cookbook, OR:
2. Mirror the public supermarket to your internal one and handle conflicts when updating your mirror.

Where I think ChefDK could be improved is:

• have a nicer syntax for specifying a supermarket cookbook source
• when there are conflicts, trace all possible dependency paths and don't error if the conflicting cookbooks cannot affect the dependency solution.

What I would prefer to avoid is, silently merging the two dependency graphs from the two sources, as this would mean you could get a cookbook from one source when you expected it to come from the other one.

[shruthi-venkateswaran]

Our internal supermarket has only internal cookbooks. The conflicting 'chef' cookbook is quite different from the opensource one and made solely for the purpose of internal use. Only the name is conflicting.
The reason we want an internal supermarket is to avoid maintaining all the dependencies on internal cookbooks stored in internal source code repos and thus less of git sources everywhere.
Options we are considering:
Mirror the public supermarket to the internal one and handle conflicts when updating your mirror.
Or
Enforce some kind of namespacing for internal cookbooks uploaded to internal supermarket

Any thoughts/feedback?

To your point about silently merging:
it does show the source in the Policyfile lockfile for each cookbook where it was fetched from so it is quite visible and is the developer's responsibility to handle that. Although I agree we can make it more visible about the conflicts if possible, but it should not fail when the cookbooks being used are not conflicting at all.

[lamont-granquist]

It might be useful to be able to tag sources with a name and then use the name later in a cookbook statement or something like that.

There's some interesting issues and solutions here in rubygems-world:

http://stackoverflow.com/questions/16072412/ruby-bundler-multiple-sources-in-gemfile

I kinda like wrapping the gem line in a source block, that is fairly neat.

[lamont-granquist]

CVE against rubygems for an attack leveraging the ambiguity that @danielsdeleo is talking about:

http://bundler.io/blog/2014/08/14/bundler-may-install-gems-from-a-different-source-than-expected-cve-2013-0334.html