An RFC to standarize on two workflows

[adamhjk]

This RFC proposes two workflows as "supported", and abstracts
as much of their differences as possible through the use of
additional generators and functionality in the 'chef' command.

If accepted, this RFC would also serve as a guide for tool makers,
as to what the requirements are for maximum converage in the Chef
community.

While other workflows are possible, this RFCs job is to say which
ones we support officially.

[adamhjk]

A little more background here, on top of whats in the commit message.

In watching and listening to folks frustrations with workflow, it feels like you can boil it down to a few things:

1. The monolithic chef-repo works great for a lot of people.
2. The "Berkshelf Way" (as opposed to Berkshelf the tool) doesn't solve a problem everyone feels they have. Meanwhile, for those who feel like they have that problem, they <3 it.
3. We have put ourselves in an awkward spot, where much new tooling targets the Berkshelf Way implicitly.
4. Regardless of the workflow, everyone wants the testing tools to work.
5. Some folks dislike the Berkshelf implementation

This RFC proposes making two workflows "supported", which means we expect them to work all the time, be supported in documentation and the community, and for tool builders to target them.

It also proposes adding generators and workflow commands to the 'chef' command in Chef DK. This means we can make the target for documentation and tool builders much smaller, as the work of noticing which workflow you prefer is done by the tooling. Where there is overlap with knife, we are going to subsume that functionality, and make it compatible at the options level (and perhaps turn the knife plugins themselves into wrappers.)

Lets fix it! :)

[mrjcleaver]

Great!

It would be nice if you could also point to an official write up of "how to migrate from librarian-chef" as well.

Best,
Martin.

[adamhjk]

@mrjcleaver I think that's a good footnote - as of now, we aren't calling out what the tooling is under the hood at all, and that's on purpose. The goal is to get the two most common workflows (monolithic and independent software projects) as close together as we can make them, while preserving backwards compat for both. Then we can start to move forward from there.

For example, the experimental Policyfile work that @danielsdeleo is doing may very well replace Berkshelf, have a better resolution mechanism for the vendor branch pattern, and maybe even solve for librarian-chef.

So I'm +1 on having migration guides, but lets settle on the two workflows you would migrate to, first.

[failshell]

One of the issue my team is facing is relating to backward compatibility.

Examples of this:

• chef 10 => chef 11
• yum 2.x => 3.x
• Berkshelf 2.x => 3.x

We understand the need to move forward, but all these breaking changes are disruptive to our internal workflows. Since our Chef tooling is there to enable us, we feel we shouldn't have to change our workflow all the time.

So we end up delaying upgrades of these tools. Which at some point we have no way to avoid. Either we do like we did with the yum cookbook, merged 2.x and 3.x functionalities in a frankenstein internal version of it, or we do like with Berkshelf and wait and wait. Which is not helping us either, as we are diverging further from the community, which is preventing us from keeping up with some good updates.

Bottom line, we'd like to see more backward compatibility.

[miah]

Hi,

Why do we even have to re-invent the wheel here? Ruby has a system already. Its called 'rubygems'. Yes; The namespace is flat, and it is not simply a collection of Chef related tooling. This does not prevent sites like Supermarket from doing its job. While there may be "problems" (yours to define) with rubygems, we're not in the problem space alone, as we are in Chef. The entire Ruby community is there to aide you in creating and debugging Gems. If you ask #ruby about cookbooks you're going to get mostly blank stares.

metadata simply does not provide enough usable functionality to be the primary source for dependency resolution. features like suggests and obsoletes are often unused, and I don't think suggests even works.

There is no reason, other than 'talent pool' that Chef cookbooks cannot be rubygems. Lets look at some of the reasons rubygems are superior to any of this other off-road development;

1. Rubygems itself is older than Chef; 0.8.3 December 7, 2004 (128 KB) Link
2. There are books written about developing Ruby, and Gems. All of the information here is applicable.
3. Many people have written gems, the number of people who can assist greatly outweighs the number of chef cookbook authors.
4. Gems can be tested without convergence.

Now some more arguments against 'cookbooks'

1. By creating our own 'cookbook format' we've just created a walled garden. The articles written about designing cookbooks clearly don't come out of the 'programming' world; many design patterns are completely ignored and anti-patterns such as Action at a distance are embraced with welcome arms. If anything, many of these articles show you what not to do and teach you really bad habits.
2. recipes are impossible to test well. The majority of the tests simply test that Chef is being Chef. Chef already includes tests for all of its built-in resources; there is no need to test that functionality (poorly) via chefspec.
3. libraries still doesn't recursively collect files in sub-directories.
4. libraries dynamically defined namespace, especially when coupled with the DSL's.. How do I find the constant of anything?
5. attributes everything about Action at a distance applies here. Instead of taking in a 'input' we are always passing these node attributes directly. node attributes are often a 'hard coded' key eg 'node[:foo][:bar][:baz]' and that may not apply to my infrastructure.
6. templates handy for very specific tasks, but in general you code yourself into a corner. You must update your template when you update code, or change attribute names.
7. lazy evaluation should probably be on by default. lol at all the NilClass type errors you'll get (or should get but probably aren't because your doing string interpolation with nil values).

So, with all of that in mind. Lets look at Chef itself;

1. Bundled Resources, and Providers are written as PORO's, and include tests written in rspec.
2. Bundled Resources, and Providers never reference node attributes directly.
3. Argument input types are inconsistent. eg; ports, sometimes they are a String, sometimes they are a Fixnum
4. Chef::Resource and Chef::Provider have lol levels of documentation. Quick! Go on #chef and ask anybody who has written a cookbook to tell you what load_current_resource is supposed to do.
5. Chef itself doesn't even serve as a education on idiomatic Ruby. (go run Rubocop for fun; 1162 files inspected, 58201 offenses detected)

So.. where do we go from here? Should we be writing cookbooks that wrap other cookbooks that wrap global variables that don't define resources for items that are created by your package manager's %post-installation routine?

No.

The majority of these problems can be solved by simply not re-inventing the wheel. Yes, writing software is hard. Yes, the number of useful generalized software abstractions already created is a pittance.

Can we work on solving these problems, instead of adding cruft on another layer of cruft?

[erichelgeson]

One thing I ran into with my workflow was developers consuming/collaborating with my recipes. Before it was as easy as vagrant up, but then you can't use chef-zero, vagrant berks does have issues installing sometimes, etc, etc. It has been made a bit easier now that chef-dk is out to say "just kitchen converge.", kinda.

It's really the discrepancy between what supports berks, what supports zero, and the dev tooling we have has to translate so our consumers can easily have the products we create with a simple command and few dependancies. Maybe chefdk is that answer.

If we can't share our cookbooks after we have all these awesome tools what good are they?

This is probably a tangent of the workflow (TM), but important to consider. Java developers don't want to spend time resolving Ruby gem conflicts. Helps adoption in the dev world as well.

[adamhjk]

@failshell totally. One thing this proposal doesn't do (explicitly) is break back-compat for anyone.

One of the things that has fed the problems with back-compat was precisely our lack of saying a given workflow was supported long term. Once we settle, we can move forward with implementing those systems in a way that stays stable and sane. That's the hope, at least.

[adamhjk]

@miah I'm not sure this PR is the right place to revisit this, but hey - lets! :) I'm happy to talk about extending how we do things to see what happens when you port the universe to raw rubygems. I believe (and I still do, but hey - I've been wrong about all kinds of things) that for a huge segment of the population that uses and adopts Chef, they precisely don't want to be involved in the broader ruby ecosystem. They really don't want to have to deal with rubocop thinking their use of double quotes is bad, they really don't want to have to figure out bundler, etc etc.

I know the counter, which is because of that we're re-inventing the same problems.

So I have a proposal - lets build a prototype, and fix the obviously bad things (for example, you should just fix the library shit in the cookbook loader and be done with it). I'm down to try it out.

[adamhjk]

@erichelgeson I think Chef DK and something like this proposal is the answer. If we can settle on basically one set of common workflow commands, we can free ourselves up to innovate on implementation without breaking the universe. An example would be a spike of @miah's changes - those would be deep and sweeping, but if we had the common interface, she could just provide another implementation, and we could all see how we feel about using it.

[salgo]

I'm a big fan of the Berkshelf way but I can understand why people might not get on with it, especially when they're first starting out with Chef. My opinion for a while has been that there's too much magic going on when people start using Chef; they see that something worked but they don't really understand why. Really understanding what Berkshelf does surely requires that one has used the chef-repo and encountered a problem? I'd suggest for that reason that we need both to help people learn at least.

A problem that people who work with Berkshelf every day might not realise is many of my users always encounter breaking changes every time they use Berkshelf. Progress is great and the recent Berkshelf changes have been fantastic but I've had developers have issues with vagrant-berkshelf and sys admins having problems because their Berkshelf.lock format changed and just don't know why.

To be able to recommend Chef and associated tools I need to know they won't break because of a dependency at some point. At the moment I can only really be sure that the chef-repo will continue working for people. Even though I use it for almost everything, Berkshelf isn't quite there reliability wise for me so I think you should keep both.

Yes, I'm basically agreeing with you but you did invite comments!

[adamhjk]

Thanks, @salgo!

[miah]

I'm totally fine with people not wanting to follow style guides. But the examples in the community should be shining examples.

As far as cookbooks, 'jane smiths opinionated apache2' isn't really valuable to the community. It results in 'wrapper' mess ecosystem. If instead, we built generalized libraries that handled most cases we could solve many of those problems.

Imagine;

Maybe you want to load your existing configuration off disk and transform it into a hash, then merge in your local settings. Or, maybe you want to just flat out create the configuration, transform it, and then write it to disk.

Doing all of that, with only the template resource is impossible. The template resource can't do this. Yes, we can render the file, based on some pre-defined structure and passed through variables. But its really not flexible. Furthermore, the names and structure we pass to the variable is almost always going to be different. See log_dir, log_directory, log-dir, etc.

There is no reason, we can't define some generalized data structures that represent a service and use this consistently.

There is no reason we can't define a generalized resource / provider that handles much of this functionality.

'file', 'template', 'execute' are all too low level. We need higher level abstractions.

We can define a service as:

• a system with a managed running process
• a configuration
• data producer (data, logs, and cache files, etc)
• a data consumer (network interface, port configuration)

If we defined, and built a system to manage that generalized structure, we would likely solve 99% of the 'development woes' and reasons behind using wrappers.

I think chef-container does this; using containers.
I've been hacking on chef_instance (which is still in its infancy); which aims to do this.

Wrappers, and 'Berkshelf way' are the result of a problem. Rather than addressing the problem, we're trying to figure out ways to live with it.

[adamhjk]

@miah I agree with you. My experience with the idea of shared cookbooks, for example, is that the ones that actually provide great utility are the ones that provide abstractions, not the ones that provide implementations. (An uber-service abstraction is more useful than yet-another apache2 cookbook)

We're solving two different ends of the problem. Right now, we have a messy middle of workflow, which is generating a heap of frustration. We have another set of people (you, noah, dan, etc.) who are working to move the primitives forward. I'm proposing we build some unity into the workflow, work on the primitives, and then start figuring out how to move people from where they are to a better place in a way that doesn't leave anyone behind (who doesn't choose to be left.)

[danielsdeleo]

@miah I'm with @adamhjk here. What you've listed is a grab bag of problems that are IMO mostly unrelated. To be clear, I'm 100% with you on points 1-4 in "Lets look at Chef itself." Let's fix those. As for (5), I have some pretty severe disagreements with rubocop, which I won't get into here. That said, we're open to adopting a style guide for the chef code base and/or a recommended one for cookbooks, but no one has stepped forward to write one yet.

And on the "action at a distance" point, this is something I think about a lot. I have a half-written implementation of a possible improvement for that here: https://github.com/danielsdeleo/chef-data-bindings But then I think about @coderanger's argument that you don't need that at all if you just compose custom resources and providers and that's pretty convincing, too. I think it's definitely too damn hard to jump from "I can write a recipe" to I can write a resource/provider (side note, in lots of cases you only ever have one provider and enforcing that separation makes for unnecessary complexity).

[jeffbyrnes]

I'd agree with @danielsdeleo’s sentiment, and reinforce what @adamhjk says: this RFC is meant to sort out the immediate pain of not having a clear set of documented workflows (regardless of the state of the primitives that lead to those workflows & tools), which I think is an immediately attainable goal that can decrease pain for a lot of users very quickly.

I'd also agree with the sentiment that a lot of Chef users are not interested in participating in the larger Ruby community. I, for one, have very little interest in hacking on Chef’s code itself, merely in using Chef as my configuration management tool.

That's a bit like not really knowing how a plane works, but gratefully using one to get from point A to B. A bit of willful ignorance, but I have limited bandwidth to spend on things, and most of that is taken up by building internal tools & handling config mgmt. itself, not understanding how my config mgmt. tool actually works.

My cohort @eherot might have some good things to chime in with here, as well; he & I have struggled together to hack our own workflow into its current form.

[ranjib]

i do not understand why an RFC related to chef needs to formalize workflows? ChefDK != chef (rubygem that we all love). Doing this will not only bring more maintenance burden (current backlog in chef bugs, feature improvement and vast majority of cookbooks indicate the reverse). This will reduce not increase, the chances of having battle hardened software.
Omnibus is a classic example of this, most of us are still on ruby 1.9 (even if 2.1 is stable with so many awesome features). The sad part is, we used to run chef as any other ruby app, albeit with some added complexities. But post omnibus, its harder just because the reverse engineering required. Taking this to a workflow and chefdk will greatly increase those pain points.
More importantly i think its bad for the community, as this will inhibit future tool development around chef (any one who writes an alternate tool will be discouraged by the fact that there exist one or a set of recommended tool).
I understand that a lot of community members need a recommendation, all in one package and then workflows on top of that, but why that has to be part of chef core? like chef-metal, chef-server these can be dealt outside core chef, have a chef-workflow project and do these things as part of that project. which will put the onus of improving and backward compatibility to those who seeks a standard workflow.

[adamhjk]

@ranjib They are outside of core chef. They are being implemented in the Chef DK, which is pretty much by definition what "chef-workflow" would be.

The goal is actually to enable more tool development - if we can settle on a common abstraction for the most common use cases, we can swap the implementation as we need. We're not going to stop shipping Chef as a rubygem, and you can continue to use it that way. There is ample evidence over the last few weeks/months that we have a particularly acute issue with workflow specifically. I want to fix that and enable folks to both bet back to work (using chef) and start to innovate again (improving chef).

[adamhjk]

@ranjib as an example, you could see @miah being able to write the code that provides the rubygems based workflow as an implementation of the chef commands in the RFC. We could then actually try it out (hell, we could ship it) and see how we feel, rather than being bogged down in theoreticals.

[danielsdeleo]

@ranjib The issue we're solving is to make a commitment to keeping a certain way of using Chef supported in the future. In general, this has been implicit in the past. For example, we ship berks as part of ChefDK, but we never considered removing all the knife commands that implement the previous monolithic repo workflow. However, since we were not explicit about this, people became concerned that we were gonna force everyone to start using berks for everything. In other words, lack of signal being interpreted as signal.

As for Omnibus, maybe we can talk in a different venue about "its harder just because the reverse engineering required." Supporting non-omnibus installs is the only reason we've waited so long to drop Ruby 1.8, for example. And we also shipped 2.0 and 2.1 support in the minor releases immediately after those versions were released, which we wouldn't have needed to do if omnibus was the only supported install method.

Re: "this will inhibit future tool development around chef," I think the audiences are different enough that this doesn't matter. People who are willing to be early adopters, deal with compatibility breaks, etc. are a different set from people who want one stable thing that works without changes forever. I think the early adopter set will be generally more likely to be aware that they have a path to becoming a "blessed" workflow (create a pull request to update this RFC), if that is a thing they want long term.

[jblaine]

Stating the obvious: I see the overarching difficulty being that Chef is a tool that sits right in the negative space between the letters "v" and "O" in "DevOps". You have career-level developers working with Chef, development-experienced operations, operations-experienced developers, and career-level operations-only folks involved with Chef. It's a messy space to live in, and there's nothing wrong with that (cue Al Franken's unfunny Stuart Smalley character from SNL).

The RFC sounds good to me, though Ranjib's idea of a 'chef-workflow' framework as the middleware (assuming I understand him correctly amidst his other concerns) seems like it might be a worthwhile implementation approach.

[johnbellone]

@miah I agree with your point about using the Ruby ecosystem to our advantage, but possibly for different reasons. When dealing with hundreds of developers looking to take more ownership over the configuration of their application infrastructure package and release engineering becomes a big problem. I would much prefer solving this once (which we have done already) for Ruby and piggy back on that for a Chef workflow. I jive with what @adamhjk says - let's develop a workflow and test it out!

In the upcoming releases Ruby(gems) will eventually merge with bundler and it'll be a lot easier to tell developers to simply use a tool that they already understand, bundle install and possibly run the chef or berks command to manage the state against the Chef Server.

One thing that @miah brought up and that I agree with is the problems with metadata. I imagine that some of could be solved with the Chef Supermarket (e.g. validating metadata prior to releasing a cookbook) or even with a tool similar to Yeoman which would give you a simple CLI to fill out all that information during cookbook generation. Either way I'd like to see more flexibility ala Rubygem Specifications.

[damacus]

So... Berkshelf/Librarian-chef are purely part of a workflow designed to fix specific problems I may suggest taking these tools forward into the default chef-cli workflow.

Berkshelf & Librarian-chef's primary goal is to solve dependencies above all else.
The Berkshelf way, as I see it, is set of cookbook patterns that I will not cover here. Some patterns are more useful than others. I would like to break "cookbook patterns" out into a separate RFC and discussion if possible.

This post is heavily swayed to Berkshelf as this is what we use internally for our workflow and is what I have experience with. My main goal here is to suggest the way we work with Chef/Berks is not the best way, but more: "this works well for us and has many benefits, lets decide on one recommended CI way of doing this".
I find myself suggesting this workflow for many people on the IRC channel to get around issues they are having. No flames and many <3 please :)

Berkshelf?

Berkshelf does a few extra things for me past knife (I don't know about librarian-chef as I don't use it internally so if someone could fill in that gap that would be great)
Checks your metadata.rb for required attributes past knife.
name, version (VERSION file).
This seems sensible to me and something that could be rolled up into the chef tool.

Workflows

Something we've pieced together over the last year or two is a decent workflow for developing and promoting cookbooks. Every problem we acknowledged Berkshelf, put it to one side. Then found ourselves taking bits from it as @reset and team already figured out the answer.

Internal Cookbooks

Our internal workflow is a follows:

• cookbook written and added to git
• git push
• web hook fires off a job in the CI server
• CI runs automated tests
  • thor version bump
• berks upload

Some things to note here:

1. version control, always.
2. web hooks. I'm bad, as is everyone else at remembering to run the test suite. Let the butler do it for you.
3. Chef-DK runs on the integration server so we definitely have the same version of rubocop/foodcritic everywhere (thanks for this one)
4. Thor does version control for us, each release has a git tag associated with it e.g. 0.0.9 so that you can easily compare between different versions of your cookbook. I would suggest this is essential for any infrastructure-as-code.
5. berks is used to upload the cookbook, this double checks the dependencies before it is uploaded, I see no reason why this behaviour and use of the same dep-solve work can't be incorporated into Chef-cli.

A supported workflow should emphasise the tools we've chosen in ChefDK namely Rubocop and Foodcritic.

External Cookbooks

I stop using berks at this point because it gets into the way.
This is purely a knife cookbook upload. However this misses out all of the above niceties, such as... cookbook dependencies. Chef cookbook upload, again could do this for you: look for the depends in the cookbook, goes "oh no you haven't got coobook X, I found it on the supermarket, do you want me to fix that for you?"

Versions

Cookbooks that have tags on them in github are easier to pull but what's even easier is if you don't need to rely on github at all for cookbooks. Supermarket superseeds this in that we just pull it down from there with a proper changelog(!). So I now believe that "problem" has been fixed by a massive community site overhaul. Internal cookbooks and version constraints are handled by the above "internal" process

Environments

This post is now getting to long but...
Environments have version pins e.g.

	test	development	production
unicorn-monitoring	~> 0.1.17	~> 0.1.17	= 0.1.19
VLE	= 0.0.18	= 0.0.18	= 0.0.18
nyan-cat-pictures	= 0.0.17	= 0.0.21	= 0.0.17

This should be heavily encouraged as it is a powerful part of chef. Rolling back parts of the infrastructure that you suddenly realise you've made a bad change to because that packages isn't quite ready yet is a life saver.

There are ways of setting this workflow up in a Windows environment that I haven't covered here, so we should try and make this process as external-tool agnostic as possible.

[stevendanna]

"two reasons" -> "three reasons" :)

[danielsdeleo]

@damacus I think your last statement sums it up, "we should try and make this process as external-tool agnostic as possible"--this is why the RFC doesn't mention berks or any specific tooling. What you've written up is an excellent way to use the single cookbook per repo workflow to achieve desirable outcomes leveraging existing APIs and tooling.

What we want to make clear in this RFC is that the way you organize your code will continue to be supported, even if/when tools and APIs change. For a concrete example, see #20 which proposes a new API which enables (what I believe to be) a simpler and more powerful way to move a collection of artifacts (a cookbook set and run list) through your various deployment stages. If that API (and supporting tooling) existed and you were using it, many of the specifics of how you use Chef would change but the overall shape would be exactly the same.

[damacus]

@danielsdeleo that's exactly what I was trying to get at here. The existing tooling, berks et al are all solving problems that we could fix internally.

What I tried to propose above was a set way of doing it but pick your own tool. Those tools should work together, and/or not interfere with each other. I'm guessing berkshelf people get bugged by people using librarian-chef and vice versa. A standard metadata format may/would help this: YARFC? (yet another rfc?) I think that would compliment #20

I think this RFC process has come at a good point in time, we are maturing as a community and product set. At a minimum I'd like to see this as a suggested tool chain/workflow:

• Testing - Chefspec, serverspec
• Linting - foodcritic, rubocop
• CI - Jenkins, TeamCity
• Version Control - Heavily weighted in favour of Git
• Versioning of artefacts - cookbook number schema, SemVer
• Environment promotion

There may be a simpler non-Ci way of doing this, however developments such as ChefDK make this much easier.
Maybe the next stages of this RFC process are to bring in well written guides from the community into athe central LearnChef resource that's been vetted by the community? As per the community sessions on IRC we could have something similar with reference material.

What are your thoughts on the dep resolution going forward? ( or should we stay clear of that issue in here? )

[reset]

@ranjib I don't agree that creating an RFC will deter people from creating additional workflows or tooling around those workflows. When I came to the community in 2009 there was pretty clear and documented (well enough) path for how to work with Chef. It didn't work and it was broken so I came up with something new. People can still do that if they believe one of these two paths don't work.

[jonlives]

I think there's definitely a benefit for the community in Chef explicitly documenting tool-agnostic supported workflows. I note that @adamhjk's PR does not advocate any specific tooling for either of the options and I think this is a good thing.

One of the main advantages of Chef as I see it is the lack of any truly canonical workflow, but this can also make it hard for non-expert level Chef users to figure out any kind of best practice for getting up and running - there are a number of community developed workflows in relatively common use including the Berkshelf way, the "spork" way and others, but these have always been community developed and hence there has never been any guarantee that Chef and the way it allows you to organise your cookbooks etc won't change in a way that renders those workflows inoperable.

I must admit, with the increasing popularity of repo-per-cookbook, I'd been concerned myself in the past that in future monolithic repos might not be supported, and there was never any real way to tell whether or not that was likely to be the case. On the other hand, I've never wanted Chef to define a single canonical "Chef way".

Essentially, what I see this PR as doing is defining two broad workflows that Chef will commit to supporting in current and future versions. I see a number of specific benefits here:

• Beginner & intermediate Chef users who don't want, know how to, or have time to, develop an entirely bespoke workflow will have a good starting point.
• Users of a specific workflow style can be assured that it will still be around and supported in future (and should that ever change, it will be documented)
• Tooling developers such as @reset and myself can know that Chef will continue to support those two workflow styles in future releases, which makes planning for the future significantly easier.

I agree with @reset here that this should not deter people from creating their own workflows or tooling in the future - I see this more as Chef simply committing to support two of the most common workflow styles without specifying the tooling used to do so.

I think this PR strikes a good balance between recommending (and supporting) a couple of solid starting points for Chef workflows without being too proscriptive - I'm giving this a big 👍

[hartmantis]

While I haven't personally had any reason to go back to a monolithic repo since Berks/Librarian first showed up, I understand that it fits some folks' needs better, so it's rad that this is happening. 👍

My only concern/question is if the "officially supported workflow" label is likely to make the monolithic repo problematically seductive to newer users. There are reasons many of us switched to individual repos, some of which (at least, at ${PREVIOUS_EMPLOYER}) aren't really evident until you've got an unmanageable monolith full of straight community cookbooks, forked community cookbooks, and custom cookbooks. And unraveling all that isn't an easy migration.

We just need to be careful of threading a needle where the workflows are both supported, but their respective use cases/advantages/disadvantages aren't ignored in the name of them being both supported.

[adamhjk]

@RoboticCheese right - right now, I think it's almost impossible to have a conversation about the merits and flaws of different workflows, because what winds up happening is one side or the other feels unsafe ("Is the berkshelf way the only way now? Cause I'll quit!"). Once we have a solid implementation of both thats as close together as we can make it, we can start talking about why to choose one over the other, how to migrate between them, etc.

[hartmantis]

@adamhjk Makes sense and sounds good. Thanks for driving this forward!

[stevendanna]

Will we still support using the Ruby DSL for things like roles and environments here?

[adamhjk]

I hadn't even thought about it, but yes - we should support both in the
generator. Should we be defaulting to json or ruby?

On Mon, Aug 4, 2014 at 8:41 AM, Steven Danna notifications@github.com
wrote:

In great-workflow.md:

+ +$ git add snazzy +$ git commit -a -m "One snazzy cookbook" +
+
+### How to create your Chef policy (everything that isn't a cookbook)
+
+To create a new type of policy document, from the top of your chef-repo:
+
+ +$ chef generate THING name +
+Note: None of these generators exist yet. They should subsume the functionality of the attendent knife X commands, and be intelligent about which workflow we've chosen.
+
+This will result in a blank, JSON formatted policy document created in the appropriate

Will we still support using the Ruby DSL for things like roles and
environments here?

—
Reply to this email directly or view it on GitHub
https://github.com/opscode/chef-rfc/pull/34/files#r15762152.

[damacus]

Would prefer Ruby for readability if possible please.

[sethvargo]

The problem to consider with Ruby is that's it's interpreted. It's the same problem as metadata.rb as metadata.json. Consider:

name 'cookbook_foo'
version File.read(File.expand_path('../VERSION', __FILE__)).chomp

This will read a local file named VERSION to get the version. This will work, and when you upload to the server, you will get the compiled JSON version like this:

{
  "name": "cookbook_foo",
  "version": "1.2.3"
}

Ruby gives us the ability to "depend" on external systems, which is both good and bad. I could see situations where using Ruby could actually cause us problems.

I'm 👍 on Ruby files, but this is something to consider.

[adamhjk]

So JSON by default, ruby by option?

Adam

On Mon, Aug 4, 2014 at 10:09 AM, Seth Vargo notifications@github.com
wrote:

In great-workflow.md:

+ +$ git add snazzy +$ git commit -a -m "One snazzy cookbook" +
+
+### How to create your Chef policy (everything that isn't a cookbook)
+
+To create a new type of policy document, from the top of your chef-repo:
+
+ +$ chef generate THING name +
+Note: None of these generators exist yet. They should subsume the functionality of the attendent knife X commands, and be intelligent about which workflow we've chosen.
+
+This will result in a blank, JSON formatted policy document created in the appropriate

The problem to consider with Ruby is that's it's interpreted. It's the
same problem as metadata.rb as metadata.json. Consider:

name 'cookbook_foo'version File.read(File.expand_path('../VERSION', FILE)).chomp

This will read a local file named VERSION to get the version. This will
work, and when you upload to the server, you will get the compiled JSON
version like this:

{
"name": "cookbook_foo",
"version": "1.2.3"}

Ruby gives us the ability to "depend" on external systems, which is both
good and bad. I could see situations where using Ruby could actually cause
us problems.

I'm [image: 👍] on Ruby files, but this is something to consider.

—
Reply to this email directly or view it on GitHub
https://github.com/opscode/chef-rfc/pull/34/files#r15766953.

[reset]

@jtimberman that's a great point. I think you've uncovered a bug with 'knife diff'. It should compile the Ruby artifacts in memory before performing a diff. The compiled version should be generated in a consistent way to allow for this (if it isn't already).

Jamie Winsor
@resetexistence

On Aug 4, 2014, at 10:46 PM, Joshua Timberman notifications@github.com wrote:

In great-workflow.md:

+ +$ git add snazzy +$ git commit -a -m "One snazzy cookbook" +
+
+### How to create your Chef policy (everything that isn't a cookbook)
+
+To create a new type of policy document, from the top of your chef-repo:
+
+ +$ chef generate THING name +
+Note: None of these generators exist yet. They should subsume the functionality of the attendent knife X commands, and be intelligent about which workflow we've chosen.
+
+This will result in a blank, JSON formatted policy document created in the appropriate
Since we're talking about workflows, a nice thing about using JSON, e.g., roles, is that the knife diff command works with JSON, but not ruby DSL. This is an essential part of our internal workflow of "make change to .json, knife diff, reconcile unrecognized differences, knife upload."

—
Reply to this email directly or view it on GitHub.

[stevendanna]

I prefer JSON as the default because I am them more confident that the item I am uploading to the server is the same as the item I have on disk. However, if knife-diff and other tools were to learn about the the Ruby DSL and handle it well, I wouldn't care as much.

That being said, I think a prerequisite to defaulting to Ruby is that we support Ruby for all policy types. Currently data bag items do not have a Ruby format as far as I know. To me, the default repo setup should present a user with a consistent experience when editing policy artifacts.

[jtimberman]

Data bag items can be specified using the Ruby DSL, as knife data bag from file uses the object loader in Chef::Knife::Core::ObjectLoader. Personally I would never recommend someone use this, as I think it can lead to "wtf how did that get in my data?" questions.

[sethvargo]

Right, but ChefFS doesn't support Ruby objects - I think that's the big(ger) problem here.

[coderanger]

@sethvargo If this is a statement of desired workflow, we should just fix that if needed :)

[adamhjk]

Ok - from the 8/21 meeting, we are making a few more edits to this document, and then I'm going to . Next IRC meeting.

[adamhjk]

Speak now or send a PR :)

[adamhjk]

@coderanger and @jonlives - this is Decided. :) Please to merge and do the needful.

[jonlives]

@adamhjk @coderanger nice, on it!

[jonlives]

Merged as RFC-019