Add expeditor #1213
Add expeditor #1213
Conversation
tduffield
force-pushed
the
tduffield/add-expeditor-config
branch
from
987c5fb to
80c4c5f
Compare
tduffield
force-pushed
the
tduffield/add-expeditor-config
branch
from
80c4c5f to
384cfff
Compare
|
This is cool! |
|
@vinyar I will post more about that later (its not a public repository). :) I just need to get this code in place to facilitate the migration. |
|
@tduffield It not being public seems like a reason to not merge this yet since we (those not at Chef Software) have no idea if this is a good change or not other than taking your word for it. |
|
@coderanger Expeditor is a replacement for the stuff that is currently going on behind the scenes including our version bumping, building triggering, etc. While the code that is executing the stuff is not public, this is actually more transparency than we had before. |
|
Okay, so why does that need to in a private repo such that no one else can look at it? |
|
If y'all want to use private tech for your own projects, that's cool, but it doesn't fly on stuff that touches the community :) |
|
@coderanger The reason it isn't public is because the code is specific to Chef. I'll need to walk through and make sure there isn't any information that we don't want exposed (i.e. internal URLs, etc). |
|
Okay, that sounds like a good thing to do and then we can loop back on this 👍 |
|
To try and explain my resistance here a bit, I'm increasingly concerned about how much hidden stuff there is in the CI and build process for Chef as a project. I am eternally thankful that Chef Software as a company is willing to devote resources towards managing builds and testing for the FOSS side of the equation, but I think we should ensure that even if Chef Software was unable to continue that for any reason, the community could take over (within reason, I suspect AIX and Solaris builds would hit the cutting room floor pretty quickly for example). So previously we had this stuff basically one-off'd in each repo but in a way that I could go plug in to a new Jenkins tomorrow if I had to. By shifting this logic to an external, private gem we're in a much more precarious position as a project. I agree it's functionally equivalent and nothing here will reduce code quality or harm users in any way, but there is a deeper existential concern on my mind. Slippery slope is a shitty argument and you would be 100% right to call me out as assuming the worst in a situation that doesn't call for it, but still, I worry. |
|
And as a secondary concern, our CI process is getting immensely complex. This would probably be more on the simplification side than anything but changes in gnarly code get my attention more than other places since any changes will be the basis for future progress. See also: the gemfile management system which kind of ... got away from us. |
|
@coderanger so I had a chance to talk with some folks internally. While there is nothing "special" per-say about the Chef Expeditor code, it is heavily customized for Chef Software. As such, it's not something that we necessarily want to support as an open source project. Additionally, there is more than just version bumping occurring in Expeditor, and we've identified some surfaces that we'd rather not publish out to the world. As a middle ground, we've given the core maintainers group read access to the repository. Any maintainer will be able to see the code and how it operates. Is that sufficient? |
|
@tduffield That's a little less bad, but if I don't have the legal right to use that code (which would be the case if I had read access to it under a proprietary license) then we're still in the same position of the project being explicitly proprietary instead of "accidentally" proprietary. And I don't think that is okay, the precedent of us having not gotten around to fully documenting/opening some of our backend-y tasks is very different from purposefully making Chef's release process require code that is private to Chef Software. |
|
@coderanger Nothing that Expeditor does is required to release Chef or ChefDK. Expeditor is a chat ops utility that Chef Software uses to automate otherwise manual tasks including version bumping. If Expeditor were to disappear or go offline, you would still be able to ship Chef or ChefDK, you would just have to perform tasks like version bumping manually. |
|
@coderanger as to your point about fully documenting our backend-y tasks, that is what https://github.com/chef/chef-rfc/blob/master/rfc086-chef-oss-project-policies.md was also intended to cover. In that document we outline a great deal about how Chef and ChefDK are built and released. |
|
Okay, so let's zoom out a bit. Chef Software Inc and Chef the open-source project exist in kind of a delicate dance. Chef Software owns the trademark on "Chef" (and a few related things), owns copyright on some but not all of the Chef code, and sponsors development time on Chef using some of its staff. In the abstract, the two should be at least somewhat separable, the project should be able to exist without the company and vice versa. In reality things are more intertwined than that, but as a major project contributor that operates outside the company, I try to ensure this kind of project continuity plan is at least mildly protected since there may come a time when Chef Software doesn't have the resources to support the project to the degree it does today (I'll leave out the "acquired by Oracle who then paints a bullseye on the FOSS community" scenario though that's one I worry about too, just less so). As a matter of how things have evolved over the years, Chef Software has also provided some services to the project that are more capital or maintenance intensive, like producing and hosting installers (which, again, super grateful for and hope will continue). At least initially this build farm being private was simply a matter of Chef Software owning the physical devices and paying for the time of all the people involved in release engineering, but this wasn't part of the day-to-day development workflow (with Travis running our day-to-day CI stuffs) and anyone could build the same installers on their own hardware if they needed to run a release (indeed, many of the early installers were hand-built on someone's laptop) so even if Chef Software was unable to build or host the installers for us, the work to replicate the 99% use case (read: Debuntu and EL) was some If we wanted to take this as an opportunity to make things more safe rather than less, how about you move all the secret-y strings in to environment variables and then open the repo under a FOSS license with a clear note that no user support will be provided and it is only open for project continuity purposes. Alternatively we could do something like the old Chef Server split, make a minimalist FOSS version of whatever scripting system Expeditor uses (looks like some form of named hook scripts from the contents of the PR) and make that available. This might be a 10 line shell script, or it might not, I don't actually know what Expeditor does so I am assuming it does a lot more than that (again, because if it was a simple project I am assuming you wouldn't have a desire to keep it proprietary). Hope that explains my viewpoint in a little more depth. And again, this is not intended to be a critique of Chef Software's involvement in Chef as a project, just trying to make sure the project stays in a healthy state. |
|
I pretty much entirely disagree with your summary of the situation around CI, actually. There's almost no proprietary code in the stack; there's a Jenkins instance that sits on a bunch of expensive hardware that we pay to host, and it runs some very trivial shell scripts (virtually all of which is released either in chef/chef, here, or in omnibus). We then pay for an Artifactory license to host the artefacts (again. because it's convenient), and give Fastly some large amount of money to CDN those artefacts. The end. I don't see what can really be changed here besides some waffly crap about donating hardware and money to a foundation that would be de-facto controlled by Chef because all of the time, money and resources would come from Chef, and honestly that's a waste of everyone's time. The reason Expeditor is not open source is because it's utterly uninteresting and provides no-one with any benefit if we do so, and because we don't want to commit the resources to supporting it for other people, which we must do for it to be open source. We can drop it and run, but that's not how Chef operates, and I don't think you mean to encourage that behaviour. If Chef ceases to be, the velocity of changes and releases drops to a point where the version bumping and notification of CI, which is 100% of what this bot does, becomes sufficiently negligible that a human can do it. |
|
@thommay The current minimal state of "behind the scenes magic" is exactly why I called this out as concerning to me. If Expeditor is minimal enough to not really move the needle then so be it, but I'm not a big fan of that as an overall plan. Maybe document what it does to enough of a degree that it could be replaced easily? That seems like a good thing anyway if this is going to be part of our of project. That would make it a lot easier to judge the full impact of this change on the project. |
|
We wrote an RFC documenting exactly that, and Tom linked to it above. |
|
That doesn't explain anything about what that config file is or why it appears there is a second bundle install going on. I have no way to judge if anything in this PR is a bug or not. I can certainly make guesses, something like that shell script snippet is run from Jenkins during each build as well as some kind of Slack notification but it's a little weird to me to have code in this project's build flow that I can't really change because I don't know where it gets called. Even just explaining what Expeditor does in here (more specifically than "it bumps versions") would be something for the community to go on. As a concrete example, if I make a change to the platforms in the gemfile should I update the install script? |
Expeditor is the next-generation utility that we'll use for version bumping. It will include additional features that we don't currently have like the ability to not bump a version on a specific PR. Signed-off-by: Tom Duffield <tom@chef.io>
tduffield
force-pushed
the
tduffield/add-expeditor-config
branch
from
384cfff to
4cc31a3
Compare
Signed-off-by: Tom Duffield <tom@chef.io>
tduffield
force-pushed
the
tduffield/add-expeditor-config
branch
from
4cc31a3 to
2fd5b54
Compare
|
@coderanger does 2fd5b54 supply the necessary context? |
|
👍 That helps. And would I be correct in saying that if you ignore all the notification features, the effective change to an imaginary Jenkins config would be |
Signed-off-by: Tom Duffield <tom@chef.io>
|
@coderanger yes, what Expeditor is doing is effectively no different than what we're doing today. I also moved stuff entirely into the single file so that things are extra clear about what is part of Chef and what Expeditor is doing. |
|
@coderanger I also explicitly declared defaults and added some more context to the config.yml file for you. |
Signed-off-by: Tom Duffield <tom@chef.io>
tduffield
force-pushed
the
tduffield/add-expeditor-config
branch
from
b7b5b14 to
b56f7a9
Compare
|
👍 Just, again, for clarity: the actual version bump itself is a feature of expeditor so this won't use the bump tasks in Rake anymore? |
|
@coderanger correct. After we formally move to expeditor, we'd remove that code. |
|
Good enough for me :) |
Please note: the VERSION file is not technically in use yet. I just needed to add it to ensure everything was in place. Signed-off-by: Tom Duffield <tom@chef.io>
tas50
added
Expeditor: Skip Changelog
Description
Expeditor is the next-generation utility that we'll use for version bumping. It will include additional features that we don't currently have like the ability to not bump a version on a specific PR.
Check List