Proposal: Build isolation

[smarterclayton]

Administrators should be able to run builds on dedicated machines in the cluster. Those machines may not be able to reach other nodes on the cluster, and may have different security policies than regular machines.

Cluster admins should have the ability to enforce this as a blanket rule. It should probably be possible for them to alter the rules for particular namespaces.

Questions:

• Should the end user be able to refine the selector?
• Should the namespace selector be taken into account? (Assumed admin configured, but may be disjoint selector)
• How do taints and tolerationa apply?

Propose global config for selector. Propose build controller not be "special" and have to be an admission controller - instead, have permission to exceed node selector (or have permission to target specific selectors). I would prefer not to make this a new "special" admission controller.

related card: https://trello.com/c/O4CNExeB/792-13-allow-for-a-separate-set-of-build-nodes-ops-rfe

[smarterclayton]

@bparees

[smarterclayton]

Proposal 2: implement this with taints and tolerations. Add ACL support for tolerations by using a virtual resource for roles that checks that the user has access to "create" a toleration matching the toleration name. This would be added to project node selector. This then enables us to move to a fully generic toleration model.

[bparees]

Need a reference to what taints+tolerations are.

[bparees]

whatever solution we come up with here, it needs to allow users to express a node preference so we can solve use cases like "i want my build to run on a high CPU machine" and "i want my build to run on a fast disk machine".

So in my view the admin is setting up the pool of build machines, but the user needs to have the ability to select from within that pool.

[smarterclayton]

I like proposal 2 a lot more

[smarterclayton]

Why does it have to support that?

[bparees]

Because I think it's an important use case and I would not want to design a system in which the build farm is assumed to be homogeneous. You don't think it's important?

there might be other divisions of build nodes as well, such as "has external network access" vs "doesn't have external network access", or geographic divisions.

[smarterclayton]

Not sure that's an end user decision in most cases. In any case, it would require an API change and would be covered by taints, so it's just something we may choose to enable later.

[jorgemoralespou]

After reading about taints and tolerations I agree that user will not need to select which node will run the build, but rather a toleration (via an admission controller) that can be feed by any consideration like type of user, group they belong, type of build,...

One question somehow related: Is there a proposal to make builds happen in pods? Or is this work just to make builds isolated as we can not make builds isolated in pods?

[bparees]

@jorgemoralespou yes we have a card to investigate getting the actual assembly containers to be run as pods.

[bparees]

After reading about taints and tolerations I agree that user will not need to select which node will run the build, but rather a toleration (via an admission controller) that can be feed by any consideration like type of user, group they belong, type of build,...

not sure what this comment was directed at, but i agree as well, i wasn't suggesting users select the node, only that users select the "type" of node (from within the larger bucket of "build-capable" nodes).

anyway i agree that taints/tolerations look like the right solution for this problem (thanks for the link, @jorgemoralespou), as long as we can apply the tolerations to the build pod in such a way that they are cluster controlled, not user controlled (until we get into the sub-types i described).

Do we need to solve the sub-type problem now? perhaps not. but i'd like to at least be confident the solution we're proposing can ultimately encompass that use case.

[smarterclayton]

I agree, I'm just not sure yet we have the tools to do the right thing for users.

[bparees]

Shouldn't part of the goal of this proposal be to define those tools (or determine if we have them), then?

[smarterclayton]

I think it's mostly orthogonal. By "not sure we have the tools" I meant "to support users directly entering node selectors, tolerations, or generally knowing anything about the info". Advanced users may know about the cluster, and be capable of knowing about tolerations. But how do they find those? Get told by an admin via email? Falls into environments? Part of the namespace? Either way, today users get no choice. At best they will get a choice for node selection under the guise of either a toleration or a node label. I don't think we can justify introducing a third way to select nodes, but perhaps environment will do that.

[bparees]

related trello card: https://trello.com/c/O4CNExeB

[jorgemoralespou]

@jorgemoralespou yes we have a card to investigate getting the actual
assembly containers to be run as pods.

@benparees Is there a link? Links help, as you've observed above.

[bparees]

@jorgemoralespou https://trello.com/c/UFDLM6Jb/997-3-first-class-build-scheduling-r-d