Support predefined shared builder in kpack

[showpune]

Hi,

When to create a new service, we need to create a default builder for the quick use. As the builder is very big ( about 7G), it will take more than 5 minutes to create it using builder CRD in kpack, most of the time is token for copy the builder image to container registry.

We just wonder if we can create a builder without clusterstack and clusterstore, just refer to a predefined builder, so we can just reuse the predefined builder without create a new one use stack/store in container registry, only when we want to customize the builder, we can copy the builder to the container registry.

To use the share container regitry, the definition of the builder in k8s will looks like

apiVersion: kpack.io/v1alpha2
kind: ClusterBuilder
spec:
tag: paketobuildpacks/builder

Is it supported in kpack

[tomkennedy513]

kpack does not support creating a builder without a clusterstack. As of 0.11.0 you can create without a clusterstore but the buildpack resources will need to be available to satisfy the order defined in the builder spec. I am a little confused by the use case though, because oif the builder is the same as an existing one, why not just use the existing one? As for the the tag, you can make it whatever you want.

Also, If two builders are being written to the same repo, you will not be writing 7gb to the registry with each builder since it will be able to just reference layers that already exist in the registry. Are you relocating buildpacks/store imaegs and clusterstack images to your registry? If not, then kpack will have to stream those layers to your registry when writing the builder which can take a bit of time.

[chenbh]

kpack intentionally moved away from the idea of using prebuilt builders, this was actually the topic of the very first RFC (discussion for this in #439).

There's a project design/architecture problem in that kpack is designed to be declarative, but if we introduce the ability to use prebuilt builders, then we have have to make part of the system reactive. The main symptom of this problem is how we do automatic rebasing of images.

Kpack tries to do everything declaratively, this means that everything is modelled as a kubernetes resource and it is up to the user to modify these resources to reflect a change in the real world: If a new version of a buildpack is released, the user must update the ClusterStore (or the Buildpack resource in >0.11.0) manually. Similarly, if a user wants to update the stack, they must interact with the ClusterStack resource. This way, we can easily tell if an app image requires rebasing because we are able to track all of these resources and act upon it as soon as they change.

As a side note, by offloading this responsibility of "syncing" to the user, kpack gets to take shortcuts and keep it performant even on clusters with lots of workloads.

However, if we were to support prebuilt builders and still have some sort of automatic rebasing, the builder image would need to be a floating tag (myrepo/my-builder:latest) instead of an immutable digest (myrepo/my-builder@sha256:1234). And in order to determine if a rebase is needed, we would need to constantly watch the builder image and detect if the digest changes. This breaks the responsibilities of kpack as we would rather have the user to be in charge of syncing the external world to the kpack resources.

Caveats:

1. But what if I don't care about triggering changes on new builder images? What if I want to use an immutable digest?

   Then the Build resource is perfect, you will be the one to determine when to generate a new build

2. But isn't there already precedent for reactive design in the git source? It has to poll periodically to determine if a new commit was made right?

   Yes. And we don't like it. This is a functionality that we need, but should really be abstracted into another concept (like the Monitor concept in the RFC) rather than keeping it in core kpack

[tomkennedy513]

There is an interesting idea in this that we haven't really talked much about historically which is the idea of a multicluster kpack where you have some control cluster that has the buildpacks and stacks and builders and then you have worker clusters that do the builds. That's straightforward to accomplish today with the Build resource but not really with the Image resource.

Reintroducing a "remote builder" resource via a tag is potentially a way to enable this type of functionality without the needing all clusters to have write access to the builder registry. Agreed that the need to poll the registry would be a departure from the declarative aspects of kpack, but we do have some polling concepts today in the form of git polling when providing a branch to the Image resource.

[driv]

So I have no way to just use the same builders I'm already using on my local machine out of the box? Seeing how easy it was to build with pack I thought kpack would be just a way to bring this simplicity into the cluster.

What do I need to define in kpack to be able to replicate pack build spring-boot-buildpack --path . --builder paketobuildpacks/builder:base for a standard springboot application?

[driv]

NVM, I just found BuildPacks https://github.com/buildpacks-community/kpack/blob/main/docs/buildpacks.md

[edmorley]

Hi! We (Heroku) publish our own builder images, which can then be consumed by Pack CLI users and Heroku's own internal CNB build system etc.

However, at present it's not possible for kpack users to consume these without recreating the builder definition as custom kpack resources.

This means anyone using kpack has to then manually keep their definitions in sync with eg:
https://github.com/heroku/cnb-builder-images/blob/main/builder-24/builder.toml

This seems like a significant limitation of the current design?

Context: This came up in heroku/buildpacks-procfile#243