Success stories from companies who adopted rules_oci

[alexeagle]

We are excited to help companies and developers build and test container images at scale using Bazel.

If you are currently using rules_oci, we want to hear about your overall experience. How did it help your project or organization?

Or just give a shout-out to the project. We mostly read about issues, so the maintainers would really appreciate your positive feedback too.

Please feel free to post your company and/or open source repository here.
We look forward to hearing from you!

[josephglanville]

We are using rules_oci at Glide right now with Kotlin/JVM and soon with Typescript in production.

We migrated from rules_docker pretty painlessly and haven't had any issues since.

Compared to rules_docker it's way easier to keep working thanks to it's minimal dependencies and the generally simpler model makes it easier to reason about how our images are built and how our code gets executed.

Highly recommand anyone on the fence to make the switch. Big thanks to everyone that has contributed!

[jungleraptor]

We're using rules_oci at Swift Navigation for building C++ and Go applications, and plan to expand our use cases in the future.

Our primary reason for switching from rules_docker was the support for building multi-arch images.

We're very happy with the results so far. The transition was trivial for us.

The rules_oci team and contributors have done an excellent job. Thanks to all involved!

[mattyclarkson]

We have evaluated rules_oci at Arm Ltd to transition away from rules_docker. The evaluation went extremely well and we were impressed by the quality of the project.

We will use rules_oci to create images to provide a developer environment for our engineers. As such, we need to install various .deb packages. We built a small extension that enables download of those:

deb = use_extension(":extensions.bzl", "deb")
deb.download(
    name = "dc",
    group = "bc",
    revision = "2+b1",
    version = "1.07.1",
    cpus = {
        "amd64": "sha256-83l0ioDpG+njq/uYFC7lPoNZMLOlNzvdK1P1ojS1W5M=",
        "arm64": "sha256-85cJFJFIUnEbvZ0uS9lwXnn61qLhI+ZTwAfByLXWHbc=",
    },
)
use_repo(deb, "dc")

This has made it much easier to download multi-arch .deb packages for installation. We would happily upstream that to rules_oci or, to keep this project complexity low, a separate rules_deb.

Our eventual goal is to transition to Bazel from other build systems and not require so may .deb as the developer tools will be resolved hermetically from Bazel.

[josephglanville]

It would be awesome if this could be combined with the work to extract the package manager from distroless as this is a very desired feature. It's possible to build the world with Bazel but it's outside the resources of much of the Bazel community to do so, Debian provides a much more scalable way for many of us to consume upstream software in combination with Bazel.

[alexeagle]

With #309 we are already committing to adding a bit of scope in rules_oci to make talking to package managers easier, so I think that FR makes sense. Making a new rules_deb would be confusing as rules_pkg is meant to be the way to produce them. Here, you just want to "convert" them into tar layers IIUC.

[alexeagle]

The other option is to talk to rules_pkg maintainers as they already have deb rules over there. Makes sense to me that it should support fetching them if it supports publishing them.

[josephglanville]

It takes a bit more than just fetching, as can be seen in Distroless you also need a solution for pinning etc. So while some of this could be implemented in rules_pkg there is likely to be a requirement for a separate set of glue ala Distroless to handle pinning to a snapshot etc.

[thesayyn]

This gap is already filled by https://github.com/GoogleContainerTools/rules_distroless

[hanikesn]

We migrated our mono repo with ~170 images from rules_docker to rules_oci, the migration was painless and after some optimization and due to integration with bazels caching etc. we we're able to improve overall build times off our repo by about 30%. it also made our packaging rules much simpler as we can now rely on the DefaultInfo and PackageFilesInfo providers to configure the contents and layout of our images.

The only downsides are resource consumption and reliability of krane/zot, but nothing that couldn't be handled. Overall a great set of rules.

[josephglanville]

What reliability issues have you ran into and are they purely related to resource consumption or are there other failure modes that the ruleset should be hardened against?

[alexeagle]

Note that crane now has a disk mode for serving the registry, which @thesayyn is trying out and will probably fix the resource usage.

[thesayyn]

FWIW, we redesigned oci_image in 2.x that should be super fast compared to 1.x

[uhlajs]

We are using rules_oci at ISEE for building Python and C/C++ application. See also: Enabling language-agnostic distributed simulation for autonomous driving with Bazel

We migrated from rules_docker really easily. Thanks to the documentation and examples, it is really simple to use. We also love the way how rules_oci is written/designed (with KISS principle). This makes the contribution/bug-fixing way easier compare to the rules_docker!

As was already mentioned, the only downside for us is crane resource consumption. Also, it would be great to extend Images containing Python applications
with:

TODO: document how to get the third-party pypi dependencies into a separate layer beneath the application.

However, we do understand that the developing rules_oci is very time-consuming. We really appreciate the hard work behind rules_oci and also behind aspect-build/bazel-lib, thank you for that!

[wskinner-dd]

@uhlajs did you develop an internal solution for getting third-party pypi dependencies into separate layers? I'll be working on this soon and anything you can share would be much appreciated.

[uhlajs]

We do have some prototype, but nothing stable or general yet. But have a look at aspect-build/bazel-examples!256.

[aran]

We ported our startup code to rules_oci from rules_docker. Certainly an improvement. Definitely we are very grateful for the team's excellent quality work in this space.

Issues we had to work through:

1. Extracting digests for kubernetes manifests: Extract container image hash for use during later build steps #60
2. Hand-implementing special tagging to be able to continue using skaffold: Feature request for Bazel support: Support containers built with rules_oci  GoogleContainerTools/skaffold#8886
3. Wrangling multiplatform differences between rules_docker and rules_oci and our deployment process. We ended explicitly using @{}_linux_amd64 targets for 3rd-party multiplatform images because we cross-compile images and can't send an arm64 image built on a mac to a registry to be run on an amd64 server. So needed to be more explicit about platforms. It also seemed like maybe the oci_pull image did not respect --cpu. Not sure if it works properly with --platforms.
4. Slight API differences between different oci_{} rules. oci_pull supports platforms, oci_image supports os and architecture but only when not on a base, oci_image does not directly support multi-platform images we need oci_image_index, and then push uses the platform. If I want to pull a 3rd-party image, add a layer, and then push it somewhere else, I think I need to break out each platform because of the intermediate oci_image/oci_image_index split. Also, oci_push concatenated repository vs. oci_pull image/repository/registry.
5. It's cleaner to use pkg_tar than container_layer, but pkg_tar is a pain, e.g. not applying strip_prefix to deps, so there's still some hassle (that lives outside this library) to actually express image layouts.

[SanjayVas]

We managed to migrate our project, but we hit some significant difficulties in the process.

• The example java_image is entirely insufficient. It does not do any of the following, which we needed to figure out and implement in our own rules.
  • Copy the runfiles tree so we could access data dependencies.
  • Set the RUNFILES_DIR environment variable.
  • Copy native libraries.
  • Set the java.library.path property.
• oci_push does not support Make var substitution, so we had to implement that using our own version of expand_template.
• There is no equivalent rule for pushing a bundle of images (see How to push image to multiple repositories? #248), so we had to implement our own using rules_multirun.
• oci_image uses much more disk space (see oci_image uses significantly more disk space than container_image from rules_docker #439) which broke our GitHub Actions CI.
  • Need to resort to hacks such as https://github.com/marketplace/actions/maximize-build-disk-space or pay for larger runners.
  • Have to delete the blobs from the output tree so the Bazel cache can fit the 10 GiB GitHub Actions cache limit.

[alexeagle]

Note, for the runfiles tree (https://github.com/world-federation-of-advertisers/common-jvm/blob/main/build/rules_oci/defs.bzl#L141) you might want to try https://github.com/aspect-build/bazel-lib/blob/main/docs/tar.md which does support runfiles.

[SanjayVas]

@alexeagle rules_pkg also has pending runfiles support (see bazelbuild/rules_pkg#754).

While I know that rules_oci is supported by Aspect and uses that underneath, we're trying to avoid direct dependencies on anything from aspect-build when there's a bazelbuild equivalent. We're currently already relying on rules_pkg for our custom rules for Kubernetes Kustomize.

[SanjayVas]

As an additional note, performance is much slower for both oci_image and oci_push compared to their rules_docker counterparts. If I deleted the layer tars for a container_image, the regeneration appeared to be just a matter running tar. I spotted #365 which seems to be for addressing oci_image.

The oci_push comparison is actually against pushing using a container_bundle, i.e. pushing a whole bundle of images to the same registry. If I had already built the targets (bazel build), running them took only 2m44s with sequential = True. The same using oci_push w/rules_multirun takes 15m21s.

[marvin-hansen]

I've migrated from rules_docker to rules_oci 1x already a few months ago, but then got bus with other stuff so this is a bit of a retrospective.

The migration was painless. Support for private GCP registry is excellent, CI builds are really fast compared to before, and the oci rules never budged. Zero incident to this day. I've debugged any number of Bazel issues, but not once the oci rules.

The only thing I would like to point out is that the oci multi-arch configuration is a bit verbose for my taste so I made a custom macro for my use case to keep things concise. In any case, rules oci is a much valued addition to the Bazel ecosystem and I'm quite positive about the upcoming V2 rules.

Thanke to the entire team for maintaining this project.