Create rules for supported distributions

[TylerHelmuth]

Description

This Issue is a continuation of open-telemetry/oteps#229.

The Collector SIG has had continuous discussion around the distributions we support, if we should support them, if we should support more, and what components should be included in the existing distributions. This is causing the SIG time and effort, but also the lack of clear rules could confuse users who are looking to use or contribute to a distribution. The goal off this issue will be to:

1. Determine the criteria for when the community will support a distribution
2. If Contrib fits those criteria, determine the rules for what components are included in the Contrib Distribution
3. If Core fits those criteria, determine the rules for what components are included in the Core Distribution

We may need to also determine how the Release pipeline will change to be able to scale to more distros. At the moment it already struggles with 2.

Current criteria based on the discussion in this issue:

• To honor commitments made by OpenTelemetry to other Open Source projects the Collector SIG should support at least 1 distribution that meets those commitments. (IDK what those commitments are, need input from others to more explicitly list out the impact of those commitments.)
• Distributions supported by the Collector SIG should serve a specific purpose and those purposes should have minimal overlap.
• Distributions supported by the Collector SIG should meet general needs and not be too niche.
• Distributions supported by the Collector SIG should only target the needs of the OpenTelemetry project.
• Distributions supported by the Collector SIG are not required to be production ready and may be focused on development and proof of concept use cases. The distribution should clearly indicate whether or not the Collector SIG considers it to be production ready.
• Distributions supported by the Collector SIG must only include components from the opentelemetry-collector and opentelemetry-collector-contrib repositories.
• Distributions supported by the Collector SIG should have a clearly defined list of criteria for which components are included.
• Distributions supported by the Collector must include the following assets except where the specific purpose of the distribution is naturally associated with a subset of these assets. In such cases, it should be clearly stated which assets are skipped and why. Additional asset may be included if the distro desires:
  • Binaries for linux_amd64, linux_arm64, windows_amd64 and darwin_arm64
  • linux_amd64 and linux_arm64 container images
  • Packages to be used with Linux distributions (apk, RPM, deb), Mac OS (brew) for each distributed binary.

[codeboten]

We may need to also determine how the Release pipeline will change to be able to scale to more distros. At the moment it already struggles with 2.

One idea I haven't explored but comes to mind is to split the goreleaser workflow per distribution. I've not tested this, but it should be doable

[TylerHelmuth]

Being able to release each distribution independently sounds like a good idea to be able to scale to more distributions.

[jpkrohling]

Can you clarify what's to be understood by "support"? What's expected from the community if we say we support a distribution?

[TylerHelmuth]

I think support implies all the duties covered by the Approver and Maintainer role requirements. In addition (bc I don't see it explicitly listed in the membership responsibilities), support means that the Collector SIG is the owner and maintainer of the binaries/images of the different Collector distributions and is responsible for the pipeline that produces those artifacts.

[TylerHelmuth]

Determine the criteria for when the community will support a distribution

Here is my initial attempt at an answering:

• To honor commitments made by OpenTelemetry to other Open Source projects the Collector SIG should support at least 1 distribution that meets those commitments. (IDK what those commitments are, need input from others to more explicitly list out the impact of those commitments.)
• Distributions supported by the Collector SIG should make using the Collector easier.
• Distributions supported by the Collector SIG should serve a specific purpose and those purposes should have minimal overlap.
• Distributions supported by the Collector SIG should meet general needs and not be too niche.
• Distributions supported by the Collector SIG must only include components from the opentelemetry-collector and opentelemetry-collector-contrib repositories.
• Distributions supported by the Collector SIG should have a clearly defined list of criteria for which components are included.
• Distributions supported by the Collector must include:
  • Binaries for linux_386, linux_amd64, linux_arm64, linux_ppc64le, windows_386, windows_amd64, windows_ppc64le, darwin_amd64, darwin_arm64, and darwin_ppc64le. (Looking at the release assets I don't actually think we provide this today despite claiming we do in our README.
  • linux_386, linux_amd64, linux_arm64, and `linux_ppc64le container images
  • Packages to be used with Linux distributions (apk, RPM, deb), Mac OS (brew) for each distributed binary.

[jpkrohling]

IDK what those commitments are

IIRC, those were listed in the original blog post that announced OTel, and included only Prometheus, Jaeger, and Zipkin, apart from OpenTracing and OpenCensus.

Other than that, your proposal looks good to me.

[MovieStoreGuy]

One thing I wouldn't mind some more clarification on is:

Distributions supported by the Collector SIG should serve a specific purpose and those purposes should have minimal overlap.

Does that mean the collector SIG sets the direction for another's SIG (say, K8s and Lambda) distribution?
Or conversely, what should be the expected level of involvement from those SIGs if we manage the distribution for them?

I know it isn't explicitly said, but is the intent that the supported distributions are intended the fulfil needs of the open telemetry project only? Meaning we are not managing distributions for vendors or other ecosystems?

[TylerHelmuth]

Does that mean the collector SIG sets the direction for another's SIG

I don't think it has to. The Collector SIG should be free to decide which specific purposes it sees as valuable enough to warrant supporting a distro. If that corresponds with the needs of another OpenTelemetry SIG that is a bonus, but it shouldn't be a requirement.

what should be the expected level of involvement from those SIGs if we manage the distribution for them?

I don't think we should view it as managing distributions for another SIG. We should only support a distribution if we see value in it for our users. If another SIG also benefits from it and wants to help support that is a bonus, but we should not rely on others to support these distributions.

I know it isn't explicitly said, but is the intent that the supported distributions are intended the fulfill needs of the open telemetry project only? Meaning we are not managing distributions for vendors or other ecosystems?

Good point, we should explicitly call this out. How about Distributions supported by the Collector SIG should only target the needs of the OpenTelemetry project.

[jpkrohling]

I don't think we should view it as managing distributions for another SIG

They are OpenTelemetry users, no matter which SIG they consume "first". Given that our distributions will have only components from either core or contrib, the support will end up on us anyway, no matter the SIG that proposed the distribution. Perhaps we could provide guidance to other SIGs into which components they can rely on?

[TylerHelmuth]

@jpkrohling I agree. My comment's intent is to say that we should not treat other OTel SIGs differently from other users. I don't think we need to create any SIG-specific guidance either; any guidance we give should apply to SIGs and end-users alike.

[TylerHelmuth]

To touch on points 2 and 3 in the issue for a bit:

Based on the discussed criteria so far, the Core distribution would no longer be supported. I believe it fails to pass these checks:

• Distributions supported by the Collector SIG should make using the Collector easier.
  • I feel like we often see users post about trouble using a component that doesn't exist in the Core repository. We frequently mention to use Contrib or build their own distribution.
• Distributions supported by the Collector SIG should serve a specific purpose and those purposes should have minimal overlap.
  • Although I am interested if others disagree, but I feel like Core doesn't have a specific purpose. If the original purpose was to meet our commitments, Contrib is already doing that. If the intent was to provide a smaller image, my response is that "to provide a smaller image" isn't a strong enough reason, plus it leads to the issue discussed in the previous bullet.

In contrast, I believe Contrib should continue to be supported. Although we haven't defined its set list of criteria yet, I believe Contrib's meets (or can be made to meet) all the criteria discussed so far. Most importantly it allows us to keep our promises, makes the Collector easy to use, servers a specific purpose*, and meets general user needs.

• We need decide a final purpose statement, but I see Contrib's purpose as being a readily available, easy solution for trying out any component that the Collector SIG supports. Contrib is the solution to "I want to try out the Collector" and I believe it gets to claim this purpose instead of Core since it contains all the components and Core does not.

[jpkrohling]

There are two parts of contrib that I don't like and would make me uncomfortable supporting it more officially:

1. it's too big: having everything in it makes it unsuitable for most use-cases. For instance, a sidecar with contrib is not up for consideration. Sure, it's tolerable for most use-cases, but it's not serving a specific purpose.
2. it has everything, including things we haven't touched or seen working (for any definition of "working") in a long time. I'm unsure if we have active maintainers and code owners for half of the components there. As such, I would be hesitant to call it "supported".

[TylerHelmuth]

Sure, it's tolerable for most use-cases, but it's not serving a specific purpose.

I see it's specific purpose to be a dev tool - something that helps users get started playing around with the Collector. Contrib provides an easy, out-of-the-box solution for any user to try out any component they read about. Anything less than Contrib means we'll have to guess which components users want to try out, defeating the purpose of a dev tool. Thinking about it another way, if we only supported Core or only supported the proposed k8s distro how would a user try out the ngixreceiver (chosen randomly as an example)? Would it be up to them to build their own distro? I feel like that requirement goes against the concept that our supported distributions should make using the collector easy.

it has everything, including things we haven't touched or seen working (for any definition of "working") in a long time.

That is a fair concern, but feels like a Contrib repo issue and not a distribution concern. We should write the Contrib criteria to be strict with the Stability requirements and hold both the Contrib repo and distro accountable. If we don't feel confident in a Contrib repo component we should deal with that as part of maintaining Contrib because it has larger implications than just our distros, it affects anyone using that component.

[djaglowski]

Distributions supported by the Collector must include:

We might consider some distributions that are naturally associated with subset of assets. For example, a windows distribution that contains several windows-only components (eventlog, performancecounters, etc). Should we consider adding a caveat for this? Roughly, "except where specific purpose of the distribution is naturally associated with a subset of these assets. In such cases, it should be clearly stated which assets are skipped and why".

[TylerHelmuth]

@djaglowski that is a super good point. I like the idea of making that criteria less strict so that we don't limit our options. How about:

• Distributions supported by the Collector must include the following assets except where the specific purpose of the distribution is naturally associated with a subset of these assets. In such cases, it should be clearly stated which assets are skipped and why:
  • Binaries for linux_386, linux_amd64, linux_arm64, linux_ppc64le, windows_386, windows_amd64, windows_ppc64le, darwin_amd64, darwin_arm64, and darwin_ppc64le. (Looking at the release assets I don't actually think we provide this today despite claiming we do in our README.
  • linux_386, linux_amd64, linux_arm64, and `linux_ppc64le container images
  • Packages to be used with Linux distributions (apk, RPM, deb), Mac OS (brew) for each distributed binary.

[mx-psi]

I feel like we should be more conservative on the assets we require for new distributions: while we ensure we can build binaries for these architectures/OS but we don't run the test suite for most of these 'flavors'. It's clear to me that linux_amd64 and windows_amd64 should be required, and it seems like linux_arm64 should be too even if we don't test it just because of expected usage, but the rest should not be a requirement.

[TylerHelmuth]

@mx-psi can you describe more about why being less strict with the release assets is a good thing. Is it to make it easier to accept new distributions that might struggle to provide certain assets?

[mx-psi]

Is it to make it easier to accept new distributions that might struggle to provide certain assets?

That's one part of it, I also just don't think it's a good idea to provide untested assets and I don't want to force new distributions to suffer that maintenance burden. For most of the assets listed we don't (and can't without significant pain) test them on CI and our ability to resolve issues specific to those artifacts is limited (IMO even on Windows we struggle to provide good support).

[TylerHelmuth]

@mx-psi would you distill the current proposal down to:

• Distributions supported by the Collector must include the following assets except where the specific purpose of the distribution is naturally associated with a subset of these assets. In such cases, it should be clearly stated which assets are skipped and why. Additional asset may be included if the distro desires:
  • Binaries for linux_amd64, linux_arm64, windows_amd64 and darwin_arm64
  • linux_amd64 and linux_arm64 container images
  • Packages to be used with Linux distributions (apk, RPM, deb), Mac OS (brew) for each distributed binary.

[mx-psi]

@TylerHelmuth That makes sense to me 👍

[mx-psi]

I feel like we often see users post about trouble using a component that doesn't exist in the Core repository. We frequently mention to use Contrib or build their own distribution.

I think this is a problem with having more than one distribution, not with core specifically. We also see that with ADOT distros (e.g. see open-telemetry/opentelemetry-collector-contrib/issues/13163 or open-telemetry/opentelemetry-collector-contrib/issues/8109). I don't see this as a reason for removing core, since the problem will still happen.

Core doesn't have a specific purpose. If the original purpose was to meet our commitments, Contrib is already doing that. If the intent was to provide a smaller image, my response is that "to provide a smaller image" isn't a strong enough reason, plus it leads to the issue discussed in the previous bullet.

The purpose of Core is IMO to fulfill use cases that fully rely on non-commercial open source observability solutions. There are some components that we could add (e.g. more processors) to make this more aligned with this purpose, but I think it does it very well.

[jpkrohling]

@TylerHelmuth, during the SIG call today, you mentioned that this issue is reaching a consensus that contrib is going to be supported and core dropped. Based on the rules mentioned here, would you mind listing the two distributions and why/why not they will be supported?

In my view, we should rather slim down contrib to include the new components we think the community is missing the most from core, instead of promoting contrib as is and dropping core.

[TylerHelmuth]

I think this is a problem with having more than one distribution, not with core specifically. We also see that with ADOT distros (e.g. see open-telemetry/opentelemetry-collector-contrib#13163 or open-telemetry/opentelemetry-collector-contrib#8109). I don't see this as a reason for removing core, since the problem will still happen.

That is fair. Maybe the criteria Distributions supported by the Collector SIG should make using the Collector easier is a little weak? I could see dropping that requirement as Distributions supported by the Collector SIG should serve a specific purpose and those purposes should have minimal overlap implies making the Collector easy to use.

The purpose of Core is IMO to fulfill use cases that fully rely on non-commercial open source observability solutions. There are some components that we could add (e.g. more processors) to make this more aligned with this purpose, but I think it does it very well.

I think that is a reasonable purpose. I'd propose changing the name to better fit that purpose though.

[TylerHelmuth]

Continuing this Issue's discussion, here are my proposed criteria for what should be included in Contrib:

1. All components from opentelemetry-collector and opentelemetry-collector-contrib that have at least 1 signal at Alpha stability or higher.
2. Components that are marked Unmaintained will be kept in the distribution for six months. After six months of being unmaintained the component will be removed from the distribution. (This rule directly follows the procedure currently documented in the Unmaintained stability.)

[TylerHelmuth]

@jpkrohling this comment here is my take on why Contrib meets the criteria we've been discussing and why Core doesn't.

For Core, I think it comes down to the criteria Distributions supported by the Collector SIG should serve a specific purpose and those purposes should have minimal overlap. In my reflection of Core I was not able to come up with a purpose that met this criteria. @mx-psi has suggested a purpose ("to fulfill use cases that fully rely on non-commercial open source observability solutions" that feels like it it could work. If we all agree on that purpose then Core meets the criteria and we'd keep it.

For Contrib, I am very much in favor of the criteria mentioned here in order to fulfill what I see as its purpose ("being a readily available, easy solution for trying out any component that the Collector SIG supports."). We cannot predict which components a user would want to try and to try to predict renders the purpose unachievable.

[jpkrohling]

I like that. Are we going to have a list of those distributions somewhere, as a table perhaps? If so, we should explicitly call that out, stating that contrib is useful for X and Y, but not for Z.

[TylerHelmuth]

We haven't landed on the presentation of the outcome of this discussion yet but a table seems reasonable.

[dmitryax]

I think one important use case we need to support is gateway mode which needs only the components defined in the core repo, specifically OTLP receiver and OTLP exporter. Jaeger/Zipkin/OpenCensus maybe even unnecessary. I think it would be easier to define if we align on a rule: core components = components from core repository.

But I also agree with @mx-psi about distribution to fulfill use cases that fully rely on non-commercial open source observability solutions. That one can be another one in between core and contrib probably

[jpkrohling]

I think having an OTLP-only distribution is reasonable, and the proposed rules do not prevent that distribution from happening, as long as we still have at least one production-quality distribution with Zipkin, Jaeger, and Prometheus.

[TylerHelmuth]

I think one important use case we need to support is gateway mode

I think having an OTLP-only distribution is reasonable

I'm all for new distributions, but for the context of this issues lets restrict the conversation to answer the questions:

1. If Contrib fits those criteria, determine the rules for what components are included in the Contrib Distribution
2. If Core fits those criteria, determine the rules for what components are included in the Core Distribution

My answer to 1 can be found here and here and once #376 is merged I'll open a PR to solidify 1.

Lets focus on 2. The conversation at this point is a little spread out (the downside to discussing 3 topics in 1 issue), but I think the current state is:

Core purpose:

• To fulfill use cases that fully rely on non-commercial open source observability solutions.
• To fulfill criteria 1 from [admin] Add criteria for distributions #376 (Support at least one distribution that is recommended for production which includes support for Prometheus, Jaeger, Zipkin, and OpenCensus.)

Is there an additional purpose around being the gateway mode distribution or otlp-only distribution? If so let's clearly define that. In particular otlp-only distribution purpose for Core conflicts with currently proposed purpose.

Core component criteria still needs solidified.

@mx-psi @jpkrohling @dmitryax as proponents of the Core distribution and you agree and a Core purpose and suggest the criteria for which components would be included?

[jpkrohling]

I think there's still something to be said about the purpose of the core, but I can't articulate yet exactly what's missing. I agree with the purposes you added there and agree that an OTLP-only or Gateway mode needs to be discussed at a later time.

Currently, core is composed of:

• components from the core repository. We thought about moving them out, but @bogdandrutu was quite opinionated in having them there as implementations trying the APIs
• components for interoperability with the wider open source observability ecosystem, as defined by the original promises made by the OTel project: Prometheus, Jaeger, Zipkin.
• components from contrib that we believe are useful to most users of the collector (pprofextension, hostmetricsreceiver, ...)

Converting that into reasonable criteria would be tricky: I don't expect most collector users to need the Kafka receiver/exporter, and I don't see that as fitting any of those criteria. But I'm not sure we want to remove that component from the core either.

[TylerHelmuth]

components from contrib that we believe are useful to most users of the collector (pprofextension, hostmetricsreceiver, ...)

I believe this is the tricky part. Trying to guess which components most users for the collector find useful feels impossible (to me at least), but thats what Core currently represents - our attempt. We saw that attempt challenged by #337. I think guessing what users want will always leave us open to challenges.

Picking the components based on non-commercial open source observability solutions does feel doable. If that changes Core components thats ok to me.

[atoulme]

Do we have numbers on downloads of the core distro? Maybe docker pulls? Can we compare to contrib pulls? Would that help know what is used and determine a way forward?

[TylerHelmuth]

They are both 10M+ pulls according to DockerHub. That isn't super surprising since the helm charts use Contrib by default and the Operator uses Core by default and both images have been around for a long time.

[dmitryax]

non-commercial open source observability solutions sounds good to me for core, but it can include too much. So maybe it includes production-ready components for non-commercial open source observability solutions? Then we need to specify production readiness :) But maybe that can be kept on Collector SIG judgment for now?

[TylerHelmuth]

Then we need to specify production readiness :) But maybe that can be kept on Collector SIG judgment for now?

I'd like to avoid "Production Ready" like the plague since there is the OTEP happening about it lol Let's call out our opinion specifically:includes components for non-commercial open source observability solutions that the Collector SIG recommends for Production.

[dmitryax]

Sounds good to me :)

[jpkrohling]

like the plague since there is the OTEP happening about it

I think I heard about that one.

includes components for non-commercial open source observability solutions that the Collector SIG recommends for Production

Yes or no:

• SkyWalking
• Zipkin
• Loki
• InfluxDB
• Syslog

Those are the observability-related ones. There are also the storage-related ones, not necessarily on the observability space: Clickhouse, Elasticsearch, OpenSearch, Parquet, Cassandra, ...

[TylerHelmuth]

I have opened a PR to solidify Contrib component rules: #382

[TylerHelmuth]

We have now merged PRs to publish our distribution rules and the component criteria for contrib. Do you want to keep this issue open for the continued discussion of Core or should it move to its own issue?

[jpkrohling]

Both options are fine by me, with a slight preference for keeping things here.

[mx-psi]

Which rules is core not fulfilling? I think we just need to put #360 (comment) in writing, but otherwise it seems to me like core fulfills every rule stated on the issue?

[TylerHelmuth]

@mx-psi I believe the remaining work is to create a list of rules for which components will be included in the Core distribution and document them.

[evan-bradley]

I think having core include all components that concern non-commercial, open-source technologies would make sense, but would also make it pretty large, especially for a distribution called "core". The potential alternatives I see are:

1. An OTLP-only distribution that also includes a basic set of processors/connectors/extensions that would likely be a little larger than a gateway distribution. I think this would make sense as a "core" distribution and align well with the components in core/contrib, but would require removing components from the existing distribution.
2. A distribution that includes support for Prometheus, Jaeger, Zipkin, and OpenCensus. This would also require removing some components, and I think wouldn't necessarily have obvious utility considering the Jaeger and OpenCensus formats should be less common as time goes on.
3. Something along the current lines of the current state of the distribution that includes components that we determine useful. I agree that there's no practical way to provide guidelines for this, so in my opinion we should avoid taking this route.

In my opinion, 1 probably aligns best with a distribution called "core", but would require a more difficult transition and would very likely be generally less useful than a distribution that simply focuses on open source technologies. I think having a larger number of officially-supported distributions would make a stripped-down core more palatable, but on the other hand the option for users to build their own distribution with OCB makes it easier to justify a larger core.

[jpkrohling]

OpenCensus is archived, Jaeger components are now gone in favor of OTLP, Prometheus will eventually adopt native OTLP ingestion. I'd have the core as we have today, as it will eventually converge into an OTLP-only distribution. In the meantime, we still send the clear message that the Collector aims to play nice with the tools that existed when it was created.

That said, I intend to create a new repository under my private namespace offering niche distributions, such as the load-balancing distribution, sidecar (OTLP-only), and others.

[codeboten]

OpenCensus is archived, Jaeger components are now gone in favor of OTLP, Prometheus will eventually adopt native OTLP ingestion.

I suspect the receivers for these formats will live on for some time.

Should we mark the core distribution as "legacy" or "deprecated" in favour of an OTLP only distribution? I guess choosing what processors or connectors are included in even the smallest of distro becomes ambiguous.

[jpkrohling]

Should we mark the core distribution as "legacy" or "deprecated" in favour of an OTLP only distribution?

Let's use a more honorable name then: "classic" :-)

[woody1872]

Some thoughts if it's okay... purely as a user of the collector, and not as a maintainer or holder any other "official" role.

I really like the core distro. I think it should focus on being very slim and OTLP focused - it has huge value and is often all that's needed, at least my own experience. I just took another look through what's included currently in core and I actually think it's excellent how it is - it's perfect for "core" use-cases as I expect it was always intended.

Where I work, and I'm not representing them here in any way, it's our explicit goal when using the collector to try and use OTLP only as much as possible. So I think a very slim, "production-ready" distro like core holds tremendous value. This is especially true as projects like Jaeger and Prometheus move to support OTLP natively - not sure about Zipkin, I've never used it.

I personally like the contrib distro as a dev/testing "tool". Sometimes you want to achieve a goal using the collector, and you're not quite sure exactly how to do it or what's the best and most efficient way. This is the beauty of the collector and how flexible and configurable it is. By having a distro like contrib, it allows easy and quick development of new collector configurations without having to touch the builder (ocb) or worry about what to include etc. When it's time for production, I'm a believer in only having the components you need, as I'm sure most others are as well, and I'd scoff at using a distro with more components in it than I actually need.

I apologies if that didn't really add anything new to what's already been said. I just wanted to contribute my own thoughts.

[jpkrohling]

This was discussed today at the SIG. I'm OK with the current set of rules, and I think both contrib and core fulfill them, while still opening the door for new distributions. I would love to see distributions having owners like we have components, but that's an improvement that can be discussed later (and only if we see the actual need for that).

From my perspective: