Dependency pinning between packages post 1.0

[owais]

Now that we've hit 1.0 for most packages, I think we should not pin stable opentelemetry-* packages to exact versions. Right now, most opentelemetry-* packages now pin other stable opentelemetry-* packages to 1.0.0. For example,

https://github.com/open-telemetry/opentelemetry-python/blob/v1.0.0/opentelemetry-distro/setup.cfg#L43-L44
https://github.com/open-telemetry/opentelemetry-python/blob/v1.0.0/opentelemetry-sdk/setup.cfg#L44
https://github.com/open-telemetry/opentelemetry-python/blob/v1.0.0/opentelemetry-instrumentation/setup.cfg#L44
https://github.com/open-telemetry/opentelemetry-python/blob/v1.0.0/exporter/opentelemetry-exporter-otlp-proto-grpc/setup.cfg#L44-L46

This can be very annoying for users. For example,

Assuming we continue to release all packages in unison so every release contains a new version of every package and all stable packages for a release have the same version number.

• opentelemetry-sdk v1.2 introduced a new feature in v1.2 along with a bug.
• opentelemetry-distro v1.3 does not really need the new SDK feature but is forced to install SDK 1.3 because of exact dep pinning.
• This forces users to use SDK 1.3 which still contains the bug or downgrade everything to 1.1 not allowing users to take advantage of new features in Distro 1.3.

Instead of pinning exact deps for stable packages, we should pin >=1.N, <2.0 where N is the minimum minor version required by the package. For example, opentelemetry-sdk should continue to depend on opentelemetry-api>=1.0,<2.0 until it needs a specific feature/fix shipped in a later version of the API package. If the API package introduces an additive change in 1.6 that SDK could take advantage of then SDK 1.6 would finally update it's dependency on API to >=1.6<2.0.

We should allow users to mix and match any official and community packages together as long as long as they are API compatible. Users should be able to use the hypothetical versions api 1.5, sdk 1.10,exporter 1.14, distro 1.15 together. However, exporter 1.15 may introduce a change that depends on sdk 1.13. In that case, user would be forced to use sdk >=1.13 only when using exporter >=1.15.

Considerations

Extra maintenance burden

This does introduce extra burden as maintainers will now have to carefully update dependencies based on what actual changes were made since last release instead of blindly bumping the next minor version everywhere. This is a big change in the workflow but I think it's worth it given how much flexibility this will provide to the Python ecosystem for OpenTelemetry. With some workflow changes such as expecting contributors to update dependencies for each PR if required, we should be able to reduce this burden.

[codeboten]

I agree that we should make this change, it's actually painful to change all the dependencies for every release.

Although i don't know if our tooling would give us a good way to ensure compatibility hasn't broken today.

[aabmass]

We definitely need to do this, especially with our beta versioning for metrics. Because of the exact requirement, it is currently impossible to install the metrics beta packages with any instrumentations.

$ pip install opentelemetry-instrumentation-flask opentelemetry-api==1.10a0
...
...
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of opentelemetry-api to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install opentelemetry-api==1.10a0, opentelemetry-instrumentation-flask==0.12b0, opentelemetry-instrumentation-flask==0.13b0, opentelemetry-instrumentation-flask==0.14b0, opentelemetry-i
nstrumentation-flask==0.15b0, opentelemetry-instrumentation-flask==0.16b0, opentelemetry-instrumentation-flask==0.16b1, opentelemetry-instrumentation-flask==0.17b0, opentelemetry-instrumentation-flas
k==0.18b0, opentelemetry-instrumentation-flask==0.18b1, opentelemetry-instrumentation-flask==0.19b0, opentelemetry-instrumentation-flask==0.20b0, opentelemetry-instrumentation-flask==0.21b0 and opent
elemetry-instrumentation-flask==0.22b0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested opentelemetry-api==1.10a0
    opentelemetry-instrumentation-flask 0.22b0 depends on opentelemetry-api==1.3.0
    The user requested opentelemetry-api==1.10a0
    opentelemetry-instrumentation-flask 0.21b0 depends on opentelemetry-api==1.2.0
    The user requested opentelemetry-api==1.10a0
    opentelemetry-instrumentation-flask 0.20b0 depends on opentelemetry-api==1.1.0
    The user requested opentelemetry-api==1.10a0
    opentelemetry-instrumentation-flask 0.19b0 depends on opentelemetry-api==1.0.0
    The user requested opentelemetry-api==1.10a0
    opentelemetry-instrumentation-flask 0.18b1 depends on opentelemetry-api==1.0.0rc1
    The user requested opentelemetry-api==1.10a0
    opentelemetry-instrumentation-flask 0.18b0 depends on opentelemetry-api==1.0.0rc1
    The user requested opentelemetry-api==1.10a0
    opentelemetry-instrumentation-flask 0.17b0 depends on opentelemetry-api==0.17b0
    The user requested opentelemetry-api==1.10a0
    opentelemetry-instrumentation-flask 0.16b1 depends on opentelemetry-api==0.16b1
    The user requested opentelemetry-api==1.10a0
    opentelemetry-instrumentation-flask 0.16b0 depends on opentelemetry-api==0.16b0
    The user requested opentelemetry-api==1.10a0
    opentelemetry-instrumentation-flask 0.15b0 depends on opentelemetry-api==0.15b0
    The user requested opentelemetry-api==1.10a0
    opentelemetry-instrumentation-flask 0.14b0 depends on opentelemetry-api==0.14b0
    The user requested opentelemetry-api==1.10a0
    opentelemetry-instrumentation-flask 0.13b0 depends on opentelemetry-api==0.13b0
    The user requested opentelemetry-api==1.10a0
    opentelemetry-instrumentation-flask 0.12b0 depends on opentelemetry-api==0.12b0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies