[Feature Request] Review options for allowing protobuf 3.x to work with this library

[cretz]

Is your feature request related to a problem? Please describe.

Many libraries have not upgraded to 4.x while many have (including us). Depending on libraries that haven't is causing a problem for users. Most libraries seem to know they need to move to protobuf 4.x but just haven't done it.

Describe the solution you'd like

Consider options for supporting both. This could likely take the form of:

• Generating proto 3.x form in addition to proto 4.x form
• Changing generation script to have the __all__ from the __init__ of the proto packages switch to importing the 3.x generated code if determined to be on 3.x
• Adding a CI step that confirms proto 3 works with tests and is generated properly
• Experiment with ways to have 4.x be the default depended on version but that 3.x works. This may prove to be a bit difficult (can we have an "extra" that downgrades? can we ask users to force downgrade/shade?). IIRC Poetry doesn't support transitive overrides for example.
  • I don't think we want 4.x to be an optional extra, I think we want it default because of a bug in 3.x w/ C static state sharing across imports which harms our sandbox.

[cretz]

open-telemetry/opentelemetry-python#2880 (comment) mentions that we may be able to just generate with proto 3.20+. Need to try this.

[cretz]

This is complicated by the fact that we use https://pypi.org/project/grpcio-tools/ to generate our protobufs and the last version that supports proto 3.x (https://pypi.org/project/grpcio-tools/1.48.2/) doesn't have a Python 3.11 build. So code can't even be generated on latest Python when older proto library is in use.

[schibrikov]

Thanks a lot for working on this @cretz !

Do you have any idea on when this might be released?

[cretz]

@schibrikov - I do not, but hopefully soon. Beta 4 should be released soon (but no exact date)