fix: Fixed the compatibility with conda, protobuf 4 and pyext protobuf

[Ark-kun]

Fixes: #470
Fixes: googleapis/python-aiplatform#3129

[conventional-commit-lint-gcf]

🤖 I detect that the PR title and the commit message differ and there's only one commit. To use the PR title for the commit history, you can use Github's automerge feature with squashing, or use automerge label. Good luck human!

-- conventional-commit-lint bot
https://conventionalcommits.org/

[parthea]

Is it possible to add a test that fails? I'm not able to reproduce the issue.

[happy-qiao]

Could you elaborate how this change can fix pyext issue? Essentially, the PR doesn't seem to change the value of exported variables: "repeated_composite_types", "repeated_scalar_types", "map_composite_types".

Is it for the case when upb and pyext both exists and the change makes it default to pyext? But pyext is deprecated, if default to pyext, will it raise other issues?

[Ark-kun]

@parthea

Is it possible to add a test that fails?

Do you have Conda tests environments?

I'm not able to reproduce the issue.

I think you've added repro steps to the internal doc.

Also there are repro steps in the linked issue: #470

[Ark-kun]

Could you elaborate how this change can fix pyext issue? Essentially, the PR doesn't seem to change the value of exported variables: "repeated_composite_types", "repeated_scalar_types", "map_composite_types".

Let's carefully examine the execution flow. The code does change the items of the lists and the number of those items.

Is it for the case when upb and pyext both exists

Yes. This is the exact case where you and our customers have issues.

>>> from google._upb import _message as msg1
>>> msg1.RepeatedCompositeContainer
<class 'google._upb._message.RepeatedCompositeContainer'>

>>> from google.protobuf.pyext import _message as msg2
>>> msg2.RepeatedCompositeContainer
<class 'google.protobuf.pyext._message.RepeatedCompositeContainer'>

The root cause of this proto-plus bug is that proto-plus does not recognize pyext types when ubp types exist. But conda's protobuf defaults to pyext despite ubp existing.

and the change makes it default to pyext?

No. The change does not change the default. It adds the pyext types to the lists.

But pyext is deprecated

pyext is used by real customers who want to use Vertex AI. pyext is used by default in protobuf installed by conda.

[parthea]

Please could you add a comment in the code which explains the reason that we need to include both upb and pyext types? You mentioned this in the PR comments but it should be included in code comments as well.

Please could you also fix the failing presubmits ? This PR needs to be updated to cater for #474 which was merged

Otherwise, LGTM

[parthea]

Would you be able to refactor the code to reduce duplication? For example, can we use a helper function instead of having repeated code?

[parthea]

Move this import statement to the top of the file since we're importing it as _message_pyext