Transient error "native provider reuse budget has been exceeded"

[ZhiminXiang]

What happened?

We have seen many transient errors like below

│     message: 'observe failed: cannot schedule a native provider during observe: e2eb787b-fd7f-4297-8110-fa8a64f45146:       │
│       cannot schedule native Terraform provider process: native provider reuse budget                                       │
│       has been exceeded: invocationCount: 112, ttl: 100' 

When this error happened, the Synced status of the related managed resource became unready.
It could last a couple of minutes for a managed resource, and then the resource would be recovered from the state.

This error makes our tests very flaky, and also causes much confusion for our users.

How can we reproduce it?

I am not really sure how to consistently reproduce this issue since the issue is transient.

What environment did it happen in?

• Crossplane Version: 1.12.0
• Provider Version: v0.33.0
• Kubernetes Version: v1.24.14
• Kubernetes Distribution: GKE

[jeanduplessis]

@ZhiminXiang version v0.35.0 of the provider was built with improvements to Upjet that address this error. Could you upgrade your provider and let us know if this is still a problem for you?

[toastwaffle]

We saw some errors similar to this when we upgraded to v0.35.0, but for us they weren't transient; the provider got completely stuck until we restarted it.

I've created crossplane/upjet#261 to add a little bit of extra logging, as I suspect the inUse count is somehow not getting decremented in some situations.

[machecazzon]

Started to observe this error suddenly as well xpkg.upbound.io/upbound/provider-gcp:v0.38.1

CannotObserveExternalResource
cannot schedule a native provider during observe: 5e656f02-149c-4ff5-9009-dc82b0c91b18: cannot schedule native Terraform provider process: native provider reuse budget has been exceeded: invocationCount: 120, ttl: 100

The only difference on my environment is now use function pipelines 🤔

[toastwaffle]

Could you upgrade to v0.40.0 or later? The new non-forking architecture mostly solves this I think

[machecazzon]

Still observing. Thinks cooled down and I had notice that sqladmin api went quota missing due to i believe a wrong project on database resource.

I can surely give v0.41.1 a try. But right now want to observe things a bit as I've never observed this error before changing to pipelines.

[github-actions]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[github-actions]

This issue is being closed since there has been no activity for 14 days since marking it as stale. If you still need help, feel free to comment or reopen the issue!