kyaml: fatal error: concurrent map read and map write

[pst]

I'm using kustomize/api to provide a Terraform provider for Kustomize. Presumably as a result of Terraform/GRPC concurrency there is a fatal error caused by concurrent map read and map write from within the kyaml code base around determining if a resource is namespace scoped.

The downstream issue has full Terraform debug output logs. But the relevant parts seem to be:

fatal error: concurrent map read and map write

goroutine 58 [running]:
runtime.throw(0x2cd0a08, 0x21)
	runtime/panic.go:1116 +0x72 fp=0xc0124988f0 sp=0xc0124988c0 pc=0x1035312
runtime.mapaccess2(0x29d7b00, 0xc0082ab770, 0xc0124989a0, 0x2, 0xc01139ea10)
	runtime/map.go:469 +0x25b fp=0xc012498930 sp=0xc0124988f0 pc=0x100f5bb
sigs.k8s.io/kustomize/kyaml/openapi.IsNamespaceScoped(...)
	sigs.k8s.io/kustomize/kyaml@v0.10.7/openapi/openapi.go:270
sigs.k8s.io/kustomize/api/resid.Gvk.IsNamespaceableKind(0x0, 0x0, 0xc01139e9f8, 0x2, 0xc01139ea10, 0xe, 0x3e6da60)
	sigs.k8s.io/kustomize/api@v0.6.9/resid/gvk.go:219 +0xf9 fp=0xc0124989d0 sp=0xc012498930 pc=0x1b37339
sigs.k8s.io/kustomize/api/resid.ResId.EffectiveNamespace(0x0, 0x0, 0xc01139e9f8, 0x2, 0xc01139ea10, 0xe, 0xc0113a6a40, 0x1c, 0xc0113a6a60, 0x11, ...)
	sigs.k8s.io/kustomize/api@v0.6.9/resid/resid.go:120 +0x5a fp=0xc012498a68 sp=0xc0124989d0 pc=0x1b37eda

Kustomize version

Relevant lines from go.mod:

sigs.k8s.io/kustomize/api v0.6.9
sigs.k8s.io/kustomize/kyaml v0.10.7

I'm blocked from updating to a higher api version due to #3614

[liggitt]

/kind bug

[Shell32-Natsu]

@natasha41575 Looks like relate to OpenAPI in kyaml.

[alapidas]

Seeing this same issue a la the kustomize provider in Terraform.

[alapidas]

@pst Can you add a mutex around the krusty code as a workaround? Not sure what kind of performance hit serializing these operations may have.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[pst]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[pst]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ljakimczuk]

@alapidas FYI, about the performance concerns you had, we have been hit by this in one of our environments due to its network conditions.

We use Flux with the build serialization workaround implemented, and the repository it reconciles relies on the kustomize Helm Chart plugin. Because of the sometimes poor network conditions of this particular environment, the build struggles to pull the Helm Chart, sometimes sevirely resulting in a failure, sometimes it just takes longer because many segments are lost and must be re-transmitted. In result, the Flux Kustomization that struggles to build due to the pulling problem, blocks other Kustomization sometimes for a very long time, below is the sample of reconciliation times we noticed for failed cases.

...
Reconciliation failed after 11m24.043533326s, next try in 30s | name=flux
Reconciliation finished in 11m43.475090566s, next run in 10s | name=collection
...
Reconciliation failed after 11m53.681076341s, next try in 30s | name=flux
Reconciliation finished in 11m53.989449191s, next run in 10s | name=collection
...
Reconciliation failed after 11m33.807809068s, next try in 30s | name=flux
Reconciliation finished in 11m52.965351964s, next run in 10s | name=collection
...
Reconciliation failed after 30m28.306259106s, next try in 30s | name=flux
Reconciliation finished in 30m46.210455889s, next run in 10s | name=collection
...
Reconciliation failed after 13m16.751290663s, next try in 30s | name=flux
Reconciliation finished in 13m24.166936029s, next run in 10s | name=collection
...
Reconciliation failed after 7m51.220454452s, next try in 30s | name=flux
Reconciliation finished in 7m51.714838107s, next run in 10s | name=collection
...
Reconciliation failed after 12m6.895618115s, next try in 30s | name=flux
Reconciliation finished in 12m7.952321057s, next run in 10s | name=collection
...
Reconciliation failed after 32m48.289605576s, next try in 30s | name=flux
Reconciliation finished in 32m48.477273626s, next run in 10s | name=collection
...

[HirazawaUi]

/reopen
This problem still exists, is there anyone who can help to solve :)

[k8s-ci-robot]

@HirazawaUi: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen
This problem still exists, is there anyone who can help to solve :)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.