fatal error: concurrent map read and map write

[olfway]

Sometimes I got this error on terraform plan:

❯ terraform plan
Acquiring state lock. This may take a few moments...
kustomization_resource.kube-dns["apps_v1_Deployment|kube-system|coredns"]: Refreshing state... [id=f2811881-9588-452a-8c0c-6c61a1ad1538]
kustomization_resource.kube-dns["rbac.authorization.k8s.io_v1_ClusterRole|~X|system:coredns"]: Refreshing state... [id=d8e40558-bff9-43d0-8097-95cf81cb825e]
kustomization_resource.kube-dns["~G_v1_Service|kube-system|kube-dns"]: Refreshing state... [id=4aea7d8f-f7e2-4252-a67d-6796b410330a]

Error: rpc error: code = Canceled desc = context canceled



Error: rpc error: code = Canceled desc = context canceled



Error: rpc error: code = Canceled desc = context canceled



Error: rpc error: code = Canceled desc = context canceled

Next time it usually works without errors

Terraform v0.14.7
+ provider registry.terraform.io/hashicorp/aws v3.30.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.0.2
+ provider registry.terraform.io/kbst/kustomization v0.4.1

[olfway]

I managed to get plugin output on errors
One log is terraform plan and another one is terraform refresh

kustomize-concurrent-plan.log: concurrent map writes
kustomize-concurrent-refresh.log: concurrent map read and map write

This was with the latest version:

provider registry.terraform.io/kbst/kustomization v0.4.2

[pst]

Thanks for reporting this, I will look into it.

[pst]

It seems this is an upstream error out of my control. I filed an issue.

[olfway]

Thanks for looking, I hope it will be fixed upstream

[rodrigorato]

I was able to get around this issue by introducing dependencies so that the affected resources are not planned concurrently. Perhaps this comment can be of use if anybody else hits this blocker.

In the meantime, thank you both for your work and I hope this gets fixed upstream! 😄

[LaurentLesle]

I am getting the same error from the traces when doing a for_each on a data "kustomization_overlay"
my workaround is to put -parallelism=1 in the meantime

[alapidas]

+1, seeing this as well

[pst]

I added a mutex to prevent multiple Kustomize runs in parallel as suggested by @alapidas in the upstream bug. This is in any case preferrable over having to set --parallelism=1 for the entire Terraform configuration. At least other data sources and resources can still be handled in parallel this way.

The bug does not show on every run for me. So I would appreciate testing if this workaround helps with your use cases. You will have to build the provider from source to test this. If it works, I'm happy to cut a bugfix release to make it available through the registry.

[pst]

I released v0.4.3. In my testing the issue did not show anymore with the mutex in place. https://github.com/kbst/terraform-provider-kustomization/releases/tag/v0.4.3

[alapidas]

Sorry for the late reply - I implemented almost the same thing, except I put the lock inside runKustomizeBuild to protect against arbitrary callers. That said, we ended up just changing the parallelism to 1 for plan and apply runs so that we didn't have to set up a pipeline to build the plugin ourselves. With this though, we can back that out. Thanks for shipping a fix so quickly.