Pulumi fails to recreate Hashicorp Vault resources after provider change

[sfc-gh-jsirak]

What happened?

We recently moved all our pulumi resources that configure Hasicorp Vault from the default provider to an explicit provider. As part of the move, pulumi recreated some resource. However, pulumi didn't recreate the resources in Vault even though its state shows that it has recreated it. This caused some of our resources to be silently deleted without any warning.

Steps to reproduce

The change can be reproduced using a dev vault server:

1. Create a vault dev server and grab the vault address and vault
2. Create a new pulumi project and stack
3. Create a vault provider and a vault policy

Pulumi Go code

      
     package main
      
      import (
          "os"
          "github.com/pulumi/pulumi-vault/sdk/v5/go/vault"
          "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
      )
      
      func main() {
          pulumi.Run(func(ctx *pulumi.Context) error {
      
  	        vaultAddr := os.Getenv("VAULT_ADDR")
  	        token := os.Getenv("VAULT_TOKEN")
      
  	        _, err := vault.NewProvider(ctx, "vault", &vault.ProviderArgs{
  		        Address: pulumi.String(vaultAddr),
  		        Token:   pulumi.ToSecret(pulumi.String(token)).(pulumi.StringOutput),
  	        })
  	        if err != nil {
  		        return err
  	        }
      
  	        _, err = vault.NewPolicy(ctx, "aws_policy_test", &vault.PolicyArgs{
  		        Name:   pulumi.String("test"),
  		        Policy: pulumi.String("path \"secret/data/test/*\" {\n  capabilities = [\"read\"]\n}\n"), 
                      })
  	        if err != nil {
  		        return err
  	        }
      
  	        return nil
          })
      }

4. Run pulumi up and create the resources
5. List the policies in vault to confirm the new policy is created

$ vault policy list                                                                                                                                                                
default
test
root

6. Update the code to make the policy resource use an explicit provider

Pulumi Go code

      
     package main
      
      import (
          "os"
          "github.com/pulumi/pulumi-vault/sdk/v5/go/vault"
          "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
      )
      
      func main() {
          pulumi.Run(func(ctx *pulumi.Context) error {
      
  	        vaultAddr := os.Getenv("VAULT_ADDR")
  	        token := os.Getenv("VAULT_TOKEN")
      
  	        provider, err := vault.NewProvider(ctx, "vault", &vault.ProviderArgs{
  		        Address: pulumi.String(vaultAddr),
  		        Token:   pulumi.ToSecret(pulumi.String(token)).(pulumi.StringOutput),
  	        })
  	        if err != nil {
  		        return err
  	        }
      
  	        _, err = vault.NewPolicy(ctx, "aws_policy_test", &vault.PolicyArgs{
  		        Name:   pulumi.String("test"),
  		        Policy: pulumi.String("path \"secret/data/test/*\" {\n  capabilities = [\"read\"]\n}\n"), 
                      }, pulumi.Provider(provider))
  	        if err != nil {
  		        return err
  	        }
      
  	        return nil
          })
      }

7. Run Pulumi up to update the provider for the policy

Pulumi Up Output

 Previewing update (test)

  View Live: https://app.pulumi.com/joseph-sirak/pulumi-test/test/previews/3f885ad4-9419-4e66-81ca-338abab52dce
  
    pulumi:pulumi:Stack: (same)
      [urn=urn:pulumi:test::pulumi-test::pulumi:pulumi:Stack::pulumi-test-test]
      +-vault:index/policy:Policy: (replace)
          [id=test]
          [urn=urn:pulumi:test::pulumi-test::vault:index/policy:Policy::aws_policy_test]
          [provider: urn:pulumi:test::pulumi-test::pulumi:providers:vault::default::db6df12a-408f-469b-9e54-23fdd6eac9a7 => urn:pulumi:test::pulumi-test::pulumi:providers:vault::vault::60fc4153-af69-4b84-83b1-989c810e94b1]
        - id    : "test"
          name  : "test"
          policy: "path \"secret/data/test/*\" {\n  capabilities = [\"read\"]\n}\n"
  Resources:
      +-1 to replace
      2 unchanged
  Do you want to perform this update? yes
  Updating (test)
  
  View Live: https://app.pulumi.com/joseph-sirak/pulumi-test/test/updates/10
  
    pulumi:pulumi:Stack: (same)
      [urn=urn:pulumi:test::pulumi-test::pulumi:pulumi:Stack::pulumi-test-test]
      +-vault:index/policy:Policy: (replace)
          [id=test]
          [urn=urn:pulumi:test::pulumi-test::vault:index/policy:Policy::aws_policy_test]
          [provider: urn:pulumi:test::pulumi-test::pulumi:providers:vault::default::db6df12a-408f-469b-9e54-23fdd6eac9a7 => urn:pulumi:test::pulumi-test::pulumi:providers:vault::vault::60fc4153-af69-4b84-83b1-989c810e94b1]
          id    : "test"
          name  : "test"
          policy: "path \"secret/data/test/*\" {\n  capabilities = [\"read\"]\n}\n"
  Resources:     
      +-1 replaced
      2 unchanged
  

8. List policies in vault to see if the policy still exists

$vault policy list  
 default
 root

Expected Behavior

The expected behavior is that the resource will not only get deleted but also recreated. (In the ideal case, pulumi determines that the explicit and default provider point to the same vault and simply update the state without recreating the resource)

Actual Behavior

The resource marked to be replaced gets deleted but not recreated despite the pulumi state recording that it has be created

Versions used

v3.34.1

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[Frassle]

Wonder if this is a similar issue to pulumi/pulumi-aws#2009 where the vault provider told the engine it had created a new resource, but actually it was the same ID as the existing resource, so when the engine then went to delete the old resource nothing was left.

[iwahbe]

Were sorry for the confusion. If you don't want to create a replace when switching resources, you can use aliases. Otherwise you can set deleteBeforeReplace.

@pulumi/platform-providers Is there anything we are missing?

[stack72]

@iwahbe i believe this is also related to the work @Frassle and I were talking about with pulumi/pulumi-aws#2009

It might be worth him looking at this to see if that's the case

[danielrbradley]

Is this basically the same issue as pulumi/pulumi#6078 – essentially a rename, but at the provider level?

[Frassle]

Is this basically the same issue as pulumi/pulumi#6078 – essentially a rename, but at the provider level?

Partially. But in the case where you really do want to do a provider replace here it should be able to handle recreating vault policies.

[Frassle]

Yup this is exactly the same as pulumi/pulumi-aws#2009. policy creating is idempotent so if you set an explicit name (and the go code attached to this ticket does that) then the provider will return OK for the create even though it hasn't actually created a new resource, its just the same as the old resource. pulumi/pulumi#9903 and pulumi/pulumi#9909 will fix this.