Alternative ways to store terraform state

[turkenh]

In the initial implementation, we are planning to store terraform states as annotations for the managed resource.

ETCD (i.e. annotation) will be authoritative in terms of the state and we avoid making any changes/edits on it except stripping out sensitive attributes. This is the most conservative approach in terms of using terraform just like as an end-user.

However, we want to investigate whether we can safely (re)build the state information from the spec, status, and kind information of the resource and completely remove the annotation.

To better describe the problem, see the following tfstate example:

{
  "version": 4,
  "terraform_version": "1.0.4",
  "serial": 5,
  "lineage": "f36b8453-5686-97c3-af3e-6a5c655c5fbf",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "aws_vpc",
      "name": "instance_vpc_hasan",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "schema_version": 1,
          "attributes": {
            "arn": "arn:aws:ec2:us-west-2:609897127049:vpc/vpc-0b396ddd9469f29e5",
            "assign_generated_ipv6_cidr_block": false,
            "cidr_block": "10.0.0.0/16",
            "default_network_acl_id": "acl-08e8f435b74af56e8",
            "default_route_table_id": "rtb-0eb8f2b9933d31175",
            "default_security_group_id": "sg-034d7899a57871f29",
            "dhcp_options_id": "dopt-0c388774",
            "enable_classiclink": false,
            "enable_classiclink_dns_support": false,
            "enable_dns_hostnames": false,
            "enable_dns_support": true,
            "id": "vpc-0b396ddd9469f29e5",
            "instance_tenancy": "default",
            "ipv6_association_id": "",
            "ipv6_cidr_block": "",
            "main_route_table_id": "rtb-0eb8f2b9933d31175",
            "owner_id": "609897127049",
            "tags": {
              "Name": "ExampleVPCInstanceByHasan"
            },
            "tags_all": {
              "Name": "ExampleVPCInstanceByHasan"
            }
          },
          "sensitive_attributes": [],
          "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ=="
        }
      ]
    }
  ]
}

• resources[0].instances[0].attributes field, is a combination of
  • spec.forProvider
  • status.atProvider
  • external_name annotation (id here)
• resources[0].instances[0].sensitive_attributes field would be available in the connection secret (if required during apply)
• resources[0].type, resources[0].name and resources[0].provider would also be available during reconciliation.

Two caveats here:

• spec.forProvider contains the desired state, not the last applied state which would be different that what tfstate typically contains.
• We are not sure how to terraform client behaves when other arbitrary information like lineage, serial and resources[0].instances[0].private exactly the same as the original one which would require to persist them as well.

[muvaf]

I believe constructing the state from CR would make it easier to understand the general flow since it'll be very similar to existing providers and also it's one less computed thing that we have to persist (on top of external-name).

spec.forProvider contains the desired state, not the last applied state which would be different that what tfstate typically contains.

In the beginning of the reconciliation, we can print the state as we know it and execute terraform show command which will use only the fields that are needed for read operations and, i.e. only external-name in most cases, so once we start reconciliation the tfstate file will be refreshed to show the current observation, hence we'd have desired in CR and observed in tfstate file in the disk at that point.

We are not sure how to terraform client behaves when other arbitrary information like lineage, serial and resources[0].instances[0].private exactly the same as the original one which would require to persist them as well.

Here is the documentation that explains what these fields mean.

The "lineage" is a unique ID assigned to a state when it is created. If a lineage is different, then it means the states were created at different times and its very likely you're modifying a different state.

Every state has a monotonically increasing "serial" number. If the destination state has a higher serial, Terraform will not allow you to write it since it means that changes have occurred since the state you're attempting to write.

I believe we can use UUID of the CR for lineage and serial can just be an arbitrary number because from what I understand, it's read in the beginning and terraform checks whether it's still the same before writing the resulting state.

$ echo "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==" | base64 -d
{"schema_version":"1"}

I think resources[0].instances[0].private is also something we have information about.

[muvaf]

I just tested removing everything except id from attributes and ran terraform refresh and it refreshed the state with the latest status. I also tested giving arbitrary serial and lineage and it still successfully refreshes. So, I believe if we just put everything we got as state to the disk and then run terraform refresh, it'll get us what we want.

[turkenh]

This could be related just in case we end up needing to store the whole state for whatever reason: hashicorp/terraform#28603