Manage clusters without using a local asset_dir #595

dghubble · 2019-12-05T09:13:32Z

Allow generated assets (TLS materials, manifests) to be securely distributed to controller node(s) via file provisioner (i.e. ssh-agent) as an assets bundle file, rather than relying on assets being locally rendered to disk in an asset_dir and then securely distributed
Change asset_dir from required to optional. Left unset, asset_dir defaults to "" and no assets will be written to files on the machine that runs terraform apply
Enhancement: Managed cluster assets are kept only in Terraform state, which supports different backends (GCS, S3, etcd, etc) and optional encryption. terraform apply accesses state, runs in-memory, and distributes sensitive materials to controllers without making use of local disk (simplifies use in automation systems)
Enhancement: Improve asset unpack and layout process to position etcd certificates and control plane certificates more cleanly, without unneeded secret materials

User Guide

If you still want to write assets to a local directory during terraform apply, keep your asset_dir setting. If you want to skip writing assets locally (recommended), remove asset_dir (or set to "") in your next cluster definition. To access the kubeconfig,

resource "local_file" "kubeconfig" {
  content  = module.mycluster.kubeconfig-admin
  filename = "/home/user/.kube/configs/kubeconfig"
}

Details

Terraform file provisioner support for distributing directories of contents (with unknown structure) has been limited to reading from a local directory, meaning local writes to asset_dir were required.
Track upstream features that could allow asset_dir to be optional #585 discusses the problem and newer or upcoming Terraform features that might help.
Observation: Terraform provisioner support for single files works well, but iteration isn't viable. We're also constrained to Terraform language features on the apply side (no extra plugins, no shelling out) and CoreOS / Fedora tools on the receive side.
Take a map representation of the contents that would have been splayed out in asset_dir and pack/encode them into a single file format devised for easy unpacking. Use an awk one-liner on the receive side to unpack. In pratice, this has worked well and its rather nice that a single
assets file is transferred by file provisioner (all or none)

* Allow generated assets (TLS materials, manifests) to be securely distributed to controller node(s) via file provisioner (i.e. ssh-agent) as an assets bundle file, rather than relying on assets being locally rendered to disk in an asset_dir and then securely distributed * Change `asset_dir` from required to optional. Left unset, asset_dir defaults to "" and no assets will be written to files on the machine that runs terraform apply * Enhancement: Managed cluster assets are kept only in Terraform state, which supports different backends (GCS, S3, etcd, etc) and optional encryption. terraform apply accesses state, runs in-memory, and distributes sensitive materials to controllers without making use of local disk (simplifies use in CI systems) * Enhancement: Improve asset unpack and layout process to position etcd certificates and control plane certificates more cleanly, without unneeded secret materials Details: * Terraform file provisioner support for distributing directories of contents (with unknown structure) has been limited to reading from a local directory, meaning local writes to asset_dir were required. #585 discusses the problem and newer or upcoming Terraform features that might help. * Observation: Terraform provisioner support for single files works well, but iteration isn't viable. We're also constrained to Terraform language features on the apply side (no extra plugins, no shelling out) and CoreOS / Fedora tools on the receive side. * Take a map representation of the contents that would have been splayed out in asset_dir and pack/encode them into a single file format devised for easy unpacking. Use an awk one-liner on the receive side to unpack. In pratice, this has worked well and its rather nice that a single assets file is transferred by file provisioner (all or none) Rel: poseidon/terraform-render-bootstrap#162

* Original tutorials favored including the platform (e.g. google-cloud) in modules (e.g. google-cloud-yavin). Prefer naming conventions where each module / cluster has a simple name (e.g. yavin) since the platform is usually redundant * Retain the example cluster naming themes per platform

* Originally, generated TLS certificates, manifests, and cluster "assets" written to local disk (`asset_dir`) during terraform apply cluster bootstrap * Typhoon v1.17.0 introduced bootstrapping using only Terraform state to store cluster assets, to avoid ever writing sensitive materials to disk and improve automated use-cases. `asset_dir` was changed to optional and defaulted to "" (no writes) * Typhoon v1.18.0 deprecated the `asset_dir` variable, removed docs, and announced it would be deleted in future. * Remove the `asset_dir` variable Cluster assets are now stored in Terraform state only. For those who wish to write those assets to local files, this is possible doing so explicitly. ``` resource local_file "assets" { for_each = module.bootstrap.assets_dist filename = "some-assets/${each.key}" content = each.value } ``` Related: * poseidon/typhoon#595 * poseidon/typhoon#678

* Originally, poseidon/terraform-render-bootstrap generated TLS certificates, manifests, and cluster "assets" written to local disk (`asset_dir`) during terraform apply cluster bootstrap * Typhoon v1.17.0 introduced bootstrapping using only Terraform state to store cluster assets, to avoid ever writing sensitive materials to disk and improve automated use-cases. `asset_dir` was changed to optional and defaulted to "" (no writes) * Typhoon v1.18.0 deprecated the `asset_dir` variable, removed docs, and announced it would be deleted in future. * Add Terraform output `assets_dir` map * Remove the `asset_dir` variable Cluster assets are now stored in Terraform state only. For those who wish to write those assets to local files, this is possible doing so explicitly. ``` resource local_file "assets" { for_each = module.yavin.assets_dist filename = "some-assets/${each.key}" content = each.value } ``` Related: * #595 * #678

* Originally, poseidon/terraform-render-bootstrap generated TLS certificates, manifests, and cluster "assets" written to local disk (`asset_dir`) during terraform apply cluster bootstrap * Typhoon v1.17.0 introduced bootstrapping using only Terraform state to store cluster assets, to avoid ever writing sensitive materials to disk and improve automated use-cases. `asset_dir` was changed to optional and defaulted to "" (no writes) * Typhoon v1.18.0 deprecated the `asset_dir` variable, removed docs, and announced it would be deleted in future. * Add Terraform output `assets_dir` map * Remove the `asset_dir` variable Cluster assets are now stored in Terraform state only. For those who wish to write those assets to local files, this is possible doing so explicitly. ``` resource local_file "assets" { for_each = module.yavin.assets_dist filename = "some-assets/${each.key}" content = each.value } ``` Related: * poseidon/typhoon#595 * poseidon/typhoon#678

* Originally, poseidon/terraform-render-bootstrap generated TLS certificates, manifests, and cluster "assets" written to local disk (`asset_dir`) during terraform apply cluster bootstrap * Typhoon v1.17.0 introduced bootstrapping using only Terraform state to store cluster assets, to avoid ever writing sensitive materials to disk and improve automated use-cases. `asset_dir` was changed to optional and defaulted to "" (no writes) * Typhoon v1.18.0 deprecated the `asset_dir` variable, removed docs, and announced it would be deleted in future. * Add Terraform output `assets_dir` map * Remove the `asset_dir` variable Cluster assets are now stored in Terraform state only. For those who wish to write those assets to local files, this is possible doing so explicitly. ``` resource local_file "assets" { for_each = module.yavin.assets_dist filename = "some-assets/${each.key}" content = each.value } ``` Related: * poseidon#595 * poseidon#678

dghubble force-pushed the terraform-apply-without-asset-dir branch from cbb43f2 to 81f2d83 Compare December 5, 2019 09:20

dghubble added 2 commits December 5, 2019 01:24

dghubble force-pushed the terraform-apply-without-asset-dir branch from 81f2d83 to d9c7a9e Compare December 6, 2019 06:59

dghubble changed the title ~~Introduce cluster creation without local writes to asset_dir~~ Manage clusters without using a local asset_dir (optional) Dec 6, 2019

dghubble changed the title ~~Manage clusters without using a local asset_dir (optional)~~ Manage clusters without using a local asset_dir Dec 6, 2019

dghubble merged commit d9c7a9e into master Dec 6, 2019

dghubble mentioned this pull request Dec 6, 2019

Track upstream features that could allow asset_dir to be optional #585

Closed

dghubble deleted the terraform-apply-without-asset-dir branch December 6, 2019 07:10

dghubble mentioned this pull request Oct 17, 2020

Remove asset_dir variable and optional asset writes poseidon/terraform-render-bootstrap#222

Merged

dghubble mentioned this pull request Oct 17, 2020

Remove asset_dir variable and optional asset writes #854

Merged

dghubble mentioned this pull request Nov 22, 2020

Marking assets_dist as a sensitive output in typhoon as well #884

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Manage clusters without using a local asset_dir #595

Manage clusters without using a local asset_dir #595

dghubble commented Dec 5, 2019 •

edited

Loading

Manage clusters without using a local asset_dir #595

Manage clusters without using a local asset_dir #595

Conversation

dghubble commented Dec 5, 2019 • edited Loading

User Guide

Details

dghubble commented Dec 5, 2019 •

edited

Loading