diff --git a/README.md b/README.md index 113294eb5..0c43f77dc 100644 --- a/README.md +++ b/README.md @@ -15,43 +15,9 @@ kubectl crossplane install provider crossplane/provider-tf-aws:v0.2.1 You can see the API reference [here](https://doc.crds.dev/github.com/crossplane-contrib/provider-tf-aws). -## Developing +## Contributing -Run code-generation pipeline: -```console -make prepare.azurerm -go run cmd/generator/main.go -``` - -Run against a Kubernetes cluster: - -```console -make run -``` - -Build, push, and install: - -```console -make all -``` - -Build image: - -```console -make image -``` - -Push image: - -```console -make push -``` - -Build binary: - -```console -make build -``` +Please see the [Adding New Resources](/docs/adding-resources.md) guide. ## Report a Bug diff --git a/cmd/generator/main.go b/cmd/generator/main.go index cb3b56f05..8f3a98624 100644 --- a/cmd/generator/main.go +++ b/cmd/generator/main.go @@ -56,7 +56,7 @@ const ( "aws_kinesis_analytics_application": {}, }*/ -var alphaIncludedResource = map[string]struct{}{ +var includedResources = map[string]struct{}{ // VPC "aws_vpc": {}, @@ -153,7 +153,7 @@ func main() { // nolint:gocyclo groups := map[string]map[string]*schema.Resource{} for name, resource := range aws.Provider().ResourcesMap { - if _, ok := alphaIncludedResource[name]; !ok { + if _, ok := includedResources[name]; !ok { // Skip if not included continue } diff --git a/docs/adding-resources.md b/docs/adding-resources.md new file mode 100644 index 000000000..ac3f5ca38 --- /dev/null +++ b/docs/adding-resources.md @@ -0,0 +1,449 @@ +# Adding New Resources + +| Please note: the steps provided here is subject for changes since Terrajet is still alpha and does not guarantee a stabilized interface yet. | +| --- | + +Adding a new resource to [Terrajet] based providers is a very straightforward +process which is basically figuring out and providing the required custom +configuration for the resource. + +## Steps to Follow + +1. **Include Resource:** + + Add the Terraform resource name to the list of resources + (`includedResources`) in the [generator main.go]. + + For example, to add [aws_iam_access_key] we add [this line]. + + +2. **Create and Register Group Config File:** (if not exist) + +3. Create the custom config file for the group: + + ```console + GROUP= + RESOURCE= + mkdir config/$GROUP + cat << EOF > config/$GROUP/config.go + package $GROUP + + import ( + "github.com/crossplane-contrib/terrajet/pkg/config" + ) + + func init() { + config.Store.SetForResource("$RESOURCE", config.Resource{}) + } + EOF + ``` + + For `aws_iam_access_key` example: + + ``` + GROUP=iam + RESOURCE=aws_iam_access_key + ``` + +4. Register this custom configuration package in custom/register.go. See + [this](https://github.com/crossplane-contrib/provider-tf-aws/blob/main/config/register.go#L11) + for iam as an example. + +5. **Add custom configuration** in `config/$GROUP/config.go`: + + 1. [Configure external name](#external-name). + + 2. [Cross resource referencing](#cross-resource-referencing). + + 3. [Configure Connection Secret Keys](#additional-sensitive-fields-and-custom-connection-details). + + 4. [Late initialization configuration](#late-initialization-behavior). + +6. Run code-generation: + +7. Run Terrajet code-generation pipeline: + + ```console + go run cmd/generator/main.go + ``` + +8. Run code-generation for Controller Tools and Crossplane Tools: + + ```console + make generate + ``` + +9. Run against a Kubernetes cluster: + + ```console + kubectl apply -f package/crds + make run + ``` + +## Custom Configuration + +| Please note: the steps provided here is subject for changes since Terrajet is still alpha and does not guarantee a stabilized interface yet. | +| --- | + +[Terrajet] generates as much as it could using the available information in the +Terraform resource schema. This includes an XRM-conformant schema of the +resource, controller logic, late initialization, sensitive data handling etc. +However, there are still couple of information that requires some input +configuration which could easily be provided by checking the Terraform +documentation of the resource: + +- [External name] +- [Cross Resource Referencing] +- [Additional Sensitive Fields and Custom Connection Details] +- [Late Initialization Behavior] + +### External Name + +Crossplane uses an annotation in managed resource CR to identify the external +resource which is managed by Crossplane. See [the external name documentation] +for more details. The format and source of the external name depends on the +cloud provider; sometimes it could simply be the name of resource +(e.g. S3 Bucket), and sometimes it is an auto-generated id by cloud API +(e.g. VPC id ). This is something specific to resource, and we need some input +configuration for terrajet to appropriately generate a resource. + +Since Terraform already needs the same identifier to import a resource, most +helpful part of resource documentation is the [import section]. + +This is [the struct that holds the External Name configuration]: + +```go +// ExternalName contains all information that is necessary for naming operations, +// such as removal of those fields from spec schema and calling Configure function +// to fill attributes with information given in external name. +type ExternalName struct { + // SetIdentifierArgumentFn sets the name of the resource in Terraform argument + // map. + SetIdentifierArgumentFn SetIdentifierArgumentFn + + // OmittedFields are the ones you'd like to be removed from the schema since + // they are specified via external name. You can omit only the top level fields. + // No field is omitted by default. + OmittedFields []string + + // DisableNameInitializer allows you to specify whether the name initializer + // that sets external name to metadata.name if none specified should be disabled. + // It needs to be disabled for resources whose external name includes information + // more than the actual name of the resource, like subscription ID or region + // etc. which is unlikely to be included in metadata.name + DisableNameInitializer bool +} +``` + +Comments explain the purpose of each field but let's clarify further with some +examples. + +Checking the [import section of aws_vpc], we see that this resource is being +imported with `vpc id`. When we check the [arguments list] and provided +[example usages], it is clear that this **id** is not something that user +provides, rather generated by AWS API. Hence, we need to disable name +initializer, which simply sets the external-name annotation to `metadata.name` +of the resource. + +```go +DisableNameInitializer: true +``` + +Since we have no related fields in the [arguments list] that could be used to +build the external-name, we don't need to omit any fields (`OmittedFields`) or +need to use external name to set some arguments (`SetIdentifierArgumentFn`). +Hence, we end up the following external name configuration for `aws_vpc` +resource: + +```go +ExternalName: config.ExternalName{ + // Set to true explicitly since the value is calculated by AWS. + DisableNameInitializer: true, +}, +``` + +Let's check another resource, [aws_s3_bucket] which requires some other +configuration. Reading the [import section of s3 bucket] we see that bucket is +imported with its **name** which is provided with the [bucket] argument. +We can just use the CR name as the bucket name, and we don't have to disable +name initializer as we did above. + +However, since we are using metadata name as `bucket` argument, we need the +following two: + +- Fill `bucket` attribute using external-name annotation, so that Terraform + knows the value we want to provide: + + ```go + // BucketExternalNameConfigure configures bucket name. + func BucketExternalNameConfigure(base map[string]interface{}, externalName string) { + base["bucket"] = name + } + ``` + +- Omit `bucket` and `bucket_prefix` from the crd spec, so that we don't have + multiple inputs for the same thing (name of the bucket): + ```go + OmittedFields: []string{ + "bucket", + "bucket_prefix", + }, + ``` + +Hence, we end up the following external name configuration for `aws_s3_bucket` +resource: + +```go +const ( + // SelfPackagePath is the golang path for this package. + SelfPackagePath = "github.com/crossplane-contrib/provider-tf-aws/config/s3" +) + +// BucketSetIdentifierArgument configures bucket name. +func BucketSetIdentifierArgument(base map[string]interface{}, name string) { + base["bucket"] = name +} + +func init() { + config.Store.SetForResource("aws_s3_bucket", config.Resource{ + ExternalName: config.ExternalName{ + SetIdentifierArgumentFn: BucketSetIdentifierArgument, + OmittedFields: []string{ + "bucket", + "bucket_prefix", + }, + }, + }) +} +``` + +Please note, you can always check configuration of existing resources as further +examples. See the existing configuration for [VPC resource] and [s3 Bucket]. + +### Cross Resource Referencing + +Crossplane uses cross resource referencing to [handle dependencies] between +managed resources. For example, if you have an iam User defined as a Crossplane +managed resource, and you want to create an Access Key for that user, you would +need to refer to the User CR from the Access Key resource. This is handled by +cross resource referencing. + +See how the [user] referenced at `forProvider.userRef.name` field of the +Access Key in the following example: + +```yaml +apiVersion: iam.aws.tf.crossplane.io/v1alpha1 +kind: User +metadata: + name: sample-user +spec: + forProvider: {} +--- +apiVersion: iam.aws.tf.crossplane.io/v1alpha1 +kind: AccessKey +metadata: + name: sample-access-key +spec: + forProvider: + userRef: + name: sample-user + writeConnectionSecretToRef: + name: sample-access-key-secret + namespace: crossplane-system +``` + +Historically, reference resolution method were written by hand which requires +some effort, however, with the latest Crossplane code generation tooling, it is +now possible to [generate reference resolution methods] by just adding some +marker on the fields. Now, the only manual step for generating cross resource +references is to provide which field of a resource depends on which information +(e.g. `id`, `name`, `arn` etc.) from the other. + +In Terrajet, we have a [configuration] to provide this information for a field: + +```go +// Reference represents the Crossplane options used to generate +// reference resolvers for fields +type Reference struct { + // Type is the type name of the CRD if it is in the same package or + // . if it is in a different package. + Type string + // Extractor is the function to be used to extract value from the + // referenced type. Defaults to getting external name. + // Optional + Extractor string + // RefFieldName is the field name for the Reference field. Defaults to + // Ref or Refs. + // Optional + RefFieldName string + // SelectorFieldName is the field name for the Selector field. Defaults to + // Selector. + // Optional + SelectorFieldName string +} +``` + +For a resource that we want to generate, we need to check its argument list in +Terraform documentation and figure out which field needs reference to which +resource. + +Let's check [iam_access_key] as an example. In the argument list, we see the +[user] field which requires a reference to a IAM user. So, we need to the +following referencing configuration: + +```go +References: config.References{ + "user": config.Reference{ + Type: "User", + }, +}, +``` + +Please note the value of `Type` field needs to be a string representing the Go +type of the resource. Since, `AccessKey` and `User` resources are under the same +go package, we don't need to provide the package path. However, this is not +always the case and referenced resources might be in different package. In that +case, we would need to provide the full path. [Referencing] to a [kms key] from +`aws_ebs_volume` resource is a good example here: + +```go +References: map[string]config.Reference{ + "kms_key_id": { + Type: "github.com/crossplane-contrib/provider-tf-aws/apis/kms/v1alpha1.Key", + }, +}, +``` + +One important point here is, some fields contains arrays or maps. In this case, +we need to point that with a `[*]` after the name of array or map. You can check +[s3_import bucket] field of [aws_rds_cluster] resource as an example where +[s3_import field is a list] in terraform schema: + +```go +"s3_import[*].bucket_name": { + Type: "github.com/crossplane-contrib/provider-tf-aws/apis/s3/v1alpha1.Bucket", +}, +``` + +### Additional Sensitive Fields and Custom Connection Details + +Crossplane stores sensitive information of a managed resource in a Kubernetes +secret, together with some additional fields that would help consumption of the +resource, a.k.a. [connection details]. + +In Terrajet, we already [handle sensitive fields] that are marked as sensitive +in Terraform schema and no further action required for them. However, we still +have some custom configuration API that would allow marking additional fields as +sensitive (e.g. just if we encounter a field that is not marked properly) and +also to [add additional keys with custom values] no matter they are sensitive or +not. + +### Late Initialization Behavior +Terrajet runtime automatically performs late-initialization during +an [`external.Observe`] call with means of runtime reflection. +State of the world observed by Terraform CLI is used to initialize +any `nil`-valued pointer parameters in the managed resource's `spec`. +In most of the cases no custom configuration should be necessary for +late-initialization to work. However, there are certain cases where +you will want/need to customize late-initialization behaviour. Thus, +Terrajet provides an extensible [late-initialization customization API] +that controls late-initialization behaviour. + +The associated resource struct is defined [here](https://github.com/crossplane-contrib/terrajet/blob/c9e21387298d8ed59fcd71c7f753ec401a3383a5/pkg/config/resource.go#L91) as follows: +```go +// LateInitializer represents configurations that control +// late-initialization behaviour +type LateInitializer struct { + // IgnoredFields are the canonical field names to be skipped during + // late-initialization + IgnoredFields []string +} +``` +Currently, it only involves a configuration option to specify +certain `spec` parameters to be ignored during late-initialization. +Each element of the `LateInitializer.IgnoredFields` slice represents +the canonical path relative to the parameters struct for the managed resource's `Spec` +using `Go` type names as path elements. As an example, with the following type definitions: +```go +type Subnet struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + Spec SubnetSpec `json:"spec"` + Status SubnetStatus `json:"status,omitempty"` +} + +type SubnetSpec struct { + ForProvider SubnetParameters `json:"forProvider"` + ... +} + +type DelegationParameters struct { + // +kubebuilder:validation:Required + Name *string `json:"name" tf:"name,omitempty"` + ... +} + +type SubnetParameters struct { + // +kubebuilder:validation:Optional + AddressPrefix *string `json:"addressPrefix,omitempty" tf:"address_prefix,omitempty"` + // +kubebuilder:validation:Optional + Delegation []DelegationParameters `json:"delegation,omitempty" tf:"delegation,omitempty"` + ... +} +``` +If you would like to have the late-initialization library *not* to process the +`SubnetParameters.AddressPrefix`parameter field, then the +following configuration where we specify the relative parameter field path is sufficient: +```go +config.Store.SetForResource("azurerm_subnet", config.Resource{ + LateInitializer: config.LateInitializer{ + IgnoredFields: []string{"AddressPrefix"}, + }, +}) +``` +Please note also that, unlike fieldpath specifications, +path elements are `Go` type names, and you should not use +array specifiers when denoting slice of struct types. +As an example if you would like to ignore `DelegationParameters.Name` +fields during late-initialization, the relative name to be used is: +`Delegation.Name`. + +[comment]: <> (References) + +[Terrajet]: https://github.com/crossplane-contrib/terrajet +[generator main.go]: /cmd/generator/main.go +[aws_iam_access_key]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_access_key +[this line]: https://github.com/crossplane-contrib/provider-tf-aws/blob/d2c0024f18b5760e4d2222c405ad0501c63ee0b2/cmd/generator/main.go#L108 +[this]: https://github.com/crossplane-contrib/provider-tf-aws/blob/main/config/register.go#L11 + +[External name]: #external-name +[Cross Resource Referencing]: #cross-resource-referencing +[Additional Sensitive Fields and Custom Connection Details]: #additional-sensitive-fields-and-custom-connection-details +[Late Initialization Behavior]: #late-initialization-behavior +[the external name documentation]: https://crossplane.io/docs/v1.4/concepts/managed-resources.html#external-name +[import section]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_access_key#import +[the struct that holds the External Name configuration]: https://github.com/crossplane-contrib/terrajet/blob/c9e21387298d8ed59fcd71c7f753ec401a3383a5/pkg/config/resource.go#L58 +[aws_vpc]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc +[import section of aws_vpc]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc#import +[arguments list]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc#argument-reference +[example usages]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc#example-usage +[aws_s3_bucket]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket +[import section of s3 bucket]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket#import +[bucket]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket#bucket +[VPC resource]: https://github.com/crossplane-contrib/provider-tf-aws/blob/d2c0024f18b5760e4d2222c405ad0501c63ee0b2/config/ec2/config.go#L188-L191 +[s3 Bucket]: https://github.com/crossplane-contrib/provider-tf-aws/blob/d2c0024f18b5760e4d2222c405ad0501c63ee0b2/config/s3/config.go#L37-L42. +[handle dependencies]: https://crossplane.io/docs/v1.4/concepts/managed-resources.html#dependencies +[user]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_access_key#user +[generate reference resolution methods]: https://github.com/crossplane/crossplane-tools/pull/35 +[configuration]: https://github.com/crossplane-contrib/terrajet/blob/24f186f45e70808768aa0b7abd4fa82e4f446f3f/pkg/config/field.go#L5 +[iam_access_key]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_access_key#argument-reference +[Referencing]: https://github.com/crossplane-contrib/provider-tf-aws/blob/5509c10d768622c3631615947cf7b18086a58aa3/config/ebs/config.go#L27-L31 +[kms key]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ebs_volume#kms_key_id +[s3_import bucket]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/rds_cluster#bucket_name +[aws_rds_cluster]: https://github.com/crossplane-contrib/provider-tf-aws/blob/5509c10d768622c3631615947cf7b18086a58aa3/config/rds/config.go#L50 +[s3_import field is a list]: https://github.com/hashicorp/terraform-provider-aws/blob/fce7062f70caa92a78bca629ffe441c088456418/aws/resource_aws_rds_cluster.go#L287 +[connection details]: https://crossplane.io/docs/v1.4/concepts/managed-resources.html#connection-details +[handle sensitive fields]: https://github.com/crossplane-contrib/terrajet/pull/77 +[add additional keys with custom values]: https://github.com/crossplane-contrib/terrajet/pull/121 +[`external.Observe`]: https://github.com/crossplane-contrib/terrajet/blob/c9e21387298d8ed59fcd71c7f753ec401a3383a5/pkg/controller/external.go#L177 +[late-initialization customization API]: https://github.com/crossplane-contrib/terrajet/blob/c9e21387298d8ed59fcd71c7f753ec401a3383a5/pkg/resource/lateinit.go#L50