Create Bucket error, dial not working

[achetronic]

What happened?

When a bucket manifest is deployed from a Kind cluster, it is not working

How can we reproduce it?

• Create a clean Kind cluster, v1.21
• Deploy the operator
• Deploy the provider-aws
• Deploy the ProviderConfig for aws credentials
• Deploy the Bucket manifest

What environment did it happen in?

Crossplane version: latest from Helm official Chart (1.10.0)
AWS provider version: 0.22
Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-14T02:36:39Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-21T23:01:33Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

OS:

PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Bucket manifest including status:

apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  annotations:
    crossplane.io/external-name: madnesscorp-amazing-app
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"s3.aws.crossplane.io/v1beta1","kind":"Bucket","metadata":{"annotations":{},"name":"madnesscorp-amazing-app"},"spec":{"deletionPolicy":"Orphan","forProvider":{"acl":"private","locationConstraint":"eu-east-1","tagging":{"tagSet":[{"key":"CreatedBy","value":"Crossplane"}]}},"providerConfigRef":{"name":"aws-config"}}}
  creationTimestamp: "2022-10-26T20:18:12Z"
  generation: 1
  name: madnesscorp-amazing-app
  resourceVersion: "2166"
  uid: d0cd8c81-7258-41d9-8242-f4662f4a9652
spec:
  deletionPolicy: Orphan
  forProvider:
    acl: private
    locationConstraint: eu-east-1
    tagging:
      tagSet:
      - key: CreatedBy
        value: Crossplane
  providerConfigRef:
    name: aws-config
status:
  atProvider:
    arn: ""
  conditions:
  - lastTransitionTime: "2022-10-26T20:18:14Z"
    message: 'observe failed: failed to query Bucket: operation error S3: HeadBucket,
      exceeded maximum number of attempts, 3, https response error StatusCode: 0,
      RequestID: , HostID: , request send failed, Head "https://madnesscorp-amazing-app.s3.eu-east-1.amazonaws.com/":
      dial tcp: lookup madnesscorp-amazing-app.s3.eu-east-1.amazonaws.com on 10.96.0.10:53:
      no such host'
    reason: ReconcileError
    status: "False"
    type: Synced

Possible related issue:

kubernetes-sigs/kind#2383

[MisterMX]

That doesn't look like a provider issue to me and seems to be more connected to your kind instance or your network configuration.

Have you checked if other pods have connection issues as well?

[achetronic]

That doesn't look like a provider issue to me and seems to be more connected to your kind instance or your network configuration.

Have you checked if other pods have connection issues as well?

Hello @MisterMX
I tried the same with minikube with the same result, so I started to debug a bit deeper. The problem is on my manifest, with a region that does not exist. IMHO, the provider should check if the region exist for the S3 service. Following, the manifest:

apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  name: madnesscorp-amazing-app
spec:
  deletionPolicy: Orphan
  providerConfigRef:
    name: aws-config
  forProvider:
    acl: private
    locationConstraint: eu-east-1
    tagging:
      tagSet:
        - key: CreatedBy
          value: Crossplane

everything started to work again when I changed it to us-west-1

Should I open a PR to check the behavior? I mean, checking it before trying to create the resource
Thank you

[MisterMX]

I am not sure how we can handle that properly. And even than I am a bit skeptical if we should even consider that since the general idea of the Crossplane providers is to pass the user input (spec) directly to the external API (here: AWS) and return any error message encountered. IMHO it is the user's responsibility to provide meaningful input.

@haarchri any opinions on that?

[haarchri]

have the same feeling - the User ist responsible to provide the input

[achetronic]

Hello there, sorry for the delay

TBH I think the same, but in this specific case, when you fill the region with a bad param, the error that is throwing is related to the connection, and that is not good for debugging purposes.

Isn't the AWS API responding anything when the region is bad?

[MisterMX]

Isn't the AWS API responding anything when the region is bad?

Not that I know. As said, we are returning any error encountered as-is. It may depend on the AWS API you are talking to.

[achetronic]

Isn't the AWS API responding anything when the region is bad?

Not that I know. As said, we are returning any error encountered as-is. It may depend on the AWS API you are talking to.

mmm I see. Well, tbh I agree with you that is the ownership of people to write a well manifest. At the same time I would like to have more representative errors but this time is AWS blame. However, I would write documentation for this to help people with this edge cases. If you agree, I can write it of course

[MisterMX]

However, I would write documentation for this to help people with this edge cases. If you agree, I can write it of course.

Yes, I think this would be useful. I believe it would be best to add a note to the locationConstraint property. Feel free to create a PR and assign me.

[achetronic]

Hello, thank you for the answer.

do you mean to document it adding the comment into this example manifest?
https://github.com/crossplane-contrib/provider-aws/blob/master/examples/s3/bucket.yaml#L12

Or you mean at code level? :)

[MisterMX]

No, you have to add the documentation in the API definition at https://github.com/crossplane-contrib/provider-aws/blob/master/apis/s3/v1beta1/bucket_types.go#L38

[achetronic]

Hello there, @MisterMX

I did the changes on the comments mentioned, hope they are well. Please ping me if something is not understandable :)

#1575

[achetronic]

I think it is worth to close this issue, as the documentation was already merged?
WDYT @MisterMX? :)

[MisterMX]

Fixed in #1575