S3 Connection not using host parameter

[sbone1]

Apache Airflow version

2.2.4

What happened

Airflow ignores the 'host' part of the s3 connection string and just uses s3.amazonaws.com

s3.amazonaws.com is appended to the bucket name, making it try to contact amazon instead of onprem s3.

What you think should happen instead

it should use the host provided in the connection string

How to reproduce

make a new connection string in the UI, and try to use that with the S3GetBucketTaggingOperator.
it should show the connection pointing to s3

like so:

list_tags_s3_bucket = S3GetBucketTaggingOperator( task_id="s3_list_tags_bucket", bucket_name="testbucket", aws_conn_id=os.environ.get("S3_CONNECTION_TEST"))

Operating System

debian buster k8s

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==3.0.0
apache-airflow-providers-celery==2.1.0
apache-airflow-providers-cncf-kubernetes==3.0.2
apache-airflow-providers-docker==2.4.1

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[Taragolis]

@sbone1 Are you sure that you pass parameters to connection correctly?

I've checked on Airflow 2.2.3 and apache-airflow-providers-amazon==3.0.0 and endpoint set correctly.

import os
from airflow.providers.amazon.aws.hooks.s3 import S3Hook

os.environ["AIRFLOW_CONN_S3_DUMMY_CONNECTION"] = "aws://?host=https://example.org"

conn = Connection.get_connection_from_secrets("s3_dummy_connection")
assert conn.host == '', "AWS Connection expected empty host"
assert conn.extra_dejson["host"] == 'https://example.org'

hook = S3Hook(aws_conn_id="s3_dummy_connection", region_name="eu-west-1")
assert hook.conn is hook.get_conn()
assert hook.conn.meta.endpoint_url == "https://example.org", hook.conn.meta.endpoint_url

hook = S3Hook(aws_conn_id=None, region_name="eu-west-1")
assert hook.conn.meta.endpoint_url == "https://s3.eu-west-1.amazonaws.com", hook.conn.meta.endpoint_url

[potiuk]

Converting to discussion until @sbone1 explains the doubt of @Taragolis

[Taragolis]

@potiuk I found almost the same issue #17833. I think there is two reason why users confuses and make this mistake.

First of all the extra parameter named as host but actually it expected endpoint_url with schema (http, https), DNS/IP, and additional HTTP URI such as port, path.

Second that Amazon Hook uses default fields in Connection UI, so users assume that host field might use as endpoint_uri.

I will have a look, and try to makes a changes in connection parameters in UI and rename host to endpoint_url

[potiuk]

Feel free @Taragolis .