HTTP_PROXY settings do not recognize Worker Pods within NO_PROXY CIDR range

[softyoungha]

Official Helm Chart version

1.13.1 (latest released)

Apache Airflow version

2.9.1

Kubernetes Version

1.26.4

Helm Chart configuration

It think there is no related Helm chart configuration.

Docker Image customizations

I am currently using HTTP proxy and NO_PROXY settings.

ARG PROXY_SERVER
ENV HTTP_PROXY=$PROXY_SERVER \
     HTTPS_PROXY=$PROXY_SERVER \
     http_proxy=$PROXY_SERVER \
     https_proxy=$PROXY_SERVER
ENV NO_PROXY="localhost,127.0.0.1,10.0.0.0/8"

What happened

Airflow webserver is unable to retrieve logs from the worker pods.

10.240.5.168
*** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers, workers and triggerer) have the same 'secret_key' configured in 'webserver' section and time is synchronized on all your machines (for example with ntpd)
See more at https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key
*** Could not read served logs: Client error '403 Forbidden' for url 'http://10.240.5.168:8793/log/dag_id=sample/run_id=manual__2024-05-23T07:18:59.298882+00:00/task_id=sourcing/attempt=1.log'
For more information check: https://httpstatuses.com/403

403 Forbidden is an error response caused by sending the request through the proxy.

What you think should happen instead

Because the worker pod's address is included in the NO_PROXY CIDR 10.0.0.0/8, requests to the worker pod should not go through the proxy. However, requests are still being sent to the proxy.

This behavior is due to httpx not supporting the CIDR range in the NO_PROXY settings.

httpx: NO_PROXY is not working for IP range/mask

airflow/airflow/utils/log/file_task_handler.py

Lines 87 to 105 in 482c0be

	def _fetch_logs_from_service(url, log_relative_path):
	# Import occurs in function scope for perf. Ref: https://github.com/apache/airflow/pull/21438
	import httpx
	from airflow.utils.jwt_signer import JWTSigner
	timeout = conf.getint("webserver", "log_fetch_timeout_sec", fallback=None)
	signer = JWTSigner(
	secret_key=conf.get("webserver", "secret_key"),
	expiration_time_in_seconds=conf.getint("webserver", "log_request_clock_grace", fallback=30),
	audience="task-instance-logs",
	)
	response = httpx.get(
	url,
	timeout=timeout,
	headers={"Authorization": signer.generate_signed_token({"filename": log_relative_path})},
	)
	response.encoding = "utf-8"
	return response

In the method airflow.utils.log.file_task_handler._fetch_logs_from_service, the webserver uses httpx to fetch logs from worker pods.

Since requests supports NO_PROXY CIDR, simply changing the httpx part to requests in the function works well.

def _fetch_logs_from_service(url, log_relative_path):
    # Import occurs in function scope for perf. Ref: https://github.com/apache/airflow/pull/21438
    import requests

    from airflow.utils.jwt_signer import JWTSigner

    timeout = conf.getint("webserver", "log_fetch_timeout_sec", fallback=None)
    signer = JWTSigner(
        secret_key=conf.get("webserver", "secret_key"),
        expiration_time_in_seconds=conf.getint("webserver", "log_request_clock_grace", fallback=30),
        audience="task-instance-logs",
    )
    response = requests.get(
        url,
        timeout=timeout,
        headers={"Authorization": signer.generate_signed_token({"filename": log_relative_path})},
    )
    response.encoding = "utf-8"
    return response

How to reproduce

When a URL is directly input for testing in airflow.utils.log.file_task_handler._fetch_logs_from_service, it can be observed that the request bypasses the NO_PROXY's CIDR and utilizes the proxy instead

import os
import logging
from airflow.utils.log.file_task_handler import _fetch_logs_from_service

# activate httpx logging
logging.basicConfig(level=logging.DEBUG)
logging.getLogger("httpcore").setLevel(logging.DEBUG)

# set proxy
os.environ["http_proxy"] = "http://proxy.example.com"
os.environ["https_proxy"] = "http://proxy.example.com"
os.environ["no_proxy"] = "localhost,10.0.0.0/8"

# test _fetch_logs_from_service
res = _fetch_logs_from_service(
    "http://10.240.5.168:8793/log/dag_id=sample/run_id=manual__2024-05-23T07:18:59.298882+00:00/task_id=sourcing/attempt=1.log",
    "dag_id=sample/run_id=manual__2024-05-23T07:18:59.298882+00:00/task_id=sourcing/attempt=1.log"
)
print(res)
# [2024-05-24T04:41:52.081+0900] {_trace.py:45} DEBUG - connect_tcp.started host='proxy.example.com' port=80 local_address=None timeout=5 socket_options=None
# [2024-05-24T04:41:52.090+0900] {_trace.py:45} DEBUG - connect_tcp.failed exception=ConnectError(gaierror(-2, 'Name or service not known'))

Anything else

After wondering why this was happening, it appears that the URLPattern matching logic in httpx does not take CIDR into account.

Below is an example code:

import os
from httpx._utils import get_environment_proxies

os.environ["http_proxy"] = "http://proxy.example.com"
os.environ["https_proxy"] = "http://proxy.example.com"
os.environ["no_proxy"] = "localhost,10.0.0.0/8"

httpx_proxy_info = get_environment_proxies()
print(httpx_proxy_info)
# {
#       'http://': 'http://proxy.example.com',
#       'https://': 'http://proxy.example.com',
#       'all://localhost': None,
#       'all://10.0.0.0/8': None     <- no_proxy
# }

from httpx._utils import URLPattern
from httpx._urls import URL

url_pattern = URLPattern("all://10.0.0.0/8")
print(url_pattern.matches(URL("http://10.0.0.0")))
# True
print(url_pattern.matches(URL("http://10.240.5.168")))
# False

1. httpx._utils.get_environment_proxies: get proxies information from env vars
2. httpx._client.BaseClient._get_proxy_map: proxy -> mounts property
3. httpx._client.BaseClient._transport_for_url: match urls here
4. but CIDR not supports

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[Taragolis]

Feel free raise a PR