Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log server on celery worker does not work in IPv6-only setup #24755

Closed
1 of 2 tasks
zartstrom opened this issue Jun 30, 2022 · 3 comments · Fixed by #24846
Closed
1 of 2 tasks

Log server on celery worker does not work in IPv6-only setup #24755

zartstrom opened this issue Jun 30, 2022 · 3 comments · Fixed by #24846
Labels
area:core kind:bug This is a clearly a bug

Comments

@zartstrom
Copy link
Contributor

zartstrom commented Jun 30, 2022

Apache Airflow version

2.2.5

What happened

I deployed the Airflow helm chart in a Kubernetes cluster that only allows IPv6 traffic.
When I want to look at a task log in the UI there is this message:

*** Fetching from: http://airflow-v1-worker-0.airflow-v1-worker.airflow.svc.cluster.local:8793/log/my-dag/my-task/2022-06-28T00:00:00+00:00/1.log
*** Failed to fetch log file from worker. [Errno 111] Connection refused

So the webserver cannot fetch the logfile from the worker.

This happens in my opinion because the gunicorn application listens to 0.0.0.0 (IPv4), see code or worker log below, and the inter-pod communication in my cluster is IPv6.

~ » k logs airflow-v1-worker-0 -c airflow-worker -p
[2022-06-30 14:51:52 +0000] [49] [INFO] Starting gunicorn 20.1.0
[2022-06-30 14:51:52 +0000] [49] [INFO] Listening at: http://0.0.0.0:8793 (49)
[2022-06-30 14:51:52 +0000] [49] [INFO] Using worker: sync
[2022-06-30 14:51:52 +0000] [50] [INFO] Booting worker with pid: 50
[2022-06-30 14:51:52 +0000] [51] [INFO] Booting worker with pid: 51
 
 -------------- celery@airflow-v1-worker-0 v5.2.3 (dawn-chorus)
--- ***** ----- 
-- ******* ---- Linux-5.10.118-x86_64-with-glibc2.28 2022-06-30 14:51:53
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         airflow.executors.celery_executor:0x7f73b8d23d00
- ** ---------- .> transport:   redis://:**@airflow-v1-redis-master.airflow.svc.cluster.local:6379/1
- ** ---------- .> results:     postgresql://airflow:**@airflow-v1-pgbouncer.airflow.svc.cluster.local:6432/airflow_backend_db
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> default          exchange=default(direct) key=default
                

[tasks]
  . airflow.executors.celery_executor.execute_command

What you think should happen instead

The gunicorn webserver should (configurably) listen to IPv6 traffic.

How to reproduce

No response

Operating System

Debian GNU/Linux 10 (buster)

Versions of Apache Airflow Providers

No response

Deployment

Other 3rd-party Helm chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@zartstrom zartstrom added area:core kind:bug This is a clearly a bug labels Jun 30, 2022
@potiuk
Copy link
Member

potiuk commented Jun 30, 2022

Would you like to fix it - you seem to know, where and how and you have nice environment to test the fix in. And you will not have to wait for someone to pick it up - this is one of the best ways you can contribute back to the software you get for free and join the community of more than 2100 contributors.

@zartstrom
Copy link
Contributor Author

zartstrom commented Jul 2, 2022

-- edited --

Hi @potiuk, I'll try to craft a PR, thanks for the inspiration :)

I have one question, maybe you can help or point me to the right source:
What is the best way to create a airflow docker image with a change in the source code?

Answer: Look in the documentation for building the image and there for section building from github.
You can run the build with build arg

  --build-arg CONSTRAINTS_GITHUB_REPOSITORY="my-account/airflow"

What I tried before: I did a tiny code change (hardcoded the log server to some other number, 8888), and then built an image with

DOCKER_BUILDKIT=1 docker build . -f Dockerfile --pull --tag my-image:0.0.1

But this builds from the latest pypi version of airflow, doh.

When I deployed an image built with the that command with a helm chart to our Kubernetes Cluster nothing changed and my code change was not visible of course.

@potiuk
Copy link
Member

potiuk commented Jul 2, 2022

Yeah. This is good observation. The Dockerfile defaults are meant to be useful for people who want to customize the image and use the "released" version of Airflow. What you should look at as a "developer" is breeze. Breeze has a build-prod-image command that uses local sources to build the image.

If you are curious - you might run it with --dry-run to understand which build-args should be passed to get the source build - and you can also look more at https://airflow.apache.org/docs/docker-stack/build-arg-ref.html to understand all the parameters passed. one of them is AIRFLOW_SOURCES_FROM set to . to use current sources, but there are few others.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:core kind:bug This is a clearly a bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants