Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Permissions of new log directory not being set to parent folders. #37200

Closed
1 of 2 tasks
murinmat opened this issue Feb 6, 2024 · 2 comments · Fixed by #37310
Closed
1 of 2 tasks

Permissions of new log directory not being set to parent folders. #37200

murinmat opened this issue Feb 6, 2024 · 2 comments · Fixed by #37310
Labels
area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet
Milestone
Airflow 2.8.2

Comments

@murinmat
Copy link

murinmat commented Feb 6, 2024 •
edited
Loading

Apache Airflow version

2.8.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When creating log dir/file structure, permissions are set only for the leaf folder.

As per pathlib documentation:

`Create a new directory at this given path. If mode is given, it is combined with the process’ umask value to determine the file mode and access flags. If the path already exists, FileExistsError is raised.

If parents is true, any missing parents of this path are created as needed; they are created with the default permissions without taking mode into account (mimicking the POSIX mkdir -p command).`

Only the leaf folder is created with the specified permisions. Therefore, if logging structure is specified and contains other than leaf folders (such as the default one), the permissions to not get propagated, resulting in a permission denied error if, for example, a celery worker running on a different machine (while in the same group with same GID) tries to write to a directory already created by a task created by a worker on a different machine.

What you think should happen instead?

No response

How to reproduce

  1. Use file logging in airflow.
  2. Use the default formatting of log files.
  3. Create a worker on a remote machine (in my case via celery on redis) with its specific queue.
  4. Run any task on this queue. Two things can happen:
    a.) If this task is the first in the DAG, other workers won't be able to write into logs of this attempt
    b.) If this task is not the first in the DAG, then the logs of this task won't be able to be written.

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@murinmat murinmat added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Feb 6, 2024
@boring-cyborg boring-cyborg
Copy link

boring-cyborg bot commented Feb 6, 2024

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

@murinmat murinmat changed the title Permissions of new directory not being set to parent folders. Permissions of new log directory not being set to parent folders. Feb 6, 2024
potiuk added a commit to potiuk/airflow that referenced this issue Feb 10, 2024
@potiuk
Fix permissions of parent folders for log file handler
89c0835 
When we created a new folder during log file handler directory creation
we did not change the permissions of parent folders.

Fixes: apache#37200
@potiuk potiuk mentioned this issue Feb 10, 2024
Merged
@potiuk potiuk added this to the Airflow 2.8.2 milestone Feb 10, 2024
potiuk added a commit that referenced this issue Feb 10, 2024
@potiuk
Fix permissions of parent folders for log file handler (#37310)
aae4a83 
When we created a new folder during log file handler directory creation
we did not change the permissions of parent folders.

Fixes: #37200
ephraimbuddy pushed a commit that referenced this issue Feb 20, 2024
@potiuk @ephraimbuddy
Fix permissions of parent folders for log file handler (#37310)
267f339 
When we created a new folder during log file handler directory creation
we did not change the permissions of parent folders.

Fixes: #37200
(cherry picked from commit aae4a83)
ephraimbuddy pushed a commit that referenced this issue Feb 22, 2024
@potiuk @ephraimbuddy
Fix permissions of parent folders for log file handler (#37310)
76a701d 
When we created a new folder during log file handler directory creation
we did not change the permissions of parent folders.

Fixes: #37200
(cherry picked from commit aae4a83)
@ephraimbuddy ephraimbuddy mentioned this issue Feb 22, 2024
Closed
68 tasks
@potiuk
Copy link
Member

potiuk commented Mar 26, 2024

@matejmurin01 -> can you please check your email or talk to me directly in Airflow Slack if you are there?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet
Projects
None yet
Milestone
Airflow 2.8.2
Development

Successfully merging a pull request may close this issue.

Fix permissions of parent folders for log file handler potiuk/airflow
2 participants
@potiuk @murinmat