Add pod_template_dict field to KubernetesPodOperator

[sonmezonur]

I would like to propose adding a new templated field to kubernetes pod operator. Currently, operator has pod_template_file field to create base pod spec but you need to create a file to use this feature. IMHO it might be useful to accept stream or content, thus users might easily configure the base pod spec.

Use Case

I would like to write airflow dag which clones git repository and mounts volume for cloned branch. Branch name should be parameterized so that users can specify it while running the DAG. Example code:

templated_pod = k8s.V1Pod(
      metadata=k8s.V1ObjectMeta(),
      spec=k8s.V1PodSpec(
          containers=[],
          init_containers=[
            k8s.V1Container(
              name='git-clone',
              image='registry.k8s.io/git-sync:v3.1.1',
              args=[
                  f'--repo=git@github.com:airflow/some_repo.git',
                  f'--branch={{ params.get("repo_branch", "master") }}',
                  '--root=/tmp/git',
                  '--dest=gitclone',
                  '--ssh=true',
                  '--wait=120',
                  '--one-time=true',
              ],
              volume_mounts=[self.git_sync_mount, self.ssh_key_mount],
              security_context=k8s.V1SecurityContext(run_as_user=65533),
              resources=k8s.V1ResourceRequirements(
                  requests={
                      "cpu": "300m",
                      "memory": "512Mi",
                  },
                  limits={
                      "cpu": "300m",
                      "memory": "512Mi",
                  },
              ),
          
          ],
          volumes=[],
      ),
)
serialized_pod = yaml.dump(
       pod_generator.PodGenerator.serialize_pod(templated_pod)
)
operator = KubernetesPodOperator(pod_template_content=serialized_pod, ...)

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[potiuk]

I like the idea, but it needs at least some unit tests to be considered as worthy merging. Without tests it's prone to regressions.

[sonmezonur]

I like the idea, but it needs at least some unit tests to be considered as worthy merging. Without tests it's prone to regressions.

Added testcase

[sonmezonur]

My first proposal was to add content string (in YAML format) but while writing unit tests, I realized that a pod template dictionary might be easier to use. What do you think? Refactored the code in my last commit. PTAL

[hussein-awala]

I wonder if we should support both types V1Pod and dict 🤔
I tested it and it seems like when we use V1Pod the template fields are not resolved. I will check if there is a simple solution for that.

[sonmezonur]

My first intention was to make full_pod_spec field templated instead of adding a new field. But templater does not resolve V1Pod as you referred. I might have a simple solution, will try once I have some time 🤞

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[hussein-awala]

It looks good, LGTM

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.