Add Snowpark operator and decorator

[sfc-gh-jdu]

---

closes: #24456

This is the first step of adding Snowpark-related operators and decorators. We will add more in the following PRs (e.g., running snowpark in virtual env). This PR also includes unit tests, system tests and documentation changes.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[sfc-gh-jdu]

Could anyone please take a look? Thanks!

[potiuk]

Could anyone please take a look? Thanks!

I think there is a problem with dependencies that causes PROD build to fail

[potiuk]

Or simply rebase might fix it.

[sfc-gh-jdu]

Could anyone please take a look? Thanks!

I think there is a problem with dependencies that causes PROD build to fail

Yeah I checked CI failures last week, it seems unrelated. I rebased once but got some new failures you saw here. @jscheffl Thanks for rebasing it!

[sfc-gh-jdu]

Or simply rebase might fix it.

Or simply rebase might fix it.

All merge gates are green now. Can you please take a look when you get a chance? Thanks!

[jscheffl]

I have no real expertise with Snowflake/Snowpark, generally the cod elooks OK but can not judge functional correctness. Second eye review requested before merge.

[potiuk]

LGTM as well. Though also - I do not know details of Snowflake integration.

Speaking of which.....

Maybe it is a good idea that Snowflake team implements System Tests and system test dashboard similar to the ones that Amazon, Google. Astronomer and Teradata did (see https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards )

We had a talk about it at Airflow Summit (recording will be published soon) where the authors of such dashboard explained what benefit it brings to them and that might be a great way to make sure that the quality of integration with Snowflake is maintained over time.

See https://airflowsummit.org/sessions/2024/hello-quality-building-cis-to-run-providers-packages-system-tests/ for description of the talk (recordings should be avaoilable at this link soon)

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

[sfc-gh-jdu]

LGTM as well. Though also - I do not know details of Snowflake integration.

Speaking of which.....

Maybe it is a good idea that Snowflake team implements System Tests and system test dashboard similar to the ones that Amazon, Google. Astronomer and Teradata did (see https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards )

We had a talk about it at Airflow Summit (recording will be published soon) where the authors of such dashboard explained what benefit it brings to them and that might be a great way to make sure that the quality of integration with Snowflake is maintained over time.

See https://airflowsummit.org/sessions/2024/hello-quality-building-cis-to-run-providers-packages-system-tests/ for description of the talk (recordings should be avaoilable at this link soon)

Yes! @mik-laj already told me that and I'm working on such a dashboard now

[potiuk]

Yes! @mik-laj already told me that and I'm working on such a dashboard now

👀 👀 👀 👀 👀 👀 👀 👀