Add Open Search Provider

[cjames23]

This provider would be a pre-requisite for adding functionality to the Amazon provider for the AWS managed service for OpenSearch. This was discussed in a different PR which will be blocked by this PR now. both PRs would be pre-work for completing #33619 and adding Open Search log integration.

Open questions to more experienced contributors:

1. For system test DAG to succeed would an OpenSearch domain need to be setup and if so what is the approach for standing up such infrastructure for integration tests?
2. Any advice on mypy for some of the operator parameters? The goal is to allow users to either pass in the arguments directly or by passing in a class in the case of Search which works but in the case of adding a document the end user of the Operator needs to subclass the Document class. I am not sure what the best approach is for the typing for that parameter.

Email sent for Lazy Consensus to the dev-list.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[vincbeck]

For system test DAG to succeed would an OpenSearch domain need to be setup and if so what is the approach for standing up such infrastructure for integration tests?

System tests are not triggered as part of the CI. They are just used as example and also as system test/integration test if anybody wants to execute it (and as such, needs to setup the infra). The approach taken by the community is your write system tests in the Airflow repository but you execute them on your own with your infra set up. As an example, in the Amazon provider package, all system tests rely on AWS services (thus AWS account). These system tests are not triggered as part of Airflow CI but AWS team trigger frequently and results are accessible through a dashboard.

[cjames23]

@eladkal any advice on the failure that is happening with the CI image build step and the static checks? I ran breeze static checks against all files locally and do not receive the same failure.

[cjames23]

Yeah, I saw that, but the only changes in the Amazon package were some whitespace and punctuation in a docstring. Better done separate, but I didn't think it was worth blocking over it.

Those changes were because pre-commit checks failed locally for those things after a merge pull.

[ferruzzi]

What an odd static check to fail, I can't see how it could be related. When the workflow ends I'll bump it and see if it was just flake. It may also be broken in main, it happens some times.

[cjames23]

If the checks fail after this merge from main do you want me to include any fixes here for the breeze mypy issue? I think it just needs to be changed from ignore[misc] to ignore[attr-defined]. Not sure about the unit tests that are failing though from core.

[eladkal]

If the checks fail after this merge from main do you want me to include any fixes here for the breeze mypy issue? I think it just needs to be changed from ignore[misc] to ignore[attr-defined]. Not sure about the unit tests that are failing though from core.

We have identified the issue
#34941 should fix it

[potiuk]

Fixed #34941 and few other issues that caused broken main. Rebasing/resolving conflicts (generally regenerating breeze images with. breeze setup regenerate-images should fix it.

[eladkal]

🎉 🎉 🎉 🎉 🎉 🎉

[ferruzzi]

Nice, congrats!

[pateash]

Hi @cjames23,
Just a quick question regarding this one.
as we know OpenSearch is just a fork of Elasticsearch which happened couple of years ago,
Can we not use Elastic search operators for connecting to OpenSearch,
and if these hooks and operators are very similar to elastic search operators.