Exploring collaboration for async support in Apache Airflow

[shubham22]

Hey aiobotocore maintainers and @thehesiod,
First off, big kudos for your amazing work on aiobotocore! It's been super helpful for enabling async boto calls, and we really appreciate it.

I'm an Apache Airflow PM at AWS, and my team recently began using aiobotocore to add asynchronous functionality to Amazon integrations available in Apache Airflow. However, we've identified a potential issue that customers may encounter regarding the pinning down of botocore versions, such as unable to use latest functionality of botocore, delays in getting bug fixes, etc. In an effort to address this concern and optimize our integration, we're very interested in learning about the challenges you face in supporting new botocore versions, your CI testing processes for aiobotocore, and potential opportunities for collaboration between our teams.

Please note that I'm reaching out as an Apache Airflow contributor, not as a representative of the AWS Python SDK team. Our primary goal is to enhance the Apache Airflow ecosystem by exploring the integration of aiobotocore for better asynchronous support.

Would you be up for a quick call to chat about this? I believe that through collaboration and knowledge sharing, we can create an even better user experience for our mutual users. TIA!

cc: @o-nikolas

[thehesiod]

hello! @jettify is the owner of this repo but I've been doing a lot of work on it in the last few years. I've had limited time due to my regular job and having two little ones :) So the main issue with supporting newer botocore versions is time and the amount of work it takes to reliably bump versions. I've documented a bit of it here: https://github.com/aio-libs/aiobotocore/blob/master/CONTRIBUTING.rst#how-to-upgrade-botocore One needs to do a lot of source diffing to see sections that have changed and been added to see if we need to apply/re-implement them on our side. If we don't do this there can be a lot of subtle bugs. Another way to make this easier would be a way for us to adapt and run all the botocore tests which would require some thought and may be impossible.

There are two other long term goals to get rid of this completely in order of preference:

1. have the botocore library support async
2. swap to the aws c++ library as a backend

if you'd like a meeting you can ping me with my handle on gmail

[terricain]

Cool, always happy to help out if needed! Also nice to hear async support is on the roadmap for botocore 😄

[thehesiod]

btw I have to shout out to @terrycain he's been a tremendous help as well and has a related project, aioboto3.

[thehesiod]

btw check out this thread: #1014 I think a good idea would be having a github action that proposes PRs if there are just json/non conflicting related changes.

[terricain]

I did something similar awhile ago to raise PRs on aioboto3 when aiobotocore had released new versions, should really go and revive that :)

[baggiponte]

Hi there, I saw you released a new version that supports the latest botocore (and boto3 and awscli) versions! Congrats.

I was wondering if aiobotocore should follow a similar release schedule. It's messy AF but right now aiobotocore works with either 1.29.162 (2.5.1, latest version) or 1.29.76 (2.5.0) - none of the other botocore versions in between (almost 100!) are supported.

I know it's ugly, but what do you think of mirroring botocore minor releases? Or could botocore version constraints be relaxed?

[thehesiod]

if you look at the PR there are code changes, they're not cross compatible.