missing client field endpoint_ruleset_resolver in botocore 1.28.0

[abivolmv]

Describe the bug
With botocore 1.28.0 (released 14h ago) aiobotocore 2.4.0 will fail when creating client because botocore 1.28.0 has a new required positional field endpoint_ruleset_resolver.
Tested locally on Windows and on Lambda (linux py 3.8)

The same code we have works with botocore 1.27.95

  File "C:\Users\xyz\repos\xyz-api\venv2\lib\site-packages\aiobotocore\session.py", line 26, in __aenter__
    self._client = await self._coro
  File "C:\Users\xyz\repos\xyz-api\venv2\lib\site-packages\aiobotocore\session.py", line 193, in _create_client
    client = await client_creator.create_client(
  File "C:\Users\xyz\repos\xyz-api\venv2\lib\site-packages\aiobotocore\client.py", line 70, in create_client
    service_client = cls(**client_args)
TypeError: __init__() missing 1 required positional argument: 'endpoint_ruleset_resolver'

Checklist

• I have reproduced in environment where pip check passes without errors
• I have provided pip freeze results
• I have provided sample code or detailed way to reproduce
• I have tried the same code in botocore to ensure this is an aiobotocore specific issue
• I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue
• I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

pip freeze results

Environment:

• Python Version: 3.8, 3.9
• OS name and version: Windows 11, Linux (lambda)

Additional context
aioboto3==10.1.0
aiobotocore==2.4.0
aiofiles==0.8.0
aiohttp==3.8.1
aioitertools==0.8.0
aiosignal==1.2.0
altgraph==0.17.2
ansicon==1.89.0
async-timeout==4.0.2
asyncmock==0.4.2
atomicwrites==1.4.0
attrs==21.2.0
aws-cdk.assets==1.137.0
aws-cdk.aws-acmpca==1.137.0
aws-cdk.aws-apigateway==1.137.0
aws-cdk.aws-applicationautoscaling==1.137.0
aws-cdk.aws-athena==1.137.0
aws-cdk.aws-autoscaling==1.137.0
aws-cdk.aws-autoscaling-common==1.137.0
aws-cdk.aws-autoscaling-hooktargets==1.137.0
aws-cdk.aws-certificatemanager==1.137.0
aws-cdk.aws-cloudformation==1.137.0
aws-cdk.aws-cloudfront==1.137.0
aws-cdk.aws-cloudfront-origins==1.137.0
aws-cdk.aws-cloudwatch==1.137.0
aws-cdk.aws-cloudwatch-actions==1.137.0
aws-cdk.aws-codebuild==1.137.0
aws-cdk.aws-codecommit==1.137.0
aws-cdk.aws-codeguruprofiler==1.137.0
aws-cdk.aws-codepipeline==1.137.0
aws-cdk.aws-codestarnotifications==1.137.0
aws-cdk.aws-cognito==1.137.0
aws-cdk.aws-dynamodb==1.137.0
aws-cdk.aws-ec2==1.137.0
aws-cdk.aws-ecr==1.137.0
aws-cdk.aws-ecr-assets==1.137.0
aws-cdk.aws-ecs==1.137.0
aws-cdk.aws-efs==1.137.0
aws-cdk.aws-eks==1.137.0
aws-cdk.aws-elasticloadbalancing==1.137.0
aws-cdk.aws-elasticloadbalancingv2==1.137.0
aws-cdk.aws-events==1.137.0
aws-cdk.aws-events-targets==1.137.0
aws-cdk.aws-globalaccelerator==1.137.0
aws-cdk.aws-glue==1.137.0
aws-cdk.aws-iam==1.137.0
aws-cdk.aws-kinesis==1.137.0
aws-cdk.aws-kinesisfirehose==1.137.0
aws-cdk.aws-kms==1.137.0
aws-cdk.aws-lambda==1.137.0
aws-cdk.aws-lambda-event-sources==1.137.0
aws-cdk.aws-logs==1.137.0
aws-cdk.aws-route53==1.137.0
aws-cdk.aws-route53-targets==1.137.0
aws-cdk.aws-s3==1.137.0
aws-cdk.aws-s3-assets==1.137.0
aws-cdk.aws-s3-notifications==1.137.0
aws-cdk.aws-sam==1.137.0
aws-cdk.aws-secretsmanager==1.137.0
aws-cdk.aws-servicediscovery==1.137.0
aws-cdk.aws-signer==1.137.0
aws-cdk.aws-sns==1.137.0
aws-cdk.aws-sns-subscriptions==1.137.0
aws-cdk.aws-sqs==1.137.0
aws-cdk.aws-ssm==1.137.0
aws-cdk.aws-stepfunctions==1.137.0
aws-cdk.aws-stepfunctions-tasks==1.137.0
aws-cdk.cloud-assembly-schema==1.137.0
aws-cdk.core==1.137.0
aws-cdk.custom-resources==1.137.0
aws-cdk.cx-api==1.137.0
aws-cdk.lambda-layer-awscli==1.137.0
aws-cdk.lambda-layer-kubectl==1.137.0
aws-cdk.lambda-layer-node-proxy-agent==1.137.0
aws-cdk.region-info==1.137.0
aws-xray-sdk==2.10.0
blessed==1.19.0
boto==2.49.0
boto3==1.25.0
botocore==1.28.0
cattrs==1.8.0
certifi==2021.10.8
chalice==1.26.3
chardet==3.0.4
charset-normalizer==2.0.9
click==8.0.3
colorama==0.4.4
constructs==3.3.168
coverage==5.5
demjson==2.2.4
dirtyjson==1.0.7
ecdsa==0.17.0
frozenlist==1.2.0
future==0.18.2
idna==2.10
iniconfig==1.1.1
inquirer==2.8.0
jinxed==1.1.0
jmespath==0.10.0
jsii==1.56.0
mock==4.0.3
multidict==5.2.0
mypy-extensions==0.4.3
packaging==21.3
pefile==2021.9.3
Pillow==8.2.0
pluggy==1.0.0
psutil==5.8.0
publication==0.0.3
py==1.11.0
pyasn1==0.4.8
pyinstaller==4.10
pyinstaller-hooks-contrib==2022.2
pyparsing==3.0.7
pytest==6.2.5
pytest-asyncio==0.17.2
pytest-cov==3.0.0
pytest-html==3.1.1
pytest-metadata==2.0.1
python-dateutil==2.8.2
python-editor==1.0.4
python-jose==3.3.0
pywin32-ctypes==0.2.0
PyYAML==6.0
qrcode==7.3.1
readchar==2.0.1
requests==2.25.1
rsa==4.8
s3transfer==0.6.0
six==1.16.0
toml==0.10.2
typing-extensions==3.10.0.2
unittest-xml-reporting==3.0.4
urllib3==1.25.11
watchtower==3.0.0
wcwidth==0.2.5
wrapt==1.13.3
yarl==1.7.2

[leandrorebelo]

I hope that solution will be to update the botocore to the newest version.

[ragulpr]

Thanks for reporting and thanks for all work team! I can verify the same issue with botocore 1.28.1 and aiobotocore 2.4.0

...
  File "/usr/local/lib/python3.8/dist-packages/fsspec/asyn.py", line 111, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/fsspec/asyn.py", line 96, in sync
    raise return_result
  File "/usr/local/lib/python3.8/dist-packages/fsspec/asyn.py", line 53, in _runner
    result[0] = await coro
  File "/usr/local/lib/python3.8/dist-packages/s3fs/core.py", line 863, in _ls
    files = await self._lsdir(path, refresh)
  File "/usr/local/lib/python3.8/dist-packages/s3fs/core.py", line 631, in _lsdir
    await self.set_session()
  File "/usr/local/lib/python3.8/dist-packages/s3fs/core.py", line 473, in set_session
    self._s3 = await s3creator.__aenter__()
  File "/usr/local/lib/python3.8/dist-packages/aiobotocore/session.py", line 26, in __aenter__
    self._client = await self._coro
  File "/usr/local/lib/python3.8/dist-packages/aiobotocore/session.py", line 193, in _create_client
    client = await client_creator.create_client(
  File "/usr/local/lib/python3.8/dist-packages/aiobotocore/client.py", line 70, in create_client
    service_client = cls(**client_args)
TypeError: __init__() missing 1 required positional argument: 'endpoint_ruleset_resolver'

aiobotocore 2.4.0 requires botocore<1.27.60,>=1.27.59, but you'll have botocore 1.28.1 which is incompatible.
Successfully installed aiobotocore-2.4.0 aiohttp-3.8.3 aioitertools-0.11.0 aiosignal-1.2.0 async-timeout-4.0.2 attrs-22.1.0 botocore-1.28.1 
...

Sorry for not providing more details but that's all I can for now.

But it should be noted that aiobotocore 2.4.0 requires botocore<1.27.60 so this may not really be a aiobotocore issue (did not verify if fixing botocore version is possible/fixes problem). See also boto/botocore#2796 for supposed fix for botocore<1.28

[thehesiod]

you cannot mix and match botocore and aiobotocore versions, that is why we have strict version requirements

[abivolmv]

you cannot mix and match botocore and aiobotocore versions, that is why we have strict version requirements

actually you can. If you shouldn't that's a different question ^^

We fixed this by hardfixing the botocore version. Until now we just had aioboto3==x.x.x and it seems botocore version 1.28.0 was picked up from some other package when doing pip install -r xxx.txt .

But it should be noted that aiobotocore 2.4.0 requires botocore<1.27.60

yes you are right. It is more an issue for future that needs to be taken care of. For the next version of aiobotocore.

[thehesiod]

well, if you obey rules ya cannot per setup.py requirements, if you want to be a rule breaker you have to deal with the consequences :) Each botocore bump is very time consuming and detailed to catch issues like these. In fact in the last attempt to do a minor patch bump it brought forth a new auth codepath that is going to require lots of new async plumbing :(

[abivolmv]

Ah, not the rule breaker by far ^^ We didn't specifically request for botocore upgrade but probably some package from above did it and overrode aiobotocore's requirements. As mentioned we did not have botocore nor aiobotocore in the requirements file until this popped up and we had to do it.
I read on contributing guide about botocore version bump , it seems really a tedious task. Anywhere we could vote for botocore to implement aio ?

[glarione]

Absence of this feature from botocore means no use of aiobotocore with minio on-prem service. Any endpoint_url used in config will be overridden by old versioned botocore 😿

[thehesiod]

Ah, not the rule breaker by far ^^ We didn't specifically request for botocore upgrade but probably some package from above did it and overrode aiobotocore's requirements. As mentioned we did not have botocore nor aiobotocore in the requirements file until this popped up and we had to do it. I read on contributing guilde about botocore version bump , it seems really a tedious task. Anywhere we could vote for botocore to implement aio ?

this is why it's critical to use something like pipenv/poetry.

yep! see #36 and boto/botocore#458

[thehesiod]

another option is for this project to swap to the cpp lib...another huge task

[thehesiod]

is there anything else needed for this issue or just a version bump?

[abivolmv]

"Just" a version bump please.

this is why it's critical to use something like pipenv/poetry.

Tried Pipenv and it didn't work well for us. Though it was 2020 and last release then was in 2018 and there was even an issue asking if this project was dead. Now I see the releases are often and project is alive. Maybe it's worth an another try as the concept was nice (npm like).
Poetry worked well in other projects I worked so it is in to-do for the project in case too.