-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Moto server hangs with the latest version of werkzeug (v2.2.1) #5341
Comments
Just confirmed this myself in my own CI pipeline. The critical change was between v2.1.2 (works) and v2.2.0 (hangs) of that package. Everything else in my environment was kept the same. |
Important to note that the hang doesn't happen at moto_server startup, it happens at the first request (in my case, a create_bucket() call). When it hangs, the moto_server process will take up 100% cpu. |
Can confirm that. Suggested downgrade to 2.1.2 worked for me |
It was the same issue, but after fix update, on create_bucket() call it fais with this error message: Failed: [undefined]botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://127.0.0.1:5000/ -- edit by @bblommers - removed to clean the thread a bit |
Just to note there are a few projects now copying this workaround. Some show up as xref's in PR ( #5330 ). Wonder if it makes sense to include this version constraint in |
The breakage of werkzeug was a combination of things - werkzeug introduced some bugs, and also exposed some bugs in Moto. For now, Moto has pinned the werkzeug-dependency to < 2.2.x until we have time to resolve this completely. As a debugging note: |
Thanks for the additional context! 🙏 No worries. Appreciate all of you taking time to carefully work through these issues 🙂 |
See: getmoto/moto#5341 git-svn-id: file:///srv/repos/svn-community/svn@1300704 9fca08f4-af9d-4005-b8df-a31f2cc04f65
See: getmoto/moto#5341 git-svn-id: file:///srv/repos/svn-community/svn@1300704 9fca08f4-af9d-4005-b8df-a31f2cc04f65
@bblommers |
As far as I can tell, that's not possible @fmigneault. Given this config:
, PIP will always try to resolve FYI, if you're interested in this sort of thing, people have been discussing whether Python/PIP should support this possibility: https://discuss.python.org/t/adding-a-default-extra-require-environment/4898/125 |
@bblommers from setuptools import setup, find_packages
setup(
name="test-pkg",
version="1.0",
packages=find_packages(),
install_requires=["werkzeug>=0.5"],
extras_require={"server": ["werkzeug>=0.5,<2.2.0"]},
) Install with extra $ pip install ".[server]"
Processing c:\users\francis\appdata\local\temp\package
Preparing metadata (setup.py) ... done
Collecting werkzeug>=0.5
Downloading Werkzeug-2.2.2-py3-none-any.whl (232 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 232.7/232.7 kB 4.7 MB/s eta 0:00:00
Downloading Werkzeug-2.1.2-py3-none-any.whl (224 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 224.9/224.9 kB 6.9 MB/s eta 0:00:00
Building wheels for collected packages: test-pkg
Building wheel for test-pkg (setup.py) ... done
Created wheel for test-pkg: filename=test_pkg-1.0-py3-none-any.whl size=994 sha256=96634c3eda4a48d4d746ebe0d6cd681b53a077ec32cfc37a05c175e45f42e4bc
Stored in directory: C:\Users\Francis\AppData\Local\Temp\pip-ephem-wheel-cache-x8wj_b8a\wheels\cf\ec\59\95d02b995d018c2613fda5938f1d242e630a1571a0d964b249
Successfully built test-pkg
Installing collected packages: werkzeug, test-pkg
Successfully installed test-pkg-1.0 werkzeug-2.1.2 Uninstall all to retry without extra. $ pip uninstall test-pkg Werkzeug
Found existing installation: test-pkg 1.0
Uninstalling test-pkg-1.0:
Would remove:
c:\users\francis\.conda\envs\test-pip\lib\site-packages\test_pkg-1.0.dist-info\*
Proceed (Y/n)? y
Successfully uninstalled test-pkg-1.0
Found existing installation: Werkzeug 2.1.2
Uninstalling Werkzeug-2.1.2:
Would remove:
c:\users\francis\.conda\envs\test-pip\lib\site-packages\werkzeug-2.1.2.dist-info\*
c:\users\francis\.conda\envs\test-pip\lib\site-packages\werkzeug\*
Proceed (Y/n)? y
Successfully uninstalled Werkzeug-2.1.2 Install without extra, $ pip install .
Processing c:\users\francis\appdata\local\temp\package
Preparing metadata (setup.py) ... done
Collecting werkzeug>=0.5
Using cached Werkzeug-2.2.2-py3-none-any.whl (232 kB)
Collecting MarkupSafe>=2.1.1
Downloading MarkupSafe-2.1.1-cp38-cp38-win_amd64.whl (17 kB)
Building wheels for collected packages: test-pkg
Building wheel for test-pkg (setup.py) ... done
Created wheel for test-pkg: filename=test_pkg-1.0-py3-none-any.whl size=994 sha256=c17a3e67c0b46b0bd051924f08005a5e9fe396b08c1cc019e6def70f5db3dcc5
Stored in directory: C:\Users\Francis\AppData\Local\Temp\pip-ephem-wheel-cache-wj7bvsmi\wheels\cf\ec\59\95d02b995d018c2613fda5938f1d242e630a1571a0d964b249
Successfully built test-pkg
Installing collected packages: MarkupSafe, werkzeug, test-pkg
Successfully installed MarkupSafe-2.1.1 test-pkg-1.0 werkzeug-2.2.2 |
Looks like that is a bug that was fixed recently. I was on an older version of PIP,
Upgrading PIP to Having said that: I think a longterm fix, where we can use Moto with any werkzeug-version, is almost ready, so I'm going to focus on that instead. |
This should now be fixed as of moto >= 4.0.8.dev17. Please let us know if you run into any other issues with this version though! |
Thanks Bert! 🙏 Is there a final (non-dev) release planned? |
There's no date planned @jakirkham, but somewhere this week. I'd like to wait until we get some positive feedback that the dev-version works as expected... (hint, hint) 🙂 |
@bblommers |
Awesome - thanks for verifying @fmigneault |
@jakirkham Moto 4.0.8 has now been released. |
Just want to say the new release worked great 😀 Thanks Bert! 🙏 |
Relates to getmoto/moto#5341
interesting, in our project we had moto[server]==4.0.8 + werkzeug==2.2.2 and I was seeing hangs. Going back to 4.0.5 fixes it |
@thehesiod Do you have a reproducible test case, or a specific API call that always hangs? |
will try today to get one |
aha, the moto server endpoint is returning a 500 during a bucket paginate operation causing it to retry multiple times and eventually time out...digging deeper
|
Just tested build of 4.0.8 wirh --deselect/--ignore all units which are using ============================================================================= warnings summary =============================================================================
../../../../../usr/lib/python3.8/site-packages/requests/__init__.py:109
/usr/lib/python3.8/site-packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (None)/charset_normalizer (3.0.0) doesn't match a supported version!
warnings.warn(
tests/test_core/test_auth.py::test_access_denied_with_temporary_credentials
tests/test_rds/test_rds.py::test_create_database_replica_cross_region
tests/test_rds/test_rds.py::test_create_db_instance_without_availability_zone
tests/test_rds/test_rds.py::test_create_db_instance_with_availability_zone
/usr/lib/python3.8/site-packages/botocore/client.py:625: FutureWarning: The rds client is currently using a deprecated endpoint: rds.amazonaws.com. In the next minor version this will be moved to rds.us-east-1.amazonaws.com. See https://github.com/boto/botocore/issues/2705 for more details.
warnings.warn(
tests/test_core/test_context_manager.py::test_context_manager_returns_mock
tests/test_core/test_moto_api.py::test_reset_api
tests/test_core/test_moto_api.py::test_data_api
tests/test_core/test_request_mocking.py::test_passthrough_requests
tests/test_sqs/test_sqs.py::test_create_queues_in_multiple_region
tests/test_sqs/test_sqs.py::test_get_queue_with_prefix
/usr/lib/python3.8/site-packages/botocore/client.py:625: FutureWarning: The sqs client is currently using a deprecated endpoint: us-west-1.queue.amazonaws.com. In the next minor version this will be moved to sqs.us-west-1.amazonaws.com. See https://github.com/boto/botocore/issues/2705 for more details.
warnings.warn(
tests/test_core/test_nested.py: 1 warning
tests/test_ses/test_ses_sns_boto3.py: 5 warnings
tests/test_sns/test_publish_batch.py: 2 warnings
tests/test_sns/test_publishing_boto3.py: 43 warnings
tests/test_sns/test_subscriptions_boto3.py: 1 warning
tests/test_sqs/test_sqs.py: 109 warnings
tests/test_sqs/test_sqs_multiaccount.py: 1 warning
/usr/lib/python3.8/site-packages/botocore/client.py:625: FutureWarning: The sqs client is currently using a deprecated endpoint: queue.amazonaws.com. In the next minor version this will be moved to sqs.us-east-1.amazonaws.com. See https://github.com/boto/botocore/issues/2705 for more details.
warnings.warn(
tests/test_emr/test_emr_boto3.py::test_run_job_flow_in_multiple_regions
/usr/lib/python3.8/site-packages/botocore/client.py:625: FutureWarning: The elasticmapreduce client is currently using a deprecated endpoint: eu-west-1.elasticmapreduce.amazonaws.com. In the next minor version this will be moved to elasticmapreduce.eu-west-1.amazonaws.com. See https://github.com/boto/botocore/issues/2705 for more details.
warnings.warn(
tests/test_events/test_events_integration.py::test_send_to_sqs_fifo_queue
tests/test_events/test_events_integration.py::test_send_to_sqs_queue
tests/test_events/test_events_integration.py::test_send_to_sqs_queue_with_custom_event_bus
/usr/lib/python3.8/site-packages/botocore/client.py:625: FutureWarning: The sqs client is currently using a deprecated endpoint: eu-central-1.queue.amazonaws.com. In the next minor version this will be moved to sqs.eu-central-1.amazonaws.com. See https://github.com/boto/botocore/issues/2705 for more details.
warnings.warn(
tests/test_events/test_events_integration.py::test_send_to_sqs_fifo_queue
/home/tkloczko/rpmbuild/BUILD/moto-4.0.8/moto/events/models.py:217: UserWarning: To let EventBridge send messages to your SQS FIFO queue, you must enable content-based deduplication.
warnings.warn(
tests/test_glue/test_partition_filter.py::test_get_partitions_expression_unknown_column
tests/test_glue/test_partition_filter.py::test_get_partitions_expression_int_column
tests/test_glue/test_partition_filter.py::test_get_partitions_expression_decimal_column
tests/test_glue/test_partition_filter.py::test_get_partitions_expression_string_column
tests/test_glue/test_partition_filter.py::test_get_partitions_expression_date_column
tests/test_glue/test_partition_filter.py::test_get_partitions_expression_timestamp_column
tests/test_glue/test_partition_filter.py::test_get_partition_expression_warnings_and_exceptions
/home/tkloczko/rpmbuild/BUILD/moto-4.0.8/moto/glue/utils.py:353: UserWarning: Expression filtering is experimental
warnings.warn("Expression filtering is experimental")
tests/test_kms/test_kms_boto3.py::test_sign_invalid_signing_algorithm
/home/tkloczko/rpmbuild/BUILD/moto-4.0.8/moto/kms/responses.py:623: UserWarning: The SigningAlgorithm-parameter is ignored hardcoded to RSASSA_PSS_SHA_256 for client.sign()
warnings.warn(
tests/test_kms/test_kms_boto3.py::test_sign_and_verify_ignoring_grant_tokens
/home/tkloczko/rpmbuild/BUILD/moto-4.0.8/moto/kms/responses.py:618: UserWarning: The GrantTokens-parameter is not yet implemented for client.sign()
warnings.warn(
tests/test_kms/test_kms_boto3.py::test_sign_and_verify_ignoring_grant_tokens
/home/tkloczko/rpmbuild/BUILD/moto-4.0.8/moto/kms/responses.py:666: UserWarning: The GrantTokens-parameter is not yet implemented for client.verify()
warnings.warn(
tests/test_kms/test_kms_boto3.py::test_verify_invalid_signing_algorithm
/home/tkloczko/rpmbuild/BUILD/moto-4.0.8/moto/kms/responses.py:676: UserWarning: The SigningAlgorithm-parameter is ignored hardcoded to RSASSA_PSS_SHA_256 for client.verify()
warnings.warn(
tests/test_sns/test_publishing_boto3.py::test_publish_to_sqs_in_different_region
tests/test_sqs/test_sqs.py::test_create_queues_in_multiple_region
/usr/lib/python3.8/site-packages/botocore/client.py:625: FutureWarning: The sqs client is currently using a deprecated endpoint: us-west-2.queue.amazonaws.com. In the next minor version this will be moved to sqs.us-west-2.amazonaws.com. See https://github.com/boto/botocore/issues/2705 for more details.
warnings.warn(
tests/test_sqs/test_sqs.py::test_receive_message_with_xml_content
/usr/lib/python3.8/site-packages/botocore/client.py:625: FutureWarning: The sqs client is currently using a deprecated endpoint: eu-west-2.queue.amazonaws.com. In the next minor version this will be moved to sqs.eu-west-2.amazonaws.com. See https://github.com/boto/botocore/issues/2705 for more details.
warnings.warn(
tests/test_sqs/test_sqs.py::test_get_queue_attributes_no_param
/usr/lib/python3.8/site-packages/botocore/client.py:625: FutureWarning: The sqs client is currently using a deprecated endpoint: ap-northeast-3.queue.amazonaws.com. In the next minor version this will be moved to sqs.ap-northeast-3.amazonaws.com. See https://github.com/boto/botocore/issues/2705 for more details.
warnings.warn(
tests/test_sqs/test_sqs.py::test_message_becomes_inflight_when_received
/usr/lib/python3.8/site-packages/botocore/client.py:625: FutureWarning: The sqs client is currently using a deprecated endpoint: eu-west-1.queue.amazonaws.com. In the next minor version this will be moved to sqs.eu-west-1.amazonaws.com. See https://github.com/boto/botocore/issues/2705 for more details.
warnings.warn(
tests/test_sqs/test_sqs.py::test_send_message_to_fifo_without_message_group_id
tests/test_sqs/test_sqs.py::test_send_messages_to_fifo_without_message_group_id
tests/test_sqs/test_sqs.py::test_maximum_message_size_attribute_default
tests/test_sqs/test_sqs.py::test_maximum_message_size_attribute_fails_for_invalid_values
tests/test_sqs/test_sqs.py::test_send_message_fails_when_message_size_greater_than_max_message_size
/usr/lib/python3.8/site-packages/botocore/client.py:625: FutureWarning: The sqs client is currently using a deprecated endpoint: eu-west-3.queue.amazonaws.com. In the next minor version this will be moved to sqs.eu-west-3.amazonaws.com. See https://github.com/boto/botocore/issues/2705 for more details.
warnings.warn(
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================================= short test summary info ==========================================================================
SKIPPED [2] ../../../../../usr/lib/python3.8/site-packages/_pytest/unittest.py:367: Only run test with external server
SKIPPED [3] ../../../../../usr/lib/python3.8/site-packages/_pytest/unittest.py:367: We only want to test ServerMode here
================================================ 6663 passed, 5 skipped, 23 deselected, 199 warnings in 1423.59s (0:23:43) ================================================= BTW: is it really necessary to use |
issue is due to this line: https://github.com/spulec/moto/blob/master/moto/s3/models.py#L1616, it's throwing an exception because it's copying a very complicated structure which has stuff like threading locks and what not in it.
|
Here is what I do:
then in a different terminal I run
curl "http://localhost:5555/"
Expected behavior:
The
curl
should immediately return a result.Actual behavior:
The
moto_server
hangs. The process is at 100% CPU usage so maybe it is busy waiting for something (just a guess).If
werkzeug
is manually downgraded to v2.1.0 (pip install Werkzeug=2.1.0
) or previous everything works as expected.Here is my complete
pip freeze
:The text was updated successfully, but these errors were encountered: