Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP serve connection failed hyper::Error(Parse(Method)) #627

Closed
songtianyi opened this issue May 23, 2023 · 9 comments · Fixed by #1313
Closed

HTTP serve connection failed hyper::Error(Parse(Method)) #627

songtianyi opened this issue May 23, 2023 · 9 comments · Fixed by #1313

Comments

@songtianyi
Copy link

I'm running two nodes in ubuntu, the msg output are reporting some error

2023-05-23 14:11:57 Parity Polkadot
2023-05-23 14:11:57 ✌️  version 0.9.42-9b1fc27cec4
2023-05-23 14:11:57 ❤️  by Parity Technologies <admin@parity.io>, 2017-2023
2023-05-23 14:11:57 📋 Chain specification: Rococo Local Testnet
2023-05-23 14:11:57 🏷  Node name: Alice
2023-05-23 14:11:57 👤 Role: AUTHORITY
2023-05-23 14:11:57 💾 Database: RocksDb at /tmp/relay/alice/chains/rococo_local_testnet/db/full
2023-05-23 14:11:57 ⛓  Native runtime: rococo-9420 (parity-rococo-v2.0-0.tx21.au0)
2023-05-23 14:12:01 🏷  Local node identity is: 12D3KooWKkCMHCA9Xm4YQkEwfRmbMsL9EoqrRH2wk8qnfQKP4rzi
2023-05-23 14:12:01 💻 Operating system: linux
2023-05-23 14:12:01 💻 CPU architecture: x86_64
2023-05-23 14:12:01 💻 Target environment: gnu
2023-05-23 14:12:01 💻 CPU: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
2023-05-23 14:12:01 💻 CPU cores: 8
2023-05-23 14:12:01 💻 Memory: 31078MB
2023-05-23 14:12:01 💻 Kernel: 5.19.0-1025-aws
2023-05-23 14:12:01 💻 Linux distribution: Ubuntu 22.04.2 LTS
2023-05-23 14:12:01 💻 Virtual machine: yes
2023-05-23 14:12:01 📦 Highest known block at paritytech/polkadot#234
2023-05-23 14:12:01 〽️ Prometheus exporter started at 127.0.0.1:9615
2023-05-23 14:12:01 Running JSON-RPC HTTP server: addr=0.0.0.0:9944, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"]
2023-05-23 14:12:01 Running JSON-RPC WS server: addr=127.0.0.1:41935, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"]
2023-05-23 14:12:01 🏁 CPU score: 1.07 GiBs
2023-05-23 14:12:01 🏁 Memory score: 4.98 GiBs
2023-05-23 14:12:01 🏁 Disk score (seq. writes): 325.41 MiBs
2023-05-23 14:12:01 🏁 Disk score (rand. writes): 64.07 MiBs
2023-05-23 14:12:01 ⚠️  The hardware does not meet the minimal requirements for role 'Authority' find out more at:
https://wiki.polkadot.network/docs/maintain-guides-how-to-validate-polkadot#reference-hardware
2023-05-23 14:12:01 👶 Starting BABE Authorship worker
2023-05-23 14:12:01 Accepting new connection 1/100
2023-05-23 14:12:01 HTTP serve connection failed hyper::Error(Parse(Method))
2023-05-23 14:12:03 Received msg before first active leaves update. This is not expected - message will be dropped. msg=DetermineUndisputedChain { base: (231, 0xba812bbbdf33535ec043fdd0192d58f243072d4631d259f19f2076f25f988d27), block_descriptions: [BlockDescription { block_hash: 0xb703a2c7249e23b1fb83de662fab2a14fb59747b7c7990a9cc847540b1d5282c, session: 40, candidates: [] }, BlockDescription { block_hash: 0xc89d5c1933b4f0e4a6dacd8c0075b24d60209b385dcd19b5f6195aa27fce70d6, session: 41, candidates: [] }, BlockDescription { block_hash: 0xe2bcb85f56e5af6da5d282796c415bd62c77ffafea9092cb1c4fefc8f1ce203b, session: 42, candidates: [] }], tx: Sender { complete: false } }
2023-05-23 14:12:03 Call to `DetermineUndisputedChain` failed error=DetermineUndisputedChainCanceled(Canceled)
2023-05-23 14:12:06 💤 Idle (0 peers), best: paritytech/polkadot#234 (0xe2bc…203b), finalized paritytech/polkadot#231 (0xba81…8d27), ⬇ 0 ⬆ 0
2023-05-23 14:12:08 Accepting new connection 1/100
2023-05-23 14:12:08 HTTP serve connection failed hyper::Error(Parse(Method))
2023-05-23 14:12:08 Accepting new connection 1/100
@bkchr
Copy link
Member

bkchr commented May 23, 2023

CC @niklasad1

@niklasad1
Copy link
Member

niklasad1 commented May 23, 2023

Hey, I think you can just ignore this log (this is most likely a client that is misbehaving)

It means that the HTTP request had an invalid HTTP method parsed, https://success.outsystems.com/documentation/11/reference/errors_and_warnings/errors/invalid_http_method_error/

We will downgrade this log to debug when paritytech/polkadot#7211 is merged.

Can you enable RUST_LOG="hyper,jsonrpsee=trace" and just paste the output so I can double check?

@xlc
Copy link
Contributor

xlc commented May 23, 2023

Screenshot 2023-05-24 at 10 59 10 AM

we also getting a lot of those in our subway instance

@niklasad1
Copy link
Member

niklasad1 commented May 24, 2023

Again, these warnings comes from bad HTTP requests and you can ignore it (unless it's a hyper/socket issue)

jsonrpsee v0.19 will downgrade all these to DEBUG/TRACE but we could register these in the substrate prometheus metrics to get a feeling how many HTTP requests are denied.

@songtianyi
Copy link
Author

I just deleted the storage in the base path and restarted the node, the errors are gone.

@niklasad1
Copy link
Member

niklasad1 commented May 24, 2023

@songtianyi is your node publicly exposed?

This may occur if something sends an invalid HTTP request to the RPC server and removing "the storage" in the base path shouldn't matter.

@Sophia-Gold Sophia-Gold transferred this issue from paritytech/polkadot Aug 24, 2023
claravanstaden pushed a commit to Snowfork/polkadot-sdk that referenced this issue Dec 8, 2023
@frikinomad
Copy link

image

I am getting the same error, any tips on how to resolve this now ?

@niklasad1
Copy link
Member

niklasad1 commented Dec 28, 2023

It's most likely just a bad HTTP request sent by the client, the error indicates the HTTP method could not be parsed.

Can you enable RUST_LOG="hyper,jsonrpsee=trace" on the node and just paste the output so I can double check?

@frikinomad
Copy link

image

P.S. I am running substrate node on wsl & running polkadot-js app on windows

github-merge-queue bot pushed a commit that referenced this issue Jan 23, 2024
This is a rather big change in jsonrpsee, the major things in this bump
are:
- Server backpressure (the subscription impls are modified to deal with
that)
- Allow custom error types / return types (remove jsonrpsee::core::Error
and jsonrpee::core::CallError)
- Bug fixes (graceful shutdown in particular not used by substrate
anyway)
   - Less dependencies for the clients in particular
   - Return type requires Clone in method call responses
   - Moved to tokio channels
   - Async subscription API (not used in this PR)

Major changes in this PR:
- The subscriptions are now bounded and if subscription can't keep up
with the server it is dropped
- CLI: add parameter to configure the jsonrpc server bounded message
buffer (default is 64)
- Add our own subscription helper to deal with the unbounded streams in
substrate

The most important things in this PR to review is the added helpers
functions in `substrate/client/rpc/src/utils.rs` and the rest is pretty
much chore.

Regarding the "bounded buffer limit" it may cause the server to handle
the JSON-RPC calls
slower than before.

The message size limit is bounded by "--rpc-response-size" thus "by
default 10MB * 64 = 640MB"
but the subscription message size is not covered by this limit and could
be capped as well.

Hopefully the last release prior to 1.0, sorry in advance for a big PR

Previous attempt: paritytech/substrate#13992

Resolves #748, resolves
#627
franciscoaguirre pushed a commit to paritytech/xcm that referenced this issue Jan 25, 2024
This is a rather big change in jsonrpsee, the major things in this bump
are:
- Server backpressure (the subscription impls are modified to deal with
that)
- Allow custom error types / return types (remove jsonrpsee::core::Error
and jsonrpee::core::CallError)
- Bug fixes (graceful shutdown in particular not used by substrate
anyway)
   - Less dependencies for the clients in particular
   - Return type requires Clone in method call responses
   - Moved to tokio channels
   - Async subscription API (not used in this PR)

Major changes in this PR:
- The subscriptions are now bounded and if subscription can't keep up
with the server it is dropped
- CLI: add parameter to configure the jsonrpc server bounded message
buffer (default is 64)
- Add our own subscription helper to deal with the unbounded streams in
substrate

The most important things in this PR to review is the added helpers
functions in `substrate/client/rpc/src/utils.rs` and the rest is pretty
much chore.

Regarding the "bounded buffer limit" it may cause the server to handle
the JSON-RPC calls
slower than before.

The message size limit is bounded by "--rpc-response-size" thus "by
default 10MB * 64 = 640MB"
but the subscription message size is not covered by this limit and could
be capped as well.

Hopefully the last release prior to 1.0, sorry in advance for a big PR

Previous attempt: paritytech/substrate#13992

Resolves paritytech/polkadot-sdk#748, resolves
paritytech/polkadot-sdk#627
bgallois pushed a commit to duniter/duniter-polkadot-sdk that referenced this issue Mar 25, 2024
This is a rather big change in jsonrpsee, the major things in this bump
are:
- Server backpressure (the subscription impls are modified to deal with
that)
- Allow custom error types / return types (remove jsonrpsee::core::Error
and jsonrpee::core::CallError)
- Bug fixes (graceful shutdown in particular not used by substrate
anyway)
   - Less dependencies for the clients in particular
   - Return type requires Clone in method call responses
   - Moved to tokio channels
   - Async subscription API (not used in this PR)

Major changes in this PR:
- The subscriptions are now bounded and if subscription can't keep up
with the server it is dropped
- CLI: add parameter to configure the jsonrpc server bounded message
buffer (default is 64)
- Add our own subscription helper to deal with the unbounded streams in
substrate

The most important things in this PR to review is the added helpers
functions in `substrate/client/rpc/src/utils.rs` and the rest is pretty
much chore.

Regarding the "bounded buffer limit" it may cause the server to handle
the JSON-RPC calls
slower than before.

The message size limit is bounded by "--rpc-response-size" thus "by
default 10MB * 64 = 640MB"
but the subscription message size is not covered by this limit and could
be capped as well.

Hopefully the last release prior to 1.0, sorry in advance for a big PR

Previous attempt: paritytech/substrate#13992

Resolves paritytech#748, resolves
paritytech#627
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants