Use maximum allowed response size for request/response protocols #5753

AndreiEres · 2024-09-18T11:06:12Z

Description

Adjust the PoV response size to the default values used in the substrate.
Fixes #5503

Integration

The changes shouldn't impact downstream projects since we are only increasing the limit.

Review Notes

You can't see it from the changes, but it affects all protocols that use the POV_RESPONSE_SIZE constant.

Protocol::ChunkFetchingV1
Protocol::ChunkFetchingV2
Protocol::CollationFetchingV1
Protocol::CollationFetchingV2
Protocol::PoVFetchingV1
Protocol::AvailableDataFetchingV1

Increasing timeouts

polkadot-sdk/polkadot/node/network/protocol/src/request_response/mod.rs

Lines 126 to 129 in fae1537

    
           /// This timeout is based on what seems sensible from a time budget perspective, considering 6 
        
           /// second block time. This is going to be tough, if we have multiple forks and large PoVs, but we 
        
           /// only have so much time. 
        
           const POV_REQUEST_TIMEOUT_CONNECTED: Duration = Duration::from_millis(1200);

I assume the current PoV request timeout is set to 1.2s to handle 5 consecutive requests during a 6s block. This setting does not relate to the PoV response size. I see no reason to change the current timeouts after adjusting the response size.

However, we should consider networking speed limitations if we want to increase the maximum PoV size to 10 MB. With the number of parallel requests set to 10, validators will need the following networking speeds:

5 MB PoV: at least 42 MB/s, ideally 50 MB/s.
10 MB PoV: at least 84 MB/s, ideally 100 MB/s.

The current required speed of 50 MB/s aligns with the 62.5 MB/s specified in the reference hardware requirements. Increasing the PoV size to 10 MB may require a higher networking speed.

sandreim

PRDoc ?

sandreim · 2024-09-19T09:02:27Z

polkadot/node/network/protocol/src/request_response/mod.rs

-/// limit might have more severe effects.
-const POV_RESPONSE_SIZE: u64 = MAX_POV_SIZE as u64 + 10_000;
+/// Same as what we use in substrate networking.
+const POV_RESPONSE_SIZE: u64 = 16 * 1024 * 1024;


I think substrate should export a constant and we should use it here.

sandreim · 2024-09-19T09:04:16Z

The current required speed of 50 MB/s aligns with the 62.5 MB/s specified in the reference hardware requirements. Increasing the PoV size to 10 MB may require a higher networking speed.

Yes, this is worst case scenario, when all blocks you need to recover are full. We'll likely have to raise specs for networking,

dmitry-markin · 2024-09-19T12:37:28Z

polkadot/node/network/protocol/src/request_response/mod.rs

-/// limit might have more severe effects.
-const POV_RESPONSE_SIZE: u64 = MAX_POV_SIZE as u64 + 10_000;
+/// Same as what we use in substrate networking.
+const POV_RESPONSE_SIZE: u64 = MAX_RESPONSE_SIZE;


Is it safe to not base this value on MAX_POV_SIZE?

It's okay, I suppose, because we make these changes to bump it.

sandreim · 2024-09-19T14:19:41Z

The current required speed of 50 MB/s aligns with the 62.5 MB/s specified in the reference hardware requirements. Increasing the PoV size to 10 MB may require a higher networking speed.

Yes, this is worst case scenario, when all blocks you need to recover are full. We'll likely have to raise specs for networking,

Did a bit of math in #5334 (comment) . looks like we should be fine.

prdoc/pr_5753.prdoc

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

paritytech-cmd-bot-polkadot-sdk · 2024-09-30T16:03:51Z

Successfully created backport PR for stable2407:

[stable2407] Backport #5753 #5873

# Description Adjust the PoV response size to the default values used in the substrate. Fixes #5503 ## Integration The changes shouldn't impact downstream projects since we are only increasing the limit. ## Review Notes You can't see it from the changes, but it affects all protocols that use the `POV_RESPONSE_SIZE` constant. - Protocol::ChunkFetchingV1 - Protocol::ChunkFetchingV2 - Protocol::CollationFetchingV1 - Protocol::CollationFetchingV2 - Protocol::PoVFetchingV1 - Protocol::AvailableDataFetchingV1 ## Increasing timeouts https://github.com/paritytech/polkadot-sdk/blob/fae15379cba0c876aa16c77e11809c83d1db8f5c/polkadot/node/network/protocol/src/request_response/mod.rs#L126-L129 I assume the current PoV request timeout is set to 1.2s to handle 5 consecutive requests during a 6s block. This setting does not relate to the PoV response size. I see no reason to change the current timeouts after adjusting the response size. However, we should consider networking speed limitations if we want to increase the maximum PoV size to 10 MB. With the number of parallel requests set to 10, validators will need the following networking speeds: - 5 MB PoV: at least 42 MB/s, ideally 50 MB/s. - 10 MB PoV: at least 84 MB/s, ideally 100 MB/s. The current required speed of 50 MB/s aligns with the 62.5 MB/s specified [in the reference hardware requirements](https://wiki.polkadot.network/docs/maintain-guides-how-to-validate-polkadot#reference-hardware). Increasing the PoV size to 10 MB may require a higher networking speed. --------- Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> (cherry picked from commit 0c9d8fe)

paritytech-cmd-bot-polkadot-sdk · 2024-09-30T16:03:56Z

Successfully created backport PR for stable2409:

[stable2409] Backport #5753 #5874

# Description Adjust the PoV response size to the default values used in the substrate. Fixes #5503 ## Integration The changes shouldn't impact downstream projects since we are only increasing the limit. ## Review Notes You can't see it from the changes, but it affects all protocols that use the `POV_RESPONSE_SIZE` constant. - Protocol::ChunkFetchingV1 - Protocol::ChunkFetchingV2 - Protocol::CollationFetchingV1 - Protocol::CollationFetchingV2 - Protocol::PoVFetchingV1 - Protocol::AvailableDataFetchingV1 ## Increasing timeouts https://github.com/paritytech/polkadot-sdk/blob/fae15379cba0c876aa16c77e11809c83d1db8f5c/polkadot/node/network/protocol/src/request_response/mod.rs#L126-L129 I assume the current PoV request timeout is set to 1.2s to handle 5 consecutive requests during a 6s block. This setting does not relate to the PoV response size. I see no reason to change the current timeouts after adjusting the response size. However, we should consider networking speed limitations if we want to increase the maximum PoV size to 10 MB. With the number of parallel requests set to 10, validators will need the following networking speeds: - 5 MB PoV: at least 42 MB/s, ideally 50 MB/s. - 10 MB PoV: at least 84 MB/s, ideally 100 MB/s. The current required speed of 50 MB/s aligns with the 62.5 MB/s specified [in the reference hardware requirements](https://wiki.polkadot.network/docs/maintain-guides-how-to-validate-polkadot#reference-hardware). Increasing the PoV size to 10 MB may require a higher networking speed. --------- Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> (cherry picked from commit 0c9d8fe)

eskimor · 2024-10-01T08:10:05Z

I assume the current PoV request timeout is set to 1.2s to handle 5 consecutive requests during a 6s block. This setting does not relate to the PoV response size. I see no reason to change the current timeouts after adjusting the response size.

This is not correct. It is set to that value because of synchronous backing. There we have a very small overall time budget. This is no longer true for asynchronous backing.

Considering that this is a hard timeout where we completely drop the response if exceeded, it might make sense to think about bumping it a bit. This would add robustness, especially with concurrent requests. @AndreiEres Given that a single fetch should take around 200ms it probably is still ok. I don't see any harm in bumping though. In any case let's test this with Gluttons on Kusama, Lot's of 10MB PoVs ... let's see what happens.

With the number of parallel requests set to 10

Those numbers should add up. If we have 10 parallel requests, than the timeout should be around 2s.

Backport #5753 into `stable2409` from AndreiEres. See the [documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md) on how to use this bot.  Co-authored-by: Andrei Eres <eresav@me.com>

Update POV_RESPONSE_SIZE

fae1537

AndreiEres force-pushed the AndreiEres/issue5503 branch from 5f02c93 to fae1537 Compare September 18, 2024 11:17

AndreiEres marked this pull request as ready for review September 18, 2024 13:58

AndreiEres changed the title ~~Update POV_RESPONSE_SIZE~~ Use maximum allowed response size for request/response protocols Sep 18, 2024

sandreim approved these changes Sep 19, 2024

View reviewed changes

Extract into a constant

754c829

AndreiEres requested a review from dmitry-markin September 19, 2024 10:19

AndreiEres added I9-optimisation An enhancement to provide better overall performance in terms of time-to-completion for a task. T8-polkadot This PR/Issue is related to/affects the Polkadot network. labels Sep 19, 2024

dmitry-markin reviewed Sep 19, 2024

View reviewed changes

dmitry-markin approved these changes Sep 19, 2024

View reviewed changes

Add prdoc

bf368e4

AndreiEres mentioned this pull request Sep 19, 2024

Increase max_pov_size to 10MB #5334

Open

Update pr doc

35149df

sandreim reviewed Sep 19, 2024

View reviewed changes

prdoc/pr_5753.prdoc Outdated Show resolved Hide resolved

Update prdoc/pr_5753.prdoc

a48876a

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

AndreiEres added this pull request to the merge queue Sep 19, 2024

Merged via the queue into master with commit 0c9d8fe Sep 19, 2024
182 of 207 checks passed

AndreiEres deleted the AndreiEres/issue5503 branch September 19, 2024 16:38

AndreiEres added the A4-needs-backport Pull request must be backported to all maintained releases. label Sep 30, 2024

paritytech-cmd-bot-polkadot-sdk bot mentioned this pull request Sep 30, 2024

[stable2407] Backport #5753 #5873

Closed

paritytech-cmd-bot-polkadot-sdk bot mentioned this pull request Sep 30, 2024

[stable2409] Backport #5753 #5874

Merged

AndreiEres had a problem deploying to subsystem-benchmarks September 30, 2024 16:23 — with GitHub Actions Failure

AndreiEres mentioned this pull request Oct 1, 2024

Bump PoV request timeout #5885

Open

s0me0ne-unkn0wn mentioned this pull request Oct 1, 2024

Do not enforce PoV size hard limit in config #5887

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use maximum allowed response size for request/response protocols #5753

Use maximum allowed response size for request/response protocols #5753

AndreiEres commented Sep 18, 2024 •

edited

Loading

sandreim left a comment

sandreim Sep 19, 2024

sandreim commented Sep 19, 2024

dmitry-markin Sep 19, 2024

AndreiEres Sep 19, 2024

sandreim commented Sep 19, 2024

paritytech-cmd-bot-polkadot-sdk bot commented Sep 30, 2024

paritytech-cmd-bot-polkadot-sdk bot commented Sep 30, 2024

eskimor commented Oct 1, 2024 •

edited

Loading

	/// This timeout is based on what seems sensible from a time budget perspective, considering 6
	/// second block time. This is going to be tough, if we have multiple forks and large PoVs, but we
	/// only have so much time.
	const POV_REQUEST_TIMEOUT_CONNECTED: Duration = Duration::from_millis(1200);

Use maximum allowed response size for request/response protocols #5753

Use maximum allowed response size for request/response protocols #5753

Conversation

AndreiEres commented Sep 18, 2024 • edited Loading

Description

Integration

Review Notes

Increasing timeouts

sandreim left a comment

Choose a reason for hiding this comment

sandreim Sep 19, 2024

Choose a reason for hiding this comment

sandreim commented Sep 19, 2024

dmitry-markin Sep 19, 2024

Choose a reason for hiding this comment

AndreiEres Sep 19, 2024

Choose a reason for hiding this comment

sandreim commented Sep 19, 2024

paritytech-cmd-bot-polkadot-sdk bot commented Sep 30, 2024

paritytech-cmd-bot-polkadot-sdk bot commented Sep 30, 2024

eskimor commented Oct 1, 2024 • edited Loading

AndreiEres commented Sep 18, 2024 •

edited

Loading

eskimor commented Oct 1, 2024 •

edited

Loading