fix(BlocksService): refactor api.derive for performance, and add historicApi to BlocksService #699

TarikGul · 2021-10-07T16:43:41Z

Performance bump

Benchmarks were done against /blocks where the main known regression took place. There are 3 versions we are testing here: v9.1.9, v9.2.0, and this current branch which I will denote as 'beta'.

// This benchmark reflects sidecar with an empty cache in all respects
4 threads 12 connections 2 mins
v9.1.9 -> Completed Reqs: 162, Avg Latency: 8.61s
v9.2.0 -> Completed Reqs: 24, Avg Latency: 45.08s
beta -> Completed Reqs: 55, Avg Latency: 22.75s

// The below benchmarks reflects sidecar with a filled cache
4 threads 6 connections 2 mins
v9.1.9 -> Completed Reqs: 239, Avg Latency: 1.98s
v9.2.0 -> Completed Reqs: 71, Avg Latency: 7.86s
beta -> Completed Reqs: 147, Avg Latency: 3.86s

4 threads 4 connections 2 mins
v9.1.9 -> Completed Reqs: 255, Avg Latency: 1.88s
v9.2.0 -> Completed Reqs: 107, Avg Latency: 5.94s
beta -> Completed Reqs: 211, Avg Latency: 2.28s

`Api.at()`

Local Benchmarks were also ran against instantiating api.at(). Initially for a particular runtime if the information isn't cached inside of the api it will take about 3 seconds too initialize. But after its cached it usually takes 1-50 milliseconds to recall.

All instances of api.query....at() in BlocksService are now using a historicApi to get any information, regarding api.query. During this I found a bug where we were getting the wrong transactionByteFee for historic runtimes because the api was always using the latest metadata. This is also now fixed, and all e2e tests for kusama reflect this change.

`api.derive`

I noticed api.derive was a large area of regression for sidecar while doing local isolated benchmarks, therefore I took the method apart and abstracted the things we need from it in order to speed the request time up.

extra notes

I fixed the mockApi for tests in PalletsStakingProgressService.spec.ts. Moving forward until all services are switched to using the historicApi the mockApi enviornment will be going through some adjustments. Once the final Service is fixed up, the mockApi will be readjusted to have a generated historicApi, as well as mockApi.

Part 2: #702

src/services/blocks/BlocksService.ts

emostov · 2021-10-08T23:18:20Z

src/services/blocks/BlocksService.ts

@@ -496,13 +507,13 @@ export class BlocksService extends AbstractService {
 		}

 		const multiplier =
-			await api.query.transactionPayment?.nextFeeMultiplier?.at(parentHash);
+			await historicApi.query.transactionPayment?.nextFeeMultiplier();


From what I understand we may actually need (await api.at(parentHash)).query.transactionPayment?.nextFeeMultiplier()

In order to make sure we are fetching at the paren block

Ahh yes, you are totally correct! good catch

sweet, I actually reverted this bit to its original form, as I cant instantiate another api.at, without throwing a ton of websocket errors in our tests.

src/services/blocks/BlocksService.ts

emostov · 2021-10-08T23:24:02Z

Left a few questions but will come back for a full review

All instances of api.query....at() in BlocksService are now using a historicApi to get any information, regarding api.query. During this I found a bug where we were getting the wrong transactionByteFee for historic runtimes because the api was always using the latest metadata. This is also now fixed, and all e2e tests for kusama reflect this change.

Is api.at only helpful for querys or does it also help for metadata consts and rpc calls?

TarikGul · 2021-10-08T23:28:33Z

Left a few questions but will come back for a full review

All instances of api.query....at() in BlocksService are now using a historicApi to get any information, regarding api.query. During this I found a bug where we were getting the wrong transactionByteFee for historic runtimes because the api was always using the latest metadata. This is also now fixed, and all e2e tests for kusama reflect this change.

Is api.at only helpful for querys or does it also help for metadata consts and rpc calls?

Thanks!

So yea api.at does not have an rpc methods decorated in it. This is the ApiDecoration<ApiType extends ApiType> type for the historicApi

niklasad1 · 2021-10-11T06:38:09Z

e2e-tests/endpoints/kusama/blocks/2786879.json

@@ -148,7 +148,7 @@
            "info": {
                "weight": "195000000",
                "class": "Normal",
-                "partialFee": "3333333"
+                "partialFee": "166666666"


why is this partialFee changed?

I guess because the historialApi is used or the calculate fee bug....

Yup so I found a bug for calculating fees in regards to historic blocks. I fixed it in this PR thats why I changed the e2e tests too for partialFee responses.

niklasad1

LGTM

insipx · 2021-10-11T11:42:31Z

src/services/blocks/BlocksService.ts

-			api,
-			parentHash
-		);
+		this.blockWeightStore[specVersion] ||= this.getWeight(historicApi);


TIL OR assignments are a thing

insipx

LGTM, have the same question as niklasad1 though

TarikGul added 11 commits October 6, 2021 19:31

fix: refactor api.derive.chain in /blocks

b115756

add historicApi

538e294

remove bench test

f51e81b

tests: adjust tests to accept historicalApi

00989cd

refactor blocks to use historicApi

f1ff0ef

fix: fetchevents bug

5601ac8

fix: update historic kusama blocks e2e tests

31628c2

put back --silent

12e9486

fix: lint

0f90d24

fix: tests for pallets staking progress

0196992

cleanup

dd07544

TarikGul mentioned this pull request Oct 8, 2021

fix(tests): restructure mockApi tests to integrate with historical api #702

Merged

20 tasks

TarikGul requested review from niklasad1, emostov, dvdplm, insipx, maciejhirsz and jsdw October 8, 2021 07:58

TarikGul mentioned this pull request Oct 8, 2021

Port over all api.query.<pallet>.<method>.at() -> api.at() historic api #698

Closed

10 tasks

emostov reviewed Oct 8, 2021

View reviewed changes

src/services/blocks/BlocksService.ts Outdated Show resolved Hide resolved

emostov reviewed Oct 8, 2021

View reviewed changes

src/services/blocks/BlocksService.ts Show resolved Hide resolved

emostov self-requested a review October 8, 2021 23:24

optimize getWeights, remove expandMetadata

cece658

niklasad1 reviewed Oct 11, 2021

View reviewed changes

niklasad1 approved these changes Oct 11, 2021

View reviewed changes

insipx reviewed Oct 11, 2021

View reviewed changes

insipx approved these changes Oct 11, 2021

View reviewed changes

fix: nexttFeeMultiplier get by parent hash

3e529f0

TarikGul mentioned this pull request Oct 11, 2021

StorageData:: Unable to decode storage system.events:: createType(Vec<EventRecord>) #701

Closed

TarikGul added 2 commits October 11, 2021 09:39

add ? to nextFeeMultiplier

ca48342

lint

fb8a01a

TarikGul merged commit 5861cb1 into master Oct 11, 2021

TarikGul deleted the tarik-refactor-derive branch October 11, 2021 14:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(BlocksService): refactor api.derive for performance, and add historicApi to BlocksService #699

fix(BlocksService): refactor api.derive for performance, and add historicApi to BlocksService #699

TarikGul commented Oct 7, 2021 •

edited

Loading

emostov Oct 8, 2021

emostov Oct 8, 2021

TarikGul Oct 8, 2021

TarikGul Oct 11, 2021

emostov commented Oct 8, 2021

TarikGul commented Oct 8, 2021 •

edited

Loading

niklasad1 Oct 11, 2021

TarikGul Oct 11, 2021 •

edited

Loading

niklasad1 left a comment

insipx Oct 11, 2021

insipx left a comment

fix(BlocksService): refactor api.derive for performance, and add historicApi to BlocksService #699

fix(BlocksService): refactor api.derive for performance, and add historicApi to BlocksService #699

Conversation

TarikGul commented Oct 7, 2021 • edited Loading

Performance bump

Api.at()

api.derive

extra notes

emostov Oct 8, 2021

Choose a reason for hiding this comment

emostov Oct 8, 2021

Choose a reason for hiding this comment

TarikGul Oct 8, 2021

Choose a reason for hiding this comment

TarikGul Oct 11, 2021

Choose a reason for hiding this comment

emostov commented Oct 8, 2021

TarikGul commented Oct 8, 2021 • edited Loading

niklasad1 Oct 11, 2021

Choose a reason for hiding this comment

TarikGul Oct 11, 2021 • edited Loading

Choose a reason for hiding this comment

niklasad1 left a comment

Choose a reason for hiding this comment

insipx Oct 11, 2021

Choose a reason for hiding this comment

insipx left a comment

Choose a reason for hiding this comment

TarikGul commented Oct 7, 2021 •

edited

Loading

`Api.at()`

`api.derive`

TarikGul commented Oct 8, 2021 •

edited

Loading

TarikGul Oct 11, 2021 •

edited

Loading