Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Historical by timestamp #2584

Merged
merged 15 commits into from
Nov 22, 2024
Merged

Historical by timestamp #2584

merged 15 commits into from
Nov 22, 2024

Conversation

stwiname
Copy link
Collaborator

@stwiname stwiname commented Nov 7, 2024 •
edited
Loading

Description

Historical indexing currently indexes using block height, this PR provides a new option to allow using the block timestamp instead. This allows indexing multichain projects with a couple of caveats, rewinding doesnt currently work as it requires synchronisation between the indexers, this also applies to unfinalized blocks.

It will use the unix timestamp in MS instead of the block height, it retains the same column called __block_range.

Because of this change a lot of internal functions that used the block height as an argument or return value have been converted into the Header type, which has also been extended to include an optional timestamp. This means that the right value is selected only where necessary.

The query service has also been updated. It will detect the historical type and depending on the mode it will change between the blockHeight option and timestamp. These behave the same just provide different names.

Other changes

  • Drop support for cockroach db, this was no longer maintained and as far as aware not used.
  • Deprecated the disable-historical flag and replaced it with historical which is of type false | "height" | "timestamp".

TODO

  • Test rewinding and unfinalized blocks
  • Throw error when trying to rewind and multichain as its not currently supported
  • Throw error when trying to use unfinalized blocks with multichain as its currently not supported

Fixes #2493

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update Update docs with historical timestamp documentation#569

Checklist

  • I have tested locally
  • I have performed a self review of my changes
  • Updated any relevant documentation Update docs with historical timestamp documentation#569
  • Linked to any relevant issues
  • I have added tests relevant to my changes
  • Any dependent changes have been merged and published in downstream modules
  • My code is up to date with the base branch
  • I have updated relevant changelogs. We suggest using chan

@stwiname stwiname force-pushed the timestamp-based-historical branch from d3c53d4 to e22fda5 Compare November 7, 2024 02:31
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 7, 2024 •
edited
Loading

Coverage report for .

Caution

Test run failed

St.❔
 Category Percentage Covered / Total
🟡 Statements
66.13% (-3.98% 🔻)
 20108/30408
🟡 Branches
79.56% (+1.27% 🔼)
 2697/3390
🟡 Functions
68.6% (+6.1% 🔼)
 1123/1637
🟡 Lines
66.13% (-3.98% 🔻)
 20108/30408
Show new covered files 🐣
St.❔
 File Statements Branches Functions Lines
🟢
... / migrate.ts
 85.34% 16.67% 100% 85.34%
🟢
... / migrate-abis.controller.ts
 95% 87.5% 100% 95%
🟢
... / constants.ts
 100% 100% 100% 100%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / ethereum.ts
 82.28% 83.33% 100% 82.28%
🟢
... / migrate-manifest.controller.ts
 88% 73.33% 88.89% 88%
🟢
... / migrate-mapping.controller.ts
 100% 100% 100% 100%
🟢
... / migrate-controller.ts
 92.41% 84.62% 100% 92.41%
🟢
... / migrate.fixtures.ts
 100% 100% 100% 100%
🟢
... / migrate-schema.controller.ts
 97.23% 79.49% 100% 97.23%
🔴
... / types.ts
 0% 0% 0% 0%
🟢
... / createProject.fixtures.ts
 84.48% 62.5% 100% 84.48%
🟢
... / config.ts
 100% 100% 100% 100%
🟢
... / index.ts
 100% 100% 100% 100%
🔴
... / moduleLoader.ts
 56% 40% 100% 56%
🟢
... / types.ts
 100% 100% 100% 100%
🔴
... / utils.ts
 0% 0% 0% 0%
🟢
... / networkFamily.ts
 92.59% 88.89% 100% 92.59%
🟡
... / IPFSHTTPClientLite.ts
 75.74% 79.17% 88.89% 75.74%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / utils.ts
 92% 66.67% 100% 92%
🟡
... / admin.controller.ts
 77.24% 96.3% 78.57% 77.24%
🔴
... / blockRange.ts
 46.67% 100% 50% 46.67%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / SubqueryProject.ts
 84.14% 50% 71.43% 84.14%
🟢
... / core.module.ts
 100% 50% 100% 100%
🟢
... / dictionary.fixtures.ts
 100% 100% 100% 100%
🟡
... / monitor.service.ts
 69.61% 85.88% 94.12% 69.61%
🔴
... / sandbox.service.ts
 33.33% 50% 25% 33.33%
🟡
... / baseCache.service.ts
 62.12% 88.89% 66.67% 62.12%
🔴
... / baseStoreModel.service.ts
 57.41% 66.67% 80% 57.41%
🟢
... / cacheable.ts
 89.66% 75% 100% 89.66%
🟡
... / csvStore.service.ts
 77.78% 80% 80% 77.78%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / cacheMetadata.ts
 88.11% 82.61% 93.33% 88.11%
🟢
... / index.ts
 100% 100% 100% 100%
🔴
... / metadata.ts
 53.01% 71.43% 55.56% 53.01%
🟢
... / utils.ts
 100% 100% 100% 100%
🟢
... / cacheModel.ts
 86.33% 83.12% 85% 86.33%
🟢
... / index.ts
 100% 100% 100% 100%
🟡
... / model.ts
 77.36% 73.91% 81.82% 77.36%
🟢
... / utils.ts
 100% 100% 100% 100%
🟡
... / cachePoi.ts
 76.84% 89.47% 85.71% 76.84%
🟢
... / index.ts
 100% 100% 100% 100%
🔴
... / poi.ts
 41.07% 71.43% 44.44% 41.07%
🟢
... / setValueModel.ts
 85.26% 83.78% 85.71% 85.26%
🟡
... / storeCache.service.ts
 63.35% 84.38% 73.33% 63.35%
🔴
... / storeModel.service.ts
 56.79% 50% 57.14% 56.79%
🟢
... / types.ts
 100% 100% 100% 100%
🟢
... / utils.ts
 87.5% 75% 100% 87.5%
🟢
... / worker.core.module.ts
 100% 100% 0% 100%
🔴
... / worker.monitor.service.ts
 41.46% 100% 0% 41.46%
🔴
... / meta.controller.ts
 53.33% 100% 0% 53.33%
🟡
... / meta.module.ts
 60% 100% 0% 60%
🟡
... / process.ts
 61.29% 66.67% 40% 61.29%
🔴
... / foreceClean.init.ts
 47.62% 100% 0% 47.62%
🔴
... / reindex.init.ts
 33.33% 100% 0% 33.33%
🟢
... / string.ts
 100% 100% 100% 100%
🔴
... / datasourceProcessors.ts
 0% 0% 0% 0%
🔴
... / index.ts
 0% 0% 0% 0%
🔴
... / types.ts
 0% 0% 0% 0%
🔴
... / networking.ts
 31.58% 100% 0% 31.58%
Show files with reduced coverage 🔻
St.❔
 File Statements Branches Functions Lines
🟢
... / index.ts
89.09% (+6.14% 🔼)
42.86% (-23.81% 🔻)
100% (+50% 🔼)
89.09% (+6.14% 🔼)
🔴
... / delete.ts
48.72% (-5.57% 🔻)
 100%
50% (+50% 🔼)
48.72% (-5.57% 🔻)
🔴
... / promote.ts
44.19% (-4.53% 🔻)
 100%
50% (+50% 🔼)
44.19% (-4.53% 🔻)
🟡
... / add.ts
61.29% (-0.78% 🔻)
 100%
50% (+50% 🔼)
61.29% (-0.78% 🔻)
🔴
... / publish.ts
25.58% (-63.75% 🔻)
100% (+42.86% 🔼)
50% (-50% 🔻)
25.58% (-63.75% 🔻)
🔴
... / add-chain-controller.ts
39.62% (-0.45% 🔻)
 66.67% 30%
39.62% (-0.45% 🔻)
🟢
... / codegen-controller.ts
88.52% (-0.86% 🔻)
84.72% (+9.14% 🔼)
 100%
88.52% (-0.86% 🔻)
🟢
... / generate-controller.ts
97.75% (+0.1% 🔼)
93.24% (-0.79% 🔻)
 100%
97.75% (+0.1% 🔼)
🟡
... / init-controller.ts
76.7% (+6.21% 🔼)
67.65% (-0.35% 🔻)
61.11% (+7.78% 🔼)
76.7% (+6.21% 🔼)
🟢
... / project-controller.ts
80.72% (-16.54% 🔻)
50% (-10% 🔻)
75% (-25% 🔻)
80.72% (-16.54% 🔻)
🔴
... / publish-controller.ts
13.56% (-66.69% 🔻)
100% (+28.89% 🔼)
0% (-100% 🔻)
13.56% (-66.69% 🔻)
🔴
... / models.ts
50% (-49.44% 🔻)
94.12% (+1.81% 🔼)
30.77% (-69.23% 🔻)
50% (-49.44% 🔻)
🟢
... / model.ts
93.14% (-4.57% 🔻)
 100%
73.33% (+1.9% 🔼)
93.14% (-4.57% 🔻)
🔴
... / models.ts
47.06% (-52.94% 🔻)
 100%
0% (-100% 🔻)
47.06% (-52.94% 🔻)
🟡
... / load.ts
61.22% (+2.04% 🔼)
47.06% (-8.5% 🔻)
 66.67%
61.22% (+2.04% 🔼)
🟡
... / github-reader.ts
65.52% (-23.77% 🔻)
85.71% (+10.71% 🔼)
71.43% (-11.9% 🔻)
65.52% (-23.77% 🔻)
🟡
... / ipfs-reader.ts
65.08% (-20.63% 🔻)
91.67% (+28.03% 🔼)
71.43% (-11.9% 🔻)
65.08% (-20.63% 🔻)
🟡
... / local-reader.ts
67.5% (-18.55% 🔻)
100% (+37.5% 🔼)
83.33% (-16.67% 🔻)
67.5% (-18.55% 🔻)
🔴
... / base.ts
45.79% (-52.21% 🔻)
 100%
40% (-40% 🔻)
45.79% (-52.21% 🔻)
🔴
... / models.ts
26.35% (-73.65% 🔻)
 100%
0% (-100% 🔻)
26.35% (-73.65% 🔻)
🟢
... / NodeConfig.ts
88.51% (-0.1% 🔻)
79.03% (-0.6% 🔻)
85.11% (-0.94% 🔻)
88.51% (-0.1% 🔻)
🟡
... / ProjectUpgrade.service.ts
78.72% (-8.35% 🔻)
80.72% (-10.64% 🔻)
92.31% (+1% 🔼)
78.72% (-8.35% 🔻)
🟢
... / configure.module.ts
82.51% (-2.06% 🔻)
72% (+8% 🔼)
 100%
82.51% (-2.06% 🔻)
🟢
... / db.module.ts
89.06% (+0.78% 🔼)
62.5% (-20.83% 🔻)
 80%
89.06% (+0.78% 🔼)
🟢
... / SchemaMigration.service.ts
84.04% (-3.52% 🔻)
81.82% (+0.57% 🔼)
83.33% (-16.67% 🔻)
84.04% (-3.52% 🔻)
🟡
... / migration-helpers.ts
75.28% (-0.74% 🔻)
88.24% (+7.64% 🔼)
 90.91%
75.28% (-0.74% 🔻)
🟡
... / migration.ts
73.29% (-9% 🔻)
83.33% (+3.6% 🔼)
85% (-5.48% 🔻)
73.29% (-9% 🔻)
🟡
... / StoreOperations.ts
72.94% (-19.37% 🔻)
86.36% (+1.36% 🔼)
87.5% (-12.5% 🔻)
72.94% (-19.37% 🔻)
🔴
... / benchmark.service.ts
43.36% (-11.89% 🔻)
 66.67%
36.36% (-3.64% 🔻)
43.36% (-11.89% 🔻)
🔴
... / base-block-dispatcher.ts
18.33% (-29.02% 🔻)
 100%
4.35% (-5.65% 🔻)
18.33% (-29.02% 🔻)
🔴
... / block-dispatcher.ts
26.61% (-14.39% 🔻)
 66.67%
25% (-3.57% 🔻)
26.61% (-14.39% 🔻)
🔴
... / worker-block-dispatcher.ts
36.17% (-14.04% 🔻)
 100%
25% (-2.27% 🔻)
36.17% (-14.04% 🔻)
🟢
... / coreDictionary.ts
94.67% (-5.33% 🔻)
88.24% (-6.21% 🔻)
88.89% (-11.11% 🔻)
94.67% (-5.33% 🔻)
🟡
... / dictionary.service.ts
67% (-18.15% 🔻)
80% (-5.71% 🔻)
83.33% (-7.58% 🔻)
67% (-18.15% 🔻)
🟢
... / dictionaryV1.ts
88.16% (-5.06% 🔻)
74.42% (-3.84% 🔻)
 100%
88.16% (-5.06% 🔻)
🟡
... / dictionaryV2.ts
78.42% (-3.25% 🔻)
71.88% (+6.66% 🔼)
88.89% (-11.11% 🔻)
78.42% (-3.25% 🔻)
🔴
... / ds-processor.service.ts
38.12% (-23.97% 🔻)
 75%
58.33% (-29.17% 🔻)
38.12% (-23.97% 🔻)
🟡
... / dynamic-ds.service.ts
60.16% (-30.04% 🔻)
65% (-1.67% 🔻)
80% (-5.71% 🔻)
60.16% (-30.04% 🔻)
🟢
... / fetch.service.ts
92.23% (-0.06% 🔻)
87.67% (-1.06% 🔻)
88% (+1.04% 🔼)
92.23% (-0.06% 🔻)
🟢
... / inMemoryCache.service.ts
86.21% (-13.79% 🔻)
 100%
75% (-25% 🔻)
86.21% (-13.79% 🔻)
🔴
... / indexer.manager.ts
16.79% (-16.42% 🔻)
 100%
10% (-2.5% 🔻)
16.79% (-16.42% 🔻)
🟢
... / PoiBlock.ts
83.87% (-14.52% 🔻)
 96.55%
75% (-25% 🔻)
83.87% (-14.52% 🔻)
🔴
... / poi.service.ts
55.7% (+31.78% 🔼)
57.69% (-42.31% 🔻)
80% (+51.43% 🔼)
55.7% (+31.78% 🔼)
🔴
... / poiSync.service.ts
55.88% (-31% 🔻)
77.61% (+1.05% 🔼)
88.89% (-3.42% 🔻)
55.88% (-31% 🔻)
🟡
... / project.service.ts
73.54% (-6.46% 🔻)
77.46% (+7.29% 🔼)
85.19% (+1.19% 🔼)
73.54% (-6.46% 🔻)
🔴
... / sandbox.ts
42.27% (-25.71% 🔻)
62.5% (+5.36% 🔼)
28.57% (-32.97% 🔻)
42.27% (-25.71% 🔻)
🔴
... / smartBatch.service.ts
10.43% (-13.91% 🔻)
 100%
12.5% (-1.79% 🔻)
10.43% (-13.91% 🔻)
🟡
... / store.service.ts
60.52% (-5.52% 🔻)
75.41% (+6.99% 🔼)
80.65% (+23.5% 🔼)
60.52% (-5.52% 🔻)
🔴
... / entity.ts
50% (-11.54% 🔻)
 100%
50% (+50% 🔼)
50% (-11.54% 🔻)
🔴
... / store.ts
30% (-2.2% 🔻)
62.5% (-37.5% 🔻)
33.33% (+33.33% 🔼)
30% (-2.2% 🔻)
🟢
... / test.runner.ts
95.06% (-4.27% 🔻)
77.78% (+4.44% 🔼)
75% (-25% 🔻)
95.06% (-4.27% 🔻)
🔴
... / testing.service.ts
12% (-16.57% 🔻)
 100% 0%
12% (-16.57% 🔻)
🔴
... / unfinalizedBlocks.service.ts
56.21% (-27.78% 🔻)
75% (-21.72% 🔻)
92% (+0.33% 🔼)
56.21% (-27.78% 🔻)
🔴
... / worker.builder.ts
59.83% (-19.36% 🔻)
 65.38% 80%
59.83% (-19.36% 🔻)
🔴
... / worker.cache.service.ts
48.78% (-9.76% 🔻)
 100% 0%
48.78% (-9.76% 🔻)
🔴
... / worker.connectionPoolState.manager.ts
38.21% (-22.76% 🔻)
 100% 0%
38.21% (-22.76% 🔻)
🔴
... / worker.dynamic-ds.service.ts
44.44% (-17.78% 🔻)
 100% 0%
44.44% (-17.78% 🔻)
🔴
... / worker.service.ts
20.54% (-24.11% 🔻)
 100% 0%
20.54% (-24.11% 🔻)
🔴
... / worker.ts
42.08% (-0.37% 🔻)
 100% 0%
42.08% (-0.37% 🔻)
🔴
... / worker.unfinalizedBlocks.service.ts
26.79% (-73.21% 🔻)
 100%
0% (-100% 🔻)
26.79% (-73.21% 🔻)
🔴
... / logger.ts
47.89% (-16.59% 🔻)
 33.33%
40% (+20% 🔼)
47.89% (-16.59% 🔻)
🔴
... / event.listener.ts
15.15% (-65.66% 🔻)
 100%
0% (-25% 🔻)
15.15% (-65.66% 🔻)
🔴
... / health.controller.ts
37.93% (-17.24% 🔻)
 100%
0% (-50% 🔻)
37.93% (-17.24% 🔻)
🔴
... / health.service.ts
21.05% (-36.48% 🔻)
 100%
0% (-20% 🔻)
21.05% (-36.48% 🔻)
🔴
... / meta.service.ts
34.26% (-35.26% 🔻)
100% (+20% 🔼)
0% (-40% 🔻)
34.26% (-35.26% 🔻)
🔴
... / ready.controller.ts
34.78% (-21.74% 🔻)
 100%
0% (-50% 🔻)
34.78% (-21.74% 🔻)
🔴
... / ready.service.ts
37.5% (-54.17% 🔻)
 100%
0% (-66.67% 🔻)
37.5% (-54.17% 🔻)
🔴
... / forceClean.service.ts
22.08% (-4.24% 🔻)
 100% 0%
22.08% (-4.24% 🔻)
🔴
... / reindex.service.ts
18.62% (-14.71% 🔻)
 100% 0%
18.62% (-14.71% 🔻)
🔴
... / autoQueue.ts
59.49% (-28.9% 🔻)
 85.11%
70.37% (-8.8% 🔻)
59.49% (-28.9% 🔻)
🟡
... / blocks.ts
71.21% (-28.79% 🔻)
91.67% (+2.78% 🔼)
75% (-25% 🔻)
71.21% (-28.79% 🔻)
🟡
... / project.ts
64.54% (-8.84% 🔻)
57.41% (-17.17% 🔻)
 77.78%
64.54% (-8.84% 🔻)
🟢
... / reindex.ts
82.05% (+48.01% 🔼)
14.29% (-85.71% 🔻)
100% (+100% 🔼)
82.05% (+48.01% 🔼)
🔴
... / yargs.ts
40.69% (-4.8% 🔻)
 100% 14.29%
40.69% (-4.8% 🔻)
🟢
... / SubqueryProject.ts
100% (+4.48% 🔼)
66.67% (-15.69% 🔻)
100% (+14.29% 🔼)
100% (+4.48% 🔼)
🟢
... / apiPromise.connection.ts
80% (+10.92% 🔼)
61.54% (-18.46% 🔻)
66.67% (+23.81% 🔼)
80% (+10.92% 🔼)
🟡
... / block-dispatcher.service.ts
66.35% (-2.52% 🔻)
 100%
33.33% (+8.33% 🔼)
66.35% (-2.52% 🔻)
🔴
... / worker-block-dispatcher.service.ts
24.16% (-12.8% 🔻)
 100% 0%
24.16% (-12.8% 🔻)
🟢
... / substrateDictionary.service.ts
98.11% (+7.55% 🔼)
66.67% (-6.06% 🔻)
100% (+20% 🔼)
98.11% (+7.55% 🔼)
🟢
... / substrateDictionaryV1.ts
85.19% (-0.3% 🔻)
78.72% (+8.96% 🔼)
84.62% (+1.28% 🔼)
85.19% (-0.3% 🔻)
🟡
... / substrateDictionaryV2.ts
73.03% (-2.48% 🔻)
 75% 33.33%
73.03% (-2.48% 🔻)
🟢
... / fetch.module.ts
89.6% (-2.27% 🔻)
50% (-25% 🔻)
 100%
89.6% (-2.27% 🔻)
🔴
... / indexer.manager.ts
54.97% (-2.57% 🔻)
 100%
20% (+10.91% 🔼)
54.97% (-2.57% 🔻)
🟡
... / base-runtime.service.ts
72.17% (-0.15% 🔻)
76.92% (+1.92% 🔼)
77.78% (+2.78% 🔼)
72.17% (-0.15% 🔻)
🟢
... / runtimeService.ts
94.52% (+2.31% 🔼)
81.82% (-7.07% 🔻)
100% (+20% 🔼)
94.52% (+2.31% 🔼)
🟢
... / unfinalizedBlocks.service.ts
85.45% (-0.82% 🔻)
 100%
60% (+10% 🔼)
85.45% (-0.82% 🔻)
🟢
... / project.ts
88.73% (-0.16% 🔻)
 68.18% 87.5%
88.73% (-0.16% 🔻)
🟢
... / buffer.ts
90.24% (-9.76% 🔻)
91.3% (-8.7% 🔻)
90.48% (+6.27% 🔼)
90.24% (-9.76% 🔻)
🟡
... / logger.ts
69.19% (-0.41% 🔻)
48.15% (-6.85% 🔻)
78.57% (+3.57% 🔼)
69.19% (-0.41% 🔻)

Test suite run failed

Failed tests: 10/597. Failed suites: 5/107. 
  ● CLI deploy, delete, promote › Deploy to Hosted Service and Delete

    Error deploying to hosted service: The selected cluster is not supported

      77 | export function errorHandle(e: any, msg: string): Error {
      78 |   if (axios.isAxiosError(e) && e?.response?.data) {
    > 79 |     return new Error(`${msg} ${e.response.data.message ?? e.response.data}`);
         |            ^
      80 |   }
      81 |
      82 |   return new Error(`${msg} ${e.message}`);

      at errorHandle (packages/cli/src/utils/utils.ts:79:12)
      at createDeployment (packages/cli/src/controller/deploy-controller.ts:62:22)
      at Object.<anonymous> (packages/cli/src/controller/deploy-controller.test.ts:115:27)

  ● CLI deploy, delete, promote › reDeploy to Hosted Service

    Error deploying to hosted service: The selected cluster is not supported

      77 | export function errorHandle(e: any, msg: string): Error {
      78 |   if (axios.isAxiosError(e) && e?.response?.data) {
    > 79 |     return new Error(`${msg} ${e.response.data.message ?? e.response.data}`);
         |            ^
      80 |   }
      81 |
      82 |   return new Error(`${msg} ${e.message}`);

      at errorHandle (packages/cli/src/utils/utils.ts:79:12)
      at createDeployment (packages/cli/src/controller/deploy-controller.ts:62:22)
      at Object.<anonymous> (packages/cli/src/controller/deploy-controller.test.ts:159:26)


  ● Test suite failed to run

    TypeError: Class extends value undefined is not a constructor or null

      84 | }
      85 |
    > 86 | class DsPluginSandbox<P> extends Sandbox {
         |                                  ^
      87 |   constructor(option: Omit<SandboxOption, 'store'>, nodeConfig: NodeConfig) {
      88 |     super(
      89 |       option,

      at Object.<anonymous> (packages/node-core/src/indexer/ds-processor.service.ts:86:34)
      at Object.<anonymous> (packages/node-core/src/indexer/indexer.manager.ts:13:1)
      at Object.<anonymous> (packages/node-core/src/indexer/index.ts:23:1)
      at Object.<anonymous> (packages/node-core/src/db/migration-service/SchemaMigration.service.ts:8:1)
      at Object.<anonymous> (packages/node-core/src/db/migration-service/index.ts:5:1)
      at Object.<anonymous> (packages/node-core/src/db/index.ts:5:1)
      at Object.<anonymous> (packages/node-core/src/utils/graphql.ts:19:1)
      at Object.<anonymous> (packages/node-core/src/utils/index.ts:7:1)
      at Object.<anonymous> (packages/node-core/src/indexer/sandbox.ts:14:1)
      at Object.<anonymous> (packages/node-core/src/indexer/sandbox.spec.ts:6:1)


  ● Intergration test - Publish › overwrites any exisiting CID files

    [tsl] ERROR in /tmp/j9e0Gk/mocked_starter/src/types/models/Transfer.ts(61,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/j9e0Gk/mocked_starter/src/types/models/Transfer.ts(67,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createTestProject (packages/cli/src/createProject.fixtures.ts:76:5)
      at async Object.<anonymous> (packages/cli/src/commands/publish.test.ts:17:22)

  ● Intergration test - Publish › create ipfsCID file stored in local with dictiory path

    [tsl] ERROR in /tmp/j9e0Gk/mocked_starter/src/types/models/Transfer.ts(61,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/j9e0Gk/mocked_starter/src/types/models/Transfer.ts(67,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createTestProject (packages/cli/src/createProject.fixtures.ts:76:5)
      at async Object.<anonymous> (packages/cli/src/commands/publish.test.ts:17:22)

  ● Intergration test - Publish › file name consistent with manfiest file name, if -f <manifest path> is used

    [tsl] ERROR in /tmp/j9e0Gk/mocked_starter/src/types/models/Transfer.ts(61,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/j9e0Gk/mocked_starter/src/types/models/Transfer.ts(67,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createTestProject (packages/cli/src/createProject.fixtures.ts:76:5)
      at async Object.<anonymous> (packages/cli/src/commands/publish.test.ts:17:22)


  ● Cli publish › should upload appropriate project to IPFS

    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/ClaimRecord.ts(75,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/ClaimRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/ForgeRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/ForgeRecord.ts(87,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/NFT.ts(63,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/NFT.ts(69,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createMultiChainTestProject (packages/cli/src/createProject.fixtures.ts:89:5)
          at async Promise.all (index 1)
      at async Object.<anonymous> (packages/cli/src/controller/publish-controller.spec.ts:18:21)

  ● Cli publish › convert to deployment and removed descriptive field

    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/ClaimRecord.ts(75,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/ClaimRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/ForgeRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/ForgeRecord.ts(87,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/NFT.ts(63,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/NFT.ts(69,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createMultiChainTestProject (packages/cli/src/createProject.fixtures.ts:89:5)
          at async Promise.all (index 1)
      at async Object.<anonymous> (packages/cli/src/controller/publish-controller.spec.ts:18:21)

  ● Cli publish › convert js object to JSON object

    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/ClaimRecord.ts(75,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/ClaimRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/ForgeRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/ForgeRecord.ts(87,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/NFT.ts(63,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/NFT.ts(69,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createMultiChainTestProject (packages/cli/src/createProject.fixtures.ts:89:5)
          at async Promise.all (index 1)
      at async Object.<anonymous> (packages/cli/src/controller/publish-controller.spec.ts:18:21)

  ● Cli publish › Get directory CID from multi-chain project

    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/ClaimRecord.ts(75,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/ClaimRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/ForgeRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/ForgeRecord.ts(87,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/NFT.ts(63,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/cK7BFa/multi_mocked_starter/src/types/models/NFT.ts(69,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createMultiChainTestProject (packages/cli/src/createProject.fixtures.ts:89:5)
          at async Promise.all (index 1)
      at async Object.<anonymous> (packages/cli/src/controller/publish-controller.spec.ts:18:21)


  ● Promise Utils › utils.promise delay()

    expect(received).toBeLessThan(expected)

    Expected: < 1050
    Received:   1713

      10 |     const millsecDiff = new Date().getTime() - start.getTime();
      11 |     expect(millsecDiff).toBeGreaterThanOrEqual(1000);
    > 12 |     expect(millsecDiff).toBeLessThan(1050);
         |                         ^
      13 |   });
      14 |
      15 |   it('utils.promise timeout()', async () => {

      at Object.<anonymous> (packages/node-core/src/utils/promise.spec.ts:12:25)

Report generated by 🧪jest coverage report action from 5a42faa

@stwiname stwiname marked this pull request as ready for review November 12, 2024 03:47
@stwiname stwiname requested a review from yoozo November 18, 2024 22:30
@stwiname stwiname force-pushed the timestamp-based-historical branch from 864a821 to 0038ce5 Compare November 19, 2024 01:35
yoozo
yoozo reviewed Nov 19, 2024
View reviewed changes
packages/query/src/graphql/plugins/historical/PgBlockHeightPlugin.ts
} catch (e) {
/* Do nothing, default value is already set */
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If “historical” is false, nothing should be done.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was mainly to make tests work as they don't all setup metadata

packages/node-core/src/indexer/worker/worker.ts Outdated
Comment on lines 83 to 85
async function fetchBlock<R extends Header /*FetchBlockResponse*/>(height: number, specVersion: number): Promise<R> {
assert(workerService, 'Worker Not initialised');
return workerService.fetchBlock(height, {specVersion});
return (workerService as unknown as BaseWorkerService<any, R>).fetchBlock(height, {specVersion});
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

async function fetchBlock(height: number, specVersion: number): Promise<ReturnType<typeof workerService["fetchBlock"]>> {

The workerService is clear, so its return value should also be clear.

packages/node-core/src/indexer/store.service.ts

if (value && multiChain) {
throw new Error(
'Historical indexing by height is enabled and not compatible with multi-chain, to multi-chain index clear postgres schema and re-index project using --multichain'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn’t the hint here be –historical=timestamp and –multichain?

packages/node-core/src/subcommands/reindex.service.ts Outdated
Comment on lines 65 to 69
const [height, timestamp] = await Promise.all([
this.metadataRepo.find('lastProcessedHeight'),
this.metadataRepo.find('lastProcessedBlockTimestamp'),
]);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

findMany can reduce the number of IO operations.

@stwiname stwiname force-pushed the timestamp-based-historical branch from 8555aaf to a19ae83 Compare November 21, 2024 22:13
@yoozo yoozo self-requested a review November 22, 2024 03:22
@yoozo yoozo merged commit 171b43d into main Nov 22, 2024
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yoozo yoozo Awaiting requested review from yoozo

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Historical indexing support for multichain
2 participants
@stwiname @yoozo