Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(core): Make filter/sort query only hold onto node properties it needs #34747

Merged
merged 46 commits into from
Feb 15, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
7f59b33
starting point
pieh Jan 20, 2022
d22f00e
Get docker running
imjoshin Jan 20, 2022
cdb008e
Add yarn commands for docker
imjoshin Jan 20, 2022
435885a
set initial mem limit, create lot of pages
pieh Jan 21, 2022
cbb7315
Set up yarn commands for serve
imjoshin Jan 21, 2022
6793007
Merge branch 'memory-test-site' of github.com:gatsbyjs/gatsby into me…
imjoshin Jan 21, 2022
7b99cc8
Get debug commands working in docker
imjoshin Jan 21, 2022
d6127b1
Add README
imjoshin Jan 21, 2022
bde2d8c
starting point
pieh Jan 20, 2022
3f8f47a
Get docker running
imjoshin Jan 20, 2022
007ae95
Add yarn commands for docker
imjoshin Jan 20, 2022
48fa848
Set up yarn commands for serve
imjoshin Jan 21, 2022
e3f83ea
set initial mem limit, create lot of pages
pieh Jan 21, 2022
c5f812f
Get debug commands working in docker
imjoshin Jan 21, 2022
bada30c
Add README
imjoshin Jan 21, 2022
1d0c662
Add docker:stats to view status easier
imjoshin Jan 21, 2022
bdb8c97
Merge branch 'memory-test-site' of github.com:gatsbyjs/gatsby into me…
imjoshin Jan 24, 2022
27f7171
Merge branch 'master' into memory-test-site
imjoshin Jan 25, 2022
f42b723
drop unused parts of site
pieh Jan 25, 2022
ae35ac0
show progress on node creation
pieh Jan 25, 2022
72ca8bc
use GATSBY_PARALLEL_QUERY_CHUNK_SIZE if provided
pieh Jan 25, 2022
0c8c2ac
don't use gatsby-dev inspired version of gatsby
pieh Jan 25, 2022
bf3a1e5
mark nodes as dirty on each build
pieh Jan 25, 2022
e037a6c
consistently use same amount of workers 3 (+1 main process)
pieh Jan 25, 2022
60d0236
add TEMPLATES env var to control which queries to run
pieh Jan 25, 2022
e5492b1
some additions to README
pieh Jan 26, 2022
d470672
drop more unused things
pieh Jan 26, 2022
56f3c93
tmp: use ids instead of full nodes for fast filters
pieh Jan 26, 2022
22d8ad0
Remove a ton of calls to getNode
imjoshin Jan 26, 2022
cd9c891
Remove a ton of calls to getNode
imjoshin Jan 26, 2022
ebf00f4
make IGatsbyNodeIdentifiers unique
pieh Jan 27, 2022
aa5d943
Move converting to actual ndoes outside of fast filters
imjoshin Jan 27, 2022
8f85c00
Get sorting working, not performant
imjoshin Jan 28, 2022
26fd255
Revert "Get sorting working, not performant"
imjoshin Jan 28, 2022
5c49293
adjust fast filters tests
pieh Jan 28, 2022
2364c00
Get sorting working without full nodes
imjoshin Feb 2, 2022
d3f8be6
Merge branch 'master' into make-fast-filter-eq-not-retain-nodes-sorting
imjoshin Feb 7, 2022
fa6d14f
Fix fast filters test
imjoshin Feb 2, 2022
9df2312
Remove debug logging
imjoshin Feb 7, 2022
7c3c9fd
Remove additional calls to getNode
imjoshin Feb 7, 2022
3bd14c8
More optimization and comments for getGatsbyNodePartial
imjoshin Feb 7, 2022
99bd831
Add types and fix indexFields check
imjoshin Feb 9, 2022
1341fb5
Make function args an object for internal functions
imjoshin Feb 9, 2022
999e9c6
Rename deref var and move partial internals to new object on partial
imjoshin Feb 10, 2022
095e814
TS fix
imjoshin Feb 10, 2022
04fb874
Merge sets of indexFields
imjoshin Feb 10, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions benchmarks/memory/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@ A shorthand for start + connect.

Stop the container used for testing.

#### yarn docker:stats

Show a polling display of the container's docker stats.

### Gatsby

These commands are used for interfacing with gatsby.
Expand Down
36 changes: 26 additions & 10 deletions packages/gatsby/src/datastore/__tests__/run-fast-filters.js
Original file line number Diff line number Diff line change
Expand Up @@ -401,13 +401,15 @@ describe(`applyFastFilters`, () => {
const result = applyFastFilters(
createDbQueriesFromObject(filter),
[typeName],
new Map()
new Map(),
[],
[]
)
expect(Array.isArray(result)).toBe(true)
expect(result.length).toEqual(2)

result.map(node => {
expect(node.slog).toEqual(`def`)
expect(getNode(node.id).slog).toEqual(`def`)
})
})

Expand All @@ -419,13 +421,15 @@ describe(`applyFastFilters`, () => {
const result = applyFastFilters(
createDbQueriesFromObject(filter),
[typeName],
new Map()
new Map(),
[],
[]
)
expect(Array.isArray(result)).toBe(true)
expect(result.length).toEqual(2)

result.map(node => {
expect(node.deep.flat.search.chain).toEqual(300)
expect(getNode(node.id).deep.flat.search.chain).toEqual(300)
})
})

Expand All @@ -438,14 +442,16 @@ describe(`applyFastFilters`, () => {
const results = applyFastFilters(
createDbQueriesFromObject(filter),
[typeName],
new Map()
new Map(),
[],
[]
)

// Count is irrelevant as long as it is non-zero and they all match filter
expect(Array.isArray(results)).toBe(true)
expect(results.length).toEqual(1)
expect(results[0].slog).toEqual(`def`)
expect(results[0].deep.flat.search.chain).toEqual(300)
expect(getNode(results[0].id).slog).toEqual(`def`)
expect(getNode(results[0].id).deep.flat.search.chain).toEqual(300)
})

it(`supports elemMatch`, () => {
Expand All @@ -458,7 +464,9 @@ describe(`applyFastFilters`, () => {
const result = applyFastFilters(
createDbQueriesFromObject(filter),
[typeName],
new Map()
new Map(),
[],
[]
)

expect(result).not.toBe(undefined)
Expand All @@ -484,7 +492,9 @@ describe(`edge cases (yay)`, () => {
const result = applyFastFilters(
createDbQueriesFromObject(filter),
[typeName],
new Map()
new Map(),
[],
[]
)

// Sanity-check
Expand All @@ -511,7 +521,13 @@ describe(`edge cases (yay)`, () => {
await getDataStore().ready()

const run = () =>
applyFastFilters(createDbQueriesFromObject(filter), [typeName], new Map())
applyFastFilters(
createDbQueriesFromObject(filter),
[typeName],
new Map(),
[],
[]
)

expect(run).toThrow(
`Invariant violation: inconsistent node counters detected`
Expand Down
Loading