Init migration to v4 #266

soupette · 2022-01-21T08:14:02Z

Init plugin migration to Strapi v4:

Bug fixes:

Signed-off-by: soupette <cyril@strapi.io>

strapi-cla · 2022-01-21T08:14:06Z

All committers have signed the CLA.

Signed-off-by: soupette <cyril@strapi.io>

timbrandin · 2022-01-21T14:34:33Z

I have a working solution here (https://github.com/relate-app/gatsby-source-strapi/) that solves all the checkpoints except maybe cloud - not sure what that includes. Please have a look, it uses strapi graphql rather than rest which has it's limitation, i.e. not exposing types properly which is available through the schema introspection on the graphql side of things.

Hope it helps! // T

Signed-off-by: soupette <cyril@strapi.io>

src/normalize.js

Signed-off-by: soupette <cyril@strapi.io>

Only fetch data after build

Signed-off-by: soupette <cyril@strapi.io>

Extract markdown files

Signed-off-by: soupette <cyril@strapi.io>

Refactoring

Signed-off-by: soupette <cyril@strapi.io>

Content Synce

Signed-off-by: soupette <cyril@strapi.io>

Add support for user content type and file content type

Signed-off-by: soupette <cyril@strapi.io>

Update readme

Signed-off-by: soupette <cyril@strapi.io>

Fix readme

TylerBarnes · 2022-02-11T20:03:14Z

src/gatsby-node.js

+ *
+ * See: https://www.gatsbyjs.com/docs/node-apis/
+ */
+// You can delete this file if you're not using it


Probably good to remove this line :)

TylerBarnes · 2022-02-11T20:08:45Z

src/gatsby-node.js

+  const allResults = await Promise.all(
+    endpoints.map(({ kind, ...config }) => {
+      if (kind === 'singleType') {
+        return fetchEntity(config, ctx);
+      }
+
+      return fetchEntities(config, ctx);
+    })
+  );
+
+  let newOrExistingEntries;
+
+  // Fetch only the updated data between run
+  if (lastFetched) {
+    // Add the updatedAt filter
+    const deltaEndpoints = endpoints.map((endpoint) => {
+      return {
+        ...endpoint,
+        queryParams: {
+          ...endpoint.queryParams,
+          // TODO
+          filters: {
+            updatedAt: { $gt: lastFetched },
+          },
+        },
+      };
+    });
+
+    newOrExistingEntries = await Promise.all(
+      deltaEndpoints.map(({ kind, ...config }) => {
+        if (kind === 'singleType') {
+          return fetchEntity(config, ctx);
+        }
+
+        return fetchEntities(config, ctx);
+      })
+    );
+  }
+
+  const data = newOrExistingEntries || allResults;


Does this still fetch all nodes on each run? Shouldn't there be a conditional to not fetch allResults if there's a lastFetched key?

Well we need to fetch all the results in order to know which content has been deleted.

Does allResults mean all node data in Strapi? If so, needing to fetch that to determine what was deleted means the plugin doesn't really have incremental builds support.
Unless I'm misunderstanding, this code fetches all data on cold builds then on subsequent cached builds it again fetches all data and then additionally double fetches anything that changed since the first build. Is that right or am I misunderstanding?

On subsequent cached build it uses the delta fetched data to create/update nodes. The first call is only used to know which nodes should be deleted.

So on cold build we use all the data and on then we only use the other fetches to have smaller rebuilds

Nice 👍 good to hear that 😄 Does that happen inside fetchEntities? I'm having trouble finding it

Not it happens with the filer updatedAt that is set with the lastFetched constant retrieved from the cache

Thanks 👍 I get that part but I'm not sure how allResults on line 71 doesn't refetch all data on each build. On cold builds that's what pulls in all initial nodes right? I'm not following how it doesn't re-fetch all nodes again on warm builds.

allResults are always fetched it is used to know which nodes we need to delete, however on subsequent build if we have already requested the API the data we use is the result of the delta fetches from L.85.

Is that an issue? Should I make an update?

If you check L.96 you'll see that we use the delta results if it exists otherwise the full results.

Ah, yeah the problem is in which data is fetched, not in which is used. Data fetching is the slow part (especially for large sites) so if we need to refetch all results on every build then the delta isn't really helping. Ideally there would be a way to fetch deletes as part of the delta too. Then on warm builds we would skip fetching all results and just fetch the delta.

Would it be interesting to add this behaviour only when GATSBY_IS_PREVIEW is true or should we always do it?

TylerBarnes · 2022-02-11T20:10:45Z

src/gatsby-node.js

+        ...endpoint,
+        queryParams: {
+          ...endpoint.queryParams,
+          // TODO


is this still a todo?

TylerBarnes · 2022-02-11T20:14:14Z

src/gatsby-node.js

+
+      const nodeType = makeParentNodeName(ctx.schemas, uid);
+
+      const mainEntryNode = nodes.find((n) => {


Is it possible to edit a block outside of a page in Strapi? In Contentful we create node manifests for all content nodes (even if they're not top level) because it means you can press "Open Preview" from within the edit view of a nested block.

No it's not possible, only primary nodes, the one that create pages, regarding linked nodes the manifest will also be created for them as we fetch them

Signed-off-by: soupette <cyril@strapi.io>

byronwilliams · 2022-02-15T17:09:07Z

README.md

+      - [Image optimisation](#image-optimisation)
+      - [Rich text field](#rich-text-field)
+      - [Components](#components)
+      - [Dynamic zones](#dynamic-zones)


Can you add a localizations section and accompanying documentation?

cc/ @remidej @markkaylor

Would it be okay to add this in a future release?

Because I think it will be easier once we'll have updated the Gatsby corporate starter which uses i18n

That'd be a cool starter to have! I've got alpha 4 working with my test site and it's working well except for a few localization bits. Doing those in a discrete release could be a nice milestone

May I chime in here? I'm just a regular developer watching the PR and tries it in development. While a starter is great to reduce startup time and complexity it's IMHO a nice to have while undocumented features makes the lib really hard to use until you release a new starter. I don't want to sound demanding here, you do a great job on this lib, I just wanted to give some user input if you find it helpful. 💚

Update dependencies

43d484a

Signed-off-by: soupette <cyril@strapi.io>

soupette requested a review from alexandrebodin January 21, 2022 08:18

soupette added 2 commits January 21, 2022 09:24

Init migration

35e7f6b

Signed-off-by: soupette <cyril@strapi.io>

Fix relations and file download

0ff61e2

Signed-off-by: soupette <cyril@strapi.io>

soupette force-pushed the init-migration branch from e2bc8b3 to 0ff61e2 Compare January 21, 2022 11:05

remidej mentioned this pull request Jan 21, 2022

Strapi v4 POC #253

Closed

soupette added 6 commits January 21, 2022 16:59

Fix multiple images and add support for components

73f0dd8

Signed-off-by: soupette <cyril@strapi.io>

Rename index.js to gatsby-node, clean data for dynamiczone

d760c9e

Signed-off-by: soupette <cyril@strapi.io>

Init DZ support

5373051

Signed-off-by: soupette <cyril@strapi.io>

Delete old nodes

9fb72a1

Signed-off-by: soupette <cyril@strapi.io>

Fix content digest

8682f9e

Signed-off-by: soupette <cyril@strapi.io>

Add comments

fb02bbc

Signed-off-by: soupette <cyril@strapi.io>

soupette force-pushed the init-migration branch from 1da6df0 to fb02bbc Compare January 26, 2022 08:02

rroslaniec reviewed Jan 26, 2022

View reviewed changes

src/normalize.js Show resolved Hide resolved

soupette added 9 commits January 26, 2022 10:43

Fix type naming and delete only parent nodes

d522b70

Signed-off-by: soupette <cyril@strapi.io>

Only fetch data after build

2182fac

Signed-off-by: soupette <cyril@strapi.io>

Merge pull request #267 from strapi/delta-fetch

fed9db5

Only fetch data after build

Extract images for markdown field

080fdf2

Signed-off-by: soupette <cyril@strapi.io>

Extract markdown images

874ca31

Signed-off-by: soupette <cyril@strapi.io>

Fix PR feedback

dcec77f

Signed-off-by: soupette <cyril@strapi.io>

Merge pull request #268 from strapi/extract-markdown-files

0d557a7

Extract markdown files

Refacto code

f91f7fd

Signed-off-by: soupette <cyril@strapi.io>

Merge pull request #269 from strapi/refactoring

d246601

Refactoring

remidej mentioned this pull request Feb 3, 2022

Proof of concept for Strapi-v4 and Gatsby-v4 #259

Closed

soupette added 4 commits February 7, 2022 17:04

Create manifest

e54a281

Signed-off-by: soupette <cyril@strapi.io>

Change manifestId

69fab3a

Signed-off-by: soupette <cyril@strapi.io>

Merge pull request #274 from strapi/content-sync

5d180b2

Content Synce

Update plugin config and README accordingly

82d3072

Signed-off-by: soupette <cyril@strapi.io>

soupette and others added 15 commits February 9, 2022 09:21

Add support for user content type and file content type

a1cd966

Signed-off-by: soupette <cyril@strapi.io>

Fix PR feedback

f2fb197

Signed-off-by: soupette <cyril@strapi.io>

Merge pull request #276 from strapi/fix-users-content-type

e1e6116

Add support for user content type and file content type

v2.0.0-alpha.1

bba30b1

Refacto README

780acb2

Signed-off-by: soupette <cyril@strapi.io>

Set timestamps after fetching models

e1a29e0

Signed-off-by: soupette <cyril@strapi.io>

Update readme and fix restricted field

806963f

Signed-off-by: soupette <cyril@strapi.io>

Fix PR feedback

6298b76

Signed-off-by: soupette <cyril@strapi.io>

Merge pull request #279 from strapi/update-readme

aa4f32f

Update readme

Fix typo

73cf9bd

Signed-off-by: soupette <cyril@strapi.io>

v2.0.0-alpha.2

e4db837

v2.0.0-alpha.3

4f4298e

Update readme

f897fc9

Signed-off-by: soupette <cyril@strapi.io>

Fix PR feedback

a2ab208

Signed-off-by: soupette <cyril@strapi.io>

Merge pull request #282 from strapi/fix-readme

1fde793

Fix readme

TylerBarnes reviewed Feb 11, 2022

View reviewed changes

soupette changed the base branch from v4-migration to master February 14, 2022 09:46

soupette added 2 commits February 15, 2022 10:33

Fix PR feedback

e74dafd

Signed-off-by: soupette <cyril@strapi.io>

v2.0.0-alpha.4

3195a48

soupette marked this pull request as ready for review February 15, 2022 09:35

Fix conflicts

25c0ff1

Signed-off-by: soupette <cyril@strapi.io>

byronwilliams reviewed Feb 15, 2022

View reviewed changes

markkaylor approved these changes Feb 16, 2022

View reviewed changes

remidej approved these changes Feb 16, 2022

View reviewed changes

soupette merged commit 058c23c into master Feb 16, 2022

soupette deleted the init-migration branch February 16, 2022 13:07

remidej added type: feature request source: gatsby-source-strapi labels Mar 24, 2022

remidej added this to the 2.0.0 milestone Mar 24, 2022

IanSpringer mentioned this pull request Dec 27, 2022

bug(gatsby-source-strapi): Image processing breaks when using relation field in Dynamic Zone gatsby-uc/plugins#336

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Init migration to v4 #266

Init migration to v4 #266

soupette commented Jan 21, 2022 •

edited

Loading

strapi-cla commented Jan 21, 2022 •

edited

Loading

timbrandin commented Jan 21, 2022

TylerBarnes Feb 11, 2022

TylerBarnes Feb 11, 2022

soupette Feb 14, 2022

TylerBarnes Feb 14, 2022 •

edited

Loading

soupette Feb 15, 2022

TylerBarnes Feb 15, 2022

soupette Feb 16, 2022 •

edited

Loading

TylerBarnes Feb 16, 2022

soupette Feb 17, 2022

TylerBarnes Feb 17, 2022 •

edited

Loading

soupette Feb 21, 2022

TylerBarnes Feb 11, 2022

TylerBarnes Feb 11, 2022

soupette Feb 14, 2022

byronwilliams Feb 15, 2022

soupette Feb 16, 2022

remidej Feb 16, 2022

byronwilliams Feb 16, 2022

AntonKL Feb 16, 2022


		const nodeType = makeParentNodeName(ctx.schemas, uid);

		const mainEntryNode = nodes.find((n) => {

Init migration to v4 #266

Init migration to v4 #266

Conversation

soupette commented Jan 21, 2022 • edited Loading

Bug fixes:

strapi-cla commented Jan 21, 2022 • edited Loading

timbrandin commented Jan 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TylerBarnes Feb 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

soupette Feb 16, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TylerBarnes Feb 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

soupette commented Jan 21, 2022 •

edited

Loading

strapi-cla commented Jan 21, 2022 •

edited

Loading

TylerBarnes Feb 14, 2022 •

edited

Loading

soupette Feb 16, 2022 •

edited

Loading

TylerBarnes Feb 17, 2022 •

edited

Loading