File System API (Final On-Hold) See Comments #786

talves · 2017-11-09T05:37:58Z

Note: This PR is on hold, because the solution will be moved to an outside proposed mono-repo. See this repo for a working solution.

- Summary

Create an end point file system api middleware for the express server (webpack-dev-server) to be able to create a backend api in the CMS to update the example site locally.

Checkout the branch yarn and yarn start
The devServer will list two paths. The root path to the project location and the path to the site directory. These paths should be correct.

Example on windows:

Root path is D:\Git\Netlify\netlify-cms-origin\
Site path is D:\Git\Netlify\netlify-cms-origin\example

Closes #329
Closes #710
Closes #706

- Test plan

Needs a test on POSIX systems
Needs a test on MAC
Windows has been tested but another look would be good

- Description for the changelog
Features:

Able to run against the /api path at two endpoints /api/file/:id and /api/files/:path
The api will later be able to run as a stand alone middleware plugin for a local express server with no changes (written)

@Benaiah @erquhart I will explain as much as I can on this PR, so you can test quicker.

I have created a new example structure at the root of example the way Shawn likes it 😄 It has the same paths and folders as the orphan branch cms-test-backend.

Structure for deploy:

I created a config.yml file in the assets folder to use when a deploy preview kicks off from now on that will point to the orphan branch. All someone has to do on a test branch that needs a different layout is put their copy of the config in there and it will now get copied over to the root example/config.yml when the PR deploy preview is created.

API

The new backend API is called file-system in the cms. It targets the endpoints in the API middleware in the scripts/fs folder. When a webpack dev server is started, the api routes are added to the server.

Enhancements (updated: 11-8-2017)

defaults to file-system during development (should only commit changes when default changes)
copies production/config.yml and production/index.html on a production deploy (these should not change except on design changes)
copies preview/config.yml on a branch-deploy and deploy-preview (these are ok to change for your branch targets)

Notes:

I think all the backend API's should be plugins at some point for the cms, so we are not carrying around unnecessary code.

The failures on read or write do make the cms fail elegantly. Any errors will show up in the inspect of the page as 500 errors.

Hope this all makes sense and code like you are on 🔥

I will close #781 due to conflicts on my rebase of the branch.

talves · 2017-11-09T05:50:06Z

webpack.dev.js

@@ -61,6 +61,6 @@ module.exports = merge.smart(require('./webpack.base.js'), {
    historyApiFallback: true,
    disableHostCheck: true,
    headers: {"Access-Control-Allow-Origin": "*"},
-    setup: fileSystemAPIPlugin,
+    before: fileSystemAPIPlugin,


@tech4him1 good catch, I ran this for the last 4 days and did not notice the warning 👍

tech4him1 · 2017-11-09T18:47:45Z

example/assets/preview/config.yml

+backend:
+  name: github
+  repo: netlify/netlify-cms
+  branch: cms-test-backend


If we use the CMS repo on the GitHub backend for deploy previews, contributors that do not have write permissions will not be able to use our deploy previews. Right?

True. That branch is an orphan branch we setup to test the backend, so in most cases we would just have to give those access rights to contributors that need them.

Do you think this is an issue? Maybe hard to maintain?

On another note: We could always lock files in the paths for production and this one to make sure there is not an overwrite. Not sure we want that, but an idea.

@erquhart can you let me know if what Caleb is asking here is an issue, so we can address?
The solution is fine either way, just want to make sure we don't introduce any issues on deploy previews.

talves · 2017-11-11T18:41:20Z

There has been a rebase to version ~~0.7.1~~ ~~0.7.2~~ ~~0.7.3~~ 0.7.6 , so make sure when pulling this branch, you pull a clean checkout from origin.

tech4him1 · 2017-11-12T16:03:23Z

src/backends/file-system/API.js

+    const fileURL = `/file/${ path }`;
+    return this.request(fileURL, {
+      method: "DELETE",
+      params: { },


These params: { }, should be necessary.

right now there is nothing being done with them to the file-system, but maybe later in the case we want to do logging etc in next phase.

tech4him1 · 2017-11-12T16:03:52Z

src/backends/test-repo/implementation.js

@@ -26,7 +26,7 @@ function nameFromEmail(email) {
 export default class TestRepo {
  constructor(config) {
    this.config = config;
-    this.assets = [];
+    this.assets = window.repoFiles.assets || [];


Can you explain why this line is needed for this PR?

https://github.com/netlify/netlify-cms/pull/786/files#diff-1634bb1b7f4da7ca72cddcb74fef74b7R12

Allows for the loading of the assets that are truly in the assets/uploads folder. The way they would show up if you were truly hitting a real api.

jrop · 2017-11-29T21:04:37Z

I have just tested on macOS and this seems to be in working order. I for one cannot wait for this to land.

erquhart · 2017-11-29T21:08:53Z

Awesome, thanks for testing it out @jrop! We definitely want to get this in soon.

jrop · 2017-11-29T21:20:22Z

I also fired this up in my Ubuntu VM and creating a post worked fine. I don't know what kind of testing you guys are looking for, though.

talves · 2017-11-30T01:08:34Z

Thanks @jrop
I was looking to make sure paths added correctly and additions and deletions worked with the commands on those systems. I covered as much as I could to make sure the commands were compatible, so until it goes into the wild, that is the best we can do for now. I will sync it with the latest version in a while and update the rebase version up top, so we can test against the current version. 😄

Update: Rebased to version 0.7.6 , so make sure when pulling this branch, you pull a clean checkout from origin.

AstroJetson · 2017-12-07T15:33:37Z

@talves - Do you have a guess when this will land? I'm very interested in using it on a locally hosted server.

Thanks!

talves · 2017-12-07T15:39:29Z

@AstroJetson I am only a contributor, so if it is accepted @erquhart will have the details on when it will land. I believe it will get looked at after 1.0 is my best guess.

erquhart · 2017-12-07T21:26:25Z

@talves I'm pretty beat from the 1.0 rush, but I'm planning onto digging into this next week. From what I've seen it looks like a massive improvement over the in-memory repo 🎉

added a _sink example

verythorough · 2018-01-13T23:55:19Z

Deploy preview for netlify-cms-www ready!

Built with commit 70640c8

https://deploy-preview-786--netlify-cms-www.netlify.com

verythorough · 2018-01-14T00:00:18Z

Deploy preview for cms-demo ready!

Built with commit 70640c8

https://deploy-preview-786--cms-demo.netlify.com

talves · 2018-01-14T00:45:37Z

Rebased as of 01/13/2018

Important Notes:

The repository hosts both the [NetlifyCMS.org](https://www.netlifycms.org/) site and also the [cms-demo.netlify.com](https://cms-demo.netlify.com) site so there is no netlify.toml other than a temporary in this PR for testing.
The netlify.toml file must be deleted and the app console build command updated WHEN this PR goes live (this PR does break deploy preview for the website!)
This PR does require a copy of config.yml to the publish dir that will change to example from the dist folder!
There is a permanent demo branch here cms-test-backend that is targeted for the deploy-preview's backend data
Branch deploys can target their own branch if there is a need to test data within the repository. These folders have the same name (production, branch-deploy, deploy-preview) in the example/assets folder in the repository. The production config.yml and index.html file should only be changed by design, not by contributors. This is the in browser setup for cms-demo going forward.

erquhart · 2018-01-25T17:57:39Z

@talves you mentioned discussing use of this as an external backend - #1011 has landed, so it's possible. We'll be converting to Lerna and keeping "official" extensions in the repo at that point, of which this would definitely be one. I'm personally in favor of moving this (along with every backend except test-repo) into a separate package and keeping the core repo slim, but we should definitely discuss.

Thoughts?

cc/ @tech4him1 @Benaiah

talves · 2018-01-27T16:50:15Z

There was a lot of discussion on bringing the current internal backends out "into the wild" between me and @tech4him1. There are a lot of considerations and we might have thought it would be better to keep the current backends in place until we hash out a lot of these considerations.

Things to consider:

Dependencies
API methods required
Documentation
Maintenance

These are all obvious, but really should be discussed to keep the CMS focused (github)

I have the localhost already completed and will publish it soon (very soon). This alone will help us to spark conversation on if we want to keep localhost fs internal with github, but make others external. Let the discussion begin. 😁

talves · 2018-02-16T00:50:23Z

Here is the backend repository for the file-system in this PR

Checkout the cms-server branch for a stand alone server example. I plan to merge it to master. Use the master branch if it has been merged. If someone wants to set it up in a project ping me in gitter for help.

Eventually this repository will be put into the cms as a mono-repo. I will keep this PR open for discussion of that repository since it will be replacing this PR.

cc/ @tech4him1 @erquhart @Benaiah

talves · 2018-02-28T15:35:04Z

Should we Close ths PR or Leave it (mark on-hold) for reference until we have a viable mono-repo for the complete solution?

bjrn · 2018-02-28T15:38:37Z

@talves if I'd like to test the local backend for development, am I right to assume that I go to the aforementioned repo and try it from there? Will be using it with Gatsby … I guess there's no way of just setting the backend option in config.yml just yet?

talves · 2018-02-28T15:48:41Z

@bjrn yes, you can easily clone/fork my repo and run a build. You can always hit me up in Gitter if you need help setting it up. The readme instructions has most of what you need to wire it into your project.

Eventually this repo will be a mono-repo inside the CMS and will be accessible through npm, but is fully functional and tested for local development to use now if you want in a project.

erquhart · 2018-07-31T21:16:04Z

Closing as stale. @talves deleting branch as the it's backed up on your fork.

jrop · 2018-07-31T21:27:13Z

@erquhart @talves I see that this got closed. What is the state of a File-System backend going forward for 2.0? Will that likely be a community-driven module?

talves · 2018-07-31T21:44:40Z

@erquhart No need to back up. It has it's own repo(s) in my account.

@jrop Exists for v1.x. See this example repo. I will more than likely move it for v2.x when I get some time. You could always take a stab at it.

Not sure if we will keep it as a community driven backend or move it into the repo, because it needs so much extra setup on top of a project to spin up the File-System API.

cjbrooks12 · 2018-07-31T23:31:18Z

Would it make sense to maintain the CMS backend separately from the node server? The backend could then be more of a generic HTTP backend, and any service that implements the REST endpoints can then be dropped in. Separately, the node filesystem server just implements the rest interface.

cjbrooks12 · 2018-07-31T23:37:27Z

As motivation for this idea, I've been using the CMS backend portion of this for a while now in Orchid. Orchid has a development server that implements the api needed by the backend, and so local development goes through the Orchid server, rather than the node server.

talves · 2018-08-01T00:28:26Z

Would it make sense to maintain the CMS backend separately from the node server? The backend could then be more of a generic HTTP backend, and any service that implements the REST endpoints can then be dropped in. Separately, the node filesystem server just implements the rest interface.

@cjbrooks12 these endpoints are not part of the file-system backend, they just happen to be setup in a server using node in the example repo. You could use any server as long as you setup the same endpoints. Are you suggesting something different? Maybe I am misreading your suggestion.

cjbrooks12 · 2018-08-01T01:49:19Z

Right, the endpoints and their formats are just what the Node server happens to be spitting out, but if we invert how we think of this, the "filesystem backend" is just a backend that hits a REST API at a given URL, and the actual filesystem portion just happens to implement the correct API endpoints with data from the local filesystem.

But because the CMS backend is just making HTTP calls, any server that produces the same response can be used and the CMS will work just fine, which is how I've already been using it for a while. I've tossed aside the actual "filesystem" portion of your fork, the node server, and replaced it with Orchid's own development server. Orchid then re-implements (in Kotlin) the code necessary to transform its own concept of pages on disk into the JSON response that the CMS backend is expecting.

Conceivably, if the actual format that the CMS backend reads were to be formalized and documented, other tools could create ways to implement those endpoints in their own development servers. Think, a Gatsby plugin that allows you to run gatsby develop and you'd be able to point the backend at http://localhost:8000/netlify-cms and have it magically work as expected, without the need of having a dedicated gatsby-server CMS backend, and without needing to start an external service to serve the filesystem contents to Netlify CMS.

jrop · 2018-08-01T13:13:24Z

I second the idea to have a backend creation "spec" or "guide". I would potentially like to create my own backends for Netlify CMS.

talves requested review from erquhart and Benaiah November 9, 2017 05:38

talves mentioned this pull request Nov 9, 2017

File System API #781

Closed

3 tasks

talves commented Nov 9, 2017

View reviewed changes

tech4him1 reviewed Nov 9, 2017

View reviewed changes

talves force-pushed the api-file-system-final branch 2 times, most recently from 36593a3 to eae6442 Compare November 11, 2017 18:33

talves force-pushed the api-file-system-final branch 2 times, most recently from 3f61e96 to 45d26e5 Compare November 12, 2017 15:34

tech4him1 reviewed Nov 12, 2017

View reviewed changes

talves force-pushed the api-file-system-final branch 2 times, most recently from 8ab4b03 to 0ed46ba Compare November 15, 2017 22:36

talves mentioned this pull request Nov 17, 2017

WIP: Take CMS offline #827

Closed

talves force-pushed the api-file-system-final branch from 0ed46ba to f4c2f20 Compare November 30, 2017 01:40

talves added 8 commits January 13, 2018 10:44

structure for example site

cc86aad

revert to master yarn.lock

8592876

client side backend api for file system

208d664

middleware plugin to weback devServer

f26c195

deployment based on deploy context

e8c85b4

deprecated setup to before

c646853

cleanup some unused assets in example

9f3e2bb

added a _sink example

catch up yarn.lock

cd1a989

PR so deploy preview commit

bb40158

talves added 2 commits January 13, 2018 16:13

publish from the example folder

fc74a13

deploy preview also

70640c8

cjbrooks12 mentioned this pull request Jan 19, 2018

Update orchid.md netlify/staticgen#326

Merged

talves mentioned this pull request Jan 20, 2018

File-system backend fixes and support JWT #1029

Closed

This was referenced Feb 6, 2018

File Collections behave strange / don't work locally #1082

Closed

Expanding local development options #1085

Closed

talves changed the title ~~File System API (Final)~~ File System API (Final On-Hold) See Comments Feb 28, 2018

erquhart closed this Jul 31, 2018

erquhart deleted the api-file-system-final branch July 31, 2018 21:16

tech4him1 added kind: discussion status: stale labels Aug 7, 2018

ramkumarkb mentioned this pull request Feb 2, 2024

Document using database-driven backends decaporg/decap-website#4

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

File System API (Final On-Hold) See Comments #786

File System API (Final On-Hold) See Comments #786

talves commented Nov 9, 2017 •

edited

Loading

talves Nov 9, 2017

tech4him1 Nov 9, 2017

talves Nov 10, 2017

talves Nov 11, 2017

talves commented Nov 11, 2017 •

edited

Loading

tech4him1 Nov 12, 2017

talves Nov 13, 2017

tech4him1 Nov 12, 2017 •

edited

Loading

talves Nov 13, 2017 •

edited

Loading

tech4him1 Nov 13, 2017

jrop commented Nov 29, 2017

erquhart commented Nov 29, 2017

jrop commented Nov 29, 2017

talves commented Nov 30, 2017 •

edited

Loading

AstroJetson commented Dec 7, 2017

talves commented Dec 7, 2017

erquhart commented Dec 7, 2017 •

edited

Loading

verythorough commented Jan 13, 2018 •

edited

Loading

verythorough commented Jan 14, 2018 •

edited

Loading

talves commented Jan 14, 2018 •

edited

Loading

erquhart commented Jan 25, 2018 •

edited

Loading

talves commented Jan 27, 2018

talves commented Feb 16, 2018 •

edited

Loading

talves commented Feb 28, 2018

bjrn commented Feb 28, 2018

talves commented Feb 28, 2018

erquhart commented Jul 31, 2018

jrop commented Jul 31, 2018

talves commented Jul 31, 2018

cjbrooks12 commented Jul 31, 2018

cjbrooks12 commented Jul 31, 2018 •

edited

Loading

talves commented Aug 1, 2018

cjbrooks12 commented Aug 1, 2018

jrop commented Aug 1, 2018

File System API (Final On-Hold) See Comments #786

File System API (Final On-Hold) See Comments #786

Conversation

talves commented Nov 9, 2017 • edited Loading

Structure for deploy:

API

Enhancements (updated: 11-8-2017)

Notes:

talves Nov 9, 2017

Choose a reason for hiding this comment

tech4him1 Nov 9, 2017

Choose a reason for hiding this comment

talves Nov 10, 2017

Choose a reason for hiding this comment

talves Nov 11, 2017

Choose a reason for hiding this comment

talves commented Nov 11, 2017 • edited Loading

tech4him1 Nov 12, 2017

Choose a reason for hiding this comment

talves Nov 13, 2017

Choose a reason for hiding this comment

tech4him1 Nov 12, 2017 • edited Loading

Choose a reason for hiding this comment

talves Nov 13, 2017 • edited Loading

Choose a reason for hiding this comment

tech4him1 Nov 13, 2017

Choose a reason for hiding this comment

jrop commented Nov 29, 2017

erquhart commented Nov 29, 2017

jrop commented Nov 29, 2017

talves commented Nov 30, 2017 • edited Loading

AstroJetson commented Dec 7, 2017

talves commented Dec 7, 2017

erquhart commented Dec 7, 2017 • edited Loading

verythorough commented Jan 13, 2018 • edited Loading

verythorough commented Jan 14, 2018 • edited Loading

talves commented Jan 14, 2018 • edited Loading

erquhart commented Jan 25, 2018 • edited Loading

talves commented Jan 27, 2018

Things to consider:

talves commented Feb 16, 2018 • edited Loading

talves commented Feb 28, 2018

bjrn commented Feb 28, 2018

talves commented Feb 28, 2018

erquhart commented Jul 31, 2018

jrop commented Jul 31, 2018

talves commented Jul 31, 2018

cjbrooks12 commented Jul 31, 2018

cjbrooks12 commented Jul 31, 2018 • edited Loading

talves commented Aug 1, 2018

cjbrooks12 commented Aug 1, 2018

jrop commented Aug 1, 2018

talves commented Nov 9, 2017 •

edited

Loading

talves commented Nov 11, 2017 •

edited

Loading

tech4him1 Nov 12, 2017 •

edited

Loading

talves Nov 13, 2017 •

edited

Loading

talves commented Nov 30, 2017 •

edited

Loading

erquhart commented Dec 7, 2017 •

edited

Loading

verythorough commented Jan 13, 2018 •

edited

Loading

verythorough commented Jan 14, 2018 •

edited

Loading

talves commented Jan 14, 2018 •

edited

Loading

erquhart commented Jan 25, 2018 •

edited

Loading

talves commented Feb 16, 2018 •

edited

Loading

cjbrooks12 commented Jul 31, 2018 •

edited

Loading