Netlify CMS on Bitbucket

[InderdeepBajwa]

While trying to use Netlify CMS on a website that uses Bitbucket to host it's coding, I received the following error while deploying. Chris suggested me to open an issue in this repo that will even help to develop this CMS. So, here's the first issue I faced while deploying the CMS:

CMD - Hugo Server
Started building sites ...
ERROR: 2017/02/10 14:52:37 shortcode.go:288: Unable to locate template for shortcode 's' in page extending
ERROR: 2017/02/10 14:52:37 shortcode.go:288: Unable to locate template for shortcode 's' in page index

Which is basically same as on Netlify's Deployment logs but with more specified error.
2:36:24 PM: ERROR: 2017/02/10 09:06:24 shortcode.go:288: Unable to locate template for shortcode 's' in page extending
2:36:24 PM: ERROR: 2017/02/10 09:06:24 shortcode.go:288: Unable to locate template for shortcode 's' in page index
2:36:24 PM: ERROR: 2017/02/10 09:06:24 general.go:212: Error while rendering page admin/example/index.html: html/template:__admin/example/index.html: ends in a non-text context: {stateJSRegexp delimNone urlPartNone jsCtxRegexp attrNone elementScript }

What can be done for this error?
Just as prior information, I installed Netlify CMS yesterday but the timestamp on local files is 2017/02/10. Moreover, deleting the /admin folder does removes this error.

[Benaiah]

Hugo is trying to build the files in the admin folder - obviously this is going to cause problems since those files aren't designed to be fed into a site generator. I doubt there's anything that could be done to fix it in Netlify CMS - you'd need to either make hugo ignore the contents of the admin directory or add it to the site after it's been built.

[tech4him1]

Just a status update: As of 2017, BitBucket currently has no public API to modify files or create commits in a repository, although they said they were working on it as of 2014 (it seems to have stalled out).

API Reference: https://developer.atlassian.com/bitbucket/api/2/reference/
Feature Request for Commits: https://bitbucket.org/site/master/issues/8672/is-there-any-api-to-make-a-commit-bb-9954

[estebanvega]

Bitbucket marked the PR as resolved on the 2017-06-12 and presented a new endpoint for commits. Not sure though if it's enough to make it work with Netlify CMS.

[erquhart]

Great news!

[InderdeepBajwa]

@tech4him1 Bitbucket actually has a public API kind of thing that I am using to modify files or create commits. Also, you can checkout Forestry.io CMS for Hugo and Jekyll that have already been doing this.

[tech4him1]

@InderdeepBajwa Do you happen to have a link to docs or an example?

[zanedev]

@netlify team is bitbucket integration coming? how difficult would it be to add it since they appear to have the commit bitbucket api now?

[erquhart]

@zanedev it's not in active development, but we'd love to see it happen! If you're up for taking a crack at it let us know!

[zanedev]

@erquhart actually I have an immediate need for it now so I'd be happy to kick it off. I'll join the planning meeting tomorrow to hash it out

[tech4him1]

@zanedev It might be worthwhile to read through the issure we have for GitLab -- #57, as we may need some of the same problems/workarounds.

[zanedev]

thanks @tech4him1 will check it out.

[erquhart]

@zanedev that's great news! A good number of folks have been interested in this, so it'll be good to see it get some traction.

[zanedev]

Quick status update below. I will push up the cms code to my branch when it's a bit more solid on the editing experience.

• Successfully configured a custom oauth provider and authenticated netlify cms via bitbucket. To get moving faster I based it on this github custom implementation. I created a fork and made some minor modifications to work with bitbucket and have it hosted with zeit/now to verify it works but there are not a lot of changes from the original so I should look at merging it back to the main branch at some point, need to confer with @vencax on his thoughts.
• Someone will need to add the bitbucket oauth consumer info to the netlify website dashboard/UI before this goes live if we are going to provide a netlify supplied bitbucket oauth endpoint like the default github implementation currently in netlify cms.
• Duplicated and modified the github provider implementation in netlify cms src and managed to pull the posts/files through the bitbucket api and show in the editor UI. 🎉
• Now working on wrapping up commits and other editor details supported by the bitbucket api.

Any thoughts/feedback on this approach?

[erquhart]

@zanedev your approach sounds right on - really looking forward to seeing this functionality in place!

@calavera let me know your thoughts on Zane's point about providing "freebie" auth for Bitbucket users, as we do for GitHub.

[calavera]

let me know your thoughts on Zane's point about providing "freebie" auth for Bitbucket users, as we do for GitHub.

We can do that.

[erquhart]

@zanedev looks like this is already in place on Netlify for anyone who uses it.

[zanedev]

@erquhart ok good to know, I'll try to use it instead of the custom auth after I finish the editor api integration.

[zanedev]

wrong button not done yet :)

[zanedev]

I've put some time into pushing a commit and it appears the preferred way is through a form post multipart data.

This isn't ideal as github blobs, for obvious reasons but most importantly there might be some limitations like chunking up the post size for large files. Also I am not sure if it supports the base64 encoding yet. If anybody has a better idea for commits/pushing to the bitbucket api let me know...

So far I've spent some time crafting my multipart form but no dice with the api. I'll keep trying and report back.

[erquhart]

Hmm...looks like that's our only option. Guess we'll find out how well it works!

[tech4him1]

Would it be worth uploading each media file as a separate commit, so that we wouldn't have a giant request if someone had lots of media files in a single post? Each media file would be uploaded/committed first, then, if they all were successful, the main file could be committed. The only inconsistent state you would risk would be having extra media files uploaded, which in most situations should not affect anything.

[zanedev]

@tech4him1 good suggestion I believe that's how it works now but will double check..

[zanedev]

Update:

• I was able to commit/push normal text md files through netlify cms ui to the bitbucket api using the post multipart method with the "form-data" npm package so that's a step forward yay!
• The bitbucket api does not appear to support base64 encoding which is kind of a show stopper for transferring files over http. Let me know your thoughts on that... Whenever I send base64 encoded files it just commits the encoded file instead of decoding it first. And the CORS pre-flight doesn't allow the "Content-Transfer-Encoding": "base64" header so I can't tell the server that it is base64.
• Unfortunately binary files/blobs like images are not saving properly in bitbucket. I've spent a lot of time trying different formats and headers etc. The bitbucket api accepts the commit but the image is corrupted or the wrong format. I've tried sending as binary text, base64, raw, you name it nothing is working. It's most likely some combination of the above and a header combination I'm missing.

So at this point I am going to post for some help in the bitbucket forums on these issues. I'm also going to keep trying to push image files there has to be a way. I'll clean up my branch and push it soon up in case anybody else wants to take a stab at it.

[erquhart]

Thanks for killing it on this @zanedev! Let's see what the Bitbucket community comes back with.

[zanedev]

Ok bitbucket question posted here hopefully someone helps it feels like a dead end in their forums. I'll keep hacking on it and post my branch soon regardless. Determined to get to the final goal (eventually)..

[Benaiah]

@zanedev thanks so much! It's great to have some movement on this issue.

[taussoe]

@zanedev Thanks for looking into bitbucket integration! Very much appreciated. I will check out your repo and take it for a spin :)

[taussoe]

@zanedev Just tested out a few different things - seems to work pretty well. I might have resolved the image problem. First time opening the netlify cms code so i'm not sure if there's smarter ways of doing it. Just sent a pull request :)

[zanedev]

@taussoe that's great news thanks for doing that I'll try it out later tonight!

[zanedev]

@taussoe thanks again that was the trick! I merged it into my branch and I'll continue on to the next road bump now that we are over that one.

[erquhart]

@taussoe @zanedev nice!!

[taussoe]

@zanedev Yay! Glad i could help :)

[zanedev]

Okay we are in good shape! I'm doing some cleanup and writing some tests and it should be good enough to use without workflow. I'll push it up and do a PR when it's ready soon. I'll also take a look at the PRs for gitlab integration and github unit tests to make sure I'm not missing anything.

[zanedev]

I removed all editorial workflow support for now and all simple paths are working except delete file. I can't find anywhere in the api documentation for deleting a file, the src endpoint only accepts get and post. Maybe someone can help me find how to delete? I put a comment here hoping to get an answer soon.

[zanedev]

Ah thanks to @tech4him I now see the API docs say to send empty file will try that out thanks.

From the API docs:
However, when the files field contains a file path that does not have a corresponding, identically-named form field, then Bitbucket interprets that as the client wanting to replace the named file with the null set and the file is deleted instead.

[tech4him1]

@zanedev Wow, I couldn't even find that in the docs, it was just a guess! Glad it worked.

[zanedev]

@tech4him1 yeah the joys of rest api's they all seem to do whatever they feel like :)

Okay delete is now working, writing some tests and will do my PR soon.

[pogo19]

Hey, any chance for an update and possible target date for Bitbucket support?

[erquhart]

cc/ @Benaiah

[Benaiah]

Hey, just wanted to leave an update here for anyone following this issue - this is being actively worked on, in concert with the GitLab backend. Expect to see more updates on both of these PRs as that work progresses - notably, I'll be wrapping up a large design document describing plans and options for the new API shortly.

[jaredmorgs]

For my platform shortlisting process, Netlify CMS is just behind forestry.io at the moment because of the GitHub only support. I would be so delighted if the GitLab and BitBucket support was implemented soon.

I have to say that the workflow management features in Netlify CMS has me gunning for the other VCS support. It's really beautiful.

[tiltos]

Any update on this one?

[erquhart]

Very very very soon. Like this week soon.

[erquhart]

I will never ever ever ever ever promise a date again.

Here's the PR to track: #1504