(WIP) Bitbucket integration #525

zanedev · 2017-08-15T03:36:40Z

Fixes #234

- Summary

Adds bitbucket api backend support at a basic level.

Listing, Saving and Deleting works.
Files are pushed to the bitbucket api through their multipart form option.
No workflow support yet as it doesn't appear to be possible with the current bitbucket api but I will file another issue for workflow support and keep looking into separately, more important to get it working at all.
If workflow is enabled in the config, the app will operate as if workflow wasn't enabled and log a warning to the console.
Needs some work with refresh access tokens, currently it is not automatic in the code or ui so you must log out after a while and log back in manually.
isContributor check during login is skipped for now unless we want to add the potential of request looping all the user's repos to find it (see comments below)
@tech4him1 has tested that netlify.com's provided auth server works. I have been using a custom auth server:

TODO to wrap up this WIP:

get refresh token working once it's available on netlify side.
double check that we handle workflow setting properly to user
implement isContributor with looping paginated requests per @Benaiah

- Test plan

Change your cms admin config.yml file to point to a bitbucket repo. Setup oauth with bitbucket and an oauth server (or possibly netlify's oauth server). see example bitbucket client site project for example cms config

- Description for the changelog

Add basic bitbucket api backend support

- A picture of a cute animal (not mandatory but encouraged)

Photo by Waranya Mooldee on Unsplash

tech4him1 · 2017-08-15T14:53:43Z

src/backends/bitbucket/implementation.js

+    return this.api.deleteFile(path, commitMessage, options);
+  }
+
+  unpublishedEntries() {


Should these functions be removed since the API does not support editorial workflow?

Ah good catch I missed those, will remove those and push again.

tech4him1 · 2017-08-15T15:02:32Z

@zanedev I'm not too familiar with the BitBucket API -- How does your implementation check if the user has permission to access the repo (like isCollaborator in the GitHub backend)?

zanedev · 2017-08-15T15:54:34Z

@tech4him1 actually I spent quite a while trying to work out isCollaborator but couldn't figure out a way to do it with the bitbucket api. I found nowhere to check scopes/permissions on a repo like the github implementation. I thought at a minimum then we could list the users projects then querying that but it's a paginated list of all projects, it would take many queries to go through and I wasn't even sure if that meant they had the correct permissions. Anyway I figure we can come back to this when we figure out or they provide how to do it.

tech4him1 · 2017-08-15T16:33:46Z

@zanedev Does this work? You still have to implement pagination, so I don't know if its feasible, but the BB API has permission checking.

  isCollaborator(user) {
    return this.request(`/repositories/${ user.username }`, {
      params: { role: "contributor" },
    }).then((r) => {
      const repos = r.values;
      let contributor = false;
      for (const repo of repos) {
        // You may need to use `${ user.username }/${ repo.name }` here, I haven't tested it yet.
        if (repo.name === this.repo) contributor = true;
      }
      return contributor;
    }).catch(/*TODO*/)
  }

https://developer.atlassian.com/bitbucket/api/2/reference/resource/repositories/%7Busername%7D

zanedev · 2017-08-15T16:59:35Z

@tech4him1 It does get a list of repos for the user but it's paginated by 10 per page, I personally have over 40, so it would need to keep requesting and paging. If you want me to add logic to to keep requesting until it reaches the end of the list I can do that. Just thought there had to be a better way to grab it directly rather than having to page it.

tech4him1 · 2017-08-15T17:07:30Z

src/backends/bitbucket/API.js

+    const uploadPromises = [];
+    const files = mediaFiles.concat(entry);
+
+    files.forEach((file) => {


@zanedev This has been discussed before, and there are obviously pros and cons to both sides, but I wonder if it would be best to upload all the media files first, then upload the entry. That way, if there is a problem with one of the media files, the entry would not be broken. But that is an opinion, some people may prefer to have the content saved first. Anyone else?

I see no problem with doing media files first. I feel like all of these backend implementations need to be refactored to have some shared code to avoid code/logic duplication however.. Maybe it's best to do it all at once for all backends so they follow the same logic and we don't further spaghetti weave it.

@tortilaman and I were actually already talking about combining the backends: https://gitter.im/netlify/NetlifyCMS?at=5989e0e91c8697534a9c1a7d, as I totally think that is an awesome idea. He was going to make an issue for it, but I'm not sure if he got around to that yet.

Yeah I think someone would have to own it across the board to do that it would be a big effort requiring ongoing maintenance and way more testing etc. Not sure if it's worth it yet. But some sort of shared interface would be a good start at least. Typescript might be a good fit for something this delicate.

tech4him1 · 2017-08-15T22:40:32Z

A basic setup is working for me!

zanedev · 2017-08-22T04:05:46Z

isCollaborator is implemented and a graceful way to handle backends with unsupported workflow. Let me know if you have other ideas about unsupported workflow, wasn't sure the best way.

tech4him1 · 2017-08-22T13:48:52Z

src/backends/bitbucket/API.js

+    return this.request("/user");
+  }
+
+  isCollaborator(user) {


isCollaborator was changed to hasWriteAccess in #543.

Ok will update and push again

It was kind of a last minute thing, thanks!

np renamed and pushed, looking at token refresh stuff now

tech4him1 · 2017-08-30T02:12:49Z

src/backends/backend.js

@@ -277,6 +278,13 @@ class Backend {
      entry.data[filterRule.get('field')] === filterRule.get('value')
    ));
  }
+
+  supportsWorkflow() {
+    if(typeof this.implementation.supportsWorkflow === "function")


I wonder if it would be better to just add this function to all the backends instead of this?

Yeah not sure just thought it was more self contained this way at the time but since there are a few backends that don't support workflow now (unfortunately) then it makes sense to add it to all of them. btw, I hope that we can implement workflow for bitbucket and gitlab sooner than later, I think we should create separate issues so they don't fall through the cracks.

Yeah, I think we need to figure out which way is best so that we can keep our PR's the same.

I prefer this approach, but it takes us halfway to what we really need, which is some sort of capabilities record for each backend, of which editorial workflow would be one. For now, I think @tech4him1's approach is ideal in its simplicity.

@zanedev The code that I had just checks to see if editorial workflow is enabled, the user doesn't have to have any other config value.

Well, it is an actual config.yml value, but it is just the one that is used to enable editorial workflow anyway (publish_mode: editorial_workflow). Do you think there is a better way @zanedev?

yep I was mistaken in my original comment.. already pushed a version that matches yours.

OK, great. I think we should move to something more like what you had down the road, though, it just seems a little complicated for integrating right now.

btw, I hope that we can implement workflow for bitbucket and gitlab sooner than later, I think we should create separate issues so they don't fall through the cracks.

@zanedev I made an issue at #568 for discussing how to do EW on these backends.

thanks i've been meaning to do that..

scher200 · 2017-10-20T18:40:13Z

@tech4him1 @tech4him1 you guys rock. Is there anything I can help with?

kristyjy · 2017-11-30T14:54:09Z

Is this close to being ready to merge in? If I can help at all I'm willing to do so. I'd like to use Netlify CMS and not have to move our repo. :)

erquhart · 2017-11-30T16:32:33Z

Hi @scher200, @kristyjy - we definitely want to get this in, it just slowed up because focus has been on prepping 1.0, which is slated for early December.

We've also discussed having a beta release line so that work like this can get out in the wild for those that want it before it's ready to hit master. But yes, by all means come pitch in! There are some TODO items in the summary up top that I'll re-paste here:

TODO to wrap up this WIP:

get refresh token working once it's available on netlify side.

double check that we handle workflow setting properly to user

implement isContributor with looping paginated requests per @Benaiah

I'm not up to speed on the refresh token issue, but it basically means the CMS stops working when your token expires, and the user has to know to log out and log back in when this occurs. I believe @biilmann had some ideas on how we might get around this, but it's been a while.

I'm not really sure what isContributor has to do with pagination, @Benaiah can you shed some light on that?

tech4him1 · 2017-11-30T18:30:31Z

@erquhart Once we get #560 merged, that would provide at least a manual token refresh, though we would probably still want to work on an automatic one.

mikestopcontinues · 2017-12-10T16:41:25Z

Hey, what's the status of this PR? Do you need any help?

scher200 · 2017-12-11T09:42:31Z

@mikestopcontinues sorry men, I stopped bothering about this, and switched git provider.

tech4him1 · 2017-12-11T16:04:53Z

@mikestopcontinues @scher200 Now that 1.0 has been released, this will be one of our primary focuses.

mikestopcontinues · 2017-12-11T16:31:28Z

@tech4him1, awesome! Thanks!

zanedev · 2017-12-12T19:57:58Z

Yeah the only major blocker here is token refresh. @mikestopcontinues If you want to use it now there is a sample auth server repo linked in the description but you'd need to modify it to work with your bitbucket settings. And the PR is a bit out of date so you'd probably need to merge in master/1.0 code to stay up to date.

maniflames · 2018-01-04T10:59:24Z

Hi, is the merge working out?
I would like to use netlify cms with bitbucket and am willing to help!

george-oakling · 2018-01-04T16:44:28Z

Hi, me too. I am also willing to help with anything :)

tech4him1 · 2018-01-04T18:21:06Z

@zanedev If you are interested in integrating it, there is working code for a manual re-auth popup in #560.

zanedev · 2018-01-05T04:22:16Z

@tech4him1 sounds cool unfortunately I'm swamped for the next month or two.. I'll try to to get some time to at least get this PR up to date with master but there will still be the refresh token issue so not sure it's worth doing anything until that is provided on the netlify side somehow..

tech4him1 · 2018-01-05T16:10:32Z

@zanedev No problem at all! #560 would be a "manual" type fix for the refresh token issue.

…s query param for full_name

verythorough · 2018-02-02T18:44:39Z

Deploy preview for netlify-cms-www ready!

Built with commit 0dd1495

https://deploy-preview-525--netlify-cms-www.netlify.com

erquhart · 2018-02-02T19:24:54Z

Update: rebased and updated for 1.x compatibility.

verythorough · 2018-02-02T19:26:34Z

Deploy preview for cms-demo ready!

Built with commit 0dd1495

https://deploy-preview-525--cms-demo.netlify.com

george-oakling · 2018-02-09T17:04:25Z

I made some updates to Bitbucket integration API and implementation (media and hasWriteAccess), where should I post my code please? I never got it how to update foreign pull-request, so if somebody can help me with that :)

tech4him1 · 2018-02-09T19:08:31Z

@george-oakling You can make a new PR, and set the base branch to bitbucket-integration instead of master.

george-oakling · 2018-02-11T10:10:52Z

#1098

woutrbe · 2018-03-18T11:45:46Z

Is there any timeline to get this released?

erquhart · 2018-03-19T17:02:37Z

@Benaiah you mentioned Bitbucket and GitLab support coming once we have the minimal required improvements in place, can you give a rough estimate?

Benaiah · 2018-03-22T23:15:37Z

Benaiah · 2018-04-17T20:54:42Z

Update

This is still underway, but it's now planned to merge the GitLab backend first and then update and merge this PR. Quoting from my comment on the GitLab PR:

I still believe the end of April is a realistic goal for merging GitLab, but the BitBucket backend hasn't received quite as much attention yet. Once GitLab and the refactoring it's based on are merged, finishing my updates to the BitBucket PR will be my top priority. If all goes well in the implementation, it shouldn't take more than a week or two to get it updated and merged once we reach that point.

pogo19 · 2018-04-17T21:34:16Z

I still want to use NetlifyCMS with Gatsby and BitBucket. I will be happy to help with some testing.

gouravsood · 2018-04-17T23:20:12Z

I am also interested in using NetlifyCMS with Bitbucket. I can be beta tester of the this PR 👍

erquhart · 2018-05-14T17:35:11Z

To all watchers: keep an eye on the current GitLab PR, as the underlying work is prerequisite to Bitbucket support.

FullstackJack · 2018-05-17T03:59:08Z

Yes, I'd also like to test this with React-Static

jaredmorgs · 2018-06-14T05:40:15Z

OMG I'm so excited to see that the GitLab support has been released. Great work team on getting this out.

Can't wait to see what's remaining now the Bitbucket side, because I would totally select NetlifyCMS over Forestry.io

I really like the editorial flow in Netlify CMS compared to Forestry.io, so I'm keen as to test it out in beta and help resolve the PR.

lukeblair · 2018-06-27T06:37:08Z

Any progress update on this? We've got a team of people that'd love to use this CMS, but bitbucket integration is a prerequisite for our use case.

erquhart · 2018-07-20T16:44:29Z

Closing in favor of #1504.

tech4him1 reviewed Aug 15, 2017

View reviewed changes

Benaiah changed the title ~~Bitbucket integration~~ (WIP) Bitbucket integration Aug 15, 2017

zanedev self-assigned this Aug 15, 2017

tech4him1 reviewed Aug 22, 2017

View reviewed changes

tech4him1 reviewed Aug 30, 2017

View reviewed changes

tech4him1 mentioned this pull request Aug 30, 2017

WIP: Gentle re-authentication modal/popup. #560

Closed

erquhart mentioned this pull request Sep 12, 2017

Allow date/datetime fields to be formatted from config #424

Closed

verythorough mentioned this pull request Oct 12, 2017

Add instructions for auth with Netlify Identity and Git Gateway #600

Closed

10 tasks

tech4him1 mentioned this pull request Dec 25, 2017

netlify bitbucket and hugo admin not working. #967

Closed

zanedev added 4 commits February 2, 2018 13:41

Added kill switch for supporting editorial workflow or not

71beffa

removed recursive repo request nonsense. it turns out we can just pas…

7085d68

…s query param for full_name

rename isCollaborator to hasWriteAccess

7652a12

changed to simpler editorial workflow check in api constructor

7b4ed27

erquhart force-pushed the bitbucket-integration branch from b413033 to 7b4ed27 Compare February 2, 2018 18:42

1.x updates, cleanup

0dd1495

erquhart force-pushed the bitbucket-integration branch from 1476eac to 0dd1495 Compare February 2, 2018 19:23

This was referenced Apr 11, 2018

Bitbucket problem decaporg/gatsby-starter-decap-cms#70

Closed

Netlify CMS 2.0 #1280

Closed

erquhart added the release: 2.0 label Apr 17, 2018

Benaiah mentioned this pull request Apr 18, 2018

(WIP) Backend restructure - Integrations #1171

Open

erquhart closed this Jul 20, 2018

erezrokah deleted the bitbucket-integration branch February 7, 2021 15:37

(WIP) Bitbucket integration #525

(WIP) Bitbucket integration #525

Conversation

zanedev commented Aug 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tech4him1 commented Aug 15, 2017

zanedev commented Aug 15, 2017

tech4him1 commented Aug 15, 2017 • edited Loading

zanedev commented Aug 15, 2017 • edited Loading

tech4him1 Aug 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tech4him1 commented Aug 15, 2017 • edited Loading

zanedev commented Aug 22, 2017 • edited Loading

tech4him1 Aug 22, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erquhart Aug 30, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tech4him1 Aug 30, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scher200 commented Oct 20, 2017

kristyjy commented Nov 30, 2017

erquhart commented Nov 30, 2017

tech4him1 commented Nov 30, 2017

mikestopcontinues commented Dec 10, 2017

scher200 commented Dec 11, 2017

tech4him1 commented Dec 11, 2017

mikestopcontinues commented Dec 11, 2017

zanedev commented Dec 12, 2017

maniflames commented Jan 4, 2018

george-oakling commented Jan 4, 2018

tech4him1 commented Jan 4, 2018

zanedev commented Jan 5, 2018

tech4him1 commented Jan 5, 2018

verythorough commented Feb 2, 2018 • edited Loading

erquhart commented Feb 2, 2018 • edited Loading

verythorough commented Feb 2, 2018

george-oakling commented Feb 9, 2018

tech4him1 commented Feb 9, 2018

george-oakling commented Feb 11, 2018

woutrbe commented Mar 18, 2018

erquhart commented Mar 19, 2018

Benaiah commented Mar 22, 2018

Before merge

Rough timeline

After merge

Benaiah commented Apr 17, 2018

Update

pogo19 commented Apr 17, 2018

gouravsood commented Apr 17, 2018

erquhart commented May 14, 2018

FullstackJack commented May 17, 2018

jaredmorgs commented Jun 14, 2018

lukeblair commented Jun 27, 2018

erquhart commented Jul 20, 2018

zanedev commented Aug 15, 2017 •

edited

Loading

tech4him1 commented Aug 15, 2017 •

edited

Loading

zanedev commented Aug 15, 2017 •

edited

Loading

tech4him1 Aug 15, 2017 •

edited

Loading

tech4him1 commented Aug 15, 2017 •

edited

Loading

zanedev commented Aug 22, 2017 •

edited

Loading

tech4him1 Aug 22, 2017 •

edited

Loading

erquhart Aug 30, 2017 •

edited

Loading

tech4him1 Aug 30, 2017 •

edited

Loading

verythorough commented Feb 2, 2018 •

edited

Loading

erquhart commented Feb 2, 2018 •

edited

Loading