fix: replace github search api with graphql in success lifecycle method #857

babblebey · 2024-06-13T19:51:09Z

This Pull Request refactors the success lifecycle function replacing the usage of Github "Search API" with "GraphQL API".

Changes Made

Added buildAssociatedPRsQuery helper function to which builds GraphQL query for fetching associated PRs to a list of commit hash (sha). It does this by collecting a list of shas which maps down to a list of strings that leverages the GraphQL fragment enabling ability to request data of multiple commits in one request using GitObject.

/**
 * Builds GraphQL query for fetching associated PRs to a list of commit hash (sha)
 * @param {Array<string>} shas
 * @returns {string}
 */
export function buildAssociatedPRsQuery(shas) {
  return `#graphql
    query getAssociatedPRs($owner: String!, $repo: String!) {
      repository(owner: $owner, name: $repo) {
        ${shas
          .map((sha) => {
            return `commit${sha.slice(0, 6)}: object(oid: "${sha}") {
            ...on Commit {
              associatedPullRequests(first: 100) {
                nodes {
                  url
                  number
                  body
                }
              }
            }
          }`;
          })
          .join("")}
      }
    }
  `;
}

Replaced the instance of the search api consumption with the getSearchQueries method integration with the graphql api integration, which was used to retrieve all associated pull requests.

const { repository } = await octokit.graphql(
  buildAssociatedPRsQuery(shas),
  { owner, repo },
);
const associatedPRs = Object.values(repository).map(
  (item) => item.associatedPullRequests.nodes,
);

Modified test suites in acknowledgement of the changes..

Related Issue

Fixes #644

Screencast/Screenshot

screencast-bpconcjcammlapcogcnnelfmaeghhagj-2024.06.24-14_54_49.webm

lib/success.js

babblebey

I'm pretty sure this works, but can't exactly test, hence I've got to figure how to update the test for a mock .... This is where i'm kinda BLOCKED..

See the comments associated to this review first.... then hear my BLOCKER 😉

I notice the convention with the usage of the fetch-mock's sandbox().getOnce() method which mocks a request and provides a response to the immediate request provided a mock endpoint like so...

fetchMock
    .sandbox()
    .getOnce(`https://api.github.local/repos/${owner}/${repo}`, {
      full_name: `${redirectedOwner}/${redirectedRepo}`,
    }); // getOnce("<mockRequestEnpoint>", "<mockResponse>")

but my question is...

HOW EXACTLY DO I MOCK THE GRAPHQL ENDPOINT??? using similar convention 🤔

I'll need the exact appropriate mock GraphQL endpoint????

I'm not sleeping on it too 😉

UPDATE:

I found https://github.com/octokit/graphql.js?tab=readme-ov-file#writing-tests 🤔, requires some new dependencies 😞 I think now... If there's a way to do this without adding any new dependency, I'll take that

gr2m · 2024-06-14T04:23:21Z

requires some new dependencies

you don't need the @octokit/graphql dependency to mock graphql requests, you just mock the POST /graphql request like any other.

lib/success.js

Co-authored-by: Gregor Martynus <39992+gr2m@users.noreply.github.com>

babblebey · 2024-06-14T06:55:00Z

you don't need the @octokit/graphql dependency to mock graphql requests, you just mock the POST /graphql request like any other.

Oh wow, I'll give that a shot... I already done too much if this is the case in my initial attempt 🫣

gr2m · 2024-06-14T16:01:45Z

Holler if you need help with this PR

… release commits and issues solved by PR/commits comments`

…release commits and issues closed by PR/commits comments with custom URL`

… returned by search (compare sha and merge_commit_sha)`

…ciated with release commits`

…m other repo`

…ks at the bottom`

…ks at the top`

…ks with no additional releases (top)`

…ecessary"

…ecate?

gr2m

Yoooo!! Awesome PR, can't wait to give this a go! I've to run now but might get to it later tonight! Great work 💐

lib/get-search-queries.js

gr2m · 2024-06-15T03:58:11Z

I ran it locally and got the following error

[8:46:26 PM] [semantic-release] › ℹ  Start step "success" of plugin "@semantic-release/github"
[8:46:27 PM] [semantic-release] › ✘  Failed step "success" of plugin "@semantic-release/github"
[8:46:27 PM] [semantic-release] › ✘  An error occurred while running semantic-release: Error: Parse error on "commit338966" (STRING) at [4, 9]
    at file:///Users/gr2m/code/semantic-release/semantic-release/node_modules/aggregate-error/index.js:23:26
    at Array.map (<anonymous>)
    at new AggregateError (file:///Users/gr2m/code/semantic-release/semantic-release/node_modules/aggregate-error/index.js:16:19)
    at file:///Users/gr2m/code/semantic-release/semantic-release/lib/plugins/pipeline.js:55:13
    at async pluginsConfigAccumulator.<computed> [as success] (file:///Users/gr2m/code/semantic-release/semantic-release/lib/plugins/index.js:87:11)
    at async run (file:///Users/gr2m/code/semantic-release/semantic-release/index.js:218:3)
    at async Module.default (file:///Users/gr2m/code/semantic-release/semantic-release/index.js:278:22)
    at async default (file:///Users/gr2m/code/semantic-release/semantic-release/cli.js:55:5) {
  locations: [ { line: 4, column: 9 } ],
  pluginName: '@semantic-release/github'
}
AggregateError: 
    Error: Parse error on "commit338966" (STRING) at [4, 9]
        at Array.map (<anonymous>)
        at file:///Users/gr2m/code/semantic-release/semantic-release/lib/plugins/pipeline.js:55:13
        at async pluginsConfigAccumulator.<computed> [as success] (file:///Users/gr2m/code/semantic-release/semantic-release/lib/plugins/index.js:87:11)
        at async run (file:///Users/gr2m/code/semantic-release/semantic-release/index.js:218:3)
        at async Module.default (file:///Users/gr2m/code/semantic-release/semantic-release/index.js:278:22)
        at async default (file:///Users/gr2m/code/semantic-release/semantic-release/cli.js:55:5)
    at file:///Users/gr2m/code/semantic-release/semantic-release/lib/plugins/pipeline.js:55:13
    at async pluginsConfigAccumulator.<computed> [as success] (file:///Users/gr2m/code/semantic-release/semantic-release/lib/plugins/index.js:87:11)
    at async run (file:///Users/gr2m/code/semantic-release/semantic-release/index.js:218:3)
    at async Module.default (file:///Users/gr2m/code/semantic-release/semantic-release/index.js:278:22)
    at async default (file:///Users/gr2m/code/semantic-release/semantic-release/cli.js:55:5) {
  errors: [
    Error: Parse error on "commit338966" (STRING) at [4, 9]
        at file:///Users/gr2m/code/semantic-release/semantic-release/node_modules/aggregate-error/index.js:23:26
        at Array.map (<anonymous>)
        at new AggregateError (file:///Users/gr2m/code/semantic-release/semantic-release/node_modules/aggregate-error/index.js:16:19)
        at file:///Users/gr2m/code/semantic-release/semantic-release/lib/plugins/pipeline.js:55:13
        at async pluginsConfigAccumulator.<computed> [as success] (file:///Users/gr2m/code/semantic-release/semantic-release/lib/plugins/index.js:87:11)
        at async run (file:///Users/gr2m/code/semantic-release/semantic-release/index.js:218:3)
        at async Module.default (file:///Users/gr2m/code/semantic-release/semantic-release/index.js:278:22)
        at async default (file:///Users/gr2m/code/semantic-release/semantic-release/cli.js:55:5) {
      locations: [Array],
      pluginName: '@semantic-release/github'
    }
  ]

In order to run it locally, checkout the semantic-release/semantic-release repo and the semantic-release/github repo. Run npm install in both. Then in the semantic-release folder, run npm install <relative path to the github folder> (e.g. in my case on Mac OS, the folders are next to each other, I ran npm install ../github.

The way I ran it locally is as follows

Create a test repository with a package.json, set "version": "0.0.0-development"
check it out locally
Add a commit with feat: initial version or similar (git commit --allow-empty -m 'feat: initial version')
Make sure you are signed out of npm locally (run npm logout)
Create a classic automation token on npm (https://www.npmjs.com/settings//tokens/`)
Create a personal access token with the repo scope

Set the following environment variables

CI=true
GITHUB_ACTIONS=true
GITHUB_REF=master
GITHUB_SHA=<latest commit sha>
GITHUB_RUN_ID=1
GITHUB_REPOSITORY=<your full repository name (owner/repo)>
GITHUB_WORKSPACE=<path to current folder>
NPM_TOKEN=<your npm token>
GITHUB_TOKEN=<your github token>

Run the following command in your local clone of the test repository
```
<path to semantic-release folder>/bin/semantic-release.js
```

Not sure if this could be streamlined ... but we should document it somewhere for future reference and improve if we can 😁

gr2m · 2024-06-15T04:07:20Z

It's possible that this error is an edge case that only happens because I ran it locally. The commit only exist on my machine due to the local testing. When your query is sent, the commit does not exist and probably returns null or even an error. If that happens, I'd simply ignore the commit and move on to the next one

babblebey · 2024-06-21T14:38:26Z

It's possible that this error is an edge case that only happens because I ran it locally. The commit only exist on my machine due to the local testing. When your query is sent, the commit does not exist and probably returns null or even an error. If that happens, I'd simply ignore the commit and move on to the next one

@gr2m, I figured it was a GraphQL error... turned out I had a some part of the query returned string (from the buildAssociatedPRsQuery) wrongly typed... i.e. the alias text and the oid.

I have recorded and added a screencast to description above.

gr2m · 2024-06-21T17:52:31Z

Great work! I'd like to see that the success step actually posts a comment, we can pair on it during our checkin today

babblebey · 2024-06-24T14:07:38Z

Hi @gr2m, turns out the reason why the comment wasn't posting to only merged PRs and not the associated closed/fixed issue is because I failed to retrieve the body property on the graphql fetched associatedPRs, hence no PR body to parse the associated closed/fixed issueNumber from.

Kindly checkout description for updated Demo Video, also PRs and Issue below...

🤦

gr2m

Let's goo 🚀

gr2m · 2024-07-01T17:12:00Z

lib/success.js

+    query getAssociatedPRs($owner: String!, $repo: String!) {
+      repository(owner: $owner, name: $repo) {
+        ${shas
+          .map((sha) => {
+            return `commit${sha.slice(0, 6)}: object(oid: "${sha}") {
+            ...on Commit {
+              associatedPullRequests(first: 100) {
+                nodes {
+                  url
+                  number
+                  body
+                }
+              }
+            }
+          }`;
+          })
+          .join("")}
+      }
+    }


We will need to address the possibility that there might be more than 100 commits. If that occurs, we need to send multiple requests as only 100 nodes can be requested with a single GraphQL request

was this addressed in this PR? im working with a repo that has more than 100 commits and am experiencing this secondary rate error on v24.0.0

Hey @brandon-kyle-bailey

The Edge Case isn't addressed in this PR nor yet.

But, could you kindly open an issue with details of this error you're getting in regard this 100+ commit, thanks.

unfortunately this is happening on an internal repo but ill do my best! @babblebey

@babblebey im not able to reliable reproduce it at this point since re-running so will hold off on logging an issue. will revisit if it reappears!

I'm currently working on a fix in regard the edge case here #892

just capturing the relation to #867 (comment) as well

gr2m · 2024-07-01T17:22:34Z

How fitting that the release of this fix also ran into the secondary rate limit issue:
https://github.com/semantic-release/github/actions/runs/9748385825/job/26903265854
(but that was still using the old code though)

babblebey · 2024-07-01T17:42:04Z

(but that was still using the old code though)

I sigh in relief**** 😮‍💨

…ease#857)

babblebey added 3 commits June 13, 2024 19:53

chore: add loadAssociatedPRsQuery helper to generate gql query

8929f7e

feat: implement graphql in associatedPRs fetch

ce1c3e4

Merge branch 'master' into refactor/replace-search-api

d24079c

babblebey commented Jun 13, 2024

View reviewed changes

lib/success.js Outdated Show resolved Hide resolved

babblebey commented Jun 13, 2024

View reviewed changes

lib/success.js Outdated Show resolved Hide resolved

babblebey commented Jun 13, 2024

View reviewed changes

babblebey added 3 commits June 13, 2024 22:10

build: add assert to devDeps

76a2cd1

do: extracted loadAssociatedPRsQuery to separate function

b9bbd1f

do (test): current attemption situation

ca0bc4a

gr2m reviewed Jun 14, 2024

View reviewed changes

lib/success.js Outdated Show resolved Hide resolved

lib/success.js Outdated Show resolved Hide resolved

lib/success.js Outdated Show resolved Hide resolved

babblebey and others added 2 commits June 14, 2024 07:48

nits: prepend string commit to gal query aliases

f3be681

Co-authored-by: Gregor Martynus <39992+gr2m@users.noreply.github.com>

nits: rename associatedPRs gql query builder helper

1516ea3

Co-authored-by: Gregor Martynus <39992+gr2m@users.noreply.github.com>

Merge branch 'master' into refactor/replace-search-api

1eaffe0

babblebey added 14 commits June 14, 2024 18:04

fix: typo in success.js

c58c3e2

fix: address test case `Add comment and labels to PRs associated with…

6714074

… release commits and issues solved by PR/commits comments`

do: address test case `Add comment and labels to PRs associated with …

0299e79

…release commits and issues closed by PR/commits comments with custom URL`

do: address test case `Do not add comment and labels for unrelated PR…

e229ad3

… returned by search (compare sha and merge_commit_sha)`

do: address test case `Do not add comment and labels if no PR is asso…

813969b

…ciated with release commits`

do: address test case `Do not add comment and labels to PR/issues fro…

b7963d7

…m other repo`

do: address test case Ignore missing and forbidden issues/PRs

0d1f44f

do: address test case Add custom comment and labels

d8b5396

do: address test case Add custom label

703f7bb

do: address test case Comment on issue/PR without ading a label

c4f5df3

do: address test case `Editing the release to include all release lin…

5585c3b

…ks at the bottom`

do: address test case `Editing the release to include all release lin…

d336aa8

…ks at the top`

do: address test case `Editing the release to include all release lin…

4b7c9a9

…ks with no additional releases (top)`

do: address other test case except "Make multiple search queries if n…

69e67cf

…ecessary"

chore: drop getSearchQueries integration from success.js and depr…

5ebd038

…ecate?

babblebey changed the title ~~fix (refactor): replace github search api with graphql~~ refactor: replace github search api with graphql in success lifecycle method Jun 14, 2024

babblebey marked this pull request as ready for review June 14, 2024 20:47

gr2m reviewed Jun 14, 2024

View reviewed changes

lib/get-search-queries.js Outdated Show resolved Hide resolved

gr2m changed the title ~~refactor: replace github search api with graphql in success lifecycle method~~ fix: replace github search api with graphql in success lifecycle method Jun 15, 2024

babblebey added 3 commits June 19, 2024 12:53

Merge branch 'master' into refactor/replace-search-api

61eea50

chore: remove deprecated getSearchQueries helper

9b2a923

fix: modify buildAssociatedPRsQuery returned string

9c58e02

babblebey added 2 commits June 24, 2024 14:34

Merge branch 'master' into refactor/replace-search-api

737fd05

feat: retrieve body property for associatedPRs

807c400

gr2m mentioned this pull request Jul 1, 2024

HttpError: You have exceeded a secondary rate limit. semantic-release/semantic-release#2204

Closed

gr2m approved these changes Jul 1, 2024

View reviewed changes

gr2m enabled auto-merge (squash) July 1, 2024 17:13

Merge branch 'master' into refactor/replace-search-api

53c8024

gr2m merged commit be394cf into master Jul 1, 2024
6 checks passed

gr2m deleted the refactor/replace-search-api branch July 1, 2024 17:15

gr2m mentioned this pull request Jul 1, 2024

feat: allow for releasedLabels with successComment set to false #861

Open

kristof-mattei added a commit to kristof-mattei/semantic-release-github that referenced this pull request Jul 2, 2024

fix: update tests to reflect changed success method (see semantic-rel…

34ebd6e

…ease#857)

babblebey mentioned this pull request Jul 3, 2024

Edge Case: Address fetching associatedPRs from more than 100 commits with GraphQL #868

Closed

joakimen mentioned this pull request Jul 22, 2024

Verify semantic-release GitHub integration capralifecycle/liflig-cdk#272

Merged

babblebey mentioned this pull request Aug 6, 2024

feat: allow conditional skip on success and fail comments #874

Merged

jennifer-shehane mentioned this pull request Aug 28, 2024

fix: republish action cypress-io/github-action#1242

Merged

oscard0m mentioned this pull request Sep 13, 2024

refactor(glob-assets): use 'tinyglobby' instead of 'globby' as dependency #925

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: replace github search api with graphql in success lifecycle method #857

fix: replace github search api with graphql in success lifecycle method #857

babblebey commented Jun 13, 2024 •

edited

Loading

babblebey left a comment •

edited

Loading

gr2m commented Jun 14, 2024

babblebey commented Jun 14, 2024

gr2m commented Jun 14, 2024

gr2m left a comment

gr2m commented Jun 15, 2024

gr2m commented Jun 15, 2024

babblebey commented Jun 21, 2024 •

edited

Loading

gr2m commented Jun 21, 2024

babblebey commented Jun 24, 2024 •

edited

Loading

gr2m left a comment

gr2m Jul 1, 2024

brandon-kyle-bailey Aug 2, 2024

babblebey Aug 2, 2024

brandon-kyle-bailey Aug 2, 2024

brandon-kyle-bailey Aug 2, 2024

babblebey Aug 2, 2024

travi Aug 2, 2024

gr2m commented Jul 1, 2024

babblebey commented Jul 1, 2024

fix: replace github search api with graphql in success lifecycle method #857

fix: replace github search api with graphql in success lifecycle method #857

Conversation

babblebey commented Jun 13, 2024 • edited Loading

Changes Made

Related Issue

Screencast/Screenshot

babblebey left a comment • edited Loading

Choose a reason for hiding this comment

UPDATE:

gr2m commented Jun 14, 2024

babblebey commented Jun 14, 2024

gr2m commented Jun 14, 2024

gr2m left a comment

Choose a reason for hiding this comment

gr2m commented Jun 15, 2024

gr2m commented Jun 15, 2024

babblebey commented Jun 21, 2024 • edited Loading

gr2m commented Jun 21, 2024

babblebey commented Jun 24, 2024 • edited Loading

gr2m left a comment

Choose a reason for hiding this comment

gr2m Jul 1, 2024

Choose a reason for hiding this comment

brandon-kyle-bailey Aug 2, 2024

Choose a reason for hiding this comment

babblebey Aug 2, 2024

Choose a reason for hiding this comment

brandon-kyle-bailey Aug 2, 2024

Choose a reason for hiding this comment

brandon-kyle-bailey Aug 2, 2024

Choose a reason for hiding this comment

babblebey Aug 2, 2024

Choose a reason for hiding this comment

travi Aug 2, 2024

Choose a reason for hiding this comment

gr2m commented Jul 1, 2024

babblebey commented Jul 1, 2024

babblebey commented Jun 13, 2024 •

edited

Loading

babblebey left a comment •

edited

Loading

babblebey commented Jun 21, 2024 •

edited

Loading

babblebey commented Jun 24, 2024 •

edited

Loading