[BUG] Intermittent 403 RequestError in build-dashboard.js script

[anshgoyalevil]

Describe the bug.

We have a build-dashboard.js script in place that runs every midnight via the GitHub Actions Cron Job defined at https://github.com/asyncapi/website/actions/runs/8730442751/workflow#L7 along with 2 other scripts build-meetings.js and build-newsroom-videos.js.

The AsyncAPI bot then creates the PR with the resulting file changes.

There's some error in the script due to which we are reaching GitHub API limits too soon.
The reported error is: https://github.com/asyncapi/website/actions/runs/8501984640/job/23285563590#step:6:54

RequestError [HttpError]: You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 7C54:8291:1619423:2304A79:6609FFC4.

Expected behavior

There must not be any such error. Either try to find an alternate method for replacing the failing code in build-dashboard.js or fix the existing approach.

Screenshots

How to Reproduce

Check out the workflow runs here: https://github.com/asyncapi/website/actions/workflows/regenerate-meetings-and-videos.yml

🥦 Browser

None

👀 Have you checked for similar open issues?

• I checked and didn't find similar issue

🏢 Have you read the Contributing Guidelines?

• I have read the Contributing Guidelines

Are you willing to work on this issue ?

None

[NightFury742]

hey @anshgoyalevil can I work on this?

[anshgoyalevil]

@NightFury742 I am not sure if this bug still persists. Feel free to check if it exists and resolve :)

[Wck-iipi]

@anshgoyalevil I was looking at this issue and found out that this workflow is not running for quite some time now. It is giving the following error:

> @asyncapi/website@0.1.0 generate:dashboard
> node scripts/dashboard/build-dashboard.js

/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4
const *** Promise *** = require('node-fetch');
                    ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /home/runner/work/website/website/node_modules/node-fetch/src/index.js from /home/runner/work/website/website/scripts/dashboard/build-dashboard.js not supported.
Instead change the require of index.js in /home/runner/work/website/website/scripts/dashboard/build-dashboard.js to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4:21) ***
  code: 'ERR_REQUIRE_ESM'

(It is giving error for 4 months:

)
I would like to work on this issue, however, should I create new issue for this error as I couldn't find a issue for above error?

[aeworxet]

Bounty Issue's service comment

Text labels: bounty/2024-Q4, bounty/advanced, bounty/coding, bounty/upgraded
First assignment to regular contributors: 2024-09-20 00:00:00 UTC+12:00
End Of Life after: 2024-10-31 23:59:59 UTC-12:00

@asyncapi/bounty_team

The Bounty Program is not a Mentorship Program. The accepted level of Bounty Program Participants is Middle/Senior.

Regular contributors should explain in meaningful words how they are going to approach the resolution process when expressing a desire to work on this Bounty Issue.

[aialok]

Hi team,

I would like to take on this issue. I have good experience with GitHub Actions from my GSoC work, where I built the CI/CD pipeline for the JSON Schema website.

Thank you : )

[aialok]

My Plan :

Quick Fix: We can downgrade the node-fetch package it will work.

But I will try to find better solution and alternative to work with latest version.

Also we need to update github action workflows since we are using older version of node

I have tried fixing with degrading version it is working fine but we need to test it in github actions environment and find a good solutions. Also current github action packages are not up to date that need to fix.

Thank you : )

[Wck-iipi]

@aeworxet I would like to get assigned for this issue as I had already started working on it 4 days before and would be completed soon. Thanks.

[aeworxet]

@Wck-iipi

Please refrain from starting work until being assigned, trying to impose the fact that you must be assigned simply because you already did half the work you weren't assigned to in the first place.

On the other hand, you can describe in plain words what you have already done and what you are planning to do next without sharing the code. Probably this will increase your chances of being assigned to this Bounty Issue.

In any case, before sharing your approach, please specify your full name in your GitHub profile.

[sambhavgupta0705]

Ref: #3036

[jerensl]

@aeworxet I think I'm already made a fix for the node-fetch problem in this PR https://github.com/asyncapi/website/pull/3038/files#diff-cfbf694e8e8ffe40a57d3b7d5357d38fde7c3a21b22f9d6c74e4cf583f4d3a84, but the issues is seems comeback.

For the rate limiting, I think we can reduce the frequence of the cron jobs, reduce retry etc.

I also like to work on this issues if this still exist under bouty program

Some of my past contribution:
asyncapi/modelina#2067
asyncapi/modelina#2028
asyncapi/modelina#2031

[jerensl]

My Plan :

Quick Fix: We can downgrade the node-fetch package it will work.

But I will try to find better solution and alternative to work with latest version.

Also we need to update github action workflows since we are using older version of node

I have tried fixing with degrading version it is working fine but we need to test it in github actions environment and find a good solutions. Also current github action packages are not up to date that need to fix.

Thank you : )

For the issues you are mention, Im already made PR for that and this PR should fix the issues regarding of node-fetch
https://github.com/asyncapi/website/pull/3099/files

[aialok]

I dont think reducing number of cron job will resolve github rate limiting.
The problem is not because of github rate limiting but because of node fetch package.

We are still using v3 which have some esm import error.

website/scripts/dashboard/build-dashboard.js

Line 4 in 0c67226

const { Promise } = require('node-fetch');

Here is fix:

#2894 (comment)

I would love to work of this issue : )

[aialok]

My Plan :
Quick Fix: We can downgrade the node-fetch package it will work.
But I will try to find better solution and alternative to work with latest version.
Also we need to update github action workflows since we are using older version of node

I have tried fixing with degrading version it is working fine but we need to test it in github actions environment and find a good solutions. Also current github action packages are not up to date that need to fix.


Thank you : )

For the issues you are mention, Im already made PR for that and this PR should fix the issues regarding of node-fetch https://github.com/asyncapi/website/pull/3099/files

Yes, This PR might fix the issue : )

[aeworxet]

I think the issue has changed a bit since the time it was introduced.

https://github.com/asyncapi/website/actions/runs/10913294253/job/30289538132

/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4
const *** Promise *** = require('node-fetch');
                    ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /home/runner/work/website/website/node_modules/node-fetch/src/index.js from /home/runner/work/website/website/scripts/dashboard/build-dashboard.js not supported.
Instead change the require of index.js in /home/runner/work/website/website/scripts/dashboard/build-dashboard.js to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4:21) ***
  code: 'ERR_REQUIRE_ESM'
***

and will require a little more than simple juggling with cron jobs and retries.

[aialok]

I think the issue has changed a bit since the time it was introduced.

https://github.com/asyncapi/website/actions/runs/10913294253/job/30289538132

/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4
const *** Promise *** = require('node-fetch');
                    ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /home/runner/work/website/website/node_modules/node-fetch/src/index.js from /home/runner/work/website/website/scripts/dashboard/build-dashboard.js not supported.
Instead change the require of index.js in /home/runner/work/website/website/scripts/dashboard/build-dashboard.js to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4:21) ***
  code: 'ERR_REQUIRE_ESM'
***

and will require a little more than simple juggling with cron jobs and retries.

Yes, exactly.
The problem is with node-fetch package.

[jerensl]

I think the issue has changed a bit since the time it was introduced.

https://github.com/asyncapi/website/actions/runs/10913294253/job/30289538132

/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4
const *** Promise *** = require('node-fetch');
                    ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /home/runner/work/website/website/node_modules/node-fetch/src/index.js from /home/runner/work/website/website/scripts/dashboard/build-dashboard.js not supported.
Instead change the require of index.js in /home/runner/work/website/website/scripts/dashboard/build-dashboard.js to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4:21) ***
  code: 'ERR_REQUIRE_ESM'
***

and will require a little more than simple juggling with cron jobs and retries.

To give some context I made PR to fix this issues which happened first on the root of build-newsroom-videos.js but wasn't aware there other node fetch inside build-dashboard.js and made follow up PR but doesn't got merge since several month

[aialok]

I think the issue has changed a bit since the time it was introduced.
https://github.com/asyncapi/website/actions/runs/10913294253/job/30289538132

/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4
const *** Promise *** = require('node-fetch');
                    ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /home/runner/work/website/website/node_modules/node-fetch/src/index.js from /home/runner/work/website/website/scripts/dashboard/build-dashboard.js not supported.
Instead change the require of index.js in /home/runner/work/website/website/scripts/dashboard/build-dashboard.js to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4:21) ***
  code: 'ERR_REQUIRE_ESM'
***

and will require a little more than simple juggling with cron jobs and retries.

Yes, exactly. The problem is with node-fetch package.

Here is fix.
#2894 (comment)

I tested it in my local it is working fine but need to check in github actions environment.

[akshatnema]

Hey @aeworxet ,
I want to work on this issue as part of the Bounty program Q4. Do assign it to me.

[aeworxet]

@akshatnema is an AsyncAPI Maintainer specified in https://raw.githubusercontent.com/asyncapi/community/master/MAINTAINERS.yaml, so they fall under the first category in the prioritization list.

[aeworxet]

Bounty Issue's Timeline

Complexity Level	Assignment Date (by GitHub)	Start Date (by BP Rules)	End Date (by BP Rules)	Draft PR Submission	Final PR Merge Start	Final PR Merge End
Medium	2024-09-19	2024-10-07	2024-11-17	2024-10-20	2024-11-03	2024-11-17

Please note that the dates given represent deadlines, not specific dates, so if the goal is reached sooner, it's better.

Keep in mind the responsibility for violations of the Timeline.

[akshatnema]

@aeworxet There are two more GitHub issues that are subtly related to this one and still need to be done:

#3132
#3008

I would like to append them to this Bounty Issue and reclassify it to Advanced.

[aeworxet]

Upon request of the AsyncAPI Maintainer, who is also the Bounty Program Participant (@akshatnema), the Complexity Level of this Bounty Issue was reclassified to Advanced (upgraded) and its Timeline changed.

Bounty Issue was Reclassified and its Timeline changed

Complexity Level	Assignment Date (by GitHub)	Start Date (by BP Rules)	End Date (by BP Rules)	Draft PR Submission	Final PR Merge Start	Final PR Merge End
Advanced	2024-09-19	2024-10-07	2024-12-01	2024-10-27	2024-11-17	2024-12-01

Please note that the dates given represent deadlines, not specific dates, so if the goal is reached sooner, it's better.

Keep in mind the responsibility for violations of the Timeline.

[aeworxet]

Changed this Bounty Issue into an Aggregated Bounty Issue #3247 that accumulates

• [BUG] Intermittent 403 RequestError in build-dashboard.js script #2894
• [BUG] tweetLimit in community/newsroom isn't been reflected #3132
• [BUG] Responsiveness issue on the View on Github button  #3008

in it, for the sake of accounting simplification.