Pinning to Pinata disabled due to "Rate limit exceeded" #1900

RubenKelevra · 2022-01-13T23:47:44Z

OS: Windows
Version of IPFS Desktop 0.18.1

Describe the bug
Pinata integration stopped working. So it worked before and after some update in the recent past it stopped working. I thought it's a temporary outage as I saw it first. But that's quite a while ago, maybe a month or two, and it's still not working.

To Reproduce
Steps to reproduce the behavior:

-> Settings
Click "+ Add Service"
Select Pinata
Go to Pinata's API page (which is linked)
Create a new key
Add it to the dialog opened in step 3
Click on Save
Pinata's status is "Pins: Failed to fetch"

Expected behavior
The integrated service connection should work.

Screenshots

hacdias · 2022-01-14T13:05:25Z

@RubenKelevra I think it's related to your API token: it's either expired or doesn't have the required permissions (Pinning API permissions).

RubenKelevra · 2022-01-14T13:08:07Z

@RubenKelevra I think it's related to your API token: it's either expired or doesn't have the required permissions (Pinning API permissions).

Well, the token is new and has all permissions active.

hacdias · 2022-01-14T13:45:04Z

Very odd, it works well here. Did you use the correct token? It's the JWT token (secret access token) and not the API secret.

hacdias · 2022-01-21T12:52:29Z

@RubenKelevra any updates on this?

RubenKelevra · 2022-01-21T13:08:57Z

Hey @hacdias, sorry for the delay. I think the issue was related to my ISP - it's working fine now without any change 🤷

Thanks for the efforts to help me, highly appreciated!

lidel · 2022-02-02T21:01:12Z

FWIW I noticed similar problem while testing webui in ipfs-desktop and clicking between settings and files.

Poked at things and this.. is not false-negative.

Pinata is rate-limiting, and HTTP checks sent by ipfs pin remote service ls --stat (used by ipfs-webui for determining if service is in a good shape) are not successful when executed in short window of time:

$ ipfs pin remote service ls --stat                                                                                                         ~
Pinata       https://api.pinata.cloud/psa     0/0/11/0
estuary      https://api.estuary.tech/pinning 14/14/1/0
pinata-debug http://127.0.0.1:5000/psa        invalid
pinata-test  https://testapi.pinata.cloud/psa 0/0/16/5

$ ipfs pin remote service ls --stat                                                                                                         ~
Pinata       https://api.pinata.cloud/psa     invalid
estuary      https://api.estuary.tech/pinning 14/14/1/0
pinata-debug http://127.0.0.1:5000/psa        invalid
pinata-test  https://testapi.pinata.cloud/psa invalid

I suspect that if you switch from Settings to Files, and try to pin something you hit the rate limit quite often.

Low level error can be reproduced with curl

$ curl -H "Authorization: Bearer secret" "https://api.pinata.cloud/psa/pins?limit=1&status=failed"
{"count":0,"results":[]}
$ curl -H "Authorization: Bearer secret" "https://api.pinata.cloud/psa/pins?limit=1&status=failed"
{"count":0,"results":[]}
$ curl -H "Authorization: Bearer secret" "https://api.pinata.cloud/psa/pins?limit=1&status=failed"
{"count":0,"results":[]}
$ curl -H "Authorization: Bearer secret" "https://api.pinata.cloud/psa/pins?limit=1&status=failed"
{"count":0,"results":[]}
$ curl -H "Authorization: Bearer secret" "https://api.pinata.cloud/psa/pins?limit=1&status=failed"
{"count":0,"results":[]}
$ curl -H "Authorization: Bearer secret" "https://api.pinata.cloud/psa/pins?limit=1&status=failed"
{"error":{"reason":"Rate limit exceeded","details":"You have exceeded the API rate limit. Wait a minute and slow down your requests."}}

@obo20 any suggestions how we avoid triggering this rate limiter?

ipfs pin remote service ls --stat sends 4 requests (one for each status), and you seem to block around 6 requests in some short window – pretty low – perhaps we could just bump this to more reasonable throttling (eg. 30-60 requests / minute)?

RubenKelevra · 2022-02-04T14:22:33Z

@lidel yeah, I can confirm that I still have issues. It jumped back to "Failed to fetch".

So I created a new key, and when I add it shows that it was working and but it jumps to "Failed to fetch" when I turn on auto-upload.

So not sure if pinata doesn't want users to use their pin-service or if that's an IPFS issue but it's kind of annoying since I have to manually upload everything to their service.

lidel · 2022-02-07T21:10:11Z

I wonder if we need to add support for Retry-After: 60 to https://github.com/ipfs/go-pinning-service-http-client/ and ipfs pin remote CLI.

Pinata returns:

< HTTP/1.1 429 Too Many Requests
< Server: nginx/1.16.1
< Date: Mon, 07 Feb 2022 21:08:07 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 135
< Connection: keep-alive
< X-Powered-By: Express
< Vary: Origin
< Access-Control-Allow-Credentials: true
< X-RateLimit-Limit: 5
< X-RateLimit-Remaining: 0
< X-RateLimit-Reset: 1644268110
< Retry-After: 60

Open question if ipfs pin remote command hanging for a minute is a better UX than:

$ ipfs pin remote ls --service=Pinata
Error: reason: "Rate limit exceeded", details: "You have exceeded the API rate limit. Wait a minute and slow down your requests.": 429 Too Many Requests

lidel · 2022-02-07T23:12:30Z

Ok, scratch it. This won't be enough by any means.
The root cause is not --stat check. It is browsing files.

The Files screen will check and cache remote pin status for every file on the screen, batching 10 CIDs per check.

The result is cached until:

the browser tab with ipfs-webui is closed (or ipfs-desktop is shut down and started again),
.. or the file is pinned/unpinned by the user via webui.

Webui calls /api/v0/pin/remote/ls?cid=Qm1,Qm2...Qm10 and that is translated to /pins?cid=Qm1,Qm2...Qm10 at Pinning Service API exposed by Pinata. These requests are what triggers the throttling – 6 requests per minute is not enough.

We send 10 CIDs per request and I can easily generate more than 10 HTTP requests while visiting a bigger directory for the first time:

(here one can see that estuary responds, but Pinata does not because of throttling its marked as offline)

Will evaluate options (increasing the CID check batch size to 100 won't help – still throttled).

This is bare minimum to restore remote pinning with rate-limited Pinata. Details in #1900 (comment) - The expensive 'ipfs pin remote service ls --stat' and autopin MFS policy is executed only on Settings screen - Files screen does not make any requests to remote pinning service anymore. - Remote pin status check is triggered by opening "Set pinning" modal via context menu or the pin icon, or by clicking on "Pin Status" column header. – We cache aggressively and assume pins on remote service are not changed by other means. The cache lives in memore, reloading the webui will purge cached states. - Clicking on "Pin Status" header bypassess remote pin status cache and forces online status re-check for all items in current directory. - Errors returned by pinning service are now correctly reported to the user. The CLI and ipfs-webui users will now see the same error message.

this leverages createCacheBundle (src/bundles/index.js) and persistActions (src/bundles/pinning.js) for persisting remote pin states across page reloads new users should not see any functional difference, old users may need to manually go to "Set pinning" to trigger status fetch, but that is the best we can do and a fair compromise, given that before this remote pins at Pinata did not work at all due to throttling (#1900)

* refactor: on-demand remote pin status This is bare minimum to restore remote pinning with rate-limited Pinata. Details in #1900 (comment) - The expensive 'ipfs pin remote service ls --stat' and autopin MFS policy is executed only on Settings screen - Files screen does not make any requests to remote pinning service anymore. - Remote pin status check is triggered by opening "Set pinning" modal via context menu or the pin icon, or by clicking on "Pin Status" column header. – We cache aggressively and assume pins on remote service are not changed by other means. The cache lives in memory, reloading the webui will purge cached states. - Clicking on "Pin Status" header bypasses remote pin status cache and forces online status re-check for all items in current directory. - Errors returned by pinning service are now correctly reported to the user. The CLI and ipfs-webui users will now see the same error message. * refactor: IndexedDB persistence of remote pins this leverages createCacheBundle (src/bundles/index.js) and persistActions (src/bundles/pinning.js) for persisting remote pin states across page reloads new users should not see any functional difference, old users may need to manually go to "Set pinning" to trigger status fetch, but that is the best we can do and a fair compromise, given that before this remote pins at Pinata did not work at all due to throttling (#1900) * refactor(i18n): new key to force new translation added two variables, and old key had none. using a different key ensures users use English instead of old translation

lidel · 2022-02-11T21:09:39Z

@RubenKelevra recent changes in ipfs-webui v2.15.0 should restore remote pinning functionality even when using throttled service like Pinata (which I confirmed to be 6 requests per minute).

You can try it at https://webui.ipfs.io (needs CORS safelisting) for now,
it will also ship in future release of ipfs-desktop (0.19 planned next week) and go-ipfs (0.13 planned this quarter)

RubenKelevra · 2022-02-11T21:46:07Z

Thanks for the fast fix! :)

SgtPooki · 2022-05-09T18:54:19Z

@lidel can we remove the pinned flag?

acostaedg · 2022-08-22T12:36:20Z

This issue is still happening in latest kubo version:

AGENT kubo v0.15.0-rc1 d577980/docker
UI v2.15.1

When Pinata is added as remote pinning service it hits rate limit right away. Can this issue be re-opened? or if any workaround exist can someone please suggest?

@lidel

RubenKelevra closed this as completed Jan 21, 2022

lidel reopened this Feb 2, 2022

lidel changed the title Feb 2, 2022

lidel transferred this issue from ipfs/ipfs-desktop Feb 7, 2022

lidel added area/pinning Integrating pinning into GUI apps kind/bug A bug in existing code (including security flaws) need/analysis Needs further analysis before proceeding P1 High: Likely tackled by core team if no one steps up labels Feb 7, 2022

This was referenced Feb 8, 2022

remote pins: inform user when pin status check failed #1901

Closed

remote pins: remove automatic status check #1902

Closed

API v2 proposal: remove PinResults.count ipfs/pinning-services-api-spec#86

Open

BigLep assigned lidel Feb 8, 2022

This was referenced Feb 9, 2022

refactor: on-demand remote pin status checks #1903

Merged

Consider how to handle repo/issue transfer requests ipdxco/github-as-code#11

Open

lidel closed this as completed in #1903 Feb 11, 2022

lidel pinned this issue Feb 11, 2022

lidel unpinned this issue May 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pinning to Pinata disabled due to "Rate limit exceeded" #1900

Pinning to Pinata disabled due to "Rate limit exceeded" #1900

RubenKelevra commented Jan 13, 2022

hacdias commented Jan 14, 2022 •

edited

Loading

RubenKelevra commented Jan 14, 2022

hacdias commented Jan 14, 2022

hacdias commented Jan 21, 2022

RubenKelevra commented Jan 21, 2022

lidel commented Feb 2, 2022 •

edited

Loading

RubenKelevra commented Feb 4, 2022 •

edited

Loading

lidel commented Feb 7, 2022 •

edited

Loading

lidel commented Feb 7, 2022 •

edited

Loading

lidel commented Feb 11, 2022 •

edited

Loading

RubenKelevra commented Feb 11, 2022

SgtPooki commented May 9, 2022

acostaedg commented Aug 22, 2022 •

edited

Loading

Pinning to Pinata disabled due to "Rate limit exceeded" #1900

Pinning to Pinata disabled due to "Rate limit exceeded" #1900

Comments

RubenKelevra commented Jan 13, 2022

hacdias commented Jan 14, 2022 • edited Loading

RubenKelevra commented Jan 14, 2022

hacdias commented Jan 14, 2022

hacdias commented Jan 21, 2022

RubenKelevra commented Jan 21, 2022

lidel commented Feb 2, 2022 • edited Loading

RubenKelevra commented Feb 4, 2022 • edited Loading

lidel commented Feb 7, 2022 • edited Loading

lidel commented Feb 7, 2022 • edited Loading

lidel commented Feb 11, 2022 • edited Loading

RubenKelevra commented Feb 11, 2022

SgtPooki commented May 9, 2022

acostaedg commented Aug 22, 2022 • edited Loading

hacdias commented Jan 14, 2022 •

edited

Loading

lidel commented Feb 2, 2022 •

edited

Loading

RubenKelevra commented Feb 4, 2022 •

edited

Loading

lidel commented Feb 7, 2022 •

edited

Loading

lidel commented Feb 7, 2022 •

edited

Loading

lidel commented Feb 11, 2022 •

edited

Loading

acostaedg commented Aug 22, 2022 •

edited

Loading