Gateway - Rate Limiting improvements with public IPFS gateways

[vasco-santos]

Problem Statement

Public IPFS gateways have rate limiting characteristics that will affect our Cloudflare worker nft.storage Gateway. A CF Worker has a unique outgoing IP, no matter what region it will be running.

When an IPFS gateway is rate limited, we will likely not be able to use a gateway that can already have the content cached. This will cause impact on response time, specially when gateway is blocked for a long period.

Finally, we can get to a state where all our public gateways are rate limited. In this case, we will simply redirect traffic to one gateway.

Gateways characteristics

Gloudflare Gateway

• by IP when 100 requests/min
• blocks IP during 1 Hour
  • further requests will not reset the timer, i.e 1H blocked from first block

Pinata Gateway

• Currently N/A for Dedicated Gateways

ipfs.io Gateway

• More difficult to predict rate limiting characteristics, considering its infrastructure setup
• ipfs.io public gateway uses several different techniques on load balancing and rate limiting, which makes it difficult to predict the actual rate limit value.
• ipfs.io gateway starts by geo routing requests, followed by load balancing. In our use case, I predict that all requests will be routed to the same geo area.
• The rate limit is set per load balancer. For instance, same {IP_ADDR, URI} are limited at1/second or 15/minute to a particular load balancer. Then there is a global 800/s on each Load Balancer.
• It also uses bursting techniques that will start by delaying responses before it actually fails.

Scale

• IPFS gateway req/s are over 1000 (metrics from end of 2021)
• TODO: @vasco-santos to find out data transfer metrics with PL Gateway Team

Solutions

We will need to work with the teams responsible for each of the gateways mentioned and create plans on how we can improve the rate limiting situation.

Per previous conversations, having improvements on the Cloudflare Worker "client" to for example have multiple output IPs is something that will likely not be possible to achieve.

Gloudflare Gateway

TBD

Multiple options on the table:

• have a different endpoint (with a special token / or some other way of whitelist our worker) that we can have more control on rate limiting
• support “do you have this locally” — we can rely on this with CF cache API. The limitation is the user would need to be within the cf-ipfs gateway region. Looking with CF team on how we could have global cache API support to the Cloudflare IPFS Gateway cache region
• just have bigger limits on Workers

ipfs.io Gateway

State: ✅

• Rely on X-Forwarded-For as a proxy indication to the ipfs.io gateway. The ipfs.io gateway would need to whitelist allowed IPs who could perform this.

[vasco-santos]

As an update, needed ipfs.io improvements were made. Pinata improvements are not needed for the time being.

We still need to figure out next steps with Cloudflare Gateway