Request limit for images.weserv.nl: Please share your views

[laukstein]

Any plans to remove or increase the limitation?
Using images.weserv.nl for image gallery would quickly or even immediately exceed the current request limit.

Also is the limitation in Cloudflare (Cloudflare can cache in its CDN and return its cache without connecting to origin until the CDN cached file removed or expired) or while connecting origin (when Cloudflare tries to receive the resource from origin when isn't found in Cloudflare CDN)?

https://images.weserv.nl/faq/#are-there-any-limitations
Furthermore, there is a request limit, which is 700 images per 3 minutes, after which the IP-address will be blocked for 1 hour.

[andrieslouw]

The limit is in the origin, so requests that are cached by CloudFlare will not be counted.
We'll check if it is possible to increase the limit.

[laukstein]

@andrieslouw, is the request limit per visitor IP or referrer server (the domain where the images are used as <img src>) ?
Maybe it is worth to update https://images.weserv.nl/faq/#are-there-any-limitations with better description, like about the request limit against the origin.

[andrieslouw]

The limit is per visitor IP, but we'll rewrite the FAQ indeed. We're quite busy, so it may take a couple of days.

[andrieslouw]

FAQ has been rewritten & published. Let us know if there is anything else we can do to assist.

[laukstein]

Shortly saying the limitation hasn't been changed, making impossible to use images.weserv.nl for things like image gallery.

[andrieslouw]

The limitation for the service we provide on images.weserv.nl hasn't changed indeed. I'm sorry, it is best effort, and while we may increase the limit in the future, for now this is what we are comfortable with offering on our current servers, keeping the service available to everyone for free. I should've stated that when closing this issue.

You're free to use the source code on your own server(s), or on any (free) cloud platform, we'll even offer our support in case you run into trouble doing so.

I'm not sure how big the image gallery is you'd be using, but if you're willing to share the amount of images per time that seems reasonable, and fits your use case, please do so. I will run some new calculations on additional/new servers for mid-2020, bearing in mind the futures requested here (animated GIF to MP4 being one of the harder in terms of processing power & capabilities).

Recently we've had a big uptake in bandwidth & CPU-demand due to supporting animated GIF resizing, which is nearly saturating our links & CPU-cores, causing some delayed requests on busy hours.

I'll reopen this issue to invite people to share their views on the current limits we have, maybe we can agree on a better way to regulate what's available to everyone; we could, for instance, increase the time window in which we measure; so that it would be possible to do 2500 requests in 10 minutes?

[andrieslouw]

If there is anyone willing to share their wishes for a raised request limit, please do so! I'd like to make some (back of the napkin) calculations to estimate if it would be worthwhile to migrate to a new AMD Zen platform.

[raRaRa]

Can we get clarification on this paragraph:

There is a filter on the origin domain name. This means that we refuse to download images from certain websites, to prevent our service from being blocked by others. This filtering is handled by OpenDNS domain tagging.

Is origin the website where the image is being displayed on, or the website where the image is being downloaded from? If it's the website where the image is being downloaded from, what happens to popular websites such as imgur, google, etc, do they have an exception?

Thanks.

[andrieslouw]

Origin is the website provided in the ?url=, thus the website the image is downloaded from. OpenDNS provides all white- and blacklists, we don't modify those. Imgur & Google will be on a whitelist as far as I know. We only block the heaviest categories to prevent our service from being blocked. This is the Low tier, explained here: https://support.opendns.com/hc/en-us/articles/227988047-Web-Content-Filtering-and-Security

[lincolnthree]

@andrieslouw Hey! Jumping in on this thread a bit late. 2500 / 10m seems like a great compromise to me. Obviously there will always be exceptions, but 700/3m seems to be chopping things off a bit abruptly for pages with lots of smaller images that are loaded in "bursts". Obviously the dream is no limits, but I understand reality :) Feels a bit strange to be sending feedback on this item to an open-source free service -- at all.

Also trying to understand what this means, in a bit more detail (I know it's been asked about before):

Furthermore, there is a request limit per visitor IP for uncached requests, which is 700 images per 3 minutes, after which the IP-address will be blocked for 1 hour.

I'm just wondering what the definition of an "uncached" request is. At what level of cache? On your server's nginx-cache, I assume? This could also be the CDN cache? Something that needs to go to the origin?

Lastly, thanks for this great project.

[andrieslouw]

I think 2500/10m would be a good compromise. Uncached requests pass through the CloudFlare cache, and also pass through the nginx-cache. The uncached requests will hit the origin server indeed.

@kleisauke : Can we change the limit in production?

[kleisauke]

I just updated the request limit on the production servers to 2500/10m, see:
https://images.weserv.nl/quota

Commit a8efbaa and weserv/docs@b0d1e22 reflects this change in the nginx configuration and documentation.

[lincolnthree]

@kleisauke @andrieslouw Awesome! Thank you so much! PS. I was unaware of the URL to query rate limit remaining. That's very good to know.

Also, I know I said it before, but I really appreciate the fact that you do offer this project and service to the community! It's fantastic and really well designed.

[lincolnthree]

Just noticed that the rate limit starts refreshing itself slowly as time goes on (e.g. it's not a strict 2500 / 10m limit, seems like a rolling window?) This is a really neat implementation. Makes a lot of sense for throttling requests - I probably would not have thought of that :)

[kleisauke]

Thanks for your kind words. Perhaps the URL to check your current rate limit quota should also be mentioned in the FAQ. fwiw, it's highlighted here now:
https://images.weserv.nl/news/2019/09/01/introducing-api-5/#improved-rate-limiter

Just noticed that the rate limit starts refreshing itself slowly as time goes on (e.g. it's not a strict 2500 / 10m limit, seems like a rolling window?) This is a really neat implementation. Makes a lot of sense for throttling requests - I probably would not have thought of that :)

Indeed, it's a Redis backed rate limiter based on the sophisticated generic cell rate algorithm which provides a rolling time window and doesn't depend on a background drip process. The source code of this nginx module can be found here:
https://github.com/weserv/rate-limit-nginx-module

[lincolnthree]

@kleisauke Agreed, I think the FAQ is a good place. I would not have thought to check the news articles (I did try 'Googling' it.) Reading into that algorithm. Quite elegant.