How to change limit? API response for ${req.url} exceeds 4MB. This will cause the request to fail in a future version.

[700software]

What version of Next.js are you using?

11.1.0

How are you deploying your application?

Other platform

Describe the Bug

When streaming a large file in an API response, are getting this warnings on the terminal.

(this change was first introduced in PR #26831 and modified to 4mb in #26887)

API response for /api/... exceeds 4MB. This will cause the request to fail in a future version. https://nextjs.org/docs/messages/api-routes-body-size-limit

But we are not using Vercel. We are not using AWS serverless functions at all. So to throw a warning is not relevant to us.

I am concerned that we will update Next.js and our file downloads will start throwing runtime exceptions in production as wording of #26831

In a future version this scenario will throw an error.

Possible solution(s)

I propose a config option to suppress this warning (future error) for non-Vercel users of Next.js.

Alternatively, I'm wondering if middleware would make this a non-issue?

We are hoping to avoid spinning up a custom server, but will do if required.

[balazsorban44]

Hi, could you try adding a minimal reproduction that would make this easier to reason about?

[700software]

reproduction

// pages/api/test-download.js
import crypto from 'crypto'

export default async function (req, res) {
  res.setHeader('Content-Disposition', 'attachment; filename=random.dat');
  // create random buffer 10mb
  res.end(crypto.randomBytes(10 * 1024 * 1024)) // above the current 4mb limit of nextjs
  // can also be reproduced with `ReadableStream.pipe(res)`
}

[balazsorban44]

So this is actually intentional.

In your use case, sending large files from a single Next.js instance isn't performant as the user could be far away or the connection gets broken so it's better to use a CDN which is designed for handling large assets.

The error document points this out as well:

If you need to support sending large files to the client, you should consider using a dedicated media host for those assets.

https://nextjs.org/docs/messages/api-routes-body-size-limit

[700software]

@balazsorban44, Prior to opening this issue I read through all the relevant documentation. I'm aware that it is intentional.

However, in our use case, these are private files (requires authentication check), a lesser-used feature in larger application, and all users are in the same country. Therefore we could not justify a CDN. Even a dedicated media host would be overkill. As we grow this may change but not any time soon.

We could certainly start a custom server but that seems like a workaround for red tape. Plus we'd lose the npm run dev hot reloading advantage for those particular endpoints.

On the other hand, it would be so easy to add a config switch, and many developers are in use cases like ours, where, nobody benefits from overengineering. That is, until they can grow their company to a larger scale.

So, I'm just thinking through the Next.js team's motivation for adding this restriction. What your team has done makes sense in certain use cases, but not for everyone. I'm particularly concerned about the Error coming in a future version.

My team would be glad to create a PR for a config switch. What do you think?

[jaymascarenas]

My app would also greatly benefit from being able to override this restriction. We too are not using serverless functions and are ok with larger response payloads. I really like the Next API and don't want to have to create a custom server and express API.

[ImTemporaryHere]

My app would also greatly benefit from being able to override this restriction. We too are not using serverless functions and are ok with larger response payloads. I really like the Next API and don't want to have to create a custom server and express API.

fully support , I dont use vercel, there is no need to limit my app... pls.

[700software]

Again, my team would be glad to create a PR introducing a next.config.js option. I just need to know it will be merged and not rejected. :)

[Tetedeiench]

@balazsorban44 Please consider reopening the issue or participating in the conversation here.

I'm also afraid of the time where this will produce an error.

Not everyone is using Vercel / Cloud services, there is a world beyond this : i'm self-hosting on a dedicated server, thus i'm fully responsible for the resource usage on my server. There can be a warning to tell me, I have no issue with that. But I would like to be fully responsible for my actions and thus :

• Be able to disable that warning if I judge it doesn't apply
• Still be able to use that function if it fits my use case

Especially since it's an easy solution to implement without going all out for a CDN which is probably overkill in many, many situations. It adds an extra layer of complexity and cost for little to no gain in many use cases.

My personal use case is gating the downloads behind an authentication mechanism ( next-auth ).

To give you some numbers :

• The gated files are about 30MB and are downloaded 20 000 times per day in my case (which amounts to about 600GB per day)
• I don't have any performance or CPU usage issues (quite the contrary, it's really low, congrats !). Same with network. It's just working like a charm.
• I have NO complains about users having slow downloads or issue with this mechanism
• Website is working like a charm

I agree with @700software here : give us a way to disable this warning, and forbid or do whatever you want on Vercel to prevent it for being used. Restricting it for everyone is just too broad.

I personally think you're implementing a Vercel/Cloud oriented warning inside Next.js, which is not the place for that limit to be.

[crice88]

@balazsorban44 Hoping you can reopen this issue and allow a PR to be made. I do not use Vercel and have a use case in streaming large files back to the client.

Trying to load large files first in memory or moving them from S3 and back again just won't work for me.

We allow users to download their folders/files from S3 (kinda like Dropbox). When a folder download request comes in we pull from S3, create directories and zip the whole package up and stream it back to the client. For us to take that zip, push it back out to S3 and request a presigned URL is just not as performant (not to mention it can get costly) as streaming the data across.

I agree with both @Tetedeiench and @700software that there should be a way to configure this option.

[ijjk]

Hi, I think a config to disable this warning similar to the externalResolver page config is reasonable, note this warning is relevant to deployments not leveraging serverless as well. If you are hosting in Docker and you are serving files from an API route this is much less optimal than serving directly from a CDN which handles disconnects/resuming serving the file from a location closest to the user.

[Tetedeiench]

Hi, I think a config to disable this warning similar to the externalResolver page config is reasonable, note this warning is relevant to deployments not leveraging serverless as well. If you are hosting in Docker and you are serving files from an API route this is much less optimal than serving directly from a CDN which handles disconnects/resuming serving the file from a location closest to the user.

Thanks for reopening the issue ! Indeed, a CDN with auth support is maybe optimal, but it comes at a steep cost.

In my case, as my app is heavy (30MB) and sort of popular ( 20 000 downloads per day), so the cost of a CDN is out of question.

I mean, an appropriate estimate for me is 17TB / month solely on downloads. I took azure CDN as an example, it would cost me 1,335$USD per month (pricing source : https://azure.microsoft.com/en-us/pricing/details/cdn/ ).

Serving it using my own server's bandwidth and using NextJS API routes is included in the hosting cost of my own, private server, and that amounts total to 50$USD / month. Yeah, I handle the administration, backup and all... but that's fine with me, i'm already doing it, and my server has bandwidth to spare.

Nobody ever complained about download speeds and resume issues...

@700software i'm probably not the best candidate for a PR there, are you still up for it ?

[700software]

We're booked solid right now, but I'm keeping this on my radar for sure.
Technically, we 'need' the PR to happen so it gives us some motivation. 😉

[Tetedeiench]

I think there's no rush - this issue being reopened, I think we can safely bet this will not translate to an error in a future version of Nextjs :)

[greendesertsnow]

4Mb is large? Who's still using 56 kbps connection?

[crice88]

@700software and @Tetedeiench would you guys be willing to provide general feedback if I were to create a PR?

[crice88]

Here's my idea for the issue: https://github.com/vercel/next.js/compare/canary...crice88:add-body-limit-config?expand=1

[ijjk]

@crice88 feel free to open a PR with those changes and we can take a look with it opened!

[Tetedeiench]

@crice88 Thanks a lot :)

[github-actions]

This closed issue has been automatically locked because it had no new activity for a month. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.