proper use of mongo db in Next.js

[oran1248]

I saw a lot of posts and articles about this alert on MongoDB Atlas ("Connections % of configured limit has gone above 80"), but couldn't figure out how to solve it in my Next.js application.

I create my db connection outside the handler function. I used a middleware withDatabase.js:

const client = new MongoClient(process.env.MONGODB_URI, { 
    useNewUrlParser: true, 
    useUnifiedTopology: true 
});

const addDbToRequest = (handler, req, res) => {
    req.db = req.connection.db("MYDBNAME");
    return handler(req, res);
};

const withDatabase = handler => async (req, res) => {
    if (!client.isConnected()) {
        await client.connect();
    }
    req.connection = client;
    return addDbToRequest(handler, req, res);
};

export default withDatabase;

This middleware wraps the API endpoint handler.

Now, if I close the connection on every API handler when it finishes, like this:
const { connection } = req; if (connection) { connection.close(); }
Than, I'm getting an error on the second request to the same api handler:
MongoError: Topology is closed, please connect
And if i'm not closing the connection, i'm getting this alert (after a short time of use) to my email:
Connections % of configured limit has gone above 80

What is the best practices to work with MongoDB Atlas in a Next.js application?

Thanks!

[mousedown]

Hi Oran,

You can cache the DB connection as per docs on Atlas.

https://docs.atlas.mongodb.com/best-practices-connecting-to-aws-lambda/

[oran1248]

I have already read that doc, but couldn't figure out how to solve this issue, and what is the right implementation. I'm feeling like I'm the only one that use MongoDB with Next.js XD

Can you please elaborate?
I need to configure something in Atlas management site? or cache the connection in my next.js app? How can I do that?
Is it true that every time that Next.js API endpoint handler is being called, a new connection is created to mongodb atlas? and you can't avoid it because of the serverless environment?
Thank you for the response :)

BTW - I'm not using AWS lambda, I didn't even deployed my app, for now I'm only developing on my local machine but DB is on Atlas.

[mousedown]

No, you don’t need to configure on Atlas, it’s all via code. Connection can be cached between requests. I would follow their strategy exactly rather than your approach. FYI I’ve deployed via now and can confirmed this approach works. You can use ‘serverless’ package to mimic AWS set up and run locally as of on lambda, so same handler function etc.

Atlas Docs

“Define the MongoDB client connection to the MongoDB server outside the AWS Lambda handler function.

This makes the database connection available between invocations of the AWS Lambda function for the duration of the lifecycle of the function”

[oran1248]

Define the MongoDB client connection to the MongoDB server outside the AWS Lambda handler function.

So the solution is to mimic AWS set up on my local machine?
(Because if I wont, it will not cache the connection, even if the handler is defined outside the Next.js api handler)

Also, what do you mean by:

I would follow their strategy exactly rather than your approach

what is my approach? I don't have an "approach", i'm just trying to get things work with next.js........

[rafinskipg]

@mousedown I am trying to follow the same approach but I am reaching the same limit of connections. The only difference is that I do not import directly the database object, I have another abstraction on top. I can see it being cached correctly if I don't hit the reload button on the browser. While navigating normally it works, if I hit reload I get a new set of connections

[rafinskipg]

I am changing somethings and will come back with more insights

[rafinskipg]

[rafinskipg]

BUMPING!

Anyone has experienced this connection issues?

[rafinskipg]

in localhost, yeah. I am still not going to production until this is working

[oran1248]

It seems that this problem is only in dev mode. In production mode the connection is not closed. For now im ok with it because i can still continue developing.

[rafinskipg]

@mousedown @oran1248 , maybe you are right, i am looking at the production logs and the connection is being reused, while in localhost is being killed on each page reload.

[daveteu]

I believe you are facing this issue on development because everytime you edit a file, the file get recompiled, and then your server gets reloaded, creating new connection every time. Hence you have to cache the connection with the code examples below.

On production, you don't get this as you only create new connection when you restart your application.

[j0lvera]

in localhost, yeah. I am still not going to production until this is working

You could use a staging environment that replicates what's going to happen in production and see if the issue persists there. What @daveteu says might be true.

[mousedown]

Here is the code I use with success. Originally I used Mongoose, but ripped that all out. I will write and upload some scrips you can use for benchmarking, which may help others.

const MongoClient = require('mongodb').MongoClient;
let cachedDb: any = null;

export const connectToDatabase = async () => {
  if (cachedDb) {
    console.log('👌 Using existing connection');
    return Promise.resolve(cachedDb);
  }

  return MongoClient.connect(process.env.DB, {
    native_parser: true,
    useUnifiedTopology: true
  })
    .then((client) => {
      let db = client.db('dbname');
      console.log('🔥 New DB Connection');
      cachedDb = db;
      return cachedDb;
    })
    .catch((error) => {
      console.log('Mongo connect Error');
      console.log(error);
    });
};

[visakadivines]

The logs are how I noticed the errors in the first place since I wasn’t having issues when on localhost. The errors are all related to mongodb connection.

#12229 (reply in thread)

[locksiDesmond]

I hosted mine yesterday and it was working, i used mongoose. It Might be something you are missing. I also tried the caching and it also worked as well.

[daveteu]

@pnishadham are you using some kind of load testing and experience the error? Or do you face the error with just a few users?

In Vercel (and most serverless environment), you cannot test with load testing because of how serverless works.

In serverless environment, you are not just running 1 single instance of your app. You may be running several instances of a function concurrently. Hence if you use load testing, Vercel might run like 10-20 instances of a single function to serve that particular /api route you are testing.

10-20 instances of your app, each with setting of poolSize: 50 will let you hit connection limits on Atlas very easily and you will meet all kind of errors.

If you are using nextjs+Vercel normally, you will not experience the error if you use the DBCache codes as shown in the many replies here.

Load Test

I did a simple load test and I can handle 27554 page hits over 5 minutes and will not experience any error. Then again that works out to be 5,510 hits per minute or 8 million hits per day. If you have that kind of page hits then you won't have problem upgrading your Vercel and Atlas to support your usage.

I tested with the following codes.

src/lib/db.js

var mongoose;
let connection = null;
var poolsize = 10;

module.exports = async function () {
  if (
    connection === null ||
    (connection.connection.readyState !== 1 &&
      connection.connection.readyState !== 2)
    // readystate 1 === connected, 2 === connecting, I don't want to start new connection if it's already connecting.
  ) {
    mongoose = require("mongoose");
    console.log("[MONGOOSE] Creating New Connection");

    mongoose.connection.on("open", () => {
      log("[ MONGOOSE] Connected with poolSize " + poolsize);
    });

    try {
      mongoose.connect("<mongodb url>/<database>?retryWrites=true&w=majority", {
        useNewUrlParser: true,
        useUnifiedTopology: true,
        poolSize: poolsize,
      });
    } catch (err) {
      console.log("Mongoose connection error", err);
    }
    connection = mongoose; //save it to the cache variable
    return;
  } else {
    return;
  }
};

/src/pages/api/api.js

import db from '../../lib/db.js'

export default async (req, res) => {
    await db(); // get existing connection or wait for db to create new connection

    //continue with the rest of your codes.
    res.status(200).send("Everything done!")
}

[sanderfish]

@daveteu Have you tested this on Vercel? It doesn't make a difference in my setup. With your setup around 25-30 connections are being made in the Vercel CLI (locally).
Once deployed, the logs show [ MONGOOSE] Connected with poolSize 10) only 3 times. But Atlas quickly ramps up to hundreds of connections. The last spike is with your above code

[daveteu]

@sanderfish how many people accessing the website?

[sanderfish]

I'm experiencing the same issue in production after following the setup described here: https://vercel.com/guides/deploying-a-mongodb-powered-api-with-node-and-vercel

The only difference is that I am using mongoose instead of the mongodb client.

let cachedDb = null

// A function for connecting to MongoDB,
// taking a single parameter of the connection string
export default async function connectToDatabase() {
  // If the database connection is cached,
  // use it instead of creating a new connection
  if (cachedDb) {
    return cachedDb
  }

  // If no connection is cached, create a new one
  const db = await mongoose.createConnection(uri, {
    bufferCommands: false, // Disable mongoose buffering
    bufferMaxEntries: 0, // and MongoDB driver buffering
    useNewUrlParser: true,
    useFindAndModify: true,
    useCreateIndex: true,
    useUnifiedTopology: true,
  })

  // Cache the database connection and return the connection
  cachedDb = db
  return db
}

As soon as I start browsing the production site, MongoDB Atlas shows 100 - 200 active connections. When actively browsing the site, connections will ramp up to ±400, but I can't get it higher than that. When logging cachedDb, it shows that it's cached on most requests.

[rafinskipg]

wow this is being the no solution thread, @pnishadham let us know if you switch to fauna, how it works. I am still not on production with our product, we will do some metrics soon, to see what's going up

[sanderfish]

I said "in production" but actually meant "on the live server". The site isn't live yet thus traffic is low. However, after that initial spike of connections, I haven't seen as many anymore.

[daveteu]

At this point I’ve gotten so tired of mongodb and it’s connection issues that I’m considering just cutting my losses and switching to a proper serverless db like faunadb. I know this isn’t practical for most folks who already have websites in production but since I’m still building mine I think I’m just going to switch.

unfortunately, you are going to have the same problem if you were to use other databases because this has got to do with how in development you are using web pack and hot reload. Even mysql pool gets this problem. Essentially caching database connection is a skill you need to master.

Even in serverless functions (Lamda, Google Functions, Stitch), we cache our connections so a new connection doesn't get created when we call the functions.

[visakadivines]

I understand db caching is important and a good skill to know how to do well. I’m not convinced the issues I’m experiencing are linked to how I’m caching the db. I’ve followed the Vercel guide with mongo, the Atlas guide, and multiple other recommendations for caching the db and I continue to experience issues multiple times per day including but not limited to pools being force destroyed, not being able to connect on the multiple attempts, and seemingly random unhandled exceptions like “cannot read the property name of undefined” in the connect module. I think there might be an issue with either mongoose, the mongdb driver, Atlas or a combination of these. These issues aren’t unique to me. Other folks are having similar issues according to threads on mongoose. I’m not saying that there’s not a possibility that I’m doing something incorrectly, but I haven’t found it. The reason I want to switch to fauna is because it’s not connection based. I completely understand that there are going to be issues with it as well and that it’ll need to be handled properly in my application, but I’ve spent months debugging and testing different things to make mongodb work more smoothly with no success.

[daveteu]

can u share ur codes?

[sambhav-gore]

I have a graphql server implementation and when building the graphql server context, I am using the following way, and I do not see a problem of too many connection. During a load test of 5K client requests per minute, the max connections I had open was 21:

const options = {
  useNewUrlParser: true,
  useUnifiedTopology: true,
  authSource: "admin",
  useFindAndModify: "false",
};

// Create cached connection variable
const connection = {};

const DB_URI = process.env.DB_URI;

const connectDb = async () => {
  if (connection.isConnected) {
    // use cached connection when available
    return;
  }
  try {
    const dbConnection = await mongoose.connect(DB_URI, options);
    connection.isConnected = dbConnection.connections[0].readyState;
  } catch (err) {
    logger.error(`error connecting to db ${err.message || err}`);
  }
};

export { connectDb };

[sanderfish]

Interesting! It looks like you're not returning the connection itself, how do you use the connection this function determines?

[sambhav-gore]

that is mongoose magic. as long as a connection exists, mongoose would work, there was no need to pass the connection back.

[sanderfish]

🤯perhaps that's where I'm going wrong. Thanks for your example!

[sambhav-gore]

I am not using Atlas though, I made my own RS on my K8s cluster. I plan to make a staging env where I am going to use atlas. Also this is the first time I am doing the complete server setup so I may be doing something incorrect. but the DB connections do work in my application .

[devictoribero]

do you have @sambhav-gore more code I could see? I've been stuck for days and I tried many possible solutions but none work.

[mousedown]

So I did some benchmarking, and I was wrong about my setup. Connection pooling is not in effect. I suspect the below is needed, but I don't think the Lambda scope is available. I'll post an issue.

For the Node.js driver, declaring the connection variable in global scope will also do the trick. However, there is a special setting without which the connection pooling is not possible. That parameter is callbackWaitsForEmptyEventLoop which belongs to Lambda’s context object. Setting this property to false will make AWS Lambda freeze the process and any state data. This is done soon after the callback is called, even if there are events in the event loop.

[mousedown]

Edit - Benchmark hit errors at 100 connections. My Atlas tier has limit of 100, so all is absolutely tickerty boo with the code mentioned above. Lambda context is also confirmed as being correctly configured.

vercel/vercel#4548 (comment)

[Vadorequest]

https://github.com/nucleo-org/edge-next seems to use MongoDB, so it may be worth checking out how they implemented it.

[sanderfish]

That is @rafinskipg's site, who has been commenting in the thread about his issues.

[Vadorequest]

Oops, my bad!

[kaijchang]

Bump

Happens in development mode but not in production mode.

[sanderfish]

Just me..

…

On Sat, 13 Jun 2020 at 17:48, Dave ***@***.***> wrote: @sanderfish <https://github.com/sanderfish> how many people accessing the website? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12229 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXXXXGRBO757JCMGWNSJCLRWOUVRANCNFSM4MRUEAEQ> .

[daveteu]

Your app opens 25-30 connections in just 1 api call? If that then there's something wrong with your codes

[eric-burel]

I think it hasn't been quoted, but there's a doc from Mongo: https://developer.mongodb.com/how-to/nextjs-building-modern-applications

I hit the same problem and I think I've mostly solved it by following this doc. Will provide more info later.

[onderonur]

I've used nearly the same solution from the Mongo docs. But I've added a little touch.
When there is a promise which is still pending, I don't create a second promise to connect to the DB. I wait for the currently pending one. This way, if we have no DB connection and if there are simultaneous API requests, it prevents waiting for and creating multiple connections.
Here is the code;

import '@/types';
import mongoose from 'mongoose';
import { NextApiHandler, NextApiRequest, NextApiResponse } from 'next';
import { Maybe } from '@/types';
import models from '../models';

declare module 'http' {
  interface IncomingMessage {
    models: Maybe<typeof models>;
  }
}

const readyStates = {
  disconnected: 0,
  connected: 1,
  connecting: 2,
  disconnecting: 3,
};

let pendingPromise: Maybe<Promise<typeof mongoose>> = null;

// https://hoangvvo.com/blog/migrate-from-express-js-to-next-js-api-routes/
const withDb = (fn: NextApiHandler) => async (
  req: NextApiRequest,
  res: NextApiResponse,
) => {
  const next = () => {
    req.models = models;
    return fn(req, res);
  };

  const { readyState } = mongoose.connection;

  // TODO: May need to handle concurrent requests
  // with a little bit more details (disconnecting, disconnected etc).
  if (readyState === readyStates.connected) {
    return next();
  } else if (pendingPromise) {
    // Wait for the already pending promise if there is one.
    await pendingPromise;
    return next();
  }

  pendingPromise = mongoose.connect(process.env.DB, {
    useNewUrlParser: true,
    useUnifiedTopology: true,
    useCreateIndex: true,
  });

  try {
    await pendingPromise;
  } finally {
    pendingPromise = null;
  }

  next();
};

export default withDb;

[mousedown]

Recycled promises. Implemented, thanks.

[ashconnell]

100% this. I keep seeing people posting their caching code and i'm like... ummm... it caches after the promise...

Here's my simplified version using just mongodb:

import { MongoClient } from 'mongodb'

let uri = process.env.MONGO_URI
let dbName = process.env.MONGO_DB

let promise = null
let cached = null

if (!uri) throw new Error('Missing environment variable MONGO_URI')
if (!dbName) throw new Error('Missing environment variable MONGO_DB')

export async function connectToDatabase() {
  if (cached) return cached
  if (!promise) {
    promise = MongoClient.connect(uri, {
      useNewUrlParser: true,
      useUnifiedTopology: true,
    })
  }
  const client = await promise
  const db = await client.db(dbName)
  cached = {
    client,
    db,
  }
  return cached
}

Btw, this still suffers from the same problem but at least it's removed one edge-case.

[ifeltsweet]

During development, you may need to prevent connections leaking due to hot-reload by using global to cache your connection. A few solutions and examples are outlined in #7811, they are knex specific but I used exactly the same strategy to fix my MongoClient problems in development.

[ashconnell]

I can confirm that swapping this:

import { MongoClient } from 'mongodb'

let uri = process.env.MONGO_URI
let dbName = process.env.MONGO_DB

let promise = null
let cached = null

if (!uri) throw new Error('Missing environment variable MONGO_URI')
if (!dbName) throw new Error('Missing environment variable MONGO_DB')

export async function connectToDatabase() {
  if (cached) return cached
  if (!promise) {
    promise = MongoClient.connect(uri, {
      useNewUrlParser: true,
      useUnifiedTopology: true,
    })
  }
  const client = await promise
  const db = await client.db(dbName)
  cached = {
    client,
    db,
  }
  return cached
}

With this (using global):

import { MongoClient } from 'mongodb'

let uri = process.env.MONGO_URI
let dbName = process.env.MONGO_DB

if (!uri) throw new Error('Missing environment variable MONGO_URI')
if (!dbName) throw new Error('Missing environment variable MONGO_DB')

export async function connectToDatabase() {
  if (global.connection) return global.connection
  if (!global.connectionPromise) {
    global.connectionPromise = MongoClient.connect(uri, {
      useNewUrlParser: true,
      useUnifiedTopology: true,
    })
  }
  const client = await global.connectionPromise
  const db = await client.db(dbName)
  global.connection = {
    client,
    db,
  }
  return global.connection
}

Consistently dropped the number of connections to MongoDB during Next.js dev mode. Although this was without modifying any source code and triggering hot-reload etc - just browsing my Next.js site

There's still something a little iffy that I don't quite understand

[mattcarlotta]

tldr: A good (best?) solution that I've found to integrate a database has been to just create a separate node process for the API (aka a microservice). I've found the provided Next API to be woefully inflexible when you start using getStaticProps with getStaticPaths .

Problems integrating with a database

Problem 1.) Duplicate connections. Using the global solution above or declaration caching still results in duplicate connections when navigating back and forth between the same dynamic page with getStaticPaths and getStaticProps:

It not only creates duplicate connections in development on each page transition, but also (sometimes) creates gSProps n usage + dynamic page n connections in production.

If your project is only using getServerSideProps or client-side requests, then this wouldn't be an issue; however, in that case, instead of polluting the global name space, I'd recommend using Next's serverRuntimeConfig and using that as a persistent variable. It not only works in development and production, but it hints at where it's being defined (within next.config.js) and where it's supposed to be used (on the server). Again, only useful for gSSP and client-side requests as it won't work with gSProps and gSPaths!

Problem 2.) You can't build a project when you use a custom express server for Next/API. While using a custom server solves the duplicate connections issue (since you're establishing a connection before loading the app), it makes it impossible to build the project for production because you can't make API requests to an API that isn't running during build-time:

const app = next({ dev })
const handle = app.getRequestHandler()

(async () => {
  try {
    await connectToDB();
    await app.prepare();

    const server = express();

    middlewares(server);

    server.use("/api", routes);

    server.all('*', (req: Request, res: Response) => handle(req, res));

    server.listen(PORT, (err?: Error) => {
      if (err) throw err;
      console.log(`Serving app on: \x1b[1m${HOST}\x1b[0m`);
    });
  } catch (err) {
    console.log(err.toString());
    process.exit(1);
  }
})();

In addition, if you use node to start this server, then you'll have start and stop the entire app every time you make a change in the API. Using nodemon will cause hot-reloading to break and do a hard refresh. In addition, navigation to dynamic pages is painfully slow. That said, if Next could be configured to spin up the custom server before building, then this would be a viable option.

Problem 3.) Middlewares. Perhaps you want to run some sort of middlewares before each DB request, while it's possible to do in a Next API route, its rather clunky (such as having to wrap each route with several callback handlers). This results in a lot of WET code.

Problem 4.) All your eggs in one basket. If the process crashes, all services become unavailable. In addition, since all the resources share the same process, multiple complex DB queries could hog resources and result in slower initial page requests.

Recommendation

Save yourself the unnecessary headache and create a separate node process for the API and establish a connection before spinning up the servers. This eliminates having to cache your connection and do connection checks for every API request. This also allows your API to be more flexible in terms of hosting options and resource management. It also spreads out point of failures (if the API crashes, it doesn't take down your entire app and vice-versa). This is also easy enough to be used as a standalone microservice within a monorepo (with some semi-complex setup).

Example

Working example

Click to expand monorepo configuration

├── .next (app dev/prod assets)
|
├── api (api src)
|   ├── controllers 
|   ├── database
|   ├── middlewares 
|   ├── models
|   ├── routes
|   └── server.ts 
| 
├── build (api prod assets)
|
├── public
|   ├── favicon.ico
|   └── index.html
|
├── src (app src)
|   ├── components 
|   ├── pages
|   ├── styles 
|   ├── types
|   └── utils 
|
├── next.config.js
├── tsconfig.json (used for next)
└── tsconfig.api.json (used for api)

Example stats

Development changes in relation to database connections (triggering both Next and API reloads):

Production requests in relation to database connections (creating dynamic iSSG pages):

Conclusion

While this may not be an all-in-one solution (only utilizing what's available in Next), it solves all the problems above while keeping the monorepo structure, and most importantly, the amount of connections should stay the same despite the number of dynamic pages and visitors increasing over time. Unless I'm missing something, the only drawback would be the time it takes to resolve requests to an API.

[ErnestBrandi]

Is there anything new about this issue ? Even with caching the connection object, I'm hitting around 50-80 connections for only one user browsing my website...

[eric-burel]

There is probably something wrong happening in your code, the proposed solution are working rather well. Do you close the connection in case of failure? Do you have more info about your code?

[ErnestBrandi]

No I don't close connections, but checking the logs there is no error happening anyway. My solution is very much alike the one proposed by @sambhav-gore : #12229 (comment) (I'm using mongoose but I have useUnifiedTopology: true).

I am not sure what to expect anyway; there is 4 main API endpoints called while browsing, if default connection pull size is 5 it should be around 20 connections for one user I guess.

I can see through logs that connection.isConnected === true therefore mongoose.connect is not called on every API call, but the number of connections on MongoDB Atlas profiler is still huge !

[akung0324]

@ErnestBrandi - If you are still having problems could you log into https://www.mongodb.com/cloud and the click the in-app chat icon in the lower right corner to chat with a MongoDB Support representative? Then please ask for the chat to be assigned to Angela - I'm on the Atlas team and would like to help you track this down.

[mousedown]

mattcarlott's is a great solution. Thanks for that example. i really did consider going down this route, but I'd already invested time wrestling with it, and was determined to see it through. I also wanted to keep the API logs within Next's dashboard. For anyone struggling with lingering connections and wanted to keep API within, they can be kept to an absolute minimum with these key changes:

1. Ensure the connection is cached, and that you deal with the promised connection correctly. - see Next example
   [https://github.com/vercel/next.js/blob/canary/examples/with-mongodb/util/mongodb.js] (https://github.com/vercel/next.js/blob/canary/examples/with-mongodb/util/mongodb.js)

2. Ensure you have configured the client to kill idle connections. Where maxIdleTimeMS isn't supported, socketTimeoutMS will do. I use

3. Make sure you close all cursors. Previously , I was letting Mongo server do this.. but it was woeful at that. Now I make sure I 1st store the response , then manually close the cursor on each end point.

A lot of trial, error and swearing has gone into this, but i am now very happy. I hope it helps somebody else.

next-connect - 0.9.1 (For middleware)
next - 10.0.6
mongodb - 3.6.4 (Beta 4 also good)

[kayk1m]

Impressive!! Is there any same way to close cursor for findOne or updateOne methods?

or I just don't need to close them..?

[ErnestBrandi]

Those functions return a single document so there is no need to handle cursors.

[kayk1m]

thx :)

[nath-green]

I've had pretty good luck with the above solutions until I tried analysing my build with @next/bundle-analyzer. Every time I run the analyzer, my connections spike and never come down. I maxed out the 500 and it never dropped. I tried redeploying (which failed as there weren't any connections) and even upgraded from M0 to M2. I ended up deploying a fresh site that was essentially empty and had no api routes (still the connections remained). Not sure if its related but I restarted my laptop and instantly the connections cleared 🤷 Running the build locally isn't somehow keeping the connections?

I have some pages with getStaticProps that get data from Atlas. Do these connections need to be closed? As they will be called when the pages are built but then not reused? Has anybody else dealt with this before? I don't know if its related or more a generic pattern I need to follow?

[UXMcCauley]

The cursor.close() approach has been BY FAR the most helpful on this topic for me. I would constantly be cruising at 400+ connections and having to restart my app constantly and now I am under 40 connections almost always.

[ckemmler]

Don't know if that's going to be of help to anyone as this is a very old question, but this post helped me solve my original issue: just using the filesystem to cache an accessToken is doing the trick for me:

let accessToken: string | null | undefined;

try {
  accessToken = fs.readFileSync(
    path.join("/tmp", ACCESS_TOKEN_CACHE_PATH),
    "utf8"
  );
} catch (error) {
  console.log("AccessToken not initialized");
}

export const generateAuthHeader = async (refresh: boolean) => {
  const then = Date.now();
  if (!accessToken || accessToken === "" || refresh) {
    console.log("logging in again... refresh? ", refresh);
    await app.logIn(credentials);
    accessToken = app.currentUser?.accessToken;

    try {
      fs.writeFileSync(
        path.join("/tmp", ACCESS_TOKEN_CACHE_PATH),
        accessToken || "",
        "utf8"
      );
      console.log("Wrote to members cache");
    } catch (error) {
      console.log("ERROR WRITING ACCESS TOKEN TO FILE");
      console.log(error);
    }

    console.log(`-> ${Date.now() - then}`);
  } else {
    console.log("already logged in");
  }
  // Get a valid access token for the current user

  // Set the Authorization header, preserving any other headers
  return {
    Authorization: `Bearer ${accessToken}`,
  };
};

For the record, I am using Realm with GraphQL queries

[mustafaKamal-fe]

Wish me luck here please:

THIRD TIME EDITED !!

The goal of this comment/solution is:

• I would like to create as much mongoose connections as i would like to since my app deals with different databases
• I do not want to cause performance issues due to creating multiple useless database connections

I am using NEXTjs to build my API routes. Hence, my routes on production will be dealt with using LABDA Functions. This means i do not have server.js file to start my server logic (including intializing databse connections once at the start).

See this helper below. It opens new connection to a database every time i call.
Example: useDb('some-db');

Now, I don't want to get two open connections at the same time for the same databse:

useDb('my-db'); // open connection to my-db
useDb('my-db'); // open ANOTHER connection to my-db;

I created this helper that does the following:

1. Get all current OPEN databse connections within mongoose instance AND ignore
2. The first filter function seeks for connections with actuall databse within it (That is an actuall DB connection to some Database). Because mongoose can initialize a connection before actually open it. Again that is you may initialize a connection with const conn = mongoose.createConnection() then open it with conn.openUri('url-to-db-you want-to-open');'. Hence, the difference here if you inspect any connection object, you can find out if it is open by having conn.db.databaseName equals the database name & conn.readyState equals 1 (connected).
3. Now the helper returns all open connections databses as an array of strings containing databse names.
4. Check if db, the databse name i would like to open is alreay included within that array.
5. If only this db is not included in that array (that is no open connection to it), then you can go ahead and open one with the whatever configuration you may need.
   6- Otherwise, return the same connection since it's open anyway.

const useDb = async (db: string) => {
	let dbs: string[] = mongoose.connections
		.filter((conn) => {
			return conn.db && conn.db.databaseName;
		})
		.map((conn) => {
			return conn.db.databaseName;
		});

	if (!dbs.includes(db)) {
	const devURL = `mongodb://localhost:27017/${db}`;
		const prodURL = `mongodb+srv://${process.env.ATLAS_ADMIN}:${process.env.ATLAS_PWD}@sandbox.rkgga.mongodb.net/${db}?retryWrites=true&w=majority`;
		const config = {
			useNewUrlParser: true,
			useUnifiedTopology: true,
			useFindAndModify: false,
			maxPoolSize: 10,
			minPoolSize: 4,
			connectTimeoutMS: 5000,
			socketTimeoutMS: 3000,
			serverSelectionTimeoutMS: 10000,
			maxIdleTimeMS: 10000,
		};
		if (process.env.NODE_ENV === 'development') {
			return await mongoose.createConnection(devURL, config);
		} else {
			return await mongoose.createConnection(prodURL, config);
		}
	} else {
		let connPosition = dbs.indexOf(db);
                let conn = mongoose.connections[connPosition];

                if(conn.readyState === 1) {
                   return conn;
                }
                
                while (conn.readyState !== 1) {
                    console.log('Waiting to open a new connection...');
                };
                    if(process.env.NODE_ENV === 'development' ) {
                        return await mongoose.createConnection(devURL, config);
                    } else {
                        return await mongoose.createConnection(prodURL, config);
                    }
                
	}
};

NOW PLEASE, Correct me if i am wrong, Educatme if you have a better way to do this.

[mustafaKamal-fe]

Experienced devs, your elaboration is highly appreciated. BTW, in case there is not any article/guide about hwo to master proper connections with mongoose, I think we might need to start writing one!!

[Vrq]

We are following this example: https://github.com/vercel/next.js/tree/canary/examples/with-mongodb-mongoose but hit the connection limit with getStaticProps and having a few thousand pages from getStaticPaths

Has anyone from the @vercel-infra @dav-is @matheuss Team actually checked if this is the right way to go?
I assume the problem is that NextJS creates multiple serverless functions instances and while the caching works within a single instance, it won't work when you create multiple (because they completely independent)

[0xmovses]

Exactly what I've been wondering, I'm pretty sure this code does not work for real world apps. Terrible to leave this in the examples folder

[daveteu]

Understanding your infrastructure is the first step to solving problems. Knowing that vercel runs on Lambda, the solution to the problem is simply to manually open the connection whenever you need them, and close them when done.

[Vrq]

@daveteu the problem is that even though everyone knows about the problem, no one really provided a solution.
Unfortunately, manually closing the connection does not help if you run with >500 requests per second.

[daveteu]

@Vrq if you run with >500ms then you should probably upgrade to the Pro plan so you can choose the location of the lambda. You can also host your mongo at a nearby location to cutdown on connection time.

[sajalarora007]

Has anyone able to come up with any proper solution ? I am using Next's native API handlers to make API calls and my app is deployed on Vercel itself. So I use a DB middleware before each API call like below

// pages/api/sample.js
import db from 'middleware/db';
export default nextConnect().use(db).get(somehandler);

// db.js
let cached = global.mongoose;

if (!cached) 
  cached = global.mongoose = { conn: null, promise: null };

export default async function (request, response, next) {
  if (cached.conn && next)
    return next();

  if (cached.conn) 
    return cached.conn;

  if (!cached.promise) {
    const options = {
      useNewUrlParser: true,
      useUnifiedTopology: true,
      useFindAndModify: false,
      bufferCommands: false
    };

    cached.promise = mongoose.connect(MONGODB_URI, options).then(mongoose => {
      return mongoose;
    });
  }
  cached.conn = await cached.promise;
  return next ? next() : cached.conn;
}

I am reusing the above middleware function also for the ISR purpose (getStaticPaths & getStaticProps) to persist the cached connection but even after that I am constantly receiving emails from mongodb atlas that my connections reached insane 500.

Below is a sample function called by getStaticProps and getStaticPaths.

// helpers.js
import dbConnect from './db.js';

export async function getPosts() {
     await dbConnect();
     const posts = await Post.find({});
     return posts;
}

[Vrq]

@eric-burel does it also work on Vercel, or what are you using for the production deployment?
From what we see, the connection is not cached with this solution because Vercel spawns completely separate Lambda invocations, which do not share any state.

[sajalarora007]

Yeah, that's really important to know. I don't see much difference in the development mode after upgrading to Mongoose 6 honestly.

[eric-burel]

I host on Vercel yes.

You are right, this caching does not extend beyond the Lambda itself. So 3 lambdas = 3 connections. However as far as I can tell, that's the best you can do codewise. To share connections between lambdas, you need another infrastructure that is dedicated to managing connections. It's expected that a serverless will spawn as many connections as there are lambdas, since after all they are separate intances of the server.

The problem for most people, is that the connection might be duplicated even when using one lambda, because you can call it concurrently. Mongoose doesn't seem to handle that out of the box (calling connect 3 time seems to create 3 connections), and the raw Mongo connector won't do anything either as expected. This code is expected to solve this issue.

[eric-burel]

@sajalarora007 I have small doubts about your code, or at least I don't fully understand it from a quick read. It took me a while and multiple iterations to figure the "await" right in my versions. I am not certain that your code handles the state where the promise is cached, but not yet resolved.

[sajalarora007]

@eric-burel So I am using it exactly how it's over here https://github.com/vercel/next.js/blob/canary/examples/with-mongodb-mongoose/lib/dbConnect.js But did you get a chance to deploy this code on vercel yet ?

[eric-burel]

Small up on this: has anyone tried to connect to Mongo not in an API route, but in a middleware?
It ssems to be a quite tricky issue, see this discussion with Redis for instance: #34108

[leerob]

You can use Mongo's Data API.

https://www.mongodb.com/docs/atlas/api/data-api-resources/

[eric-burel]

Excellent! This is specific to the host though, it seems to be the same pattern with Redis+Redis lab (https://docs.redis.com/latest/rc/api/get-started/use-rest-api/) or with Upstash as you show in Vercel edge handler examples.

[iamsoltan]

what I detected , is that when we use next-auth , connections problems will persist !
So in my opinion,
we need to use it either with token approach only , then we will handle user data persistency with data-api , or we should use a complete other DB like aws DynamoDB ( a complete http based solution like data-api but with fully managed adapter with next-auth ).

[daniellin215]

@iamsoltan what is the impact of your connection issue? I hit connection limit of 500 easily on local host nextjs, but when deployed it does not have this problem. I am using the outlined best practices but also using next-auth. And the serverless function cold start times do affect my requests response times to about 2-3 seconds on first load which is a bad experience.

[zeemo25]

This is a known issue in nextjs api routing. See the path forward they provide here (albeit for a different db) https://www.prisma.io/docs/guides/database/troubleshooting-orm/help-articles/nextjs-prisma-client-dev-practices

[apsquared]

Has anyone found a suitable solution. This issue is causing sporadic issues with my users causing my project to stall. I do not appear to ever come close to 500 connections (as seen by MongoDB) but have at least 10-20 UnhandledPromiseRejection errors a day (with pretty low traffic).

[boredland]

It's 2024 and I think there still isn't a proper solution. In the postgres word, pg_bouncer and other external poolers make this work. For MongoDB, I know of no such thing.

[Vrq]

That's why we dropped using Next.

[boredland]

I'd rather drop Vercel and/or mongodb, but yes, this issue seems to send us down that road.

[boredland]

there actually is an external connection pooler, which seems to work quite nicely. Sadly it doesn't support authentication, so there isn't a way to make it available publicly (which is needed with vercel):
https://github.com/coinbase/mongobetween

[Luckysolanki902]

Hey, I resolved the MongoDB connection issue by configuring the connection pool limit to 1 and setting serverSelectionTimeoutMS to 5000 (to fail fast if no server is found). The code is working well for MongoDB now. Here's the updated snippet:

import mongoose from 'mongoose';

const { MONGODB_URI } = process.env;

if (!MONGODB_URI) {
  console.error('MONGODB_URI environment variable not defined.');
  throw new Error('Please define the MONGODB_URI environment variable.');
}

// Global cache to prevent multiple connections in development mode
let cached = global.mongoose;
if (!cached) {
  cached = global.mongoose = { conn: null, promise: null };
}

// Configuration to manage connection pooling and timeouts
const options = {
  bufferCommands: false,
  maxPoolSize: 1,
  minPoolSize: 1,
  serverSelectionTimeoutMS: 5000,   // Fail fast if no server is found
  socketTimeoutMS: 30000,          // Socket idle timeout
  heartbeatFrequencyMS: 10000,     // Ping server every 10 seconds
  maxIdleTimeMS: 10000,            // Reap idle connections after 10 seconds
  waitQueueTimeoutMS: 5000,        // Timeout for connection queue
};

async function connectToDatabase() {
  // Use existing connection if it's already established
  if (cached.conn && cached.conn.readyState === 1) {
    return cached.conn;
  }

  // Wait for an in-progress connection if one exists
  if (cached.promise) {
    cached.conn = await cached.promise;
    return cached.conn;
  }

  // Create a new connection if none exist
  console.info('MongoDB: Initiating a new connection...');

  cached.promise = mongoose.connect(MONGODB_URI, options).then((mongooseInstance) => {
    console.info('MongoDB: Successfully connected.');
    return mongooseInstance;
  }).catch(err => {
    console.error(`MongoDB: Connection error - ${err.message}`);
    cached.promise = null;
    throw err;
  });

  cached.conn = await cached.promise;
  return cached.conn;
}

export default connectToDatabase;

After hours of trying, this finally drastically reduced the number of connections.

While this resolved the MongoDB connection issue, I'm still experiencing problems with other APIs on the serverless platform, particularly when streaming files from AWS for downloads. Considering these issues, I’m thinking of switching to Railway. Currently, I'm on Vercel’s paid plan. Can someone please estimate the approximate cost on Railway for a similar setup?