Lambda Best Practices: EFS as Persistent Storage for functions

Mystique Unicorn App is a building new application based on microservice architectural pattern. One of their microservices need to have access to a persistent storage layer.

They have created this use-case as an analogy to explain their requirement. Let us say, we have a "Message Board", where any user can post messages and read messages of other users. This means, we need to store the messages from all the functions and generate be able to list/view of the messages. Ofcourse, we can use a database, but that is not the point there, "How can we use a persistent storage to share data among functions?"

Can you help them do that in Amazon API Gateway & AWS Lambda?

🎯Solutions

We can use AWS Elastic File System with AWS Lambda to achieve this.

In this article, we will build an architecture, similar to the one shown above - A simple message wall API using API Gateway and Lambda. Any POST message send to the api will be stored in the EFS share and any subsequent GET requests will return all the messages in the wall.

1. 🧰 Prerequisites

   This demo, instructions, scripts and cloudformation template is designed to be run in us-east-1. With few modifications you can try it out in other regions as well(Not covered here).

   • 🛠 AWS CLI Installed & Configured - Get help here
   • 🛠 AWS CDK Installed & Configured - Get help here
   • 🛠 Python Packages, Change the below commands to suit your OS, the following is written for amzn linux 2
     • Python3 - yum install -y python3
     • Python Pip - yum install -y python-pip
     • Virtualenv - pip3 install virtualenv

2. ⚙️ Setting up the environment

   • Get the application code

     git clone https://github.com/miztiik/lambda-with-efs
     cd lambda-with-efs

3. 🚀 Prepare the dev environment to run AWS CDK

   We will cdk to be installed to make our deployments easier. Lets go ahead and install the necessary components.

   # If you DONT have cdk installed
   npm install -g aws-cdk
   
   # Make sure you in root directory
   python3 -m venv .env
   source .env/bin/activate
   pip3 install -r requirements.txt

   The very first time you deploy an AWS CDK app into an environment (account/region), you’ll need to install a bootstrap stack, Otherwise just go ahead and deploy using cdk deploy.

   cdk bootstrap
   cdk ls
   # Follow on screen prompts

   You should see an output of the available stacks,

   vpc-stack
   efs-stack
   lambda-with-efs

4. 🚀 Deploying the application

   Let us walk through each of the stacks,

   • Stack: efs-stack This stack will create the Amazon EFS. There are few resources that are prerequisites to create the EFS share. This stack will create the following resources,

     • A VPC to host our EFS share - Deployed by the dependant stack vpc-stack
     • Security group for our EFS share allowing inbound TCP on ort 2049 from our VPC IP range
     • Posix user & acl 1000 - In case you want to use OS level access restrictions, these will come in handy
     • EFS Access Point to make it easier to mount to Lambda and apply resource level access restrictions
       • The default path for the access point is set to /efs

     Initiate the deployment with the following command,

     cdk deploy efs-stack

   • Stack: lambda-with-efs

     This stack: lambda-with-efs creates an REST API with a lambda backend. This lambda function will be deployed in the same VPC as our EFS share and use the same security group(TODO: Host lambda in a independant security group). The stack mounts the EFS Access point to our lambda function, there-by enabling us to read and write to our EFS share.

     Initiate the deployment with the following command,

     cdk deploy lambda-with-efs

     Check the Outputs section of the stack to access the GreetingsWallApiUrl

5. 🔬 Testing the solution

   We can use a tool like curl or Postman to query the url. The Outputs section of the respective stacks has the required information on the urls.

   $ GREETINGS_WALL_URL="https://2s9p0x3p53.execute-api.us-east-2.amazonaws.com/prod/lambda-with-efs/greeter"
   $ curl ${GREETINGS_WALL_URL}
   No message yet.
   $ curl -X POST -H "Content-Type: text/plain" -d 'Hello from EFS!' ${GREETINGS_WALL_URL}
   Hello from EFS!
   
   $ curl -X POST -H "Content-Type: text/plain" -d 'Hello again :)' ${GREETINGS_WALL_URL}
   Hello from EFS!
   Hello again :)
   
   $ curl ${GREETINGS_WALL_URL}
   Hello from EFS!
   Hello again :)
   
   $ curl -X DELETE ${GREETINGS_WALL_URL}
   Messages deleted.
   
   $ curl ${GREETINGS_WALL_URL}
   No message yet.

   You should be able observe that we were able to read, write & delete data from the EFS share.

6. 📒 Conclusion

   Here we have demonstrated how to use EFS along with AWS Lambda to create a persistent storage for your functions. This can be really helpful in a variety of situations. For example,

   • If you are running machine language inference, lambda internal storage and layers might not be enough to host all the dependant libraries. In those cases an external storage becomes a necessity.
   • Another usecase is when you want to process really huge files - unpack/zip them. Then the extra scratch space offered EFS comes handy.

   If you know of other usecases for using EFS with lambda, do let me know.

1. 🧹 CleanUp

   If you want to destroy all the resources created by the stack, Execute the below command to delete the stack, or you can delete the stack from console as well

   • Resources created during Deploying The Application
   • Delete CloudWatch Lambda LogGroups
   • Any other custom resources, you have created for this demo

   # Delete from cdk
   cdk destroy
   
   # Follow any on-screen prompts
   
   # Delete the CF Stack, If you used cloudformation to deploy the stack.
   aws cloudformation delete-stack \
       --stack-name "MiztiikAutomationStack" \
       --region "${AWS_REGION}"

   This is not an exhaustive list, please carry out other necessary steps as maybe applicable to your needs.

📌 Who is using this

This repository aims to teach best practices & advanced file system techniques to new developers, Solution Architects & Ops Engineers in AWS. Based on that knowledge these Udemy course #1, course #2 helps you build complete architecture in AWS.

💡 Help/Suggestions or 🐛 Bugs

Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional documentation or solutions, we greatly value feedback and contributions from our community. Start here

👋 Buy me a coffee

Buy me a coffee ☕.

📚 References

1. Developers guide to using Amazon EFS with Amazon ECS and AWS Fargate – Part 1

2. AWS Blog & Use Lambda & EFS to process big files

3. Use Lambda & EFS to Update Fargate Web Content

4. Use Lambda & EFS I/O Performance

5. Lambda & EFS to run ML Inference

🏷️ Metadata

Level: 300