A Simple API Powered by Snowflake

Technologies used: Snowflake, Python, Flask, AWS API Gateway, AWS Lambda, Serverless Framework

Requirements:

• Snowflake.com and Serverless.com account
• node.js, python 3, virtualenv installed
• Citibike data loaded into Snowflake
• Snowflake user authorized to access citibike data with key pair authentication

This project demonstrates how to build and deploy a custom API powered by Snowflake. It uses a simple Python Flask API service running on AWS Lambda using Serverless Framework. Connectivity to Snowflake is made via key pair authentication.

Configuration

Copy the serverless-template.yml to serverless.yml and modify the parameters according to your Snowflake configuration. Put your private key to your Snowflake user in AWS SSM is us-west-2 region under the parameter .DATA_APPS_DEMO.

Install serverless and other required node packages and configure serverless (sls) for the project.

npm install
sls login

Create a virtualenv locally and install python packages.

virtualenv venv --python=python3
source ./venv/bin/activate
pip install -r requirements.txt

Local Development

For local development you will want to use the venv previously created. This will run a local application server and connect to your Snowflake account for data access.

Start the local serverless server.

sls wsgi serve

Invocation

After successful startup, you can call the created application via HTTP:

curl http://localhost:5000/

Which should result in the following response:

{"result":"Nothing to see here", "time_ms": 0}

Deployment

Build and deploy the application to AWS. For your first time, you will have to run sls without deploy to configure the project.

sls deploy

Invocation

After successful deployment, you can call the created application via HTTP:

curl https://xxxxxxx.execute-api.us-west-2.amazonaws.com/dev/

Which should result in the following response:

{"result":"Nothing to see here", "time_ms": 0}

Scaling

By default, AWS Lambda limits the total concurrent executions across all functions within a given region to 1000. The default limit is a safety limit that protects you from costs due to potential runaway or recursive functions during initial development and testing. To increase this limit above the default, follow the steps in To request a limit increase for concurrent executions.