Skip to content

Painless deployment of your favorite MLOps technologies with Docker - Grafana, Prometheus, Airflow, Spark, Postgres, Redis, NPM, Traefik and ...

License

Notifications You must be signed in to change notification settings

keivanipchihagh/easy-peasy-deployment

Repository files navigation

Easy Peasy Deployment

Painless deployment of your favorite technologies in a snap!

Table of Contents

Toolkit Technologies

Default Grafana Dashboards

All credits goes to the creators of these dashboards. I just tweeked them a bit to make them easier to work with.

Setup

  1. Install Docker: Install Docker Engine and docker-compose.
  2. Setting up the .env: Create a .env file in the root directory (so that all the modules can access it, instead of having a seperate .env for each module) that looks something like the following:
# Grafana
GRAFANA_CONTAINER_NAME=grafana
GF_PORT=3000
GF_SECURITY_ADMIN_USER=<username>
GF_SECURITY_ADMIN_PASSWORD=<password>
GF_USERS_ALLOW_SIGN_UP=true
GF_AUTH_DISABLE_LOGIN_FORM=false

# Prometheus
PROMETHEUS_CONTAINER_NAME=prometheus
PROMETHEUS_PORT=9090

# PostgreSQL
POSTGRES_CONTAINER_NAME=postgres
POSTGRES_HOST=<machine-ip>
POSTGRES_PORT=5432
POSTGRES_DB=postgres
POSTGRES_SSL=disable
POSTGRES_USER=<username>
POSTGRES_PASSWORD=<password>
POSTGRES_EXPORTER_PORT=9187

# Redis
REDIS_CONTAINER_NAME=redis
REDIS_HOST=<machine-ip>
REDIS_PORT=6379
REDIS_PASSWORD=admin
REDIS_EXPORTER_PORT=9121

# Node-Exporter
NODE_CONTAINER_NAME=node
NODE_PORT=9100

# Cadvisor
CADVISOR_CONTAINER_NAME=cadvisor
CADVISOR_PORT=8080

# Portainer
PORTAINER_CONTAINER_NAME=portainer
PORTAINER_PORT=9000
PORTAINER_AGENT_PORT=9001

# Nginx-Proxy-Manager
NPM_CONTAINER_NAME=nginx-proxy-manager
NPM_PORT=81
  • Note 1: most of the variables spesified above have default values in-case you forget to initialize them, such as names and ports; However, that is not the case with sensitive variables such as usernames and passwords, for security reasons!
  • Note 2: By default, all exporters and agents that are dependent to a parent container are named with the prefix of the parent container for ease of identification. So just change the parent container's name and the children will follow! (eg. <parent-container> & <parent-container>-exporter, <parent-container> & <parent-container>-agent)

2.5. Replace <machine-ip> in prometheus/prometheus.yml with your .

  1. Building the toolkit: Every module comes with a build.sh. Just cd to the directory, run it with bash build.sh and that's it!

Contributing

Since this an ongoing project, you can contribute by adding more exporters or dashboards. You can also add more modules to the project. If you have any questions, feel free to open an issue.