Devops Workshop 3: Log Management

Welcome to the Devops Workshop 3! In this hands-on workshop, we will explore what is log management, why is it important and the power of the ELK Stack (Elasticsearch, Logstash, Kibana) as part of our DevOps journey. Throughout this workshop, you will learn how to harness the capabilities of ELK Stack to centralize, parse, analyze, and visualize logs generated by your applications.

Prerequisites

• Machine/VM with ubuntu 22.04
• Docker
• Git
  • Create github account
  • Setting up SSH key with GitHub for Ubuntu
• VS code IDE
• Stable Internet Connection and any browser.

Workshop environment setup

• Check if Git, Docker, and Docker Compose are installed in on the system. Open the terminal and run the following command

  $ git --version
  git version 2.25.1
  
  $ docker --version
  Docker version 20.10.17, build 100c701
  
  $ docker compose version
  Docker Compose version v2.6.0
  

• Open terminal and run following command to create a folder called workshop

  $ mkdir workshop

• Navigate to the folder workshop and clone the from your personal repo using git

  $ cd workshop

• Clone DevOps-Workshop3 repo && go inside DevOps-Workshop3 folder

  $ git clone git@github.com:UniCourt/DevOps-Workshop3.git
  $ cd DevOps-Workshop3

• To open folder in VS code editor

  $ cd ~/workshop/DevOps-Workshop3
  $ code .

Install Docker

• If docker is not installed in your Linux run the following command

  $ sudo apt-get update
  $ sudo apt-get install -y curl 
  $ curl -fsSL https://get.docker.com -o get-docker.sh
  $ sudo sh get-docker.sh

• Do docker login

  $ sudo docker login -u username --password-stdin

Docker without sudo

• To run docker commands as normal user without sudo, we need to create a group for docker and add the user to it.

1. Create the docker group

   $ sudo groupadd docker

2. Add your user to docker group

   $ sudo usermod -aG docker $USER

3. Activate the changes to groups:

   $ newgrp docker

4. Verify that you can run docker commands without sudo.

   $ docker images

Workshop Goals:

1. Log Management: Discover how to effectively manage logs in a centralized environment with Elasticsearch.
2. Log Parsing: Use Logstash to intelligently parse logs generated by the applications.
3. Data Visualization: Harness the power of Kibana to create stunning visualizations and dashboards. Make your logs accessible and easy to analyze.

Schedule

Time	Topics
09:00 - 09:30	Why do we need logs? Importance of Log Management
09:45 - 10:00	The ELK stack
10:00 - 10:45	Creating an web application with logs
10:45 - 11:30	Configure logstash
11:30 - 12:30	Setting up ELK stack using docker-compose
12:00 - 01:00	Visualing the logs in kibana
01:00 - 02:00	[Break]
02:00 - 03:30	Adding more configurations for logstash
03:00 - 03:30	Elasticsearch APIs
03:30 - 03:45	Best practices
03:45 - 04:00	[Q & A]
04:00 - 04:10	[Wrapping Up]