Agile Development with Azure Project: Building a CI/CD Pipeline

Overview

This demo project demonstrates the steps of automation with DevOps and the overall guidelines of the CI/CD pipeline. This repo represents an ML web application that is run, built, tested, and deployed through GitHub action and azure pipelines. this project depicts the important minimum steps of DevOps integration and continuous improvement (KAIZEN).

This repository is demonstrated with instructions as follows:

• Project Plan
• Trello board
• Integrating with Github Action
• Azure Pipeline setup
• Load Testing
• Log viewer
• A demo descriptive video

The overall architecture of this project is as follows:

Project Plan

• Quarterly plan Qaterly plan
• Weekly project plan for a complete year(:anger: With complexity indication by color :anger:) Weekly Plan

Trello

• Project workspace on Trello

#Instraction

Creating a repository in Github

Using your GitHub account first create a repo with azure pipeline enabled. Azure pipeline could be installed through Github Marketplace

1. Sign in to you azure account at (https://portal.azure.com)
2. Open Azure CLI and select the Bash from the cloud Shell window

3. Clone the repo into the azure cloud

mahfuzur@Azure:~$ git clone git@github.com:mahfuzmr/DevOpsCICD.git
mahfuzur@Azure:~$ cd DevOpsCICD

4. Run command to create virtual environment for your application. Then activate the created python environment

python3 -m venv ~/.flask-ml-azure
source ~/.flask-ml-azure/bin/activate

5. Run the command to install all dependencies from the requirements.txt file

make install

And the output should look like the follows:

Integration with Github Action

When the system is successfully installing all the dependencies from the requirements.txt file in step 5. It is time to create CI through GitHub Action

• After login to the created repo Click on the Action Tab and "Set up a workflow". This will create a .yml file that has the pre-written code for automation

• And when we create the .yml file and commit the build automation will be automatically triggered. The successful build automation will look as follows:

6. Create an app service which will deploy the app in Cloud Shell:

-az webapp up -n my-ml-app-service

7. Update the "URL" in the file make_predict_azure_app.sh

-X POST https://<**my-ml-app-service**.azurewebsites.net:$PORT/predict> 

9. Set your app deployed and running in a separate web browser. If the deployment is successful is will show the initial message in the browser

10. Now its time to test locally

• Build the app using app.py

   python app.py

A successful build will show as in the following depiction

• In a separate browser tab open bash from the azure CLI again and run command to check the prediction locally

A successful prediction will look like this:

Azure pipeline

At step 9 our application is successfully integrated with the azure app service, GitHub action, and generating prediction. We will now move to our continuous delivery process with the azure pipeline. The ecosystem of azure pipeline could be understand by DevOps Ecosystem

10. At this time our system is ready to go on the pipeline process. The following steps will create a successful CD process.

• Inthe azure portal create a new project
• From the setting menu connect the specific Github Repo
• Create a service connection based on the create appservice
• Head to Pipelines and click on 'New pipeline'. Update the code for created service connection
• Configure the pipeline in the .yml file.
• Based on the latest commit Azure pipeline will create a starter basic YAML file (default)
• When we save the file it will create pipeline JOB and run the process. A successful job build will look as follows:

#Now for every commit, git hub will trigger CI in Github actions and Azure pipeline will do CI and deliver the update into the webapp service(CD) through Azure Pipeline success.

Load Test

Now its time to load test. We can ensure the performance validation throug a open source load testing tools Locust. This will do GET and POST to our app service with some sample data to test the performance of our application/ appservice

• Install locast in the project file

  pip install locust

• Pust the sample data into the newly created locustfile.py file

• Set the calling parameter by selecting 20 user with 5 row as a chunks for 20 second run time.

(.myrepo) mahfuzur [ ~/DevOpsCICD ]$ locust -f locustfile.py --headless -u 20 -r 5 -t 20s

A successfull load testing will look like as :

Logs

Logs can be visible for the running app service in two different ways

1. Through azure provided api end point (https://my-ml-app-service.scm.azurewebsites.net/api/logs/docker) or
2. Running command on the cloud Shell

   (.myrepo) mahfuzur [ ~/DevOpsCICD ]$ az webapp log tail

Work flow Demo

Video demonstration