- Jenkins: Continuous Integration and Continuous Deployment (CI/CD)
- Docker: Containerization
- Kubernetes: Container Orchestration
- Prometheus: Monitoring and Alerting
- Grafana: Visualization and Dashboards
- SonarQube: Static Code Analysis
- OWASP Dependency-Check: Dependency Vulnerability Scanning
- Trivy: Container Image Vulnerability Scanning
- Node Exporter: System Metrics Collection
-
Initial Setup and Deployment
- Launch EC2 Instance
- Clone Application Code
- Install Docker
- Create Dockerfile
- Get the API Key
- Build Docker Image
-
Security Scanning
- Install SonarQube and Trivy
- Integrate SonarQube with CI/CD Pipeline
- Install OWASP Dependency Check Plugins in Jenkins
- Configure Dependency-Check Tool
-
Continuous Integration and Continuous Deployment (CI/CD) with Jenkins
- Install Jenkins
- Install Necessary Plugins in Jenkins
- Configure SonarQube Server in Jenkins
- Configure CI/CD Pipeline in Jenkins
- Add DockerHub Credentials in Jenkins
- Configure Dependency-Check and Trivy Scans in Pipeline
-
Monitoring Setup
- Install Prometheus
- Install Node Exporter
- Configure Prometheus to Scrape Metrics
- Install Grafana
- Add Prometheus Data Source in Grafana
- Import Pre-configured Dashboards in Grafana
- Configure Prometheus Plugin Integration in Jenkins
-
Kubernetes Setup
- Install Kubectl on Jenkins Machine
- Setup Master and Worker Instances
- Initialize Kubernetes on the Master Node
- Join Worker Node to Kubernetes Cluster
- Handle Config Files for Jenkins
- Install Kubernetes Plugins on Jenkins
- Install Node Exporter on Master and Worker Nodes
-Launch an AWS T2 Large Instance using the Ubuntu image. -Configure HTTP and HTTPS settings in the security group.
- Update all the packages - Clone application code repository onto the EC2 instancegit clone https://github.com/N4si/DevSecOps-Project.git
sudo apt-get update
sudo apt-get install docker.io -y
sudo usermod -aG docker $USER
newgrp docker
sudo chmod 777 /var/run/docker.sock
FROM node:16.17.0-alpine as builder WORKDIR /app COPY ./package.json . COPY ./yarn.lock . RUN yarn install COPY . . ARG TMDB_V3_API_KEY ENV VITE_APP_TMDB_V3_API_KEY=${TMDB_V3_API_KEY} ENV VITE_APP_API_ENDPOINT_URL="https://api.themoviedb.org/3" RUN yarn build
FROM nginx:stable-alpine WORKDIR /usr/share/nginx/html RUN rm -rf ./* COPY --from=builder /app/dist . EXPOSE 80 ENTRYPOINT ["nginx", "-g", "daemon off;"]
- Open a web browser and navigate to TMDB (The Movie Database) website.
- Click on "Login" and create an account.
- Once logged in, go to your profile and select "Settings."
- Click on "API" from the left-side panel.
- Create a new API key by clicking "Create" and accepting the terms and conditions.
- Provide the required basic details and click "Submit."
- You will receive your TMDB API key.
docker build --build-arg TMDB_V3_API_KEY=<your_api_key> -t netflix .
Install SonarQube and Trivy on the EC2 instance to scan for vulnerabilities.
Install SonarQube
docker run -d --name sonar -p 9000:9000 sonarqube:lts-community
To access: publicIP:9000 (by default username & password is admin)
- Integrate SonarQube with your CI/CD pipeline.
- Configure SonarQube to analyze code for quality and security issues.
Install Trivy
sudo apt-get install wget apt-transport-https gnupg lsb-release
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
echo deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update
sudo apt-get install trivy
Trivy now installed on the EC2 instance
Install Jenkins on the EC2 instance to automate deployment
sudo apt update
sudo apt install fontconfig openjdk-17-jre
java -version
sudo wget -O /usr/share/keyrings/jenkins-keyring.asc https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update
sudo apt-get install jenkins
sudo systemctl start jenkins
sudo systemctl enable jenkins
Access Jenkins in a web browser using the public IP of your EC2 instance. publicIp:8080
Eclipse Temurin Installer (Install without restart) SonarQube Scanner (Install without restart) NodeJs Plugin (Install without restart)
Configure Java and Node.js in Global Tool Configuration.
Create a token and add it to Jenkins. Go to Jenkins Dashboard → Manage Jenkins → Credentials
Install a sonar scanner in the tools.
Created a CI/CD pipeline in Jenkins to automate the application deployment
pipeline {
agent any
tools {
jdk 'jdk17'
nodejs 'node16'
}
environment {
SCANNER_HOME = tool 'sonar-scanner'
}
stages {
stage('clean workspace') {
steps {
cleanWs()
}
}
stage('Checkout from Git') {
steps {
git branch: 'main', url: 'https://github.com/N4si/DevSecOps-Project.git'
}
}
stage("Sonarqube Analysis") {
steps {
withSonarQubeEnv('sonar-server') {
sh '''$SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=Netflix \
-Dsonar.projectKey=Netflix'''
}
}
}
stage("quality gate") {
steps {
script {
waitForQualityGate abortPipeline: false, credentialsId: 'Sonar-token'
}
}
}
stage('Install Dependencies') {
steps {
sh "npm install"
}
}
}
}
- Go to "Dashboard" in your Jenkins web interface.
- Navigate to "Manage Jenkins" → "Manage Plugins."
- Click on the "Available" tab and search for "OWASP Dependency-Check."
- Check the checkbox for "OWASP Dependency-Check" and click on the "Install without restart" button.
Configure Dependency-Check Tool:
- After installing the Dependency-Check plugin, you need to configure the tool.
- Go to "Dashboard" → "Manage Jenkins" → "Global Tool Configuration."
- Find the section for "OWASP Dependency-Check."
- Add the tool's name, e.g., "DP-Check."
- Save your settings.
Click on Apply and Save here.
Now go configure → Pipeline and add this stage to your pipeline and build.
stage('OWASP FS SCAN') {
steps {
dependencyCheck additionalArguments: '--scan ./ --disableYarnAudit --disableNodeAudit', odcInstallation: 'DP-Check'
dependencyCheckPublisher pattern: '**/dependency-check-report.xml'
}
}
stage('TRIVY FS SCAN') {
steps {
sh "trivy fs . > trivyfs.txt"
}
}
Install Docker Tools and Docker Plugins:
- Go to "Dashboard" in your Jenkins web interface.
- Navigate to "Manage Jenkins" → "Manage Plugins."
- Click on the "Available" tab and search for "Docker."
- Check the following Docker-related plugins:
- Docker
- Docker Commons
- Docker Pipeline
- Docker API
- docker-build-step
- Click on the "Install without restart" button to install these plugins.
Add DockerHub Credentials:
- To securely handle DockerHub credentials in your Jenkins pipeline, follow these steps:
- Go to "Dashboard" → "Manage Jenkins" → "Manage Credentials."
- Click on "System" and then "Global credentials (unrestricted)."
- Click on "Add Credentials" on the left side.
- Choose "Secret text" as the kind of credentials.
- Enter your DockerHub credentials (Username and Password) and give the credentials an ID (e.g., "docker").
- Click "OK" to save your DockerHub credentials.
##Monitoring Setup##
Set up Prometheus and Grafana to monitor your application.
Created new t.2 medium EC2 instance for prometheus and grafana to run,
sudo useradd --system --no-create-home --shell /bin/false prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.47.1/prometheus-2.47.1.linux-amd64.tar.gz
Extract Prometheus files, move them, and create directories:
tar -xvf prometheus-2.47.1.linux-amd64.tar.gz
sudo mv prometheus-2.47.1.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-2.47.1.linux-amd64/promtool /usr/local/bin/
sudo mv prometheus-2.47.1.linux-amd64/consoles /etc/prometheus/
sudo mv prometheus-2.47.1.linux-amd64/console_libraries /etc/prometheus/
sudo mv prometheus-2.47.1.linux-amd64/prometheus.yml /etc/prometheus/
sudo mkdir /var/lib/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus
Create the prometheus.service
file to define the Prometheus service:
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus/ \
--web.listen-address=0.0.0.0:9090 \
--web.enable-lifecycle
[Install]
WantedBy=multi-user.target
Enable and start the Prometheus service:
sudo systemctl enable prometheus
sudo systemctl start prometheus
Check the status of the Prometheus service:
sudo systemctl status prometheus
Open your web browser and navigate to: http://:9090
Prometheus now running on port 9090
Node Exporter is used to collect and expose system-level metrics such as CPU, memory, disk, and network usage for Prometheus to scrape and monitor.
Create a system user for Node Exporter and download the binaries:
sudo useradd --system --no-create-home --shell /bin/false node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
Extract the tarball, move the binary, and clean up:
tar -xvf node_exporter-1.6.1.linux-amd64.tar.gz
sudo mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
rm -rf node_exporter*
Create a systemd unit file for Node Exporter:
sudo nano /etc/systemd/system/node_exporter.service
Add the following content:
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=500
StartLimitBurst=5
[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/node_exporter --collector.logind
[Install]
WantedBy=multi-user.target
Enable and start the Node Exporter service:
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
Check the status of Node Exporter:
sudo systemctl status node_exporter
Modify the prometheus.yml
file to include Node Exporter and Jenkins scraping:
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'node_exporter'
static_configs:
- targets: ['localhost:9100']
- job_name: 'jenkins'
metrics_path: '/prometheus'
static_configs:
- targets: [':']
Check the validity of your Prometheus configuration:
promtool check config /etc/prometheus/prometheus.yml
Reload Prometheus configuration without restarting:
curl -X POST http://localhost:9090/-/reload
Node exporter metrics visible on port 9100
Open your web browser and navigate to: http://:9090/targets
Node exporter and Jenkins target now showing on Prometheus
First, ensure that all necessary dependencies are installed:
sudo apt-get update
sudo apt-get install -y apt-transport-https software-properties-common
Add the GPG key for Grafana:
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
Add the repository for Grafana stable releases:
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
Update the package list and install Grafana:
sudo apt-get update
sudo apt-get -y install grafana
To automatically start Grafana after a reboot, enable the service:
sudo systemctl enable grafana-server
Then, start Grafana:
sudo systemctl start grafana-server
Verify the status of the Grafana service to ensure it's running correctly:
sudo systemctl status grafana-server
Open a web browser and navigate to Grafana using your server's IP address. The default port for Grafana is 3000. For example: http://:3000
You'll be prompted to log in to Grafana. The default username is "admin," and the default password is also "admin."
When you log in for the first time, Grafana will prompt you to change the default password for security reasons. Follow the prompts to set a new password.
To visualize metrics, you need to add a data source. Follow these steps:
- Click on the gear icon (⚙️) in the left sidebar to open the "Configuration" menu.
- Select "Data Sources."
- Click on the "Add data source" button.
- Choose "Prometheus" as the data source type.
- In the "HTTP" section, set the "URL" to
http://localhost:9090
(assuming Prometheus is running on the same server). - Click the "Save & Test" button to ensure the data source is working.
To make it easier to view metrics, you can import a pre-configured dashboard. Follow these steps:
- Click on the "+" (plus) icon in the left sidebar to open the "Create" menu.
- Select "Dashboard."
- Click on the "Import" dashboard option.
- Enter the dashboard code you want to import (e.g., code 1860).
- Click the "Load" button.
- Select the data source you added (Prometheus) from the dropdown.
- Click on the "Import" button.
You should now have a Grafana dashboard set up to visualize metrics from Prometheus.
Configure Prometheus Plugin Integration: Integrate Jenkins with Prometheus to monitor the CI/CD pipeline. Installed plugin
Import Jenkins dashboard to Grafana
Run the pipeline by clicking Build Now button on Jenkins.
The pipeline ran successfully with no errors.
To see the Sonarqube report, you can go to Sonarqube Server and go to Projects.
Dependency check results will show like below:
When you log in to Dockerhub, you will see a new image is created
Netflix clone app can now be seen running on port 8081
Install kubectl
on the Jenkins machine:
sudo apt update
sudo apt install curl
curl -LO https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client
Ensure that both master and worker instances are set up.
sudo hostnamectl set-hostname K8s-Master
sudo hostnamectl set-hostname K8s-Worker
sudo apt-get update
sudo apt-get install -y docker.io
sudo usermod -aG docker ubuntu
newgrp docker
sudo chmod 777 /var/run/docker.sock
sudo curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo tee /etc/apt/sources.list.d/kubernetes.list <
sudo kubeadm init --pod-network-cidr=10.244.0.0/16
# Exit from the root user and run the following commands:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Join the worker node to the Kubernetes cluster using the command provided by the kubeadm init
output:
sudo kubeadm join : --token --discovery-token-ca-cert-hash
- Copy the Kubernetes config file to the Jenkins master or your local file manager.
- Save it as
secret-file.txt
in a chosen folder (e.g., Documents
).
- Use this file in the Kubernetes credential section of Jenkins.
Install the necessary Kubernetes plugins on Jenkins to facilitate the integration.
- Install the Kubernetes Plugin in Jenkins.
- Go to
Manage Jenkins -> Manage Credentials
.
- Click on Jenkins Global, then Add Credentials.
- Select Add Secret File and save.
sudo useradd --system --no-create-home --shell /bin/false node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar -xvf node_exporter-1.6.1.linux-amd64.tar.gz
sudo mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
rm -rf node_exporter*
node_exporter --version
sudo vim /etc/systemd/system/node_exporter.service
Add the following content to node_exporter.service
:
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=500
StartLimitBurst=5
[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/node_exporter --collector.logind
[Install]
WantedBy=multi-user.target
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
sudo systemctl status node_exporter
Verify that Node Exporter is running on both the master and worker nodes.
sudo vim /etc/prometheus/prometheus.yml
Add the following job configurations in prometheus.yml
:
- job_name: node_export_masterk8s
static_configs:
- targets: [":9100"]
- job_name: node_export_workerk8s
static_configs:
- targets: [":9100"]
Validate the Prometheus configuration:
promtool check config /etc/prometheus/prometheus.yml
Reload the Prometheus configuration using a POST request:
curl -X POST http://localhost:9090/-/reload
Check the Prometheus targets: http://:9090/targets
By default, Node Exporter will be exposed on port 9100.
Verify that both targets are running in Prometheus.
stage('Deploy to Kubernetes') {
steps {
script {
dir('Kubernetes') {
withKubeConfig(caCertificate: '', clusterName: '', contextName: '', credentialsId: 'k8s', namespace: '', restrictKubeConfigAccess: false, serverUrl: '') {
sh 'kubectl apply -f deployment.yml'
sh 'kubectl apply -f service.yml'
}
}
}
}
}
In the Kubernetes cluster (master node), run:
kubectl get all
kubectl get svc # use anyone
Access the application using: public-ip-of-worker:service-port
Complete any necessary steps to terminate instances and clean up resources after deployment.
pipeline {
agent any
tools {
jdk 'jdk17'
nodejs 'node16'
}
environment {
SCANNER_HOME = tool 'sonar-scanner'
DOCKER_CREDENTIALS_ID = 'docker'
SONAR_CREDENTIALS_ID = 'Sonar-token'
TMDB_API_KEY = 'API_KEY'
}
stages {
stage('Clean Workspace') {
steps {
cleanWs()
}
}
stage('Checkout from Git') {
steps {
git branch: 'main', url: 'https://github.com/N4si/DevSecOps-Project.git'
}
}
stage('SonarQube Analysis') {
steps {
withSonarQubeEnv('sonar-server') {
sh '''$SCANNER_HOME/bin/sonar-scanner \
-Dsonar.projectName=Netflix \
-Dsonar.projectKey=Netflix'''
}
}
}
stage('Quality Gate') {
steps {
script {
waitForQualityGate abortPipeline: false, credentialsId: SONAR_CREDENTIALS_ID
}
}
}
stage('Install Dependencies') {
steps {
sh 'npm install'
}
}
stage('OWASP FS SCAN') {
steps {
dependencyCheck additionalArguments: '--scan ./ --disableYarnAudit --disableNodeAudit', odcInstallation: 'DP-Check'
dependencyCheckPublisher pattern: '**/dependency-check-report.xml'
}
}
stage('Trivy FS Scan') {
steps {
sh 'trivy fs . > trivyfs.txt'
}
}
stage('Docker Build & Push') {
steps {
script {
withDockerRegistry(credentialsId: DOCKER_CREDENTIALS_ID, toolName: 'docker') {
sh 'docker build --build-arg TMDB_V3_API_KEY=${TMDB_API_KEY} -t netflix .'
sh 'docker tag netflix morlo66/netflix:latest'
sh 'docker push morlo66/netflix:latest'
}
}
}
}
stage('Trivy Image Scan') {
steps {
sh 'trivy image nasi101/netflix:latest > trivyimage.txt'
}
}
stage('Deploy to Container') {
steps {
sh 'docker run -d --name netflix -p 8081:80 nasi101/netflix:latest'
}
}
stage('Deploy to Kubernetes') {
steps {
script {
withKubeConfig(caCertificate: '', clusterName: '', contextName: '', credentialsId: 'k8s', namespace: '', restrictKubeConfigAccess: false, serverUrl: '') {
sh 'kubectl apply -f Kubernetes/deployment.yml'
sh 'kubectl apply -f Kubernetes/service.yml'
}
}
}
}
}
}