Wanderlust DevSecOps Pipeline Project

WanderLust is a simple MERN travel blog website ✈ This project is aimed to help people to contribute in open source, upskill in react and also master git.

In this demo, we will see how to deploy an end to end three tier MERN stack application on EKS cluster.

Tech stack used in this project:

GitHub (Code)
Docker (Containerization)
Jenkins (CI)
OWASP (Dependency check)
SonarQube (Quality)
Trivy (Filesystem Scan)
ArgoCD (CD)
Redis (Caching)
AWS EKS (Kubernetes)
Helm (Monitoring using grafana and prometheus)

How pipeline will look after deployment:

CI pipeline to build and push
CD pipeline to update application version
ArgoCD application for deployment on EKS

SonarQube Analysis Report

Pre-requisites to implement this project:

[Note] This project will be implemented on Mumbai region (ap-south-1).

Create 1 Master machine on AWS with 2CPU, 8GB of RAM (t2.large) and 30 GB of storage and install Docker on it.

Open the below ports in security group of master machine and also attach same security group to Jenkins worker node (We will create worker node shortly)

[Note] We are creating this master machine because we will configure Jenkins master, eksctl, EKS cluster creation from here.

Install & Configure Docker by using below command, "NewGrp docker" will refresh the group config hence no need to restart the EC2 machine.

sudo apt-get update

sudo apt-get install docker.io -y
sudo usermod -aG docker ubuntu && newgrp docker
docker -v

Install and configure Jenkins (Master machine)

sudo apt update -y
sudo apt install fontconfig openjdk-17-jre -y

sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
  https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key

echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \
  https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null

sudo apt-get update -y
sudo apt-get install jenkins -y

Now, access Jenkins Master on the browser on port 8080 and configure it.
Create EKS Cluster on AWS (Master machine)
- IAM user with access keys and secret access keys
- AWSCLI should be configured (Setup AWSCLI)

    curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
    sudo apt install unzip
    unzip awscliv2.zip
    sudo ./aws/install
    aws configure

Check AWS connectivity

    aws sts get-caller-identity

Install kubectl (Master machine) (Setup kubectl )

    curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.19.6/2021-01-05/bin/linux/amd64/kubectl
    chmod +x ./kubectl
    sudo mv ./kubectl /usr/local/bin
    kubectl version --short --client

Install eksctl (Master machine) (Setup eksctl)

    curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
    sudo mv /tmp/eksctl /usr/local/bin
    eksctl version

Create EKS Cluster (Master machine)

    eksctl create cluster --name=wanderlust \
    --region=ap-south-1 \
    --version=1.30 \
    --without-nodegroup

Associate IAM OIDC Provider (Master machine)

    eksctl utils associate-iam-oidc-provider \
    --region ap-south-1 \
    --cluster wanderlust \
    --approve

Create Nodegroup (Master machine)

    eksctl create nodegroup --cluster=wanderlust \
    --region=ap-south-1 \
    --name=wanderlust \
    --node-type=t2.large \
    --nodes=2 \
    --nodes-min=2 \
    --nodes-max=2 \
    --node-volume-size=29 \
    --ssh-access \
    --ssh-public-key=mumbai_key

[Note] Make sure the ssh-public-key "mumbai_key is available in your aws account"

Setting up jenkins worker node
- Create a new EC2 instance (Jenkins Worker) with 2CPU, 8GB of RAM (t2.large) and 29 GB of storage and install java on it

    sudo apt update -y
    sudo apt install fontconfig openjdk-17-jre -y

Create an IAM role with administrator access attach it to the jenkins worker node Select Jenkins worker node EC2 instance --> Actions --> Security --> Modify IAM role
Configure AWSCLI (Setup AWSCLI)

    sudo su

    curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
    sudo apt install unzip
    unzip awscliv2.zip
    sudo ./aws/install

Generate ssh keys (Master machine) to setup jenkins master-slave

ssh-keygen

Now move to directory(.ssh) where your ssh keys are generated and copy the content of public key and paste to authorized_keys file of the Jenkins worker node (Jenkins Worker).

Now, go to the Jenkins master and navigate to Manage Jenkins --> Nodes, and click on New node

Name: Node
Type: permanent agent
Number of executors: 2
Remote root directory: /home/ubuntu
Labels: Node
Usage: Only build jobs with label expressions matching this node
Launch method: Via ssh
Host: <public-ip-worker-jenkins>
Credentials: Add --> Kind: ssh username with private key --> ID: Worker --> Description: Worker --> Username: ubuntu --> Private key: Enter directly --> Add Private key

Note: the private-ip should be master’s instance generated private which is stored in id_ed25519.

Host Key Verification Strategy: Non verifying Verification Strategy
Availability: Keep this agent online as much as possible

And your jenkins worker node is added

If any error comes open the setting icon of Node and make changes then launch agent again.

Install docker (Jenkins Worker)

sudo apt install docker.io -y
sudo usermod -aG docker ubuntu && newgrp docker

Install and configure SonarQube (Master machine)

docker run -itd --name SonarQube-Server -p 9000:9000 sonarqube:lts-community

Access the Sonarqube server on <public-ip-of-jenkins-CI>:9000

Intial username= admin and password= admin

Install Trivy (Jenkins Worker)

sudo apt-get install wget apt-transport-https gnupg lsb-release -y
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
echo deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update -y
sudo apt-get install trivy -y

Install and Configure ArgoCD (Master Machine)
- Create argocd namespace

    kubectl create namespace argocd

Apply argocd manifest

    kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Make sure all pods are running in argocd namespace

    watch kubectl get pods -n argocd

Install argocd CLI

    sudo curl --silent --location -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/download/v2.4.7/argocd-linux-amd64

Provide executable permission

    sudo chmod +x /usr/local/bin/argocd

Check argocd services

    kubectl get svc -n argocd

Change argocd server's service from ClusterIP to NodePort

    kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "NodePort"}}'

Or you can manually edit the manifest file also by going inside it, using this command:

    kubectl edit svc/argocd-server -n argocd

Confirm service is patched or not

    kubectl get svc -n argocd

Check the port where ArgoCD server is running and expose it on security groups of a worker node

port: 31797
Access it on browser, click on advance and proceed with

    <public-ip-master>:<port>

If you find error like this in the above image, then run the below the command:

    kubectl port-forward svc/argocd-server -n argocd 31797:80 --address 0.0.0.0 &

Fetch the initial password of argocd server

    kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

Username: admin
Now, go to User Info and update your argocd password

Steps to add email notification

Go to your Jenkins Master EC2 instance and allow 465 port number for SMTPS
Now, we need to generate an application password from our Gmail account to authenticate with Jenkins
- Open gmail and go to Manage your Google Account --> Security

[Important] Make sure 2 step verification must be on

Search for App password and create a app password for jenkins

[Note]: App password will be in this form atqw gvjh ifrr wdkf, make sure when you enter in jenkins it should have no space between them like atqwgvjhifrrwdkf.

Once, app password is created and go back to Jenkins Manage Jenkins --> Credentials to add username and password for email notification
Go back to Manage Jenkins --> System and search for Extended E-mail Notification
Scroll down and search for E-mail Notification and setup email notification

[Important] Enter your gmail password which we copied recently in password field E-mail Notification --> Advance

Steps to implement the project:

Go to Jenkins Master and click on Manage Jenkins --> Plugins --> Available plugins install the below plugins:
- OWASP
- Git
- SonarQube Scanner
- Docker
- Docker Pipeline
- Pipeline: Stage View
- Pipeline: Job
- Blue Ocean
Configure OWASP, move to Manage Jenkins --> Plugins --> Available plugins (Jenkins Worker)
After OWASP plugin is installed, now move to Manage Jenkins --> Tools (Jenkins Worker)
Login to SonarQube server and create the credentials for Jenkins to integrate with SonarQube
- Navigate to Administration --> Security --> Users --> Token
Now, go to Manage Jenkins --> credentials and add SonarQube credentials:
Go to Manage Jenkins --> Tools and search for SonarQube Scanner installations:
Go to Manage Jenkins --> credentials and add GitHub credentials to push updated code from the pipeline:

[Note] While adding GitHub credentials add Personal Access Token in the password field.

Go to Manage Jenkins --> System and search for SonarQube installations:

Now again, Go to Manage Jenkins --> System and search for Global Trusted Pipeline Libraries:

Note: Before putting this shared library, Fork this repository LINK, and put the project repository URL of your repository.

Login to SonarQube server, go to Administration --> Webhook and click on create

Public Ip is of Jenkins CI (Master) because SonarQube is running on it only.

Now, go to GitHub repository and under Automations directory update the instance-id field on both the updatefrontendnew.sh updatebackendnew.sh with the k8s worker's instance id (Jenkins Worker or Slave ID)
Navigate to Manage Jenkins --> credentials and add credentials for docker login to push docker image:

Create a Wanderlust-CI pipeline

The Jenkins Pipeline script you will get in my GitHub —> Jenkinsfile, before putting the pipeline script in Jenkins, do the below steps:

Fork the Repository
Go to Jenkinsfile
Make Sure you have forked and configured shared library repository for jenkinsfile which uses groovy syntax.
Name the Jenkins slave node by Node only.
The SonarQube tool which Jenkins will install should be named as Sonar as per pipeline script.

Make Sure you change the repository URL and branch as per your GitHub URL.

In place of wanderlust-backend-beta, put the name of docker image which you want and in place of harshitsahu2311 put your Dockerhub username so that it gets tagged with it.

Here also change the image name and username as previously changed.

Make Sure the second job which is Wanderlust-CD, should have same name.

Create one more pipeline Wanderlust-CD

The Jenkins Pipeline script you will get in my GitHub —> GitOps → Jenkinsfile, before putting the pipeline script in Jenkins, do the below steps:

Change the repository URL
Make sure you save the GitHub credentials with this ID only: Github-cred and change the GitHub URL as per your forked URL.
Change the email Id as per your own address in place harshitsahu6088@gmail.com.
Go to Kubernetes → Change the username in backend.yaml and frontend.yaml as per your Dockerhub username.

After all of this Our Jobs are now ready.

Now Click on Wanderlust-CI and build and then when it gets completes, Wanderlust-CD will automatically trigger and get build.

Provide permission to docker socket so that docker build and push command do not fail (Jenkins Worker)

chmod 777 /var/run/docker.sock

Go to Master Machine and add our own eks cluster to argocd for application deployment using cli
- Login to argoCD from CLI

    argocd login 65.2.186.181:31797 --username admin

[Tip] 65.2.186.181:31797 --> This should be your argocd url and after login enter your password

Check how many clusters are available in argocd

argocd cluster list

Get your cluster name

kubectl config get-contexts

Add your cluster to argocd

argocd cluster add iam-root-account@wanderlust.ap-south-1.eksctl.io --name wanderlust-eks-cluster

[Tip] iam-root-account@wanderlust.ap-south-1.eksctl.io --> This should be your EKS Cluster Name.

Once your cluster is added to argocd, go to argocd console Settings --> Clusters and verify it
Go to Settings --> Repositories and click on Connect repo

[Note] Connection should be successful

Now, go to Applications and click on New App

[Important] Make sure to click on the Auto-Create Namespace option while creating argocd application

Congratulations, your application is deployed on AWS EKS Cluster
Open port 31000 and 31100 on Nodes of EKS Cluster and Access it on browser

<worker-public-ip>:31000

Email Notification

How to monitor EKS cluster, Kubernetes components and workloads using Prometheus and Grafana via HELM (On Master machine)

Install Helm Chart

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Add Helm Stable Charts for Your Local Client

helm repo add stable https://charts.helm.sh/stable

Add Prometheus Helm Repository

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Create Prometheus Namespace

kubectl create namespace prometheus
kubectl get ns

Install Prometheus using Helm

helm install stable prometheus-community/kube-prometheus-stack -n prometheus

Verify Prometheus installation

kubectl get pods -n prometheus

Check the services file (svc) of the Prometheus

kubectl get svc -n prometheus

Expose Prometheus and Grafana to the external world through Node Port

[Important] change it from Cluster IP to NodePort after changing make sure you save the file and open the assigned nodeport to the service.

kubectl edit svc/stable-kube-prometheus-sta-prometheus -n prometheus

Verify service

kubectl get svc -n prometheus

Now, let’s change the SVC file of the Grafana and expose it to the outer world

kubectl edit svc stable-grafana -n prometheus

Check grafana service

kubectl get svc -n prometheus

Access the Grafana with <public-ip-of-master>:32108 and Prometheus with <public-ip-of-master>:31521

If it shows error like the above image, then port-forward through the below command:

  kubectl port-forward svc/stable-grafana 32108:80 -n prometheus --address 0.0.0.0 &
  kubectl port-forward svc/stable-kube-prometheus-sta-prometheus 31521:9090 -n prometheus --address 0.0.0.0 &

Prometheus

Get a password for Grafana

kubectl get secret --namespace prometheus stable-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

[Note] Username: admin

Now, view the Dashboard in Grafana

Click on Dashboards

Put the Data Source: Prometheus and Namespace: wanderlust and run for last 5 minutes.

Custom Dashboards

On google search Grafana Dashboards and go for the first link

Search for Kubernetes Dashboards

Choose any one and click on it and then Copy the ID of that Dashboard

Then go to Grafana → dashboards → New → Import

Put the Copy ID in the section and then click on Load

Choose Prometheus in the metrics and then click on Import

BOOM! Our Dashboards is ready within seconds

If you like the full implementation of project, then support me through your like and comment and also follow me on LinkedIn.

Clean Up

Delete EKS cluster

eksctl delete cluster --name=wanderlust --region=ap-south-1