Hello folks, In this post I am going to explain about the ci/cd pipeline with Jenkins as automation server , Git as version control system and Kubernetes for the production.The source code example is available here. This a series in which we’ll discuss the actual ci/cd pipeline along with the other concepts which can be invoked in the process of continuous integration and continuous deployment like SonarQube which is a code quality testing tool.
Before Going to the example let us know the basic concepts like DevOps, ci/cd , need of ci/cd in real time and the architectural flow of ci/cd.
What is DevOps ?
DevOps is a set of practices that combines software development (Dev) and IT operations (Ops). It aims to shorten the systems development life cycle and provide continuous delivery with high software quality. DevOps is complementary with Agile software development; several DevOps aspects came from Agile methodology.
The set of DevOps tools increases an organization’s ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes.
As you can see in the above image, DevOps is the combination of several stages , at each stage there are different tools that fulfills the stage functionalities.
What is CI/CD ?
Ci/Cd generally refers to the combined practices of continuous integration and either continuous delivery or continuous deployment. Basically ci/cd bridges the gaps between development and operational activities and teams by enforcing automation in building, testing and deployment of applications.
Challenges Before CI/CD ?
- Automation of build and testing process.
- Fast delivery
- Automation of application deployment
- Fast feedback loop
- Avoids the process Integration of all stages which is a hell
- Improves code quality and test ability
From the above challenges we can understand the need and importance of ci/cd in real time.
Architectural flow of CI/CD :
Basically, the process of ci/cd starts with the code committing stage in any version control system like Git. The committed code will then enters CI pipeline process, here we can include the stages as per the requirement.Generally this process includes Build stage, testing stage, Docker image generation, pushing the image into docker hub. If we wants to include the sonarqube for the purpose of code quality , we can also include it in ci process.
In the Cd process, the docker image will be pulled by the docker and will be deployed into the Kubernetes.
prerequisites:
- Jenkins
Jenkins is a free and open source automation server. It helps automate the parts of software development related to building, testing, and deploying, facilitating continuous integration and continuous delivery.If you are not yet installed Jenkins, do install from here.
2. Docker
Docker is a set of platform as a service(Paas) products that use OS-level virtualization to deliver software in packages called containers. Containers are isolated from one another and bundle their own software, libraries and configuration files; they can communicate with each other through well-defined channels.
You can use Docker Desktop (If you are using windows then windows pro version only supports this and MAC OS will support docker desktop). For windows OS other than pro version we have an option with Docker Tool Box which provides the Virtual Box,Docker and Kitematic. We can setup the docker in virtual box and can use it. Install Docker Desktop from here and Docker Tool Box here. For downloading docker in Linux and Ubuntu machines follow this.
3. Version Control
Version control helps a software team manage changes to source code over time. Version control software keeps track of every modification to the code in a special kind of database. If a mistake is made, developers can turn back the clock and compare earlier versions of the code to help fix the mistake while minimizing disruption to all team members. My suggestion is to use Git as version control system.
Examples: Git, Bit bucket
4. Kubernetes
Kubernetes is an open-source container-orchestration system for automating computer application deployment, scaling, and management. Docker Desktop includes a standalone Kubernetes server and client, as well as Docker CLI integration. The Kubernetes server runs locally within your Docker instance, is not configurable, and is a single-node cluster. If you are using docker tool box then install kubectl and minikube. After installation set the paths of both inside your system’s environment variables.
5. Docker Hub:
Docker Hub is also like a version control system where we will store the docker images inside a registry. It stores all the versions of our images. Keep an account in the docker hub with valid credentials which will be more useful in the process of CI.
6. SonarQube:
SonarQube is an open-source platform developed by SonarSource for continuous inspection of code quality to perform automatic reviews with static analysis of code to detect bugs, code smells, and security vulnerabilities on 20+ programming languages. Since, this is optional stage in CI/CD , if you want to know about SonarQube and SonarQube Quality Gate, have a look at here. In the given link I gave a full fledged clarity about them.
We can install and use the community version of the SonarQube for free here.
7. ngrok:
ngrok is a cross-platform application that enables developers to expose a local development server to the Internet with minimal effort. The software makes your locally-hosted web server appear to be hosted on a subdomain of ngrok.com, meaning that no public IP or domain name on the local machine is needed. Download the ngrok here.
CI/CD Procedure :
Let’s now dive into the Steps to Ci/Cd
- Jenkins setup:
After successful installation of Jenkins do register and login to the Jenkins dashboard with the help of a master key generated inside the directory created by Jenkins. The dashboard of Jenkins will look like this.
Explore the operations available in Jenkins before creating a pipeline to know the basic idea on Jenkins Features.
- Install the required plugins for ci/cd pipeline in the manage Jenkins section. The required plugins are, git,docker,blue ocean,maven (for spring boot),pipeline,build pipeline,delivery pipeline and sonarqube. Install the other plugins based on the framework you’ve chosen like node, java etc..
Manage Jenkins -> Manage plugins -> install the plugins
- Add the local paths of Git, shell (available inside Git folder),Maven, Docker. You can also choose the options like choose automatically , but sometimes it won’t work well and you’ll face problems.
Manage Jenkins -> Global Tool Configuration -> Add local git path until git.exe, Maven path Manage Jenkins -> Global Tool Configuration -> Add local SonarQube scanner and do remember the name given to this path in Jenkins.
Add Docker paths like below. You’ll find out those inside your environment variables.
Manage Jenkins -> Configure System ->Global properties
- Add Sonarqube and shell paths inside Configure system only.
- Set the docker details and kube config details for using Kubernetes in environment variables. Inside user directory .kube folder gets created.
- Adding Docker Hub and Git credentials:
Credentials -> system -> Global credentials -> Add credentials
(Give Username and password of your docker hub & git)We are done with setup of Jenkins and Docker.
Let’s test all of them to make sure that they are working fine with the help of a free style project in Jenkins.
- Click on New Item and choose a free style project with a name and observe the options available in the next page. Go to build step and choose Execute shell to test whether shell is working or not. Type in echo “test” in the text area,save and apply. Run this project using Build Now option and check the result. If it gives the result as “SUCCESS” then shell is working fine.
- Now Choose Source Code Management as Git, Enter the Repository URL,choose the branches to build. Goto Build step and write the command “docker build”. This build will takes the code from the git hub repo and build the image based on the Dockerfile provided in the git repo.
- For checking this functionality i’ve provided a Repository on github with java Helloworld code here. Feel free to fork it or use it directly.The result after building this step should be “Success”, Otherwise it means that you haven’t properly set the docker path or install the plugins in the Jenkins.
- In the same way you can also test the maven or another shell commands.
2. Creating a CI/CD pipeline in Jenkins
Create a New item by choosing pipeline project and give the same as you desired. In the Build triggers sections check the option “GitHub hook trigger for GITScm polling”, this will be used to build the entire project again when the user commits code on Git (Version control).
- In the pipeline section choose the option Pipeline script from SCM.Choose SCM as Git, enter the git url, Add credentials (previously set)name of the repo (if it is private),select Branches to build and script path as Jenkinsfile because inside the Git repo my Jenkinsfile is on root path itself.If it is not like that give the directory path and save it.
- In this example I am taking a spring boot application. For a node application example check here.
3. Jenkinsfile Explanation
We’ll look into the CI part of Jenkinsfile Now
pipeline{
environment {
scannerHome = tool 'SonarQubeScanner'
registry = "simhalp9/springali"
registryCredential = 'dockerhub'
dockerImage = ''
}
agent any
stages {
stage('clean') {
steps {
bat 'mvn clean'
}
}
stage('build and test') {
steps {
bat 'mvn package'
}
} stage('Sonarqube') {
steps {
script {
withSonarQubeEnv('sonarqube') {
bat "${scannerHome}/bin/sonar-scanner"
}
}
}
}
stage("SonarQube Quality Gate"){
steps {
script {
timeout(time: 5, unit: 'MINUTES') {
def qualitygate = waitForQualityGate()
if (qualitygate.status != 'OK') {
abortPipeline:true
error "Pipeline aborted due to quality gate failure: ${qualitygate.status}"
}
else {
echo "Quality gate passed"
}
}
}
}
}
stage('Building image') {
steps{
script {
dockerImage = docker.build registry + ":$BUILD_NUMBER"
}
}
}
stage('Push Image') {
steps{
script {
docker.withRegistry( '', registryCredential ) {
dockerImage.push()
}
}
}
}
}
}
Basically in a Jenkinsfile we’ll write the pipeline stages that are to be executed in an sequential manner. Let’s explore the each ans every stage now.
- Environment setup
environment {
scannerHome = tool 'SonarQubeScanner'
registry = "simhalp9/springali"
registryCredential = 'dockerhub'
dockerImage = ''
}- Here, we are getting the details of the sonarQubeScanner into scannerHome which we are set in Jenkins with the name “SonarQubeScanner”, docker hub registry name and docker hub credentials to access the registry and store the docker images.
- SonarQube quality gate is provided to get the quality gate result, if it is failed then the remaining stages will be aborted.
- Stage “Build”
stage('clean') {
steps {
bat 'mvn clean'
}
}stage('build and test') {
steps {
bat 'mvn package'
}
}
- Here, we are building the spring boot project with bat command , you can also run this commands using shell like, sh “mvn package”
- Stage “SonaQube”
stage('Sonarqube') {
steps {
script {
withSonarQubeEnv('sonarqube') {
bat "${scannerHome}/bin/sonar-scanner"
}
}
}
}
stage("SonarQube Quality Gate"){
steps {
script {
timeout(time: 5, unit: 'MINUTES') {
def qualitygate = waitForQualityGate()
if (qualitygate.status != 'OK') {
abortPipeline:true
error "Pipeline aborted due to quality gate failure: ${qualitygate.status}"
}
else {
echo "Quality gate passed"
}
}
}
}
}- These two stages are optional. But If we want to have a code quality testing then it is mandatory to write this step. In this stage we are scanning the project for checking the code quality with the help of file named as “sonar-project.properties”. More Details related to sonarqube and sonarqube quality gate are available in the later articles.
- After the successful execution of Sonarqube stage the results will be appeared on the sonarqube dashboard. Here you can findout the code quality metrics like bugs, vulnerabilities, security threats and code hotspots.
- Stage “Image Building”
stage('Building image') {
steps{
script {
dockerImage = docker.build registry + ":$BUILD_NUMBER"
}
}
}In this stage we are guiding the Jenkins to build an image with a version number , here “BUILD_NUMBER” of Jenkins will be used as a version number for the generated image and “registry” is the name of the docker image.
- Image building stage will be carried out with the help of Dockerfile which is provided in root path
FROM openjdk:8-jre-alpine
WORKDIR /app
COPY ./target/springJPA-0.0.1-SNAPSHOT.jar /app/ ENTRYPOINT ["java", "-jar", "springJPA-0.0.1-SNAPSHOT.jar"]- Let us look into the each step in detail
1. FROM will gets the openjdk:8-jre-alpine image which is the docker image of openjdk and it is useful for spring boot and java projects
2. We are creating and making working directory as /app
3. Copying the generated jar file which will be available in targets folder of the project after building stage.
Note: target folder will not be visible to you because all the actions are happening in docker
4. Starting the project with command “java -jar “generated jar filename” ”
- Stage “Image Pushing to Docker Hub”
stage('Push Image') {
steps{
script {
docker.withRegistry( '', registryCredential ) {
dockerImage.push()
}
}
}
}This stage tell the Jenkins to push the generated image to the dockerhub with the dockerhub registry credentials variable “registryCredentail” which we were set before in the Jenkins credentials. This variable consists of the Jenkins docker hub credentials name.
- Now when we hit “Build Now” in Jenkins , the pipeline stages will start executing one by one. You can see the results and progress of the stages in Jenkins dashboard.
- If you need to have a good visualization then watch it on “Blue Ocean” which is available on Jenkins Dashboard.
- We can also view the logs of each stage in Jenkins in console output option.
As of now we are creating a pipeline and configuring the required details in it, Running the Jenkinsfile which contains stages of ci/cd pipeline. It is getting the repo address, building an image and pushing the generated docker image into docker hub.
Now if we run our application, it will be accessed by the local web server. To make it available as a public URL we need to set ngrok in coordination with web hooks which is available in git.
Setting the web hooks with ngrok:
- Run the executable file of ngrok then it will opens a terminals with useful commands
- Execute the command “ngrok http 8080”. Here, http is the protocol we are using (we can also use tcp) and 8080 is the port where our application is running.
- We can also set this for our domain name like this →ngrok http foo.dev:80
- Copy the generated forwarding address for our application portby ngrok till ngrok.io.
- In your Git repo , go to settings and choose webhooks, select add webhook and paste the ngrok generated url in the payload Url place and append “/github-webhook/” to this Url. Select the events to which you want to run this Url.
- Now whenever we made any changes to the code and commit the repo, then ci/cd pipeline will be executed.
- You are all set to run your Continuous integration process now. Make any changes in the code and hit commit. Your pipeline will automatically get’s started in jenkins and you can observe the each stage visually there in the Jenkins.
- If any Stage get’s failed , remaining stages will be aborted automatically.
Wrapping up, we have installed the prerequisites, started Jenkins, installed required plugins for ci/cd, added the local paths of Git, shell, maven, Docker, Sonarqube and kubernetes to Jenkins, tested whether all are working fine in Jenkins environment with a sample Helloworld project, Created a pipeline project, Wrote a Jenkinsfile which will build and test the project, scan the project for code quality, generate a docker image based on the SonarQube quality gate, pushed the generated docker image onto docker hub, added ngrok generated forwarding address to webhooks of the repo and finally started our project ci/cd project onCommit of code.
The next part of this series ,Continuous Deployment is available here .
That’s it about Continuous Integration guys, I hope you’ll understood well.For any queries on any of my articles feel free to reach me out at narsimhulu.464@gmail.com and at LinkedIn B Narsimha
Thank You!!!