How to Set Up a CI/CD Pipeline from Scratch

Continuous Integration and Continuous Deployment (CI/CD) has become an essential practice for modern software development teams. By automating the build, test, and deployment processes, CI/CD pipelines help developers deliver code faster, more reliably, and with fewer errors. This comprehensive guide will walk you through setting up a complete CI/CD pipeline from scratch, even if you’re new to the concept.

Table of Contents

• What is CI/CD?
• Prerequisites and Tools
• Choosing Your CI/CD Tools
• Setting Up Your Code Repository
• Installing and Configuring Jenkins
• Creating Your First Pipeline
• Adding Automated Testing
• Configuring Deployment Stages
• Best Practices and Security

What is CI/CD?

CI/CD stands for Continuous Integration and Continuous Deployment. Continuous Integration is the practice of automatically merging code changes from multiple developers into a shared repository several times a day. Each integration is verified by automated builds and tests to detect errors quickly.

Continuous Deployment takes this further by automatically deploying all code changes that pass the testing phase to production. This automation eliminates manual steps, reduces human error, and allows teams to release updates more frequently and confidently.

Prerequisites and Tools

Before building your CI/CD pipeline, you’ll need to have a few things in place:

• A source code management system (Git)
• A server or cloud instance for your CI/CD tool
• Basic knowledge of command line operations
• A test environment for your application
• Administrative access to your deployment environment

For the server infrastructure, you’ll need a reliable hosting solution. Kamatera offers flexible cloud infrastructure that’s perfect for hosting CI/CD tools, with scalable resources that can grow with your pipeline needs.

Choosing Your CI/CD Tools

Several excellent CI/CD tools are available, each with unique strengths:

Jenkins

Jenkins is the most popular open-source automation server. It’s highly extensible with over 1,500 plugins and supports building, deploying, and automating any project. We’ll use Jenkins for this tutorial due to its flexibility and widespread adoption.

GitLab CI/CD

GitLab provides integrated CI/CD capabilities directly within its platform. It’s excellent if you’re already using GitLab for version control.

GitHub Actions

GitHub Actions offers CI/CD functionality natively within GitHub repositories. It’s particularly convenient for projects already hosted on GitHub.

CircleCI and Travis CI

These are cloud-based CI/CD platforms that integrate seamlessly with GitHub and Bitbucket, offering quick setup and minimal maintenance.

Setting Up Your Code Repository

First, ensure your code is in a Git repository. If you haven’t initialized one yet, navigate to your project directory and run:

git init
git add .
git commit -m "Initial commit"
git remote add origin https://github.com/yourusername/yourproject.git
git push -u origin main

Your repository structure should include configuration files for your CI/CD pipeline. Create a dedicated branch for pipeline development:

git checkout -b cicd-setup

Installing and Configuring Jenkins

Let’s install Jenkins on a Linux server. For Ubuntu/Debian systems, execute these commands:

wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
sudo sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt update
sudo apt install jenkins -y
sudo systemctl start jenkins
sudo systemctl enable jenkins

Jenkins runs on port 8080 by default. Access it by navigating to http://your-server-ip:8080. Retrieve the initial admin password:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Complete the setup wizard, install suggested plugins, and create your admin user account.

Creating Your First Pipeline

In Jenkins, create a new pipeline job:

1. Click “New Item” from the dashboard
2. Enter a name and select “Pipeline”
3. Click OK to proceed

Create a Jenkinsfile in your repository root. This file defines your pipeline stages:

pipeline {
    agent any
    
    stages {
        stage('Checkout') {
            steps {
                git branch: 'main', url: 'https://github.com/yourusername/yourproject.git'
            }
        }
        
        stage('Build') {
            steps {
                sh 'npm install'
                sh 'npm run build'
            }
        }
        
        stage('Test') {
            steps {
                sh 'npm test'
            }
        }
        
        stage('Deploy') {
            steps {
                sh './deploy.sh'
            }
        }
    }
    
    post {
        success {
            echo 'Pipeline completed successfully!'
        }
        failure {
            echo 'Pipeline failed. Please check the logs.'
        }
    }
}

Adding Automated Testing

Automated testing is crucial for maintaining code quality. Your pipeline should run multiple test types:

Unit Tests

These test individual components in isolation. Configure them to run early in your pipeline to catch issues quickly.

Integration Tests

These verify that different parts of your application work together correctly.

Security Scans

Implement security scanning tools like OWASP Dependency Check or Snyk to identify vulnerabilities in your dependencies.

If you’re looking to deepen your understanding of CI/CD practices and DevOps principles, consider taking specialized courses on Coursera, where industry experts share practical knowledge and real-world scenarios.

Configuring Deployment Stages

A robust pipeline includes multiple deployment environments:

Development Environment

Deploy automatically on every commit to the development branch. This environment is for active development and rapid iteration.

Staging Environment

Create a production-like environment for final testing. Deploy here after all tests pass in the development environment.

Production Environment

Implement manual approval gates or deploy automatically after successful staging tests. Here’s an example deployment configuration:

stage('Deploy to Production') {
    when {
        branch 'main'
    }
    steps {
        input message: 'Deploy to production?', ok: 'Deploy'
        sh 'kubectl apply -f kubernetes/production/'
        sh 'kubectl rollout status deployment/my-app'
    }
}

Best Practices and Security

Follow these best practices to maintain a secure and efficient pipeline:

Use Environment Variables

Never hardcode sensitive information. Store credentials and API keys as Jenkins credentials or environment variables:

environment {
    DATABASE_URL = credentials('database-url')
    API_KEY = credentials('api-key')
}

Implement Pipeline as Code

Keep your Jenkinsfile in version control alongside your application code. This ensures pipeline changes are tracked and reviewable.

Monitor and Log Everything

Configure proper logging and monitoring to quickly identify and resolve pipeline failures. Integrate tools like Prometheus and Grafana for visualization.

Keep Dependencies Updated

Regularly update your CI/CD tools, plugins, and dependencies to benefit from security patches and new features.

Use Parallel Execution

Speed up your pipeline by running independent stages in parallel:

stage('Parallel Tests') {
    parallel {
        stage('Unit Tests') {
            steps {
                sh 'npm run test:unit'
            }
        }
        stage('Integration Tests') {
            steps {
                sh 'npm run test:integration'
            }
        }
    }
}

Implement Rollback Mechanisms

Always have a strategy to quickly roll back deployments if issues arise in production. Automate this process within your pipeline.

Setting up a CI/CD pipeline from scratch requires initial effort, but the long-term benefits far outweigh the investment. You’ll achieve faster development cycles, fewer bugs in production, and more confident deployments. Start simple, iterate based on your team’s needs, and gradually add more sophisticated features as you become comfortable with the workflow. Remember that CI/CD is not just about tools—it’s a culture of continuous improvement and automation that transforms how teams deliver software.