Application deployment example with Jenkins

15/12/2017 - CAPISTRANO, JENKINS, VAGRANT

In this example, we're going to manually deploy our application to web server. Process is simple so Jenkins will tell web server to pull our application repository from Github. See the detailed explanation below.


How it works


With the configurations below, you can:



By default the deployment process:



Optionally, the deployment process allows us to:



Server configuration


I am using two vagrant boxes, one for Jenkins and another for Web server. You can follow examples below to setup your servers.


Jenkins


Vagrantfile


# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
    config.vm.box = "ubuntu/xenial64"

    config.vm.define :vagrant do |vagrant_config|
        vagrant_config.vm.hostname = "jenkins"
        vagrant_config.vm.network "private_network", ip: "192.168.99.30"
        vagrant_config.vm.provision :shell, path: "bootstrap.sh"
    end

    config.vm.provider :virtualbox do |virtualbox_config|
        virtualbox_config.name = "Jenkins - Ubuntu 16.04 - 99.30"
    end
end

bootstrap.sh


#!/usr/bin/env bash

# BEGIN ########################################################################
echo -e "-- ------------------ --\n"
echo -e "-- BEGIN BOOTSTRAPING --\n"
echo -e "-- ------------------ --\n"

# BOX ##########################################################################
echo -e "-----------------------------------------------------------------------"
echo -e "-- Updating packages list\n"
apt-get update -y

# APACHE #######################################################################
echo -e "-----------------------------------------------------------------------"
echo -e "-- Updating Java packages list\n"
add-apt-repository ppa:openjdk-r/ppa
apt-get -y update

echo -e "-- Installing Java\n"
apt-get install -y openjdk-7-jre
apt-get install -y openjdk-7-jdk

# JENKINS ######################################################################
echo -e "-----------------------------------------------------------------------"
echo -e "-- Installing Jenkins\n"
wget -q -O - https://pkg.jenkins.io/debian/jenkins-ci.org.key | apt-key add -
sh -c "echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list"

echo -e "-- Updating packages list\n"
apt-get update -y
echo -e "-- Installing Jenkins automation server\n"
apt-get install jenkins -y

# END ##########################################################################
echo -e "-- ---------------- --"
echo -e "-- END BOOTSTRAPING --"
echo -e "-- ---------------- --"

Jenkins


Vagrantfile


# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
    config.vm.box = "ubuntu/xenial64"

    config.vm.define :vagrant do |vagrant_config|
        vagrant_config.vm.hostname = "website"
        vagrant_config.vm.network "private_network", ip: "192.168.99.40"
        vagrant_config.vm.provision :shell, path: "bootstrap.sh"
    end

    config.vm.provider :virtualbox do |virtualbox_config|
        virtualbox_config.name = "Website - Ubuntu 16.04 - 99.40"
    end
end

bootstrap.sh


#!/usr/bin/env bash

# BEGIN ########################################################################
echo -e "-- ------------------ --\n"
echo -e "-- BEGIN BOOTSTRAPING --\n"
echo -e "-- ------------------ --\n"

# VARIABLES ####################################################################
echo -e "-----------------------------------------------------------------------"
echo -e "-- Setting global variables\n"
VIRTUAL_HOST=localhost
DOCUMENT_ROOT=/var/www/html
APACHE_CONFIG=/etc/apache2/apache2.conf

# BOX ##########################################################################
echo -e "-----------------------------------------------------------------------"
echo -e "-- Updating packages list\n"
apt-get update -y

# APACHE #######################################################################
echo -e "-----------------------------------------------------------------------"
echo -e "-- Installing Apache web server\n"
apt-get install -y apache2

echo -e "-- Adding ServerName to Apache config\n"
grep -q "ServerName ${VIRTUAL_HOST}" "${APACHE_CONFIG}" || echo "ServerName ${VIRTUAL_HOST}" >> "${APACHE_CONFIG}"

echo -e "-- Updating vhost file\n"
cat > ${SITES_ENABLED}/000-default.conf <<EOF
<VirtualHost *:80>
    DocumentRoot ${DOCUMENT_ROOT}

    <Directory ${DOCUMENT_ROOT}>
        Options Indexes FollowSymlinks
        AllowOverride All
        Order allow,deny
        Allow from all
        Require all granted
    </Directory>

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
EOF

echo -e "-- Restarting Apache web server\n"
service apache2 restart

# PHP ##########################################################################
echo -e "-----------------------------------------------------------------------"
echo -e "-- Installing PHP7\n"
apt-get install -y language-pack-en-base
LC_ALL=en_US.UTF-8 add-apt-repository ppa:ondrej/php -y
add-apt-repository ppa:ondrej/php
apt-get update -y
apt-get install -y php7.1

echo -e "-- Installing PHP modules\n"
apt-get install -y libapache2-mod-php7.1

# COMPOSER #####################################################################
echo -e "-----------------------------------------------------------------------"
echo -e "-- Setting up Composer\n"
curl -sSk https://getcomposer.org/installer | php -- --disable-tls
mv composer.phar /usr/local/bin/composer

# GIT ##########################################################################
echo -e "-----------------------------------------------------------------------"
echo -e "-- Installing Git\n"
apt-get install -y git

# GITHUB #######################################################################
echo -e "-----------------------------------------------------------------------"
echo -e "-- Enabling SSH connection to GitHub\n"
mkdir -p ~/.ssh
ssh-keyscan -H github.com >> ~/.ssh/known_hosts

# SYSTEM #######################################################################
echo -e "-----------------------------------------------------------------------"
echo -e "-- Restarting Apache web server\n"
service apache2 restart

# END ##########################################################################
echo -e "-- ---------------- --"
echo -e "-- END BOOTSTRAPING --"
echo -e "-- ---------------- --"

Application


We have just two files. One for deployment process and another one is the home page of our website so the directory structure is shown below.


vagrant@dev:application$ tree
.
├── ci
│   └── deploy.sh
└── index.php

1 directory, 2 files

index.php


echo "Hello World 1";

deploy.sh


#!/bin/bash

# - VALIDATION ---------------------------------------------------------------------------
NUMBER_ONLY_REGEX="^[0-9]+$"

if [ -z ${1+x} ] || ! [[ $1 =~ $NUMBER_ONLY_REGEX ]]; then
    printf "A valid 'build id' parameter is required\n"
    exit 1
fi
BUILD_ID=$1

if [ -z ${2+x} ] || ! [[ $2 =~ $NUMBER_ONLY_REGEX ]]; then
   printf "A valid 'total releases to keep' parameter is required\n"
   exit 1
else
   if [ $2 -lt 5 ]; then
       printf "You must keep at least '5' releases\n"
       exit 1
   fi
fi
TOTAL_RELEASES_TO_KEEP=$2

if ! [[ -z ${4+x} ]] && [[ $4 =~ $NUMBER_ONLY_REGEX ]] && [ $4 -lt $BUILD_ID ]; then
    BUILD_ID=$4
fi
# ----------------------------------------------------------------------------------------

# - FUNCTIONS ----------------------------------------------------------------------------
function print_message {
    printf "\n%s\n$1\n" ----------------------

    if [ "$1" == "SUCCESS" ]; then exit 0; fi
    if [ "$1" == "FAILURE" ]; then exit 1; fi
}

function clone_repository {
    NEW_RELEASE_DIR=$1
    BRANCH_NAME=$2

    if ! [[ -d $NEW_RELEASE_DIR ]]; then
        print_message "CLONING REPOSITORY"

        git clone git@github.com:Inanzzz/hello-world.git $NEW_RELEASE_DIR
        if [ $? -ne 0 ]; then print_message "FAILURE"; fi

        git -C $NEW_RELEASE_DIR checkout $BRANCH_NAME
        if [ $? -ne 0 ]; then print_message "FAILURE"; fi
    fi
}

function create_symbolic_link {
    NEW_RELEASE_DIR=$1
    WEBSITE_ROOT_DIR=$2

    print_message "CREATING SYMBOLIC LINK"

    ln -fs $NEW_RELEASE_DIR/* $WEBSITE_ROOT_DIR
    if [ $? -ne 0 ]; then print_message "FAILURE"; fi

    printf "Linking $NEW_RELEASE_DIR to $WEBSITE_ROOT_DIR\n"
}

function remove_old_releases {
    TOTAL_RELEASES_TO_REMOVE=$1
    RELEASES_DIR=$2

    if [ "$TOTAL_RELEASES_TO_REMOVE" -gt "0" ]; then
        print_message "REMOVING OLD RELEASES"

        for dir in $(ls -tr $RELEASES_DIR | head -$TOTAL_RELEASES_TO_REMOVE); do
            rm -rf $RELEASES_DIR/$dir

            printf "Removing $RELEASES_DIR/$dir\n"
        done
    fi
}

function update_deploy_script {
    NEW_DEPLOY_SCRIPT_PATH=$1

    print_message "UPDATING DEPLOYMENT SCRIPT"

    cp $NEW_DEPLOY_SCRIPT_PATH ~
    if [ $? -ne 0 ]; then print_message "FAILURE"; fi

    printf "Copying $NEW_DEPLOY_SCRIPT_PATH to ~\n"
}
# ----------------------------------------------------------------------------------------

# - VARIABLES ----------------------------------------------------------------------------
RELEASES_DIR="/var/www/releases"
TOTAL_RELEASES=0
if [ -d $RELEASES_DIR ]; then
    TOTAL_RELEASES=`find $RELEASES_DIR -mindepth 1 -maxdepth 1 -type d | wc -l | sed 's/ //g'`
fi
TOTAL_RELEASES_TO_REMOVE=$((TOTAL_RELEASES-TOTAL_RELEASES_TO_KEEP))
WEBSITE_ROOT_DIR="/var/www/html"
NEW_RELEASE_DIR="$RELEASES_DIR/$BUILD_ID"
NEW_DEPLOY_SCRIPT_PATH="$NEW_RELEASE_DIR/ci/deploy.sh"
BRANCH_NAME=$3
# ----------------------------------------------------------------------------------------

# - INFORMATION --------------------------------------------------------------------------
print_message "DEPLOYMENT INFORMATION"

printf "Release ID: $BUILD_ID\n"
printf "Total releases: $TOTAL_RELEASES\n"
printf "Total releases to keep: $TOTAL_RELEASES_TO_KEEP\n"
printf "Total releases to remove: $TOTAL_RELEASES_TO_REMOVE\n"
printf "Branch to checkout: $BRANCH_NAME\n"
# ----------------------------------------------------------------------------------------

# - PROCESS ------------------------------------------------------------------------------
if [ -d $NEW_RELEASE_DIR ]; then
    create_symbolic_link $NEW_RELEASE_DIR $WEBSITE_ROOT_DIR
    print_message "SUCCESS"
fi

clone_repository $NEW_RELEASE_DIR $BRANCH_NAME
create_symbolic_link $NEW_RELEASE_DIR $WEBSITE_ROOT_DIR
remove_old_releases $TOTAL_RELEASES_TO_REMOVE $RELEASES_DIR
update_deploy_script $NEW_DEPLOY_SCRIPT_PATH

print_message "SUCCESS"
# ----------------------------------------------------------------------------------------

Web server setup


Install web server with vagrant and navigate to http://192.168.99.40/ just to make sure that the server is running. You should see just an empty page. All the steps below are done only once.


GitHub integration setup


ubuntu@website:~$ ssh-keygen -t rsa -b 4096 -C "inanzzz@domain.com"
ubuntu@website:~$ eval "$(ssh-agent -s)"
ubuntu@website:~$ ssh-add ~/.ssh/id_rsa
ubuntu@website:~$ cat ~/.ssh/id_rsa.pub # Copy this key and add it to GitHub
ubuntu@website:~$ ssh -T git@github.com

Create release directory


ubuntu@website:~$ sudo mkdir /var/www/releases
ubuntu@website:~$ sudo chown ubuntu:ubuntu /var/www/releases

Copy deployment script


Create a copy of deploy.sh script in web server root so that Jenkins can run it everytime when we want to deploy our application. Make sure you run chmod +x deploy.sh command as well.


Current structure


ubuntu@website:~$ ls -l
-rwxrwxr-x 1 ubuntu ubuntu 1091 Dec 10 07:15 deploy.sh

ubuntu@website:~$ ls -l /var/www
drwxr-xr-x  3 ubuntu ubuntu 4096 Dec 10 17:31 html
drwxr-xr-x 19 ubuntu ubuntu 4096 Dec 10 17:31 releases

ubuntu@website:~$ ls -l /var/www/html
total 0

ubuntu@website:~$ ls -l /var/www/releases
total 0

Test


If you want, you can manually run ubuntu@website:~$ ./deploy.sh command to simulate Jenkins's deployment process.


Jenkins server setup


Install Jenkins server with vagrant and access GUI via http://192.168.99.30:8080/ just to make sure that the server is running.


SSH integration


Jenkins server runs commands as jenkins user so we should make sure that this user can communicate with web server over SSH by using web server's user ubuntu. Switch to jenkins user and create SSH key. Copy SSH key and paste it into .ssh/authorized_keys file on web server.


ubuntu@jenkins:~$ sudo su -l jenkins
jenkins@jenkins:~$ ssh-keygen -t rsa
jenkins@jenkins:~$ cat ~/.ssh/id_rsa.pub # Copy and paste into ".ssh/authorized_keys" file on web server

After copying SSH key over web server, test if SSH connection works.


jenkins@jenkins:~$ ssh ubuntu@192.168.99.40
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-92-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

96 packages can be updated.
45 updates are security updates.

Last login: Sat Dec  9 21:46:11 2017 from 10.0.2.2
ubuntu@website:~$

Setting up deployment job


Login to Jenkins GUI via http://192.168.99.30:8080/ and do the following.


  1. Go to http://192.168.99.30:8080/newJob, enter item name as Deploy hello-world, select "Freestyle project" and hit "OK" button.

  2. Tick "This project is parameterized" option. Type branch in "Name" and master in "Default Value" text fields. Add another "String parameter" option and type release in "Name" text field.

  3. Under "Build" section, select "Execute shell" option, add ssh ubuntu@192.168.99.40 ./deploy.sh ${BUILD_NUMBER} 10 ${branch} ${release} into the text field and hit "Save" button.

Deployment test


Click "Build with Parameters" menu option on the left:



Deployment outputs


Default deployment


Started by user admin
Building in workspace /var/lib/jenkins/workspace/Deploy hello-world
[Deploy hello-world] $ /bin/sh -xe /tmp/jenkins5964394101391481029.sh
+ ssh ubuntu@192.168.99.40 ./deploy.sh 64 10

----------------------
DEPLOYMENT INFORMATION
Build ID: 64
Total releases: 0
Total releases to keep: 10
Total releases to remove: -10
Branch to checkout: master

----------------------
CLONING REPOSITORY
Cloning into '/var/www/releases/64'...
Already on 'master'
Your branch is up-to-date with 'origin/master'.

----------------------
CREATING SYMBOLIC LINK
Linking /var/www/releases/64 to /var/www/html

----------------------
UPDATING DEPLOYMENT SCRIPT
Copying /var/www/releases/64/ci/deploy.sh to ~

----------------------
SUCCESS
Finished: SUCCESS

Specific branch deployment


Started by user admin
Building in workspace /var/lib/jenkins/workspace/Deploy hello-world
[Deploy hello-world] $ /bin/sh -xe /tmp/jenkins3469822540779562185.sh
+ ssh ubuntu@192.168.99.40 ./deploy.sh 65 10 test/branch

----------------------
DEPLOYMENT INFORMATION
Build ID: 65
Total releases: 1
Total releases to keep: 10
Total releases to remove: -9
Branch to checkout: test/branch

----------------------
CLONING REPOSITORY
Cloning into '/var/www/releases/65'...
Branch test/branch set up to track remote branch test/branch from origin.

----------------------
CREATING SYMBOLIC LINK
Switched to a new branch 'test/branch'
Linking /var/www/releases/65 to /var/www/html

----------------------
UPDATING DEPLOYMENT SCRIPT
Copying /var/www/releases/65/ci/deploy.sh to ~

----------------------
SUCCESS
Finished: SUCCESS

Removing extra releases


Started by user admin
Building in workspace /var/lib/jenkins/workspace/Deploy hello-world
[Deploy hello-world] $ /bin/sh -xe /tmp/jenkins3831172603669268769.sh
+ ssh ubuntu@192.168.99.40 ./deploy.sh 60 10

----------------------
DEPLOYMENT INFORMATION
Build ID: 60
Total releases: 11
Total releases to keep: 10
Total releases to remove: 1
Branch to checkout: master

----------------------
CLONING REPOSITORY
Cloning into '/var/www/releases/60'...
Already on 'master'
Your branch is up-to-date with 'origin/master'.

----------------------
CREATING SYMBOLIC LINK
Linking /var/www/releases/60 to /var/www/html

----------------------
REMOVING OLD RELEASES
Removing /var/www/releases/49

----------------------
UPDATING DEPLOYMENT SCRIPT
Copying /var/www/releases/60/ci/deploy.sh to ~

----------------------
SUCCESS
Finished: SUCCESS

Broken release deployment


Started by user admin
Building in workspace /var/lib/jenkins/workspace/Deploy hello-world
[Deploy hello-world] $ /bin/sh -xe /tmp/jenkins7266370327453402599.sh
+ ssh ubuntu@192.168.99.40 ./deploy.sh 68 10 hello-world

----------------------
DEPLOYMENT INFORMATION
Build ID: 68
Total releases: 4
Total releases to keep: 10
Total releases to remove: -6
Branch to checkout: hello-world

----------------------
CLONING REPOSITORY
git: 'cloning' is not a git command. See 'git --help'.

----------------------
FAILURE
Build step 'Execute shell' marked build as failure
Finished: FAILURE