GitOps style CI/CD setup using GitHub Actions, ArgoCD, Helm and Kubernetes

20/03/2022 - DOCKER, GIT, GO, HELM, KUBERNETES, ARGOCD

I previously wrote about such subject but this is slightly different in that the application repository has no CI/CD related configuration files anymore. They all are stored in their dedicated repository. The reason for this separation is because we want to achieve a proper GitOps practise. In short, application and config repository are all separated from each other.


Current setup


We have four Kubernetes clusters which are argocd, dev, sbox and prod. ArgoCD runs on argocd cluster and deploys to other three clusters.


Deployment strategy



Setup


Prepare GitHub


I assume you already have an application repository called pacman.


  1. Create Docker access token called GITHUB_ACTIONS with read, write and delete permissions. Result: 8d8937fe-753b-4f2b-96bc-b3603e4ed2b0

  2. Create PAT called ARGOCD using "repo" scopes. Result: ghp_fhjkiuUYuy456frreSgrtry2

  3. Create PAT called GITHUB_ACTIONS using "repo" scopes. Result: ghp_re543tgtu7hgaTHhyh6757

  4. Create pacman repository secret called ACTIONS_TOKEN using ghp_re543tgtu7hgaTHhyh6757.

  5. Create pacman repository secret called DOCKERHUB_TOKEN using 8d8937fe-753b-4f2b-96bc-b3603e4ed2b0.

  6. Create pacman repository secret called DOCKERHUB_USER using your Docker username.

Prepare Kubernetes


$ minikube start -p argocd --vm-driver=virtualbox --memory=2000
$ minikube start -p dev --vm-driver=virtualbox --memory=2000
$ minikube start -p sbox --vm-driver=virtualbox --memory=2000
$ minikube start -p prod --vm-driver=virtualbox --memory=2000

$ kubectl config get-contexts
CURRENT   NAME     CLUSTER   AUTHINFO   NAMESPACE
*         argocd   argocd    argocd
          dev      dev       dev
          prod     prod      prod
          sbox     sbox      sbox

Prepare ArgoCD


Install ArgoCD in terminal

$ kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.2.4/manifests/install.yaml

Obtain password

$ kubectl get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
// gYzYfJ5kxzJsdZGK

Login to UI

$ kubectl port-forward svc/argd-server 8443:443
Forwarding from 127.0.0.1:8443 -> 8080

// Visit 127.0.0.1:8443 and use admin:gYzYfJ5kxzJsdZGK

Login CLI

$ argocd --insecure login 127.0.0.1:8443

Add clusters

$ argocd cluster add dev
$ argocd cluster add sbox
$ argocd cluster add prod

Add repository

This is optional. You could use "default".


$ argocd repo add https://github.com/you/config \
    --type git \
    --name config \
    --project config \
    --username you \
    --password ghp_fhjkiuUYuy456frreSgrtry2

Create project

This is optional. You could use "default".


$ argocd proj create --file ~/local/file/system/config/infra/argocd/project.yaml

Create applications

You could use ApplicationSet instead.


$ argocd app create --file ~/local/file/system/config/infra/argocd/pacman/dev.yaml
$ argocd app create --file ~/local/file/system/config/infra/argocd/pacman/sbox.yaml
$ argocd app create --file ~/local/file/system/config/infra/argocd/pacman/prod.yaml

Application repository


├── .github
│   └── workflows
│       ├── merge.yaml
│       ├── pull_request.yaml
│       └── release.yaml
├── docker
│   └── Dockerfile
├── .dockerignore
├── main.go
└── main_test.go

Files


.dockerignore

.dockerignore
.gitignore
*.md
pacman
.DS_Store

docker/
.git/
.idea/

main.go

package main

import (
	"log"
	"net/http"
	"os"
)

var ver string

func main() {
	rtr := http.DefaultServeMux
	rtr.HandleFunc("/", home{}.handle)

	addr := os.Getenv("HTTP_ADDR")

	log.Printf("%s: info: http listen and serve pacman: %s", ver, addr)
	if err := http.ListenAndServe(addr, rtr); err != nil && err != http.ErrServerClosed {
		log.Printf("%s: error: http listen and serve pacman: %s", ver, err)
	}
}

type home struct{}

func (h home) handle(w http.ResponseWriter, r *http.Request) {
	log.Printf("%s: info: X-Request-ID: %s\n", ver, r.Header.Get("X-Request-ID"))
	_, _ = w.Write([]byte("pacman" + ver))
}

main_test.go

package main

import (
	"net/http"
	"net/http/httptest"
	"testing"
)

func Test_home_handle(t *testing.T) {
	req := httptest.NewRequest(http.MethodGet, "/", nil)
	res := httptest.NewRecorder()

	home{}.handle(res, req)

	if res.Code != http.StatusOK {
		t.Error("expected 200 but got", res.Code)
	}
}

Dockerfile

FROM golang:1.17.5-alpine3.15 as build
WORKDIR /source
COPY . .
ARG VER
RUN CGO_ENABLED=0 go build -ldflags "-s -w -X main.ver=${VER}" -o pacman main.go

FROM alpine:3.15
COPY --from=build /source/pacman /pacman
EXPOSE 8080
ENTRYPOINT ["./pacman"]

pull_request.yaml

# Trigger the workflow only when:
# - a new pull request with any name/type is opened against the master, develop, hotfix/* or release/* branch
# - a commit is directly pushed to the pull request

name: Pull Request

on:
  pull_request:
    branches:
      - master
      - develop
      - hotfix/*
      - release/*

jobs:

  setup:
    runs-on: ubuntu-latest
    steps:
      - name: Use repository
        uses: actions/checkout@v2
      - name: Upload repository
        uses: actions/upload-artifact@v2
        with:
          name: repository
          path: |
            ${{ github.workspace }}/main.go
            ${{ github.workspace }}/main_test.go
            ${{ github.workspace }}/go.mod
            ${{ github.workspace }}/go.sum

  test:
    needs: setup
    runs-on: ubuntu-latest
    steps:
      - name: Use Golang 1.17
        uses: actions/setup-go@v2
        with:
          go-version: 1.17
      - name: Download repository
        uses: actions/download-artifact@v2
        with:
          name: repository
      - name: Run tests
        run: go test -v -race -timeout=180s -count=1 -cover ./...

merge.yaml

This is going to push a commit to config repository to force trigger dev deployment.


# Trigger the workflow only when:
# - an existing pull request with any name/type is merged to the master or develop branch
# - a commit is directly pushed to the master or develop branch

name: Merge

on:
  push:
    branches:
      - master
      - develop

jobs:

  setup:
    runs-on: ubuntu-latest
    outputs:
      ver: ${{ steps.vars.outputs.ver }}
    steps:
      - name: Use repository
        uses: actions/checkout@v2
      - name: Build variables
        id: vars
        run: |
          echo "::set-output name=ver::$(git rev-parse --short "$GITHUB_SHA")"
      - name: Upload repository
        uses: actions/upload-artifact@v2
        with:
          name: repository
          path: |
            ${{ github.workspace }}/docker
            ${{ github.workspace }}/.dockerignore
            ${{ github.workspace }}/main.go
            ${{ github.workspace }}/main_test.go
            ${{ github.workspace }}/go.mod
            ${{ github.workspace }}/go.sum

  test:
    needs: setup
    runs-on: ubuntu-latest
    steps:
      - name: Use Golang 1.17
        uses: actions/setup-go@v2
        with:
          go-version: 1.17
      - name: Download repository
        uses: actions/download-artifact@v2
        with:
          name: repository
      - name: Run tests
        run: go test -v -race -timeout=180s -count=1 -cover ./...

  docker:
    needs: [setup, test]
    runs-on: ubuntu-latest
    steps:
      - name: Download repository
        uses: actions/download-artifact@v2
        with:
          name: repository
      - name: Login to DockerHub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKERHUB_USER }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - name: Build and push image
        uses: docker/build-push-action@v2
        with:
          push: true
          file: docker/Dockerfile
          tags: ${{ github.repository }}:latest
          build-args: VER=${{ needs.setup.outputs.ver }}

  config:
    needs: [setup, docker]
    runs-on: ubuntu-latest
    steps:
      - name: Use config repository
        uses: actions/checkout@v2
        with:
          repository: ${{ github.repository_owner }}/config
          ref: master
          token: ${{ secrets.ACTIONS_TOKEN }}
      - name: Push commit hash to config repository
        run: |
          echo ${{ needs.setup.outputs.ver }} > infra/helm/pacman/crds/vcs/hash
          git config user.name $(git log -n 1 --pretty=format:%an)
          git config user.email $(git log -n 1 --pretty=format:%ae)
          git commit infra/helm/pacman/crds/vcs/hash -m "pacman ${{ needs.setup.outputs.ver }}"
          git push origin HEAD

release.yaml

This is going to push a commit to config repository to force trigger sbox deployment.


# Trigger the workflow only when:
# - a new release is released which excludes pre-release and draft

name: Release

on:
  release:
    types:
      - released

jobs:

  setup:
    runs-on: ubuntu-latest
    steps:
      - name: Use repository
        uses: actions/checkout@v2
      - name: Upload repository
        uses: actions/upload-artifact@v2
        with:
          name: repository
          path: |
            ${{ github.workspace }}/.docker
            ${{ github.workspace }}/.dockerignore
            ${{ github.workspace }}/main.go
            ${{ github.workspace }}/main_test.go
            ${{ github.workspace }}/go.mod
            ${{ github.workspace }}/go.sum

  test:
    needs: setup
    runs-on: ubuntu-latest
    steps:
      - name: Use Golang 1.17
        uses: actions/setup-go@v2
        with:
          go-version: 1.17
      - name: Download repository
        uses: actions/download-artifact@v2
        with:
          name: repository
      - name: Run tests
        run: go test -v -race -timeout=180s -count=1 -cover ./...

  docker:
    needs: [setup, test]
    runs-on: ubuntu-latest
    steps:
      - name: Download repository
        uses: actions/download-artifact@v2
        with:
          name: repository
      - name: Login to DockerHub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKERHUB_USER }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - name: Build and push image
        uses: docker/build-push-action@v2
        with:
          push: true
          file: docker/Dockerfile
          tags: ${{ github.repository }}:${{ github.event.release.tag_name }}
          build-args: VER=${{ github.event.release.tag_name }}

  config:
    needs: docker
    runs-on: ubuntu-latest
    steps:
      - name: Use config repository
        uses: actions/checkout@v2
        with:
          repository: ${{ github.repository_owner }}/config
          ref: master
          token: ${{ secrets.ACTIONS_TOKEN }}
      - name: Push release tag to config repository
        run: |
          echo ${{ github.event.release.tag_name }} > infra/helm/pacman/crds/vcs/tag
          git config user.name $(git log -n 1 --pretty=format:%an)
          git config user.email $(git log -n 1 --pretty=format:%ae)
          git commit infra/helm/pacman/crds/vcs/tag -m "pacman ${{ github.event.release.tag_name }}"
          git push origin HEAD

Config repository


└── infra
    ├── argocd
    │   └── pacman
    │       ├── dev.yaml
    │       ├── prod.yaml
    │       ├── project.yaml
    │       └── sbox.yaml
    └── helm
        └── pacman
            ├── Chart.yaml
            ├── crds
            │   └── vcs
            │       ├── hash
            │       └── tag
            ├── dev.yaml
            ├── prod.yaml
            ├── sbox.yaml
            └── templates
                ├── configmap.yaml
                ├── deployment.yaml
                └── service.yaml

Files


argocd/dev.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: pacman-dev

spec:
  project: pacman
  source:
    repoURL: https://github.com/you/config
    path: infra/helm/pacman
    targetRevision: HEAD
    helm:
      valueFiles:
        - dev.yaml
  destination:
    name: dev
    namespace: default
  syncPolicy:
    syncOptions:
      - ApplyOutOfSyncOnly=true
      - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true

argocd/sbox.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: pacman-sbox

spec:
  project: pacman
  source:
    repoURL: https://github.com/you/config
    path: infra/helm/pacman
    targetRevision: HEAD
    helm:
      valueFiles:
        - sbox.yaml
  destination:
    name: sbox
    namespace: default
  syncPolicy:
    syncOptions:
      - ApplyOutOfSyncOnly=true
      - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true

argocd/prod.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: pacman-prod

spec:
  project: pacman
  source:
    repoURL: https://github.com/you/config
    path: infra/helm/pacman
    targetRevision: HEAD
    helm:
      valueFiles:
        - prod.yaml
  destination:
    name: prod
    namespace: default
  syncPolicy:
    syncOptions:
      - ApplyOutOfSyncOnly=true
      - CreateNamespace=true

argocd/project.yaml

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: pacman

spec:
  destinations:
    - name: '*'
      namespace: '*'
      server: '*'
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'
  orphanedResources:
    warn: true
  sourceRepos:
    - https://github.com/you/config

helm/hash

Dedicated for dev deployment. This file is automatically updated by "merge" GitHub Actions workflow.


1234567

helm/tag

Dedicated for sbox and prod deployment. This file is automatically updated by "release" GitHub Actions workflow.


latest

helm/Chart.yaml

apiVersion: v2
name: pacman
version: 0.0.0

helm/dev.yaml

env:
  HTTP_ADDR: :8080

image:
  name: you/pacman
  tag: latest
  pull: Always

deployment:
  force: true
  replicas: 1
  container:
    name: go
    port: 8080

service:
  type: ClusterIP
  port: 8080

helm/sbox.yaml

env:
  HTTP_ADDR: :8080

image:
  name: you/pacman
  tag: ""
  pull: IfNotPresent

deployment:
  force: false
  replicas: 2
  container:
    name: go
    port: 8080

service:
  type: ClusterIP
  port: 8080

helm/prod.yaml

env:
  HTTP_ADDR: :8080

image:
  name: you/pacman
  tag: ""
  pull: IfNotPresent

deployment:
  force: false
  replicas: 2
  container:
    name: go
    port: 8080

service:
  type: ClusterIP
  port: 8080

helm/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Chart.Name }}
  namespace: default

data:
  HTTP_ADDR: {{ .Values.env.HTTP_ADDR }}

helm/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: {{ .Chart.Name }}
  namespace: default

spec:
  type: {{ .Values.service.type }}
  selector:
    app: {{ .Chart.Name }}
  ports:
    - port: {{ .Values.service.port }}
      targetPort: {{ .Values.deployment.container.port }}

helm/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Chart.Name }}
  namespace: default
  labels:
    app: {{ .Chart.Name }}

spec:
  replicas: {{ .Values.deployment.replicas }}
  selector:
    matchLabels:
      app: {{ .Chart.Name }}
  template:
    metadata:
      labels:
        app: {{ .Chart.Name }}
      {{- if .Values.deployment.force }}
      annotations:
        roller: {{ .Files.Get "crds/vcs/hash" | trim }}
      {{- end }}
    spec:
      containers:
        - name: {{ .Values.deployment.container.name }}
          image: "{{ .Values.image.name }}:{{ .Values.image.tag | default (.Files.Get "crds/vcs/tag" | trim) }}"
          imagePullPolicy: {{ .Values.image.pull }}
          ports:
            - containerPort: {{ .Values.deployment.container.port }}
          envFrom:
            - configMapRef:
                name: {{ .Chart.Name }}

CI/CD screenshots


AcroCD


Cluster


Repository


Project


Application


GitHub


Merge


Commit in config repository!



Release


Commit in config repository!



Docker


Tag