inanzzz | Managing application configurations in local and kubernetes environments

This idea originates from GitOps paradigm. I am going to show you how an application and an engineer would interact with application configurations. I have seen this approach being used at a few places and it makes sense because of the benefits listed below.

As we all know by now, application configurations often live in application repository. It tends to be a .env file or in some other form. There can also be multiple copies for each environment, so you end up having many files scattered around. Let's cut it short for now to see why I think moving away from this practise is beneficial. I will be using a Golang application in this example.

Benefits

Keeping application configurations in a centralised repository helps ease management and maintenance. There is often many duplicated configuration key-value pair for multiple environments. It would be very time consuming and error prune operation if the repository wasn't centralised.

Following separation of concerns principle at infrastructure level.

Applications become stateless and free of configuration files.

No more configuration files scattered around in application repositories.

It is a GitOps friendly approach where the application configuration is managed with Git for obvious reasons.

CI/CD pipelines don't need to manage application configurations because when an application bootstraps, it will handle everything itself.

No more hard-coded configuration values in Docker images as environment files or values.

Easy to extend if you need to introduce more environments or new key-value pairs.

Exactly same structure/pattern for all application configurations per environment.

How it works

There are two repositories.

Config - This is where all the application configurations live. Engineers use this repository to add/update/delete application configurations.

Application - This is where the application lives.

The way it works differ for "local" and "other" environments. The "local" is where engineers run applications in their PC whereas "other" is where Kubernetes runs your applications (qa, staging, sandbox, prod ...). Both have "green" and "red" sections as described below.

Local

Green - An engineer works with the local config repository clone to add/update/delete application configurations in JSON format and push the changes to GitHub. If they wish to fetch the latest changes, they just pull it. As you can imagine, we won't use this often because we don't always fiddle with configurations.

Red - When an engineer runs an application, it reads the JSON configuration file from the local config repository to map to a struct.

Other

Combining green and red, this has only one way of working and very simple. When the application runs first time, it talks to remote config repository using a GitHub client to pull the configuration to map to a struct.

Implementation

Config repository

You can use this repository to follow GitOps paradigm by keeping all the service and infrastructure configurations in it as shown below.

├── argocd
│   └── ...
├── helm
│   └── ...
├── service
│   ├── local
│   │   └── api.json
│   ├── prod
│   │   └── api.json
│   └── test
│       └── api.json
├── terraform
│   └── ...
└── ...

local/api.json

{
  "auth_url": "http://localhost:8000/api/v1/auth"
}

prod/api.json

{
  "auth_url": "http://auth:8000/",
  "auth_keys": [
	"prod_1"
  ]
}

test/api.json

{
  "auth_url": "http://auth:8000/",
  "auth_keys": [
	"test_1"
  ]
}

Service repository

This is a completely decoupled example. However, if you want to keep the type initialisation at minimum in your service, you can couple your code but I would advise against it.

├── config
│   └── store.go
├── github
│   └── client.go
├── go.mod
├── go.sum
└── main.go

main.go

package main

import (
	"context"
	"fmt"
	"log"
	"os"

	"api/config"
	"api/github"
)

type Config struct {
	AuthURL  string   `json:"auth_url"`
	AuthKeys []string `json:"auth_keys"`
}

func main() {
	// Initialise remote config storage client.
	cfgClient := github.Client{
		Token: os.Getenv("GITHUB_TOKEN"),
		Owner: os.Getenv("GITHUB_OWNER"),
		Repo:  os.Getenv("GITHUB_CONFIG_REPO"),
	}

	// Initialise config storage using remote client and local storage path.
	cfgStore := config.Store{
		Client:      cfgClient,
		ServiceEnv:  os.Getenv("SERVICE_ENV"),
		ServiceName: "api",
		RepoBase:    "../config",
		FileBase:    "/service",
	}

	// Initialise service config without any value.
	var svcConfig Config

	// Load config into service config.
	if err := cfgStore.Load(context.Background(), &svcConfig); err != nil {
		log.Fatalln(err)
	}

	fmt.Printf("%+v\n", svcConfig)
}

If you did couple your code, this is how your main.go file would look like.

package main

import (
	"context"
	"fmt"
	"log"

	"api/config"
)

type Config struct {
	AuthURL  string   `json:"auth_url"`
	AuthKeys []string `json:"auth_keys"`
}

func main() {
	// Initialise service config without any value.
	var svcConfig Config

	// Load config into service config.
	if err := config.Load(context.Background(), &svcConfig, "api"); err != nil {
		log.Fatalln(err)
	}

	fmt.Printf("%+v\n", svcConfig)
}

config/store.go

package config

import (
	"context"
	"encoding/json"
	"fmt"
	"io/ioutil"
)

// Client interfaces remote config storage layer and all remote storages must satisfy this type.
type Client interface {
	LoadConfig(ctx context.Context, cfg interface{}, path string) error
}

type Store struct {
	// Client represents remote config storage.
	Client Client
	// ServiceEnv represents the environment which service is running on.
	ServiceEnv string
	// ServiceName represents the service name which satisfies remote and local storage.
	ServiceName string
	// RepoRoot represents the absolute/relative base path to local clone of remote config storage.
	RepoBase string
	// FileRoot represents the base path to config file in local clone of remote config storage.
	FileBase string
}

// Load reads service config values from either remote storage or local clone of remote storage
// and maps to `cfg` argument which is a struct as in pointer.
func (s Store) Load(ctx context.Context, cfg interface{}) error {
	if s.ServiceEnv == "" {
		return fmt.Errorf("service environment is required")
	}

	if s.ServiceEnv == "local" {
		return s.fromLocal(cfg)
	}

	path := fmt.Sprintf("%s/%s/%s.json", s.FileBase, s.ServiceEnv, s.ServiceName)

	return s.Client.LoadConfig(ctx, cfg, path)
}

// fromLocal reads service configuration file from the local file system.
func (s Store) fromLocal(cfg interface{}) error {
	path := fmt.Sprintf("%s/%s/local/%s.json", s.RepoBase, s.FileBase, s.ServiceName)

	file, err := ioutil.ReadFile(path)
	if err != nil {
		return fmt.Errorf("read file from local environment: %w", err)
	}

	if err := json.Unmarshal(file, cfg); err != nil {
		return fmt.Errorf("unmarshal file from local environment: %w", err)
	}

	return nil
}

github/client.go

Beware GitHub API rate limiting.

package github

import (
	"context"
	"encoding/json"
	"fmt"
	"io/ioutil"

	"github.com/google/go-github/v42/github"
	"golang.org/x/oauth2"
)

type Client struct {
	Token string
	Owner string
	Repo  string
}

// LoadConfig reads config file from the remote repository and maps it to its struct which is a pointer.
func (c Client) LoadConfig(ctx context.Context, cfg interface{}, path string) error {
	if c.Token == "" {
		return fmt.Errorf("github token is required")
	}

	clt := github.NewClient(
		oauth2.NewClient(ctx, oauth2.StaticTokenSource(&oauth2.Token{AccessToken: c.Token})),
	)

	con, res, err := clt.Repositories.DownloadContents(ctx, c.Owner, c.Repo, path, nil)
	if err != nil {
		return fmt.Errorf("download content: %w", err)
	}
	if res.StatusCode != 200 {
		return fmt.Errorf("unexpected response code: %d", res.StatusCode)
	}

	file, err := ioutil.ReadAll(con)
	if err != nil {
		return fmt.Errorf("read file from github: %w", err)
	}

	if err := json.Unmarshal(file, cfg); err != nil {
		return fmt.Errorf("unmarshal file from github: %w", err)
	}

	return nil
}

You will need these environment variables or use your preferred names and values.

SERVICE_ENV: A single environment. local/test/sbox/prod/....
GITHUB_TOKEN: User personal access token
GITHUB_OWNER: GitHub account owner
GITHUB_CONFIG_REPO: GitHub config repository

Test

$ env | grep GITHUB
GITHUB_CONFIG_REPO=config
GITHUB_TOKEN=ghp_cctTG5cyYy6cyxbcFJG7u56fghyfhjFgd432f
GITHUB_OWNER=you

$ export SERVICE_ENV=local
$ go run main.go
{AuthURL:http://localhost:8000/api/v1/auth AuthKeys:[]}

$ export SERVICE_ENV=test
$ go run main.go
{AuthURL:http://auth:8000/ AuthKeys:[test_1]}