Managing Golang application secrets in local and kubernetes environments
23/02/2022 - AWS, GO, KUBERNETES
This idea originates from how we used to handle application secrets where I worked before. The way it was managed was fine but needed some enhancements in my opinion. I am going to show you how an application and an engineer would interact with application secrets.
As we all know by now, application secrets often live in applications' repository. It tends to be a .env file or environment variables in a plain text form which is not ideal for obvious reasons. In this example we are going to benefit from encryption/decryption and I will be using a Golang application. This is more secure to old way because, intruder has to steal JSON files, SSH files, your application, work out how the relevant environment variables work so on. Too much to deal with and harder to guess.
Benefits
- Keeping application secrets in a centralised service helps ease management and maintenance.
- Applications become stateless and configuration file free.
- Secrets are always kept in encrypted form in our machines.
- No more configuration files scattered around in application repositories.
- CI/CD pipelines don't need to manage application secrets because when an application bootstraps, it will handle everything itself.
- No more hard-coded secrets in Docker images as environment files or values.
- Easy to extend if you need to introduce more environments or new key-value pairs.
- Exactly same structure/pattern for all application configurations for each environments.
How it works
User will use secret push ... command to push new secrets to remote storage and use secret pull ... command to pull secrets to local environment. The important point here is that, pulling secrets only applicable for local environment because you don't want to keep hitting remote service because it would be slow as well as costly. However, all the other environments will just use remote service.
Remote secret store is where we are going to store the secrets. We will be using AWS SSM service but it could be any other service such as Vault, Azure, GCP etc. When the user pulls secrets they will be saved to local machine in a JSON file using SSH keys to encrypt values. When the application runs, this file will be read, decrypted and mapped to struct.
As mentioned earlier, the way it works differ for "local" and "other" environments. The "local" is where engineers run applications in their PC whereas "other" is where Kubernetes runs your applications (qa, staging, sandbox, prod ...). Both have "green" and "red" sections as described below.
Local
- Green - An engineer uses
secret push ...command to push secrets to remote storage and usessecret pull ...to pull secrets to local machine. Note: These will be barely used because we don't always interact with secrets. - Red - An engineer runs application and application reads the JSON secrets file, once, from local file system to map to a struct using a custom struct tag called
secret.
Other
Combining green and red, this has only one way of working and very simple. When the application runs first time, it talks to remote storage to pull the secrets to map to a struct.
Things to know
First all all, I hardcoded some values and made some code "opinionated" but you can refactor it to improve the it. The reason is because, our point here is not to deliver the most beautiful code! It is about giving you an idea on how to handle secrets.
- When typed in terminal using
pushcommand, secret value is not printed to terminal for security reasons. - Multiline secret values should be entered as single line values using
\nas line breaks. For instance, SSH keys. - Currently
pushcommand handles single key-value pair at a time. Ideally, it should handle multiple. - The format for local storage is
User_Home_Directory/.Organisation_Name/secret/Service_Environment/Service_Name.json. e.g./Users/you/.inanzzz/secret/test/app.json
Note
If you want your application to always read secrets from the JSON file when running on the local environment, you can modify the secret.go file to get rid of opinionated code. I suggest you to this because, again, you don't want to depend on a third party service while working on local environment. Maybe not for production secrets!
Secret repository
├── aws
│ ├── aws.go
│ └── kms.go
├── crypto
│ └── crypto.go
├── go.mod
├── main.go
└── secret
├── command.go
└── secret.go
Files
aws.go
package aws
import (
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
)
func NewSession() (*session.Session, error) {
return session.NewSessionWithOptions(session.Options{
Profile: "localstack",
Config: aws.Config{
Region: aws.String("eu-west-1"),
Endpoint: aws.String("http://localhost:4566"),
},
})
}
kms.go
package aws
import (
"context"
"fmt"
"strings"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/ssm"
)
type KMS struct {
client *ssm.SSM
}
func NewKMS(ses *session.Session) *KMS {
return &KMS{
client: ssm.New(ses),
}
}
// Load gets multiple secret values from the remote location using their path prefix.
func (k *KMS) Load(ctx context.Context, path string) (map[string]string, error) {
res, err := k.client.GetParametersByPathWithContext(ctx, &ssm.GetParametersByPathInput{
Path: aws.String(path),
})
if err != nil {
return nil, err
}
sec := make(map[string]string, len(res.Parameters))
for _, param := range res.Parameters {
sec[strings.TrimPrefix(*param.Name, path)] = *param.Value
}
return sec, nil
}
// Insert puts a new secret key-value pair to the remote location using its path prefix.
func (k *KMS) Insert(ctx context.Context, path, key, val string) error {
_, err := k.client.PutParameterWithContext(ctx, &ssm.PutParameterInput{
Name: aws.String(fmt.Sprintf("%s/%s", path, key)),
Tier: aws.String(ssm.ParameterTierStandard),
Type: aws.String(ssm.ParameterTypeString),
Value: aws.String(val),
Overwrite: aws.Bool(true),
})
if err != nil {
return err
}
return nil
}
crypto.go
package crypto
import (
"crypto/rand"
"crypto/rsa"
"crypto/sha256"
"crypto/x509"
"encoding/hex"
"encoding/pem"
"fmt"
"golang.org/x/crypto/ssh"
"os"
)
type Crypto struct{}
// Encrypt encrypts plain value using SSH public key.
func (c Crypto) Encrypt(plain string) (string, error) {
sshKey, err := c.sshKey("id_rsa.pub")
if err != nil {
return "", fmt.Errorf("get ssh key: %w", err)
}
pubKey, _, _, _, err := ssh.ParseAuthorizedKey(sshKey)
if err != nil {
return "", fmt.Errorf("parse authorised key: %w", err)
}
key := pubKey.(ssh.CryptoPublicKey).CryptoPublicKey().(*rsa.PublicKey)
val, err := rsa.EncryptOAEP(sha256.New(), rand.Reader, key, []byte(plain), nil)
if err != nil {
return "", fmt.Errorf("encrypt: %w", err)
}
return hex.EncodeToString(val), nil
}
// Decrypt decrypts previously encrypted value using SSH private key.
func (c Crypto) Decrypt(encoded string) (string, error) {
sshKey, err := c.sshKey("id_rsa")
if err != nil {
return "", fmt.Errorf("get ssh key: %w", err)
}
decoded, err := hex.DecodeString(encoded)
if err != nil {
return "", fmt.Errorf("decode data: %w", err)
}
pemBlock, _ := pem.Decode(sshKey)
key, err := x509.ParsePKCS1PrivateKey(pemBlock.Bytes)
if err != nil {
return "", fmt.Errorf("parse private key: %w", err)
}
val, err := rsa.DecryptOAEP(sha256.New(), rand.Reader, key, decoded, nil)
if err != nil {
return "", fmt.Errorf("decode: %w", err)
}
return string(val), nil
}
// sshKey returns either a public or private SSH key.
func (c Crypto) sshKey(file string) ([]byte, error) {
home, err := os.UserHomeDir()
if err != nil {
return nil, err
}
key, err := os.ReadFile(fmt.Sprintf("%s/.ssh/%s", home, file))
if err != nil {
return nil, err
}
return key, nil
}
command.go
package secret
import (
"context"
"encoding/json"
"fmt"
"os"
"github.com/you/secret/aws"
"github.com/you/secret/crypto"
)
type Command struct {
Crypto crypto.Crypto
Storage *aws.KMS
Action string
Environment string
Service string
SecretKey string
SecretVal string
}
// Pull talks to remote secret storage to fetch secrets, talks to crypto
// service to encrypt the values, constructs a JSON file using key-value pair
// and saves it to local path.
func (c Command) Pull(ctx context.Context) error {
if c.Environment == "" {
return fmt.Errorf("service environment variable is required")
}
list, err := c.Storage.Load(ctx, fmt.Sprintf("/%s/%s/", c.Environment, c.Service))
if err != nil {
return fmt.Errorf("load secrets: %w", err)
}
for k, v := range list {
val, err := c.Crypto.Encrypt(v)
if err != nil {
return fmt.Errorf("encrypt secret value: %w", err)
}
list[k] = val
}
data, err := json.MarshalIndent(list, "", " ")
if err != nil {
return fmt.Errorf("marshal secrets: %w", err)
}
home, err := os.UserHomeDir()
if err != nil {
return fmt.Errorf("get home directory: %w", err)
}
dir := fmt.Sprintf("%s/.inanzzz/secret/%s", home, c.Environment)
if err := os.MkdirAll(dir, 0755); err != nil {
return fmt.Errorf("create secret directory: %w", err)
}
file := fmt.Sprintf("%s/.inanzzz/secret/%s/%s.json", home, c.Environment, c.Service)
if err := os.WriteFile(file, data, 0755); err != nil {
return fmt.Errorf("create secret file: %w", err)
}
return nil
}
// Push pushes a secret key-value pair to remote secret storage.
func (c Command) Push(ctx context.Context) error {
if c.Environment == "" {
return fmt.Errorf("service environment variable is required")
}
err := c.Storage.Insert(ctx, fmt.Sprintf("/%s/%s/", c.Environment, c.Service), c.SecretKey, c.SecretVal)
if err != nil {
return fmt.Errorf("insert secret: %w", err)
}
return nil
}
secret.go
package secret
import (
"context"
"encoding/json"
"fmt"
"io/ioutil"
"os"
"reflect"
"github.com/you/secret/aws"
"github.com/you/secret/crypto"
)
type Secret struct {
Crypto crypto.Crypto
Storage *aws.KMS
Service string
Environment string
}
func (s Secret) Load(ctx context.Context, cfg interface{}) error {
switch s.Environment {
case "":
return fmt.Errorf("service environment variable is required")
case "local":
return s.fromFile(cfg)
}
return s.fromClient(ctx, cfg)
}
func (s Secret) fromFile(cfg interface{}) error {
home, err := os.UserHomeDir()
if err != nil {
return err
}
file, err := ioutil.ReadFile(fmt.Sprintf("%s/.inanzzz/secret/local/%s.json", home, s.Service))
if err != nil {
return err
}
data := make(map[string]string)
if err := json.Unmarshal(file, &data); err != nil {
return err
}
return s.bind(cfg, data, true)
}
func (s Secret) fromClient(ctx context.Context, cfg interface{}) error {
data, err := s.Storage.Load(ctx, fmt.Sprintf("/%s/%s/", s.Environment, s.Service))
if err != nil {
return fmt.Errorf("load secrets: %w", err)
}
return s.bind(cfg, data, false)
}
func (s Secret) bind(cfg interface{}, data map[string]string, dec bool) error {
configSource := reflect.ValueOf(cfg)
if configSource.Kind() != reflect.Ptr {
return fmt.Errorf("config must be a pointer")
}
configSource = configSource.Elem()
if configSource.Kind() != reflect.Struct {
return fmt.Errorf("config must be a struct")
}
configType := configSource.Type()
for i := 0; i < configSource.NumField(); i++ {
fieldTag, ok := configType.Field(i).Tag.Lookup("secret")
if !ok {
continue
}
fieldName := configType.Field(i).Name
fieldValue := configSource.FieldByName(fieldName)
if !fieldValue.IsValid() {
continue
}
if !fieldValue.CanSet() {
continue
}
enc, ok := data[fieldTag]
if !ok {
continue
}
if !dec {
fieldValue.SetString(enc)
continue
}
val, err := s.Crypto.Decrypt(enc)
if err != nil {
return fmt.Errorf("decrypt %s: %w", fieldTag, err)
}
fieldValue.SetString(val)
}
return nil
}
main.go
package main
import (
"bufio"
"context"
"flag"
"fmt"
"golang.org/x/term"
"log"
"os"
"strings"
"github.com/you/secret/aws"
"github.com/you/secret/crypto"
"github.com/you/secret/secret"
)
func main() {
ctx := context.Background()
// Initialise AWS session.
awsSes, err := aws.NewSession()
if err != nil {
log.Fatalln(err)
}
// Initialise AWS KMS service which is remote secret storage.
scrClient := aws.NewKMS(awsSes)
// Initialise secret command handler.
scrCmd := secret.Command{
Crypto: crypto.Crypto{},
Storage: scrClient,
Environment: os.Getenv("SERVICE_ENV"),
}
// Capture user input and update command handler.
flag.Parse()
scrCmd.Action = flag.Arg(0)
if flag.Arg(1) == "-svc" || flag.Arg(1) == "--svc" {
scrCmd.Service = flag.Arg(2)
}
switch scrCmd.Action {
case "pull":
if err := scrCmd.Pull(ctx); err != nil {
log.Fatalln(err)
}
fmt.Println("Pulled!")
case "push":
var err error
scrCmd.SecretKey, err = key()
if err != nil {
log.Fatalln(err)
}
scrCmd.SecretVal, err = val()
if err != nil {
log.Fatalln(err)
}
if err := scrCmd.Push(ctx); err != nil {
log.Fatalln(err)
}
fmt.Println("Pushed!")
default:
fmt.Println("Invalid action!")
}
}
// key prompts user to enter secret key which is visible on terminal.
func key() (string, error) {
fmt.Printf("Key > ")
rdr := bufio.NewReader(os.Stdin)
for {
answer, err := rdr.ReadString('\n')
if err != nil {
return "", err
}
return strings.TrimSuffix(answer, "\n"), nil
}
}
// val prompts user to enter secret value which is not visible on terminal.
func val() (string, error) {
fmt.Printf("Value > ")
raw, err := term.MakeRaw(0)
if err != nil {
return "", err
}
defer term.Restore(0, raw)
var (
prompt string
answer string
)
trm := term.NewTerminal(os.Stdin, prompt)
for {
char, err := trm.ReadPassword(prompt)
if err != nil {
return "", err
}
answer += char
if char == "" || char == answer {
return answer, nil
}
}
}
Usage
Run $ go install github.com/you/secret@latest command to install the binary to your machine so that you can start using $ secret push/pull... commands from anywhere.
Application repository
package main
import (
"context"
"fmt"
"log"
"os"
"strings"
"github.com/you/secret/aws"
"github.com/you/secret/crypto"
"github.com/you/secret/secret"
)
type Config struct {
ServiceAddress string `json:"service_address"`
MySQLPass string `secret:"mysql_pass"`
PrivateKey string `secret:"private_key"`
}
func main() {
ctx := context.Background()
cfg := Config{
ServiceAddress: "http://localhost/",
MySQLPass: "pass",
PrivateKey: "",
}
// Initialise AWS session.
awsSes, err := aws.NewSession()
if err != nil {
log.Fatalln(err)
}
// Initialise AWS KMS service which is remote secret storage.
scrClient := aws.NewKMS(awsSes)
scr := secret.Secret{
Crypto: crypto.Crypto{},
Storage: scrClient,
Service: "api",
Environment: os.Getenv("SERVICE_ENV"),
}
if err := scr.Load(ctx, &cfg); err != nil {
log.Fatalln(err)
}
fmt.Println("----")
fmt.Println("Service Address:", cfg.ServiceAddress)
fmt.Println("MySQL Password:", cfg.MySQLPass)
fmt.Println("Private Key:")
fmt.Println(strings.ReplaceAll(cfg.PrivateKey, `\n`, "\n"))
}
If you want to simplify the usage in your applications, you can hardcode some variables and dependencies in the secret repository. I am leaving this up to you but your final code would look like this instead.
package main
import (
"context"
"fmt"
"log"
"os"
"strings"
"github.com/you/secret/secret"
)
type Config struct {
ServiceAddress string `json:"service_address"`
MySQLPass string `secret:"mysql_pass"`
PrivateKey string `secret:"private_key"`
}
func main() {
cfg := Config{
ServiceAddress: "http://localhost/",
MySQLPass: "pass",
PrivateKey: "",
}
if err := secret.Load(context.Background(), &cfg, "api"); err != nil {
log.Fatalln(err)
}
// ...
}
Test
Let's pretend like we are dealing with "prod" environment secrets and our application is called "api".
$ env | grep SERVICE_ENV
SERVICE_ENV=prod
Push secrets to AWS KMS.
// Value is 123123
$ secret push --svc api
Key > mysql_pass
Value >
Pushed!
// Value is -----BEGIN RSA PRIVATE KEY-----\nvpGGduSuXsp++jPUTvQxAZMRX/Y0Q==\n-----END RSA PRIVATE KEY-----
$ secret push --svc api
Key > private_key
Value >
Pushed!
Verify their existence.
$ aws --profile localstack --endpoint-url http://localhost:4566 ssm get-parameters-by-path --path "/prod/api/"
{
"Parameters": [
{
"Name": "/prod/api/mysql_pass",
"Type": "String",
"Value": "123123",
},
{
"Name": "/prod/api/private_key",
"Type": "String",
"Value": "-----BEGIN RSA PRIVATE KEY-----\nvpGGduSuXsp++jPUTvQxAZMRX/Y0Q==\n-----END RSA PRIVATE KEY-----",
}
]
}
Given our environment is set to prod the application will run because as mentioned before, only "local" environment reads from the local storage.
api$ go run main.go
----
Service Address: http://localhost/
MySQL Password: 123123
Private Key:
-----BEGIN RSA PRIVATE KEY-----
vpGGduSuXsp++jPUTvQxAZMRX/Y0Q==
-----END RSA PRIVATE KEY-----
Now let's change the environment to local and run the application. This will fail.
api$ export SERVICE_ENV=local
api$ go run main.go
2022/03/05 18:44:32 open /Users/you/.inanzzz/secret/local/api.json: no such file or directory
exit status 1
In order to make this work, we need to push secrets first and pull then retry. I assume you pushed already!
$ secret pull --svc api
Pulled!
$ cat $HOME/.inanzzz/secret/local/api.json
{
"mysql_pass": "a64c9135...",
"private_key": "24501aa8e3f3..."
}
api$ go run main.go
----
Service Address: http://localhost/
MySQL Password: 111111
Private Key:
-----BEGIN RSA PRIVATE KEY-----
vpGGduSuXsp++jPUTvQxAZMRX/Y0Q==
-----END RSA PRIVATE KEY-----