23/02/2022 - AWS, GO, KUBERNETES
This idea originates from how we used to handle application secrets where I worked before. The way it was managed was fine but needed some enhancements in my opinion. I am going to show you how an application and an engineer would interact with application secrets.
As we all know by now, application secrets often live in applications' repository. It tends to be a .env
file or environment variables in a plain text form which is not ideal for obvious reasons. In this example we are going to benefit from encryption/decryption and I will be using a Golang application. This is more secure to old way because, intruder has to steal JSON files, SSH files, your application, work out how the relevant environment variables work so on. Too much to deal with and harder to guess.
Alternative to hex
encoder, you can use base64.URLEncoding/base64.RawStdEncoding
functions for shorter results.
User will use secret push ...
command to push new secrets to remote storage and use secret pull ...
command to pull secrets to local environment. The important point here is that, pulling secrets only applicable for local
environment because you don't want to keep hitting remote service because it would be slow as well as costly. However, all the other environments will just use remote service.
Remote secret store is where we are going to store the secrets. We will be using AWS SSM service but it could be any other service such as Vault, Azure, GCP etc. When the user pulls secrets they will be saved to local machine in a JSON file using SSH keys to encrypt values. When the application runs, this file will be read, decrypted and mapped to struct.
As mentioned earlier, the way it works differ for "local" and "other" environments. The "local" is where engineers run applications in their PC whereas "other" is where Kubernetes runs your applications (qa, staging, sandbox, prod ...). Both have "green" and "red" sections as described below.
secret push ...
command to push secrets to remote storage and uses secret pull ...
to pull secrets to local machine. Note: These will be barely used because we don't always interact with secrets.secret
.Combining green and red, this has only one way of working and very simple. When the application runs first time, it talks to remote storage to pull the secrets to map to a struct.
First all all, I hardcoded some values and made some code "opinionated" but you can refactor it to improve the it. The reason is because, our point here is not to deliver the most beautiful code! It is about giving you an idea on how to handle secrets.
push
command, secret value is not printed to terminal for security reasons.\n
as line breaks. For instance, SSH keys.push
command handles single key-value pair at a time. Ideally, it should handle multiple.User_Home_Directory/.Organisation_Name/secret/Service_Environment/Service_Name.json
. e.g. /Users/you/.inanzzz/secret/test/app.json
If you want your application to always read secrets from the JSON file when running on the local environment, you can modify the secret.go
file to get rid of opinionated code. I suggest you to this because, again, you don't want to depend on a third party service while working on local environment. Maybe not for production secrets!
├── aws
│ ├── aws.go
│ └── kms.go
├── crypto
│ └── crypto.go
├── go.mod
├── main.go
└── secret
├── command.go
└── secret.go
package aws
import (
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
)
func NewSession() (*session.Session, error) {
return session.NewSessionWithOptions(session.Options{
Profile: "localstack",
Config: aws.Config{
Region: aws.String("eu-west-1"),
Endpoint: aws.String("http://localhost:4566"),
},
})
}
package aws
import (
"context"
"fmt"
"strings"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/ssm"
)
type KMS struct {
client *ssm.SSM
}
func NewKMS(ses *session.Session) *KMS {
return &KMS{
client: ssm.New(ses),
}
}
// Load gets multiple secret values from the remote location using their path prefix.
func (k *KMS) Load(ctx context.Context, path string) (map[string]string, error) {
res, err := k.client.GetParametersByPathWithContext(ctx, &ssm.GetParametersByPathInput{
Path: aws.String(path),
})
if err != nil {
return nil, err
}
sec := make(map[string]string, len(res.Parameters))
for _, param := range res.Parameters {
sec[strings.TrimPrefix(*param.Name, path)] = *param.Value
}
return sec, nil
}
// Insert puts a new secret key-value pair to the remote location using its path prefix.
func (k *KMS) Insert(ctx context.Context, path, key, val string) error {
_, err := k.client.PutParameterWithContext(ctx, &ssm.PutParameterInput{
Name: aws.String(fmt.Sprintf("%s/%s", path, key)),
Tier: aws.String(ssm.ParameterTierStandard),
Type: aws.String(ssm.ParameterTypeString),
Value: aws.String(val),
Overwrite: aws.Bool(true),
})
if err != nil {
return err
}
return nil
}
package crypto
import (
"crypto/rand"
"crypto/rsa"
"crypto/sha256"
"crypto/x509"
"encoding/hex"
"encoding/pem"
"fmt"
"golang.org/x/crypto/ssh"
"os"
)
type Crypto struct{}
// Encrypt encrypts plain value using SSH public key.
func (c Crypto) Encrypt(plain string) (string, error) {
sshKey, err := c.sshKey("id_rsa.pub")
if err != nil {
return "", fmt.Errorf("get ssh key: %w", err)
}
pubKey, _, _, _, err := ssh.ParseAuthorizedKey(sshKey)
if err != nil {
return "", fmt.Errorf("parse authorised key: %w", err)
}
key := pubKey.(ssh.CryptoPublicKey).CryptoPublicKey().(*rsa.PublicKey)
val, err := rsa.EncryptOAEP(sha256.New(), rand.Reader, key, []byte(plain), nil)
if err != nil {
return "", fmt.Errorf("encrypt: %w", err)
}
return hex.EncodeToString(val), nil
}
// Decrypt decrypts previously encrypted value using SSH private key.
func (c Crypto) Decrypt(encoded string) (string, error) {
sshKey, err := c.sshKey("id_rsa")
if err != nil {
return "", fmt.Errorf("get ssh key: %w", err)
}
decoded, err := hex.DecodeString(encoded)
if err != nil {
return "", fmt.Errorf("decode data: %w", err)
}
pemBlock, _ := pem.Decode(sshKey)
key, err := x509.ParsePKCS1PrivateKey(pemBlock.Bytes)
if err != nil {
return "", fmt.Errorf("parse private key: %w", err)
}
val, err := rsa.DecryptOAEP(sha256.New(), rand.Reader, key, decoded, nil)
if err != nil {
return "", fmt.Errorf("decode: %w", err)
}
return string(val), nil
}
// sshKey returns either a public or private SSH key.
func (c Crypto) sshKey(file string) ([]byte, error) {
home, err := os.UserHomeDir()
if err != nil {
return nil, err
}
key, err := os.ReadFile(fmt.Sprintf("%s/.ssh/%s", home, file))
if err != nil {
return nil, err
}
return key, nil
}
package secret
import (
"context"
"encoding/json"
"fmt"
"os"
"github.com/you/secret/aws"
"github.com/you/secret/crypto"
)
type Command struct {
Crypto crypto.Crypto
Storage *aws.KMS
Action string
Environment string
Service string
SecretKey string
SecretVal string
}
// Pull talks to remote secret storage to fetch secrets, talks to crypto
// service to encrypt the values, constructs a JSON file using key-value pair
// and saves it to local path.
func (c Command) Pull(ctx context.Context) error {
if c.Environment == "" {
return fmt.Errorf("service environment variable is required")
}
list, err := c.Storage.Load(ctx, fmt.Sprintf("/%s/%s/", c.Environment, c.Service))
if err != nil {
return fmt.Errorf("load secrets: %w", err)
}
for k, v := range list {
val, err := c.Crypto.Encrypt(v)
if err != nil {
return fmt.Errorf("encrypt secret value: %w", err)
}
list[k] = val
}
data, err := json.MarshalIndent(list, "", " ")
if err != nil {
return fmt.Errorf("marshal secrets: %w", err)
}
home, err := os.UserHomeDir()
if err != nil {
return fmt.Errorf("get home directory: %w", err)
}
dir := fmt.Sprintf("%s/.inanzzz/secret/%s", home, c.Environment)
if err := os.MkdirAll(dir, 0755); err != nil {
return fmt.Errorf("create secret directory: %w", err)
}
file := fmt.Sprintf("%s/.inanzzz/secret/%s/%s.json", home, c.Environment, c.Service)
if err := os.WriteFile(file, data, 0755); err != nil {
return fmt.Errorf("create secret file: %w", err)
}
return nil
}
// Push pushes a secret key-value pair to remote secret storage.
func (c Command) Push(ctx context.Context) error {
if c.Environment == "" {
return fmt.Errorf("service environment variable is required")
}
err := c.Storage.Insert(ctx, fmt.Sprintf("/%s/%s/", c.Environment, c.Service), c.SecretKey, c.SecretVal)
if err != nil {
return fmt.Errorf("insert secret: %w", err)
}
return nil
}
package secret
import (
"context"
"encoding/json"
"fmt"
"io/ioutil"
"os"
"reflect"
"github.com/you/secret/aws"
"github.com/you/secret/crypto"
)
type Secret struct {
Crypto crypto.Crypto
Storage *aws.KMS
Service string
Environment string
}
func (s Secret) Load(ctx context.Context, cfg interface{}) error {
switch s.Environment {
case "":
return fmt.Errorf("service environment variable is required")
case "local":
return s.fromFile(cfg)
}
return s.fromClient(ctx, cfg)
}
func (s Secret) fromFile(cfg interface{}) error {
home, err := os.UserHomeDir()
if err != nil {
return err
}
file, err := ioutil.ReadFile(fmt.Sprintf("%s/.inanzzz/secret/local/%s.json", home, s.Service))
if err != nil {
return err
}
data := make(map[string]string)
if err := json.Unmarshal(file, &data); err != nil {
return err
}
return s.bind(cfg, data, true)
}
func (s Secret) fromClient(ctx context.Context, cfg interface{}) error {
data, err := s.Storage.Load(ctx, fmt.Sprintf("/%s/%s/", s.Environment, s.Service))
if err != nil {
return fmt.Errorf("load secrets: %w", err)
}
return s.bind(cfg, data, false)
}
func (s Secret) bind(cfg interface{}, data map[string]string, dec bool) error {
configSource := reflect.ValueOf(cfg)
if configSource.Kind() != reflect.Ptr {
return fmt.Errorf("config must be a pointer")
}
configSource = configSource.Elem()
if configSource.Kind() != reflect.Struct {
return fmt.Errorf("config must be a struct")
}
configType := configSource.Type()
for i := 0; i < configSource.NumField(); i++ {
fieldTag, ok := configType.Field(i).Tag.Lookup("secret")
if !ok {
continue
}
fieldName := configType.Field(i).Name
fieldValue := configSource.FieldByName(fieldName)
if !fieldValue.IsValid() {
continue
}
if !fieldValue.CanSet() {
continue
}
enc, ok := data[fieldTag]
if !ok {
continue
}
if !dec {
fieldValue.SetString(enc)
continue
}
val, err := s.Crypto.Decrypt(enc)
if err != nil {
return fmt.Errorf("decrypt %s: %w", fieldTag, err)
}
fieldValue.SetString(val)
}
return nil
}
package main
import (
"bufio"
"context"
"flag"
"fmt"
"golang.org/x/term"
"log"
"os"
"strings"
"github.com/you/secret/aws"
"github.com/you/secret/crypto"
"github.com/you/secret/secret"
)
func main() {
ctx := context.Background()
// Initialise AWS session.
awsSes, err := aws.NewSession()
if err != nil {
log.Fatalln(err)
}
// Initialise AWS KMS service which is remote secret storage.
scrClient := aws.NewKMS(awsSes)
// Initialise secret command handler.
scrCmd := secret.Command{
Crypto: crypto.Crypto{},
Storage: scrClient,
Environment: os.Getenv("SERVICE_ENV"),
}
// Capture user input and update command handler.
flag.Parse()
scrCmd.Action = flag.Arg(0)
if flag.Arg(1) == "-svc" || flag.Arg(1) == "--svc" {
scrCmd.Service = flag.Arg(2)
}
switch scrCmd.Action {
case "pull":
if err := scrCmd.Pull(ctx); err != nil {
log.Fatalln(err)
}
fmt.Println("Pulled!")
case "push":
var err error
scrCmd.SecretKey, err = key()
if err != nil {
log.Fatalln(err)
}
scrCmd.SecretVal, err = val()
if err != nil {
log.Fatalln(err)
}
if err := scrCmd.Push(ctx); err != nil {
log.Fatalln(err)
}
fmt.Println("Pushed!")
default:
fmt.Println("Invalid action!")
}
}
// key prompts user to enter secret key which is visible on terminal.
func key() (string, error) {
fmt.Printf("Key > ")
rdr := bufio.NewReader(os.Stdin)
for {
answer, err := rdr.ReadString('\n')
if err != nil {
return "", err
}
return strings.TrimSuffix(answer, "\n"), nil
}
}
// val prompts user to enter secret value which is not visible on terminal.
func val() (string, error) {
fmt.Printf("Value > ")
raw, err := term.MakeRaw(0)
if err != nil {
return "", err
}
defer term.Restore(0, raw)
var (
prompt string
answer string
)
trm := term.NewTerminal(os.Stdin, prompt)
for {
char, err := trm.ReadPassword(prompt)
if err != nil {
return "", err
}
answer += char
if char == "" || char == answer {
return answer, nil
}
}
}
Run $ go install github.com/you/secret@latest
command to install the binary to your machine so that you can start using $ secret push/pull...
commands from anywhere.
package main
import (
"context"
"fmt"
"log"
"os"
"strings"
"github.com/you/secret/aws"
"github.com/you/secret/crypto"
"github.com/you/secret/secret"
)
type Config struct {
ServiceAddress string `json:"service_address"`
MySQLPass string `secret:"mysql_pass"`
PrivateKey string `secret:"private_key"`
}
func main() {
ctx := context.Background()
cfg := Config{
ServiceAddress: "http://localhost/",
MySQLPass: "pass",
PrivateKey: "",
}
// Initialise AWS session.
awsSes, err := aws.NewSession()
if err != nil {
log.Fatalln(err)
}
// Initialise AWS KMS service which is remote secret storage.
scrClient := aws.NewKMS(awsSes)
scr := secret.Secret{
Crypto: crypto.Crypto{},
Storage: scrClient,
Service: "api",
Environment: os.Getenv("SERVICE_ENV"),
}
if err := scr.Load(ctx, &cfg); err != nil {
log.Fatalln(err)
}
fmt.Println("----")
fmt.Println("Service Address:", cfg.ServiceAddress)
fmt.Println("MySQL Password:", cfg.MySQLPass)
fmt.Println("Private Key:")
fmt.Println(strings.ReplaceAll(cfg.PrivateKey, `\n`, "\n"))
}
If you want to simplify the usage in your applications, you can hardcode some variables and dependencies in the secret repository. I am leaving this up to you but your final code would look like this instead.
package main
import (
"context"
"fmt"
"log"
"os"
"strings"
"github.com/you/secret/secret"
)
type Config struct {
ServiceAddress string `json:"service_address"`
MySQLPass string `secret:"mysql_pass"`
PrivateKey string `secret:"private_key"`
}
func main() {
cfg := Config{
ServiceAddress: "http://localhost/",
MySQLPass: "pass",
PrivateKey: "",
}
if err := secret.Load(context.Background(), &cfg, "api"); err != nil {
log.Fatalln(err)
}
// ...
}
Let's pretend like we are dealing with "prod" environment secrets and our application is called "api".
$ env | grep SERVICE_ENV
SERVICE_ENV=prod
Push secrets to AWS KMS.
// Value is 123123
$ secret push --svc api
Key > mysql_pass
Value >
Pushed!
// Value is -----BEGIN RSA PRIVATE KEY-----\nvpGGduSuXsp++jPUTvQxAZMRX/Y0Q==\n-----END RSA PRIVATE KEY-----
$ secret push --svc api
Key > private_key
Value >
Pushed!
Verify their existence.
$ aws --profile localstack --endpoint-url http://localhost:4566 ssm get-parameters-by-path --path "/prod/api/"
{
"Parameters": [
{
"Name": "/prod/api/mysql_pass",
"Type": "String",
"Value": "123123",
},
{
"Name": "/prod/api/private_key",
"Type": "String",
"Value": "-----BEGIN RSA PRIVATE KEY-----\nvpGGduSuXsp++jPUTvQxAZMRX/Y0Q==\n-----END RSA PRIVATE KEY-----",
}
]
}
Given our environment is set to prod
the application will run because as mentioned before, only "local" environment reads from the local storage.
api$ go run main.go
----
Service Address: http://localhost/
MySQL Password: 123123
Private Key:
-----BEGIN RSA PRIVATE KEY-----
vpGGduSuXsp++jPUTvQxAZMRX/Y0Q==
-----END RSA PRIVATE KEY-----
Now let's change the environment to local
and run the application. This will fail.
api$ export SERVICE_ENV=local
api$ go run main.go
2022/03/05 18:44:32 open /Users/you/.inanzzz/secret/local/api.json: no such file or directory
exit status 1
In order to make this work, we need to push secrets first and pull then retry. I assume you pushed already!
$ secret pull --svc api
Pulled!
$ cat $HOME/.inanzzz/secret/local/api.json
{
"mysql_pass": "a64c9135...",
"private_key": "24501aa8e3f3..."
}
api$ go run main.go
----
Service Address: http://localhost/
MySQL Password: 111111
Private Key:
-----BEGIN RSA PRIVATE KEY-----
vpGGduSuXsp++jPUTvQxAZMRX/Y0Q==
-----END RSA PRIVATE KEY-----