inanzzz | Managing application secrets with Hashicorp Vault

Vault is used to manage application secrets and protect sensitive information using UI, CLI or HTTP API - i.e. tokens, passwords, certificates, encryption keys so on. Vault is a centralised place to store all secrets. What we will be doing in this example is, we will create a vault docker container, store some secrets in it and access them from outside with Vault's HTTP API.

Vault components used

Our example makes use of Vault's components listed below.

Auth Methods > Token: The "Auth Methods" are responsible for authentication and assigning set of policies to a user. The "Token" auth method allows users to authenticate using a token, to create a new token, revoke secrets by token so on. It is available on /auth/token URI location.

Storage > Filesystem: The "Storage backend" represents the location for the storage of Vault's information. The "filesystem" storage backend stores data on the filesystem using a standard directory structure.

Secrets Engine > Key/Value: The "Secrets Engine" handles how to store, generate and encrypt data. The "Key/Value" (kv) secrets engine stores secrets as in classic key/value fashion within the selected storage backend.

Audit Devices > File: The "Audit Devices" keep a log of all requests and responses. The "File" audit device writes audit logs to a file.

Policy: Everything in Vault is path based. A "policy" is like a user with set of access/usage permissions. E.g. A hello-world can interact with all the secrets stored under secret/hello-world path.

Structure

├── docker-compose.yml
└── vault
    └── config
        └── config.json

Files

docker-compose.yml

In production do not set SKIP_SETCAP to true. SKIP_CHOWN=true is used because we are not copying "config.json" file into the container.

version: "3.4"

services:
  vault:
    image: vault:latest
    environment:
      - VAULT_ADDR=http://localhost:8200
      - VAULT_API_ADDR=http://0.0.0.0:8200
      - SKIP_SETCAP=true
      - SKIP_CHOWN=true
    ports:
      - 8200:8200
    cap_add:
      - IPC_LOCK
    command: server
    volumes:
      - ./vault/config:/vault/config

config.json

In production do not set disable_mlock and tls_disable to true.

{
  "storage": {
    "file": {
      "path": "vault/file"
    }
  },
  "listener": {
    "tcp": {
      "address": "0.0.0.0:8200",
      "tls_disable": true
    }
  },
  "ui": true,
  "max_lease_ttl": "8760h",
  "default_lease_ttl": "8760h",
  "disable_mlock": true
}

Installation

You can get rid of permission errors below by either using SKIP_CHOWN=true (preferred) or VAULT_LOCAL_CONFIG environment variable or copying the "config.yml" into the container or use at runtime.

$ docker-compose up --build

Starting my-vault_vault_1 ... done
Attaching to my-vault_vault_1
vault_1  | ==> Vault server configuration:
vault_1  | 
vault_1  |              Api Address: http://0.0.0.0:8200
vault_1  |                      Cgo: disabled
vault_1  |          Cluster Address: https://0.0.0.0:8201
vault_1  |               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
vault_1  |                Log Level: info
vault_1  |                    Mlock: supported: true, enabled: false
vault_1  |                  Storage: file
vault_1  |                  Version: Vault v1.1.3
vault_1  |              Version Sha: 9bc820f700f83a7c4bcab54c5323735a581b34eb
vault_1  | 
vault_1  | ==> Vault server started! Log data will stream in below:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
1859c17f18c3        vault:1.1.2         "docker-entrypoint.s…"   21 hours ago        Up 27 minutes       0.0.0.0:8200->8200/tcp   my-vault_vault_1

The UI will be accessible via http://your-address-or-ip:8200/ui.

Setup

$ docker exec -it my-vault_vault_1 sh
/ # vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            n/a
HA Enabled         false

Initialising

You will have to use three of the unseal keys below if the Vault server is re-sealed or restarted. Also use token below to login as "root" user/policy.

/ # vault operator init
Unseal Key 1: 8+cqV1Kx23V+6Vk/Q5AcS44dUUt/9EOYOHYAgUW5Nr2x
Unseal Key 2: bB1TvY6HAHohDDnBkMUOtpwIjAqmuG1O3BO+PjPINJAU
Unseal Key 3: uptS2m4t26fTzWVfHbVa9ff6mYxyGLKAoooppVWcPNhP
Unseal Key 4: F/zN9VB0gN6PqiCJz26z0o+rKrYNSm/LVrPfpDZ7R+pr
Unseal Key 5: AIzECCC2OBFU8w1WyOoYmeM/HwnMVwisnsXwAa9uAs1N

Initial Root Token: s.wGVtc0rBJlBpsPQ8xNc84aNJ

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

Unsealing

You need to run this command three times with three different "unseal keys" as mentioned above. The result should look like below.

/ # vault operator unseal
Unseal Key (will be hidden):

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.1.2
Cluster Name    vault-cluster-6e925364
Cluster ID      57a932bc-bf5a-c358-a24a-5b716cee9e5b
HA Enabled      false

Login

Use "Initial Root Token" as shown above. Alternatively you can login with UI with the same token.

/ # vault login
Token (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.wGVtc0rBJlBpsPQ8xNc84aNJ
token_accessor       kLt2G7FFemIf9YaxvPrplCAp
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

We will define a different policy later on because using "root" policy in production is not recommended.

Audit

Let's enable auditing and write logs to a file.

/ # ls -l vault/logs
total 0

/ # vault audit enable file file_path=/vault/logs/audit.log
Success! Enabled the file audit device at: file/

/ # ls -l vault/logs
total 4
-rw-------    1 vault    vault         1131 May  5 22:15 audit.log

/ # vault audit list
Path     Type    Description
----     ----    -----------
file/    file    n/a

Secrets

We will be using "static" secrets and they can be managed through the CLI, HTTP API or UI.

/ # vault secrets list -detailed
Path          Plugin       Accessor              Default TTL    Max TTL    Force No Cache    Replication    Seal Wrap    Options    Description
----          ------       --------              -----------    -------    --------------    -----------    ---------    -------    -----------
cubbyhole/    cubbyhole    cubbyhole_c051b82e    n/a            n/a        false             local          false        map[]      per-token private secret storage
identity/     identity     identity_f0455f33     system         system     false             replicated     false        map[]      identity store
sys/          system       system_ef5abc18       n/a            n/a        false             replicated     false        map[]      system endpoints used for control, policy and debugging

As you can see above we haven't got secrets engine called kv (Key/Value) yet so we need to enable it first.

/ # vault secrets enable -path=secret kv
Success! Enabled the kv secrets engine at: secret/

Let's confirm.

/ # vault secrets list -detailed
Path          Plugin       Accessor              Default TTL    Max TTL    Force No Cache    Replication    Seal Wrap    Options    Description
----          ------       --------              -----------    -------    --------------    -----------    ---------    -------    -----------
cubbyhole/    cubbyhole    cubbyhole_c051b82e    n/a            n/a        false             local          false        map[]      per-token private secret storage
identity/     identity     identity_f0455f33     system         system     false             replicated     false        map[]      identity store
secret/       kv           kv_f75b140b           system         system     false             replicated     false        map[]      n/a
sys/          system       system_ef5abc18       n/a            n/a        false             replicated     false        map[]      system endpoints used for control, policy and debugging

Secret creation

Let's create a new secret with a key of DB_PASS and value of 123123 within the secret/my-vault/prod/DB_PASS path. If we are to explain the segments, I would do this way (you are free to structure it as you wish):

my-vault {application-name}
prod {application-environment-name}
DB_PASS {environment-variable-name}

When you restart the container, your secrets still be there unless you remove the container!

CLI version

/ # vault kv put secret/my-vault/prod/DB_PASS DB_PASS=123123
Success! Data written to: secret/my-vault/prod/DB_PASS

/ # vault kv get secret/my-vault/prod/DB_PASS
===== Data =====
Key        Value
---        -----
DB_PASS    123123

HTTP API version

The v1/ is compulsory but you can enable and use different versions if you wish.

curl \
    -H "X-Vault-Token: s.wGVtc0rBJlBpsPQ8xNc84aNJ" \
    -H "Content-Type: application/json" \
    -i -X POST \
    -d '{"data":{"DB_USER":"inanzzz"}}' \
    http://your-address-or-ip:8200/v1/secret/my-vault/prod/DB_USER

HTTP/1.1 204 No Content

curl \
    -H "X-Vault-Token: s.wGVtc0rBJlBpsPQ8xNc84aNJ" \
    -i -X GET \
    http://your-address-or-ip:8200/v1/secret/my-vault/prod/DB_USER

HTTP/1.1 200 OK
{
  "request_id": "f6d8e003-2b21-ee03-11f7-22bde3668120",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 31536000,
  "data": {
    "data": {
      "DB_USER": "inanzzz"
    }
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

Policy

We have been using "root" policy which has all the permissions to interact with the Vault. This comes by default with Vault but it is highly recommended that we should either revoke all root tokens or just reduce permissions before running Vault in production. If you remove the policy, you won't be able to create new policies so make sure you know what you are doing. For more information read Root Policy page.

New policy

Policy file below gives just list access right to all directories and secret names (not the values) stored under secret/ and secret/my-vault/ paths but no other rights (this is not needed for CLI so you can omit this bit below but it is needed for UI if you really need UI). However, it gives create|read|update|delete|list access right to all directories and secrets stored under secret/my-vault/stag/ directory.

/ # vi vault/file/sys/policy/_my-vault-stag.json
{
  "path": {
    "secret/": {
      "capabilities": [
        "list"
      ]
    }
  },
  "path": {
    "secret/my-vault/": {
      "capabilities": [
        "list"
      ]
    }
  }, 
 "path": {
    "secret/my-vault/stag/*": {
      "capabilities": [
        "create",
        "read",
        "update",
        "delete",
        "list"
      ]
    }
  }
}

Let's create the policy.

/ # vault policy write my-vault-stag vault/file/sys/policy/_my-vault-stag.json
Success! Uploaded policy: my-vault-stag

Create a new token for this policy so that it can be used.

/ # vault token create -policy=my-vault-stag
Key                  Value
---                  -----
token                s.qjQxk6i7nBvq1yKx9ZK8Kkfp
token_accessor       WZJHfz2GOpMvBFbIVUOtGRFF
token_duration       8760h
token_renewable      true
token_policies       ["default" "my-vault-stag"]
identity_policies    []
policies             ["default" "my-vault-stag"]

Testing

Trying to read from secret/my-vault/prod which new policy doesn't have access to.

curl \
    -H "X-Vault-Token: s.qjQxk6i7nBvq1yKx9ZK8Kkfp" \
    -i -X GET \
    http://your-address-or-ip:8200/v1/secret/my-vault/prod/DB_USER

HTTP/1.1 403 Forbidden
{"errors":["1 error occurred:\n\t* permission denied\n\n"]}

Let's test it in CLI. We need to run vault login to login as the new policy first.

/ # vault login
Token (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.qjQxk6i7nBvq1yKx9ZK8Kkfp
token_accessor       WZJHfz2GOpMvBFbIVUOtGRFF
token_duration       8759h49m13s
token_renewable      true
token_policies       ["default" "my-vault-stag"]
identity_policies    []
policies             ["default" "my-vault-stag"]

Trying to read from secret/my-vault/prod which new policy doesn't have access to.

/ # vault kv get secret/my-vault/prod/DB_PASS
Error reading secret/my-vault/prod/DB_PASS: Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/my-vault/prod/DB_PASS
Code: 403. Errors:

* 1 error occurred:
	* permission denied

Trying to work with secret/my-vault/stag which new policy does have access to.

/ # vault kv put secret/my-vault/stag/DB_PASS DB_PASS=666666
Success! Data written to: secret/my-vault/stag/DB_PASS

/ # vault kv get secret/my-vault/stag/DB_PASS
===== Data =====
Key        Value
---        -----
DB_PASS    666666

curl \
     -H "X-Vault-Token: s.qjQxk6i7nBvq1yKx9ZK8Kkfp" \
     -i -X GET \
     http://your-address-or-ip:8200/v1/secret/my-vault/stag/DB_PASS

HTTP/1.1 200 OK
{
  "request_id": "e71bcdbf-1049-b635-1f47-91496e85e9c6",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 31536000,
  "data": {
    "DB_PASS": "666666"
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}