Hello everyone!

We have been investing plenty of personal time and energy for many years to share our knowledge with you all. However, we now need your help to keep this blog running. All you have to do is just click one of the adverts on the site, otherwise it will sadly be taken down due to hosting etc. costs. Thank you.

Vault is used to manage application secrets and protect sensitive information using UI, CLI or HTTP API - i.e. tokens, passwords, certificates, encryption keys so on. Vault is a centralised place to store all secrets. What we will be doing in this example is, we will create a vault docker container, store some secrets in it and access them from outside with Vault's HTTP API.


Vault components used


Our example makes use of Vault's components listed below.



Structure


├── docker-compose.yml
└── vault
└── config
└── config.json

Files


docker-compose.yml


In production do not set SKIP_SETCAP to true. SKIP_CHOWN=true is used because we are not copying "config.json" file into the container.


version: "3.4"

services:
vault:
image: vault:latest
environment:
- VAULT_ADDR=http://localhost:8200
- VAULT_API_ADDR=http://0.0.0.0:8200
- SKIP_SETCAP=true
- SKIP_CHOWN=true
ports:
- 8200:8200
cap_add:
- IPC_LOCK
command: server
volumes:
- ./vault/config:/vault/config

config.json


In production do not set disable_mlock and tls_disable to true.


{
"storage": {
"file": {
"path": "vault/file"
}
},
"listener": {
"tcp": {
"address": "0.0.0.0:8200",
"tls_disable": true
}
},
"ui": true,
"max_lease_ttl": "8760h",
"default_lease_ttl": "8760h",
"disable_mlock": true
}

Installation


You can get rid of permission errors below by either using SKIP_CHOWN=true (preferred) or VAULT_LOCAL_CONFIG environment variable or copying the "config.yml" into the container or use at runtime.


$ docker-compose up --build

Starting my-vault_vault_1 ... done
Attaching to my-vault_vault_1
vault_1 | ==> Vault server configuration:
vault_1 |
vault_1 | Api Address: http://0.0.0.0:8200
vault_1 | Cgo: disabled
vault_1 | Cluster Address: https://0.0.0.0:8201
vault_1 | Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
vault_1 | Log Level: info
vault_1 | Mlock: supported: true, enabled: false
vault_1 | Storage: file
vault_1 | Version: Vault v1.1.3
vault_1 | Version Sha: 9bc820f700f83a7c4bcab54c5323735a581b34eb
vault_1 |
vault_1 | ==> Vault server started! Log data will stream in below:

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1859c17f18c3 vault:1.1.2 "docker-entrypoint.s…" 21 hours ago Up 27 minutes 0.0.0.0:8200->8200/tcp my-vault_vault_1

The UI will be accessible via http://your-address-or-ip:8200/ui.



Setup


$ docker exec -it my-vault_vault_1 sh
/ # vault status
Key Value
--- -----
Seal Type shamir
Initialized false
Sealed true
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version n/a
HA Enabled false

Initialising


You will have to use three of the unseal keys below if the Vault server is re-sealed or restarted. Also use token below to login as "root" user/policy.


/ # vault operator init
Unseal Key 1: 8+cqV1Kx23V+6Vk/Q5AcS44dUUt/9EOYOHYAgUW5Nr2x
Unseal Key 2: bB1TvY6HAHohDDnBkMUOtpwIjAqmuG1O3BO+PjPINJAU
Unseal Key 3: uptS2m4t26fTzWVfHbVa9ff6mYxyGLKAoooppVWcPNhP
Unseal Key 4: F/zN9VB0gN6PqiCJz26z0o+rKrYNSm/LVrPfpDZ7R+pr
Unseal Key 5: AIzECCC2OBFU8w1WyOoYmeM/HwnMVwisnsXwAa9uAs1N

Initial Root Token: s.wGVtc0rBJlBpsPQ8xNc84aNJ

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

Unsealing


You need to run this command three times with three different "unseal keys" as mentioned above. The result should look like below.


/ # vault operator unseal
Unseal Key (will be hidden):

Key             Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.1.2
Cluster Name vault-cluster-6e925364
Cluster ID 57a932bc-bf5a-c358-a24a-5b716cee9e5b
HA Enabled false


Login


Use "Initial Root Token" as shown above. Alternatively you can login with UI with the same token.


/ # vault login
Token (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key Value
--- -----
token s.wGVtc0rBJlBpsPQ8xNc84aNJ
token_accessor kLt2G7FFemIf9YaxvPrplCAp
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]

We will define a different policy later on because using "root" policy in production is not recommended.


Audit


Let's enable auditing and write logs to a file.


/ # ls -l vault/logs
total 0

/ # vault audit enable file file_path=/vault/logs/audit.log
Success! Enabled the file audit device at: file/

/ # ls -l vault/logs
total 4
-rw------- 1 vault vault 1131 May 5 22:15 audit.log

/ # vault audit list
Path Type Description
---- ---- -----------
file/ file n/a

Secrets


We will be using "static" secrets and they can be managed through the CLI, HTTP API or UI.


/ # vault secrets list -detailed
Path Plugin Accessor Default TTL Max TTL Force No Cache Replication Seal Wrap Options Description
---- ------ -------- ----------- ------- -------------- ----------- --------- ------- -----------
cubbyhole/ cubbyhole cubbyhole_c051b82e n/a n/a false local false map[] per-token private secret storage
identity/ identity identity_f0455f33 system system false replicated false map[] identity store
sys/ system system_ef5abc18 n/a n/a false replicated false map[] system endpoints used for control, policy and debugging

As you can see above we haven't got secrets engine called kv (Key/Value) yet so we need to enable it first.


/ # vault secrets enable -path=secret kv
Success! Enabled the kv secrets engine at: secret/

Let's confirm.


/ # vault secrets list -detailed
Path Plugin Accessor Default TTL Max TTL Force No Cache Replication Seal Wrap Options Description
---- ------ -------- ----------- ------- -------------- ----------- --------- ------- -----------
cubbyhole/ cubbyhole cubbyhole_c051b82e n/a n/a false local false map[] per-token private secret storage
identity/ identity identity_f0455f33 system system false replicated false map[] identity store
secret/ kv kv_f75b140b system system false replicated false map[] n/a
sys/ system system_ef5abc18 n/a n/a false replicated false map[] system endpoints used for control, policy and debugging

Secret creation


Let's create a new secret with a key of DB_PASS and value of 123123 within the secret/my-vault/prod/DB_PASS path. If we are to explain the segments, I would do this way (you are free to structure it as you wish):


my-vault {application-name}
prod {application-environment-name}
DB_PASS {environment-variable-name}

When you restart the container, your secrets still be there unless you remove the container!


CLI version


/ # vault kv put secret/my-vault/prod/DB_PASS DB_PASS=123123
Success! Data written to: secret/my-vault/prod/DB_PASS

/ # vault kv get secret/my-vault/prod/DB_PASS
===== Data =====
Key Value
--- -----
DB_PASS 123123


HTTP API version


The v1/ is compulsory but you can enable and use different versions if you wish.


curl \
-H "X-Vault-Token: s.wGVtc0rBJlBpsPQ8xNc84aNJ" \
-H "Content-Type: application/json" \
-i -X POST \
-d '{"data":{"DB_USER":"inanzzz"}}' \
http://your-address-or-ip:8200/v1/secret/my-vault/prod/DB_USER

HTTP/1.1 204 No Content

curl \
-H "X-Vault-Token: s.wGVtc0rBJlBpsPQ8xNc84aNJ" \
-i -X GET \
http://your-address-or-ip:8200/v1/secret/my-vault/prod/DB_USER

HTTP/1.1 200 OK
{
"request_id": "f6d8e003-2b21-ee03-11f7-22bde3668120",
"lease_id": "",
"renewable": false,
"lease_duration": 31536000,
"data": {
"data": {
"DB_USER": "inanzzz"
}
},
"wrap_info": null,
"warnings": null,
"auth": null
}


Policy


We have been using "root" policy which has all the permissions to interact with the Vault. This comes by default with Vault but it is highly recommended that we should either revoke all root tokens or just reduce permissions before running Vault in production. If you remove the policy, you won't be able to create new policies so make sure you know what you are doing. For more information read Root Policy page.


New policy


Policy file below gives just list access right to all directories and secret names (not the values) stored under secret/ and secret/my-vault/ paths but no other rights (this is not needed for CLI so you can omit this bit below but it is needed for UI if you really need UI). However, it gives create|read|update|delete|list access right to all directories and secrets stored under secret/my-vault/stag/ directory.


/ # vi vault/file/sys/policy/_my-vault-stag.json
{
"path": {
"secret/": {
"capabilities": [
"list"
]
}
},
"path": {
"secret/my-vault/": {
"capabilities": [
"list"
]
}
},
"path": {
"secret/my-vault/stag/*": {
"capabilities": [
"create",
"read",
"update",
"delete",
"list"
]
}
}
}

Let's create the policy.


/ # vault policy write my-vault-stag vault/file/sys/policy/_my-vault-stag.json
Success! Uploaded policy: my-vault-stag

Create a new token for this policy so that it can be used.


/ # vault token create -policy=my-vault-stag
Key Value
--- -----
token s.qjQxk6i7nBvq1yKx9ZK8Kkfp
token_accessor WZJHfz2GOpMvBFbIVUOtGRFF
token_duration 8760h
token_renewable true
token_policies ["default" "my-vault-stag"]
identity_policies []
policies ["default" "my-vault-stag"]

Testing


Trying to read from secret/my-vault/prod which new policy doesn't have access to.


curl \
-H "X-Vault-Token: s.qjQxk6i7nBvq1yKx9ZK8Kkfp" \
-i -X GET \
http://your-address-or-ip:8200/v1/secret/my-vault/prod/DB_USER

HTTP/1.1 403 Forbidden
{"errors":["1 error occurred:\n\t* permission denied\n\n"]}

Let's test it in CLI. We need to run vault login to login as the new policy first.


/ # vault login
Token (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key Value
--- -----
token s.qjQxk6i7nBvq1yKx9ZK8Kkfp
token_accessor WZJHfz2GOpMvBFbIVUOtGRFF
token_duration 8759h49m13s
token_renewable true
token_policies ["default" "my-vault-stag"]
identity_policies []
policies ["default" "my-vault-stag"]

Trying to read from secret/my-vault/prod which new policy doesn't have access to.


/ # vault kv get secret/my-vault/prod/DB_PASS
Error reading secret/my-vault/prod/DB_PASS: Error making API request.

URL: GET http://127.0.0.1:8200/v1/secret/my-vault/prod/DB_PASS
Code: 403. Errors:

* 1 error occurred:
* permission denied

Trying to work with secret/my-vault/stag which new policy does have access to.


/ # vault kv put secret/my-vault/stag/DB_PASS DB_PASS=666666
Success! Data written to: secret/my-vault/stag/DB_PASS

/ # vault kv get secret/my-vault/stag/DB_PASS
===== Data =====
Key Value
--- -----
DB_PASS 666666

curl \
-H "X-Vault-Token: s.qjQxk6i7nBvq1yKx9ZK8Kkfp" \
-i -X GET \
http://your-address-or-ip:8200/v1/secret/my-vault/stag/DB_PASS

HTTP/1.1 200 OK
{
"request_id": "e71bcdbf-1049-b635-1f47-91496e85e9c6",
"lease_id": "",
"renewable": false,
"lease_duration": 31536000,
"data": {
"DB_PASS": "666666"
},
"wrap_info": null,
"warnings": null,
"auth": null
}