25/05/2023 - AWS, TERRAFORM
When working with Terraform locally, Terraform uses default local backend to store its state in .terraform.tfstate
file. The state file must not be committed in version control system as it might expose secret information. When some Terraform commands are executed, this file is modified. Hence reason changes to infrastructure must be available to engineers immediately. However, if the state file is managed locally, this is not possible. In order to overcome these issues, we will be using AWS S3 to store the state file and DynamoDB for locking information. The locking mechanism is an important aspect to avoid corrupting state file when it is modified concurrently. Terraform acquires a state lock to protect the state from being written by multiple users at the same time.
You should carry out these steps manually in AWS console but I will use Terraform to handle it for now. By the way your user/group needs iam:*, ec2:*, s3:*, dynamodb:*
permissions at most but prefer using least privileged permissions.
├── dynamodb.tf
├── local.tf
├── main.tf
└── s3.tf
locals {
terraform_state_bucket_name = "inanzzz-development-terraform"
terraform_state_bucket_key = "terraform.tfstate"
terraform_state_dynamodb_table = "terraform_tfstate"
}
terraform {
required_version = "~> 1.4.4"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.30.0"
}
}
}
provider "aws" {
profile = "development"
region = "eu-west-1"
}
resource "aws_s3_bucket" "terraform_state" {
bucket = local.terraform_state_bucket_name
force_destroy = true
}
resource "aws_s3_bucket_versioning" "versioning_example" {
bucket = aws_s3_bucket.terraform_state.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_public_access_block" "terraform_state" {
bucket = aws_s3_bucket.terraform_state.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
resource "aws_dynamodb_table" "terraform_state" {
name = local.terraform_state_dynamodb_table
billing_mode = "PAY_PER_REQUEST"
hash_key = "LockID"
attribute {
name = "LockID"
type = "S"
}
}
Run terraform init
and terraform apply
commands. This will create S3 bucket and DynamoDB table in AWS console. In local environment, .terraform
folder as well as .terraform.lock.hcl
and .terraform.tfstate
files will be created.
├── group.tf
└── main.tf
terraform {
required_version = "~> 1.4.4"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.30.0"
}
}
}
provider "aws" {
profile = "development"
region = "eu-west-1"
}
terraform {
backend "s3" {
profile = "development"
region = "eu-west-1"
encrypt = true
bucket = "inanzzz-development-terraform"
key = "terraform.tfstate"
dynamodb_table = "terraform_tfstate"
}
}
This is optional but we will use it to demonstrate how state file is not stored in local environment.
# resource "aws_iam_group" "dummy_team" {
# name = "dummy-team"
# }
Run terraform init
command. This won't create any resource. However, it will create .terraform
folder and .terraform.lock.hcl
file but not .terraform.tfstate
file.
Comment out aws_iam_group
block and run terraform apply
command. This will create .terraform.tfstate
file in S3 bucket and set LockID
property in DynamoDB table for locking mechanism.
From now on everytime you run terraform apply
commands, local .terraform.tfstate
file won't have any resources listed inside. Instead they will appear in .terraform.tfstate
file in S3 bucket. You can download and check its content. Also LockID
property in DynamoDB table will be updated.
Open a terminal to add a new resource (lock_team
) and run apply but do not respond to prompt.
$ terraform apply
2023/05/25 21:26:26 Enabling CSM
2023/05/25 21:26:27 Enabling CSM
aws_iam_group.dummy_team: Refreshing state... [id=dummy-team]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_iam_group.lock_team will be created
+ resource "aws_iam_group" "lock_team" {
+ arn = (known after apply)
+ id = (known after apply)
+ name = "lock-team"
+ path = "/"
+ unique_id = (known after apply)
}
Plan: 1 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value:
Open another terminal and run plan command. You should see locking error.
$ terraform plan
2023/05/25 21:26:39 Enabling CSM
2023/05/25 21:26:39 Enabling CSM
╷
│ Error: Error acquiring the state lock
│
│ Error message: ConditionalCheckFailedException: The conditional request failed
│ Lock Info:
│ ID: 12345-qwer-asdf-34e5-1234567890
│ Path: inanzzz-development-terraform/terraform.tfstate
│ Operation: OperationTypeApply
│ Who: me@home
│ Version: 1.4.4
│ Created: 2023-05-25 20:26:29.120347 +0000 UTC
│ Info:
│
│
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.
╵
For normal users, you can use policy below for S3 bucket.
data "aws_iam_policy_document" "terraform_state" {
version = "2012-10-17"
statement {
actions = [
"s3:ListBucket",
]
resources = [
"arn:aws:s3:::inanzzz-development-terraform"
]
}
statement {
actions = [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
]
resources = [
"arn:aws:s3:::inanzzz-development-terraform/terraform.tfstate"
]
}
}
resource "aws_s3_bucket_policy" "terraform_state" {
bucket = aws_s3_bucket.terraform_state.id
policy = data.aws_iam_policy_document.terraform_state.json
}