inanzzz | Using AWS S3 backend and DynamoDB for Terraform state

When working with Terraform locally, Terraform uses default local backend to store its state in .terraform.tfstate file. The state file must not be committed in version control system as it might expose secret information. When some Terraform commands are executed, this file is modified. Hence reason changes to infrastructure must be available to engineers immediately. However, if the state file is managed locally, this is not possible. In order to overcome these issues, we will be using AWS S3 to store the state file and DynamoDB for locking information. The locking mechanism is an important aspect to avoid corrupting state file when it is modified concurrently. Terraform acquires a state lock to protect the state from being written by multiple users at the same time.

Setup S3 bucket and DynamoDB

You should carry out these steps manually in AWS console but I will use Terraform to handle it for now. By the way your user/group needs iam:*, ec2:*, s3:*, dynamodb:* permissions at most but prefer using least privileged permissions.

├── dynamodb.tf
├── local.tf
├── main.tf
└── s3.tf

local.tf

locals {
  terraform_state_bucket_name    = "inanzzz-development-terraform"
  terraform_state_bucket_key     = "terraform.tfstate"
  terraform_state_dynamodb_table = "terraform_tfstate"
}

main.tf

terraform {
  required_version = "~> 1.4.4"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.30.0"
    }
  }
}

provider "aws" {
  profile = "development"
  region  = "eu-west-1"
}

s3.tf

resource "aws_s3_bucket" "terraform_state" {
  bucket        = local.terraform_state_bucket_name
  force_destroy = true
}

resource "aws_s3_bucket_versioning" "versioning_example" {
  bucket = aws_s3_bucket.terraform_state.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

dynamodb.tf

resource "aws_dynamodb_table" "terraform_state" {
  name         = local.terraform_state_dynamodb_table
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

Setup

Run terraform init and terraform apply commands. This will create S3 bucket and DynamoDB table in AWS console. In local environment, .terraform folder as well as .terraform.lock.hcl and .terraform.tfstate files will be created.

Create S3 backend

├── group.tf
└── main.tf

main.tf

terraform {
  required_version = "~> 1.4.4"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.30.0"
    }
  }
}

provider "aws" {
  profile = "development"
  region  = "eu-west-1"
}

terraform {
  backend "s3" {
    profile        = "development"
    region         = "eu-west-1"
    encrypt        = true
    bucket         = "inanzzz-development-terraform"
    key            = "terraform.tfstate"
    dynamodb_table = "terraform_tfstate"
  }
}

group.tf

This is optional but we will use it to demonstrate how state file is not stored in local environment.

# resource "aws_iam_group" "dummy_team" {
#   name = "dummy-team"
# }

Setup

Run terraform init command. This won't create any resource. However, it will create .terraform folder and .terraform.lock.hcl file but not .terraform.tfstate file.

Comment out aws_iam_group block and run terraform apply command. This will create .terraform.tfstate file in S3 bucket and set LockID property in DynamoDB table for locking mechanism.

From now on everytime you run terraform apply commands, local .terraform.tfstate file won't have any resources listed inside. Instead they will appear in .terraform.tfstate file in S3 bucket. You can download and check its content. Also LockID property in DynamoDB table will be updated.

Testing locking

Open a terminal to add a new resource (lock_team) and run apply but do not respond to prompt.

$ terraform apply
2023/05/25 21:26:26 Enabling CSM
2023/05/25 21:26:27 Enabling CSM
aws_iam_group.dummy_team: Refreshing state... [id=dummy-team]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_iam_group.lock_team will be created
  + resource "aws_iam_group" "lock_team" {
      + arn       = (known after apply)
      + id        = (known after apply)
      + name      = "lock-team"
      + path      = "/"
      + unique_id = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

Open another terminal and run plan command. You should see locking error.

$ terraform plan
2023/05/25 21:26:39 Enabling CSM
2023/05/25 21:26:39 Enabling CSM
╷
│ Error: Error acquiring the state lock
│
│ Error message: ConditionalCheckFailedException: The conditional request failed
│ Lock Info:
│   ID:        12345-qwer-asdf-34e5-1234567890
│   Path:      inanzzz-development-terraform/terraform.tfstate
│   Operation: OperationTypeApply
│   Who:       me@home
│   Version:   1.4.4
│   Created:   2023-05-25 20:26:29.120347 +0000 UTC
│   Info:
│
│
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.
╵

Note

For normal users, you can use policy below for S3 bucket.

data "aws_iam_policy_document" "terraform_state" {
  version = "2012-10-17"

  statement {
    actions = [
      "s3:ListBucket",
    ]
    resources = [
      "arn:aws:s3:::inanzzz-development-terraform"
    ]
  }

  statement {
    actions = [
      "s3:GetObject",
      "s3:PutObject",
      "s3:DeleteObject",
    ]
    resources = [
      "arn:aws:s3:::inanzzz-development-terraform/terraform.tfstate"
    ]
  }
}

resource "aws_s3_bucket_policy" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  policy = data.aws_iam_policy_document.terraform_state.json
}