Auto-unseal Vault using AWS KMS

When a Vault server is started, it starts in a sealed state and it does not know how to decrypt data. Before any operation can be performed on the Vault, it must be unsealed. Unsealing is the process of constructing the master key necessary to decrypt the data encryption key.

This tutorial demonstrates an example of how to use Terraform to provision a Vault server that can use an encryption key from AWS Key Management Services (KMS) to automatically unseal.

Challenge

Vault unseal operation requires a quorum of existing unseal keys split by Shamir's Secret sharing algorithm. This is done so that the "keys to the kingdom" won't fall into one person's hand. However, this process is manual and can become painful when you have many Vault clusters as there are now many different key holders with many different keys.

Solution

Vault supports opt-in automatic unsealing via cloud technologies: AliCloud KMS, AWS KMS, Azure Key Vault, Google Cloud KMS, and OCI KMS. This feature enables operators to delegate the unsealing process to trusted cloud providers to ease operations in the event of partial failure and to aid in the creation of new or ephemeral clusters.

This tutorial demonstrates Vault Auto Unseal using AWS KMS.

Prerequisites

This tutorial assumes the following:

• AWS account for provisioning cloud resources
• Terraform installed and basic understanding of its usage

Step 1: Provision the cloud resources

1. Clone the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this tutorial.

   $ git clone https://github.com/hashicorp/vault-guides
   

   Copy

   This repository contains supporting content for all of the Vault learn tutorials. The content specific to this tutorial can be found within a sub-directory.

2. Be sure to set your working directory to where the /operations/aws-kms-unseal/terraform-aws folder is located.

   $ cd vault-guides/operations/aws-kms-unseal/terraform-aws
   

   Copy

   The working directory should contain the provided Terraform configuration files:

   $ tree
   .
   ├── README.md
   ├── instance-profile.tf
   ├── instance.tf
   ├── main.tf
   ├── ssh-key.tf
   ├── terraform.tfvars.example
   ├── userdata.tpl
   ├── variables.tf
   └── versions.tf
   
   0 directories, 9 files
   

   Copy

3. Set your AWS Access Key ID value in an environment variable named AWS_ACCESS_KEY_ID.

   $ export AWS_ACCESS_KEY_ID = "<YOUR_AWS_ACCESS_KEY_ID>"
   

   Copy

4. Set your AWS Secret Access Key value in an environment variable named AWS_SECRET_ACCESS_KEY.

   $ export AWS_SECRET_ACCESS_KEY = "<YOUR_AWS_SECRET_ACCESS_KEY>"
   

   Copy

   Tip

   The above example uses IAM user authentication. You can use any authentication method described in the AWS provider documentation.

5. By default, AWS resources will be provisioned in the us-east-1 region. If you wish to use another AWS region, create a file named terraform.tfvars and set the appropriate aws_region and aws_zone. (Reference the provided terraform.tfvars.example.)

   Example:

   terraform.tfvars

   Copy

   aws_region = "us-west-1"
   aws_zone = "us-west-1a"
   

6. Initialize the Terraform workspace, and pull down the necessary provider resources.

   $ terraform init
   

   Copy

   Example truncated output:

   Initializing the backend...
   
   Initializing provider plugins...
   ...snip...
   Terraform has been successfully initialized!
   ...snip...
   

7. Plan the changes and review resources which will be changed by Terraform.

   $ terraform plan -out learn-vault-aws-kms.plan
   

   Copy

   Example truncated output:

   data.aws_iam_policy_document.assume_role: Reading...
   data.aws_ami.ubuntu: Reading...
   data.aws_iam_policy_document.assume_role: Read complete after 0s [id=1903849331]
   data.aws_ami.ubuntu: Read complete after 1s [id=ami-0688ba7eeeeefe3cd]
   
   Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
   symbols:
     + create
   <= read (data resources)
   
   Terraform will perform the following actions:
   ...snip...
   

8. Apply the plan by specifying the plan filename, learn-vault-aws-kms.plan. The plan will run without prompting.

   $ terraform apply "learn-vault-aws-kms.plan"
   

   Copy

   Example truncated output:

   random_pet.env: Creating...
   random_pet.env: Creation complete after 0s [id=genuine_satyr]
   tls_private_key.main: Creating...
   ...snip...
   
   Apply complete! Resources: 15 added, 0 changed, 0 destroyed.
   
   Outputs:
   
   connections = <<EOT
   Connect to Vault via SSH   ssh ubuntu@35.91.217.103 -i private.key
   Vault web interface  http://35.91.217.103:8200/ui
   
   EOT
   

   When the apply command completes, the Terraform output will display the public IP address in examples for either SSH access, or to access the Vault web UI.

Step 2: Test the auto-unseal feature

Access the EC2 instance and test the auto-unseal feature.

1. SSH into the provisioned EC2 instance.

   $ ssh ubuntu@192.0.2.1 -i private.key
   

   Copy

   Example truncated output:

   ...snip...
   Are you sure you want to continue connecting (yes/no)? yes
   

   When you are prompted, enter yes to continue.

   Note

   The prompt will change to one like ubuntu@ip-192-168-100-22:~$ when you are using the EC2 instance.

2. Set the VAULT_ADDR environment variable to address the local server.

   $ export VAULT_ADDR=http://127.0.0.1:8200
   

   Copy

3. Check the Vault server status.

   $ vault status
   

   Copy

   Example output:

   Key                      Value
   ---                      -----
   Recovery Seal Type       awskms
   Initialized              false
   Sealed                   true
   Total Recovery Shares    0
   Threshold                0
   Unseal Progress          0/0
   Unseal Nonce             n/a
   Version                  1.12.2
   Build Date               2022-11-23T12:53:46Z
   Storage Type             file
   HA Enabled               false
   

   Notice that the Initialized value is false.

4. Initialize Vault.

   $ vault operator init
   

   Copy

   Example output:

   Recovery Key 1: IWFFX9K8ot9Mirii5nxA8iWYP4Sv7r6OUyPxPpWSXolp
   Recovery Key 2: 3b7hfLAc+A778EsKIPI9c6bUd5hVyOivgMcRF89Ce++R
   Recovery Key 3: Uox5m+JQbFnQTFTWSV3n5pyycBz1BfEbtSjA6E2V40FM
   Recovery Key 4: MeD2wsJmCG5CPMX/ERN9Uwj9qDoFrYF+L8dSx/BpHE82
   Recovery Key 5: syGIEWNIKDlvYdeHJHMWqMHRSb3LnlTtFxvbKKe3A/4B
   
   Initial Root Token: hvs.oeRIhQy9AYgSpuJCSRgDxgeP
   
   Success! Vault is initialized
   
   Recovery key initialized with 5 key shares and a key threshold of 3. Please
   securely distribute the key shares printed above.
   

   The initialization generates Recovery Keys (instead of unseal keys) when using auto-unseal. Some of the Vault operations still require Shamir keys. For example, to regenerate a root token, each key holder must enter their recovery key. Similar to unseal keys, you can specify the number of recovery keys and the threshold using the -recovery-shares and -recovery-threshold flags. It is strongly recommended to initialize Vault with PGP.

5. Once you initialize the Vault server, it is operationally ready; check the Vault status again to verify that it has been initialized and unsealed.

   $ vault status
   

   Copy

   Example output:

   Key                      Value
   ---                      -----
   Recovery Seal Type       shamir
   Initialized              true
   Sealed                   false
   Total Recovery Shares    5
   Threshold                3
   Version                  1.12.2
   Build Date               2022-11-23T12:53:46Z
   Storage Type             file
   Cluster Name             vault-cluster-f06f20a2
   Cluster ID               90f4e65f-ccc1-1fae-9529-394b04cec782
   HA Enabled               false
   

   Notice that the Vault server is already unsealed (Sealed is false).

6. Restart the Vault server to ensure that Vault server gets automatically unsealed:

   $ sudo systemctl restart vault
   

   Copy

7. Verify the Vault status.

   $ vault status
   

   Copy

   Example output:

   Key                      Value
   ---                      -----
   Recovery Seal Type       shamir
   Initialized              true
   Sealed                   false
   Total Recovery Shares    5
   Threshold                3
   Version                  1.12.2
   Build Date               2022-11-23T12:53:46Z
   Storage Type             file
   Cluster Name             vault-cluster-f06f20a2
   Cluster ID               90f4e65f-ccc1-1fae-9529-394b04cec782
   HA Enabled               false
   

8. Log into Vault.

   $ vault login
   Token (will be hidden):
   

   Copy

   Provide the generated Initial Root Token value from the initialization output when prompted.

   Example output:

   Success! You are now authenticated. The token information displayed below
   is already stored in the token helper. You do NOT need to run "vault login"
   again. Future Vault requests will automatically use this token.
   
   Key                  Value
   ---                  -----
   token                hvs.oeRIhQy9AYgSpuJCSRgDxgeP
   token_accessor       lvPtNBLpyS82SYeFiSsadCFW
   token_duration       ∞
   token_renewable      false
   token_policies       ["root"]
   identity_policies    []
   policies             ["root"]
   

9. Review the Vault configuration file (/etc/vault.d/vault.hcl).

   $ cat /etc/vault.d/vault.hcl
   

   Copy

   Example output:

   storage "file" {
   path = "/opt/vault"
   }
   listener "tcp" {
   address     = "0.0.0.0:8200"
   tls_disable = 1
   }
   seal "awskms" {
   region     = "us-west-2"
   kms_key_id = "df68263c-d9ee-44e3-afca-83cfa370c605"
   }
   ui=true
   

   Notice the Vault configuration file defines the awskms stanza which sets the AWS KMS key ID to use for encryption and decryption.

At this point, you should be able to launch the Vault Enterprise UI by entering the address provided in the terraform apply outputs (e.g. http://35.91.217.103:8200/ui) and log in with your initial root token.

Step 3: Clean up

You can follow these steps to clean up resources from this hands on lab scenario.

1. Exit the EC2 SSH session by typing exit and pressing return.

   Example output:

   logout
   Connection to 35.91.217.103 closed.
   

   Copy

2. Clean up the AWS cloud resources provisioned by the Terraform configuration in Step 1 without additional approval.

   $ terraform destroy -auto-approve
   

   Copy

   Example output:

   random_pet.env: Refreshing state... [id=genuine_satyr]
   tls_private_key.main: Refreshing state... [id=5ca681e470933099a09de8adf756c722267071b1]
   null_resource.main: Refreshing state... [id=8108715646854798618]
   ...snip...
   random_pet.env: Destroying... [id=genuine_satyr]
   random_pet.env: Destruction complete after 0s
   
   Destroy complete! Resources: 15 destroyed.
   

   Copy

3. Delete the state files and SSH key generated by Terraform.

   $ rm -rf .terraform terraform.tfstate* private.key
   

   Copy

4. Unset your AWS Access Key ID and Secret Access Key environment variable values.

   $ unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY
   

   Copy

   This command is expected to produce no output.

Help and reference

• Vault 1.0: How to auto-unseal and other new features
• Seal Migration
• Vault Enterprise Auto Unseal
• Configuration: awskms Seal