Terraform workspaces with gitlab CICD

Something that I’ve always noticed ( and people have often asked for ) is a way to do simple infrastructure.

As an operations engineer, so that I can keep my infrastructure consistent, use a single manifest to create many of the same things.

CloudFormation makes this very easy, however, it’s also possible to do this with less evolved tools like Terraform. We can use Terraform ( tf ) and workspaces to do this, and add gitlab pipelines to help with the automation.

Here’s the basic workflow:

• Create a basic VPC/Subnet layout which expresses a basic layout
• Create some workspaces in TF
• Configure our variable things
• Wire up the CICD things

In this article I’ll be talking about applications in that each application is a slice of the bigger picture.

Network, Compute and each Application are all considered applications as per our pipeline build process. We can still use modules, but we’re going to use workspaces to do something slightly different, let’s take a look at how this looks:

module "network" "production" {module "network" "stage" {
 source = "../../../modules/network"
 source = "../../../modules/network"
 vpc_cidr_block = "172.20.0.0/16"
} 
 vpc_cidr_block = "172.10.0.0/16"
}

As opposed to…

module "network" {
 source = "../../../modules/network"
 vpc_cidr_block = local.operations_config_map[terraform.workspace]["network"]["cidr"]
}

Pretty close to the same thing, right? The only real difference is that the workspace is holding the delta between the environments. We can expand this to also include different AWS providers which point to different AWS accounts. The big difference that I’ve noticed in this way of doing things is that I can be hammering on a dev environment without actually impacting any other environment. Not using environments runs the risk of impacting a production thing while working on a dev thing.

This pattern can also lead to an interesting model where we actually get rid of the module model altogether since we might not actually need it. Something to think about if you’re doing some greenfield things.

Let’s dive right in with setting up our networking things. The networking bits are in my github project called raleigh. Most of this was c&p’d from other sources.

Let’s start this by taking a look at how we differentiate the tf workspaces using local variables.

locals { public_subnet_count = length(data.aws_availability_zones.available.names)
 transit_config = { 
 
 private_subnet_count = length(data.aws_availability_zones.available.names) 
 cidr = "10.128.0.0/16" 
 peer_region = "us-west-2" 
 peer_route_table_id = "rtb-b5e31fd3" 
 }

So we have this mapping, but here’s how we use it. First we set the local config for easier use later:

locals { cfg = local.operations_config_map[terraform.workspace]}

Now we use the cfg in the VPC setup:

resource "aws_vpc" "main" {
 cidr_block = local.cfg["network"]["cidr"]
}

This is how we’re going to build our network, which will include:

• VPC with a /16 CIDR
• 2 groups of subnets, Public and Private
• InternetGateway, NAT gateway ( in the first public subnet )
• Route tables
• [Optional] we can enable the routes back to the home office. Transit gateway is something we should think about to replace this.

This is our bedrock for everything else. After this point we use data sources to lookup networking information.

For example, every environment can find the appropriate VPC by doing the following lookup. We actually bake this into the modules instead of passing it through to the module because we’re baking this into the application as an absolute assumption.

Most people will do this at the implementation level instead of the module, then pass the vpc/subnet information through the module. In most cases people will only use 1 or 2 subnets because they don’t want to have to pass more than 2 things into the module, or worse yet, they’ll just hardcode the subnet id’s and hope for the best. Both are terrible ideas when we can do something better.

The concept of IaC is supposed to give us assumptions that make things easier by making things more consistent. Hardcoding things is the opposite of that. Here’s how we find our subnets; consistency leads to accuracy.

Now let’s take a look at how we use workspaces to segment the work. I do these actions in the network folder of my project where my tf bits are.

➜ raleigh git:(master) ➜ raleigh git:(master) ➜ raleigh git:(master) Created and switched to workspace "production"!➜ raleigh git:(master) defaultterraform workspace listterraform workspace new stage
Created and switched to workspace "stage"! terraform workspace new production 
You're now on a new, empty workspace. Workspaces isolate their state, so if you run "terraform plan" Terraform will not see any existing state for this configuration. terraform workspace list 
* production
stage 
* default You're now on a new, empty workspace. Workspaces isolate their state, so if you run "terraform plan" Terraform will not see any existing state for this configuration.

Easy peasy, now we have some workspaces to play with. At this point we can tf plan on any environment and see what’s about to happen.

And now for the fun stuff. This could probably be modified to work on just about any CICD environment. I just happen to have a working version that works in gitlab.

There is a design with AWS::EKS in which each cluster is bundled with a unique, custom VPC/Subnet network. This seems fine if you have small’ish rigs, however, in this case we want to allow for multiple clusters in a given network, and make sure each network is tied back to the home office.

We want to make sure that we can route from the office network ( 10.0.0.0/8 ) to the AWS infrastructure bits ( 172.0.0.0/8 ). We might have a rig in a separate AWS account for a customer on something like 172.10.0.0/16, so we want to make sure we can privately route to the cluster to do things. We also want to make sure that all of our compute resources ( EKS rigs, or whatever ) are tucked away behind the firewalls and not publicly available.

The example above shows the output from the network application which handles the VPC/Subnet objects. For this application we have 3 environments which are being provisioned.

The really fun part of this little project comes from this little bit of code:

- >-
 curl -X POST -g -H "PRIVATE-TOKEN: ${GITLAB_TOKEN}"
 --data-urlencode "body=${PLAN}"
 "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/merge_requests/${CI_MERGE_REQUEST_IID}/discussions"

It’s sending the tf plan to the comment stream on the PR, which is super helpful. Now we can see the code that is changed as well as the changes that TF is planning to pull off. Now our entire team can see what's about to go down.

I owe a huge thanks to someone on the Renovo team. I’m not going to mention his name specifically because I don’t have his consent. Let’s call him “T”. One day, in a meeting I had mentioned that I had looked into tf workspaces and didn’t think they would do what I thought they would do. At that point I thought they were garbage and wouldn’t work.

I was wrong about how tf workspaces actually worked and T called me out on it, and rightfully so. I was a little embarrassed at the time, but quickly got up to speed on the concept and eventually built out the networking, EKS and application stacks for the company using the model I’ve outlined here.

My deepest gratitude to T for helping me learn something new. I’ve been doing this for 20 years and there’s always a new adventure. Thank you, T!

Originally published at https://medium.com on January 5, 2020.