I have been thinking of creating an article to list a few of my experiences managing Terraform setups. About a month back, Slack Engineering published a post on how they use Terraform at Slack. Not only did it cover a lot of aspects I had in mind, but it also went on to provide excellent details on their Terraform journey and how they have evolved and matured over a period. The post also provided a lot of implementation details of Slack’s in-house tooling for managing IaC using Terraform. If you are interested in knowing more about using Terraform on a big scale, you might find this article interesting as I did.
My article today summarises some of the aspects this Slack article covers and provides my viewpoints on these aspects.
Using Terraform over CloudFormation for IaC
Even though Slack runs the bulk of its workloads in AWS, it also uses Digital Ocean, NS1, and GCP. That is why it chose Terraform over CloudFormation.
I have always found that comparing Terraform and CloudFormation is like comparing apples and oranges. People tend to focus only on the cloud services aspect of this comparison. Look at this page of all the Terraform providers, and you will see 53 pages of tools and services that you can provision, manage, and configure using Terraform. It is the one ring to manage all the others.
AWS multi-account strategy
The post talks about the challenges of usage limits and access control that Slack faced, which forced them to move from a single account to a multi-account AWS strategy. It also discusses how Slack uses dedicated AWS accounts for some services and teams.
A multi-account AWS strategy indeed adds a bit of complexity to your setup. However, I feel that for any decent-sized shop, a multi-account setup is the way to go. This page talks about a few of the benefits of using multiple accounts.
Orchestrating Terraform pipelines
The post provides some great details about how Slack designs Jenkins pipelines and stages, and how pipelines are chained for various environment deploys. It also discusses the challenges of regularly creating new pipelines and how Slack developed an in-house tool to quickly churn out new pipelines.
Having designed and developed Terraform pipelines in Jenkins, GitHub Actions, and CircleCI, I can certify that getting your first Terraform pipeline right is paramount. It is essential to arrive at a standard convention for naming your environments, environment files, contexts, projects, and resources, and how you use these in your Terraform code.
Once you have got the template right, you can copy and paste the pipeline to any repo and it should work with minimal changes. If you spend time creating new pipelines from scratch, you have not got your template right.
Not having a centralised Ops team to manage all Terraform code
Slack has a central team called the Cloud Foundations team, which manages the tooling and the platform for running and managing Terraform - however, individual teams own their Terraform code.
I believe this is how the setup should be. The DevOps team should focus on the platform aspect of things. Functional code and IaC should go hand in hand, if not in the same hand. The dev team not owning IaC - and another team playing catch-up with Dev to update and manage IaC - is the perfect recipe for delays, even for small teams.
Designing Terraform states
The post mentions how Slack designs its states and has some significant inputs on how it can be done at scale. It discusses having a state file per region in each child account, and a separate state file for global services like IAM and CloudFront. It also talks about keeping the number of resources managed by each state file to a minimum.
I feel there is no silver bullet solution to this, and how you design your Terraform state is unique to how you deploy and manage workloads.
Terraform backend and state file
Slack uses S3 as a backend for the state and DynamoDB to manage the state lock.
I have seen teams opt for Terraform Cloud to manage state, thinking the home-grown solution is not robust enough. This is a fine example to prove otherwise. With just a little planning and additional security in place, you can create your very own robust state management setup. If you end up creating your own state management solution, keep in mind that how you write your IaC pipeline and how it ties to the state management component is critical to get right.
Testing Terraform code
The post talks about how Slack tests Terraform code and covers a few challenges and how they tackle them. It talks about in-house tooling to review changes, do dependency management, review the change impact, and have on-demand boxes to do test runs.
Here are a few of my thoughts on testing Terraform:
- It is critical that you design your workflow so you can test your Terraform code before applying it to higher environments.
- You should be able to churn out new environments quickly - sometimes in the same AWS account - with minimal parameter changes. This again ties to how well you have designed your IaC pipeline.
- Remember: running a plan is not enough. Unless you apply your change, you cannot say that your code works.
- Terraform changes should work for an update, destroy, and fresh-provisioning scenario. If you are not testing all three scenarios, you are not testing enough.
- A mix of IaC and manual configuration is a sure-shot recipe for delays. If everything cannot be automated immediately, keep a backlog of future enhancements and document your manual changes well.
- Just because the code works in Dev or Test does not ensure it will work in Prod. Identifying these scenarios and retrofitting the fixes is vital for a robust IaC setup.
- The actual test of your Terraform code is how confidently you can apply it to Production without worrying about impact or downtime.
- Last but not least, testing IaC takes time - sometimes significantly more time than the actual change took. Plan for it.
Using Terraform modules
The post talks in detail about how Slack has developed a way to develop, manage, and consume Terraform modules. It has some significant inputs on some of the challenges and the possible solutions for a home-grown approach.
Terraform modules often evolve over time - common patterns that are repeated often get packaged as modules. This page talks about some of the common patterns of module creation. While it is possible to reference modules using a relative path and git repo links, if you are a shop that creates and uses many in-house modules, you might want to explore the Terraform Cloud (HashiCorp’s managed service offering) feature to host a Private Registry.
Challenges of upgrading Terraform versions
The post provides significant detail on how Slack has faced issues with Terraform and provider version upgrades, and how they solved them with in-house tooling. The post contains many implementation details with code snippets which can be valuable if you build such tooling.
Barring a few sporadic issues, I have not really faced many issues with Terraform or provider version upgrades - maybe because I have been predominantly exposed to greenfield setups. However, as this post stresses, having a Terraform and provider version upgrade strategy can be very useful.