In a series of blog posts we will focus on some of the best practices we use within Merapar to evolve our DevOps practices we have built around the GCP platform. Why? Because we think it’s fun to share knowledge and to learn from others in the industry.
Nothing seems better than to start your GCP deployment from a clean GCP organisation. No technical depth, organisational policies you are not aware of, etc.
Right?
However this also comes with a lot of challenges to overcome, how can you bootstrap your GCP organisation using Infrastructure as Code?
This is very much aside from any best practices you want to apply while creating your projects. Think about audit logging, least privilege access and budgets.
We often go for 3 layers in our infrastructure as code projects:
First layer, the GCP Seed Layer, in here we deploy as little as possible to be able to run the next layer. Minimum components are:
Deployment of this happens manually by someone who has enough permissions to do so and is mostly a one time thing. The terraform state is checked in, you could later add this state to the create bucket by backend migration.
This build on the idea’s as defined here: terraform-google-bootstrap
Although we do prefer to not always use these pre-build modules since they sometimes deploy more resources then you would expect. This is always the balance we are looking for, since we do use terraform-google-project-factory to create projects.
The second layer is our Project Definition Layer. Here we create all projects and folders including their security baseline. These deployments are happening using a workload identity pool with trust to Gitlab, see our other blog.
Deployment occurs automated by the CI/CD pipeline, by calling the terraform apply from within the pipeline (after a manual approval of the result of terraform plan), the state is stored in a cloud storage bucket made in the previous step. As well the service account is used as made in the previous step.
For this, we heavily use terraform-google-project-factory to create projects and provide the security baseline using any additional resource as demanded.
The third layer is the Project Infrastructure Layer, in which we are deploying all project related instances, such as VPC, Compute Engines, Pub/Sub Topics / Subscriptions, etc.
This layer is changing almost daily, and is therefore located close to the source code hosted in the projects. The deployment is automated either running on a CI/CD pipeline hosted by GitLab or customer runners within our network.
Typically we want to keep our Terraform code dry from this level, and use a tool like terragrunt to achieve this. See our other blog “Keeping your terraform DRY using terraform”
We hope that this blog helps you in understanding how to automate the complete setup of your GCP organisation. In case you are in need of any help. Feel free to reach out to Merapar.