In a series of blog posts we will focus on some of the best practices we use within Merapar to evolve our DevOps practices we have built around the GCP platform. Why? Because we think it’s fun to share knowledge and to learn from others in the industry.

Situation

We come across customers who’s preferred Git repository tool and pipelines are hosted by Gitlab.com. Where we will quickly try to deploy from Gitlab runners hosted within the GCP environment, there are always some bootstrap projects required which from a dependency point cannot use the gitlab runners hosted on GCP. The easiest example, there has to be a pipeline which deploys the gitlab runner itself.

Back in the days, we would use a service account key, securely stored in Gitlab. Where the industry has learned that we need to rotate these frequently. This work is cumbersome, and often gets forgotten when the project is in maintenance mode.

Solution

A solution which is coming up, and please note that Gitlab states as of December 2022 this feature is available in Alpha and not ready for production, is to use OpenID connect with GCP workload identity federation. 

This is a mouth full of hype words, what it means is that each job running on a Gitlab runner hosted by gitlab.com has an JSON Web Token (JWT) injected via an environment variable. 
Trust should be given to a JWT token signed by Gitlab.com including certain assertions, and as such, in exchange of a JWT token a short lived service account key can be created.

There are 2 parts to this setup, a part within GCP to setup the trust and associated service account. And a second part within the build pipeline steps.

This is depicted in the picture below:

GCP Setup

Let’s look into the required Terraform setup with which we can define the required setup on GCP.

First we will define some variables to be used when creating the other resources

locals {
  gitlab_url          = "https://gitlab.com"
  # The organisation as created within gitlab.com
  gitlab_org          = "my-first-org"

  wip_id              = "gitlab-pool"
  wipp_id             = "gitlab-provider"
}

We should create a service account, and provide permissions to create tokens:

resource "google_service_account" "sa" {
  account_id   = "gitlab-deployment-runner"
}

resource "google_project_iam_member" "sa-token-creator" {
  role    = "roles/iam.serviceAccountTokenCreator"
  member  = "serviceAccount:${google_service_account.sa.email}"
}

Additionally, you want to provide the necessary rights to the service account for the actual deployments.

We are as well required to create an identity pool and an identity pool provider:

resource "google_iam_workload_identity_pool" "wip" {
  workload_identity_pool_id = local.wip_id
}

resource "google_iam_workload_identity_pool_provider" "wipp" {
  workload_identity_pool_id          = local.wip_id
  workload_identity_pool_provider_id = local.wipp_id
  attribute_condition                = "assertion.namespace_path.startsWith(\"${local.gitlab_org}\")"
  attribute_mapping = {
    "google.subject"           = "assertion.sub",      
    "attribute.aud"            = "assertion.aud",
    "attribute.project_path"   = "assertion.project_path",
    "attribute.project_id"     = "assertion.project_id",
    "attribute.namespace_id"   = "assertion.namespace_id",
    "attribute.namespace_path" = "assertion.namespace_path",
    "attribute.user_email"     = "assertion.user_email",
    "attribute.ref"            = "assertion.ref",
    "attribute.ref_type"       = "assertion.ref_type",
  }
  oidc {
    issuer_uri        = local.gitlab_url
    allowed_audiences = [local.gitlab_url]
  }
}

Note that in the Workload Identity pool provider attribute condition you can create additional assertions which are valid for your use case. As always within the cloud, having additional assertions here promotes a least privileges and security in depth principle. For more information see https://docs.gitlab.com/ee/ci/cloud_services/google_cloud/ 

And as a last step, we need to glue the service account and workload identity together using a service account IAM binding:

resource "google_service_account_iam_binding" "sa-binding" {
  service_account_id = google_service_account.sa.name
  role               = "roles/iam.workloadIdentityUser"

  members = [ "principalSet://iam.googleapis.com/${google_iam_workload_identity_pool.wip.name}/*
  ]
}

Members can be scoped down using additional attribute referencing from attributes, see https://cloud.google.com/iam/docs/workload-identity-federation#impersonation

Gitlab runner

In order to use the policies associated with the service account, we need to execute a few actions on Gitlab, see the commands below.

test:
  image: google/cloud-sdk:slim
  script:
    - echo ${CI_JOB_JWT_V2} > .temp.jwt
    - gcloud iam workload-identity-pools create-cred-config "projects/<project number>/locations/global/workloadIdentityPools/gitlab-pool/providers/gitlab-provider"
      --service-account="gitlab-deployment-runner@<project name>.iam.gserviceaccount.com"
      --output-file=.temp.cred
      --credential-source-file=.tem.jwt
    - gcloud auth login --cred-file=`pwd`/.tem.cred

When all goes fine, the last gcloud command should have logged you in into the project. When things are misconfigured, you can verify the audit logging of the GCP project containing the identity pool for error messages, these will point you into a solution direction.

While doing so, the JWT is sent to GCP, GCP verifies that the JWT is signed correctly with the public key. The public key is exposed to GCP using the openid-connect configuration as can be seen here: https://gitlab.com/.well-known/openid-configuration. The JWT is then after validating the assertions (e.g. project limitations) exchanged for a short lived service account key.

By using this solution, you can completely move away from managed service account keys in gitlab. We have been using this solution for almost a year and did not have any issues with it although it's still in alfa. We are not using it for any production critical pipelines.