3 Ways for Sharing Parameters between Microservices

Serverless and microservice applications in general consist of multiple deployable units - stacks. Those stacks are managed with IaC tools like CloudFormation. You can use CloudFormation directly or with higher-level tools built on top of it: Serverless Framework, SAM, or CDK. No matter what you choose, those stacks have dependencies on each other.

Let's take an e-commerce application as an example. It will have a Store service communicating with the Catalog, Orders, and Payments. Therefore, the Store is dependent on those three.

Firstly, the Store service needs to know the API endpoints of the other services to communicate. This is relatively easy to solve with good old domain names. In the Store service, you can hardcode the Catalog URL, let's say catalog.example.com. For non-prod environments, it can use a subdomain matching its own environment name: on "staging" call staging.catalog.example.com, on "preprod" call preprod.catalog.example.com, and so on.

With domains, the deployment order doesn't matter. The Store just needs to handle 404 errors if the Catalog is not (yet) available. The overhead of registering a subdomain for each service and environment is minimal after it's configured once.

But even with an API endpoint, you may need additional parameters. Since that is an internal API, it will be protected best with IAM authorization. For this purpose, you create an IAM role in the Catalog service that the Store service can assume and use to sign the requests.

You can share resource names and ARNs by defining them as CloudFormation Outputs with an Export flag. Then you can reference it in another stack using the Fn::ImportValue intrinsic CloudFormation function.

AWSTemplateFormatVersion: '2010-09-09'

Resources:
  HttpApi:
    Type: AWS::ApiGatewayV2::Api
    Properties:
      ProtocolType: HTTP

Outputs:
  ApiEndpoint:
    Value: !GetAtt HttpApi.ApiEndpoint
    Export:
      Name: !Sub '${AWS::StackName}-ApiEndpoint'

AWSTemplateFormatVersion: '2010-09-09'

Parameters:
  CatalogStackNameParameter:
    Type: String

Resources:
  MyLambda:
    Type: AWS::Lambda::Function
    Properties:
	  Runtime: nodejs14.x
	  Environment:
	    Variables:
		  CATALOG_API_ENDPOINT: !ImportValue
		    'Fn::Sub': '${CatalogStackNameParameter}-ApiEndpoint'

As you can see, there are quite a few restrictions. Most of them prevent you from altering the exported output value if any other stack uses it.

Initially, that seems to make sense. You don't want the Catalog service to be removed (together with exported parameters) since the Store service relies on it, right?

Well, not always.

Take the development environment as an example. If you create an environment for the feature branch testing, which is later removed on the branch merge, it doesn't really matter if you remove the Catalog service first. You just want to purge everything.

But this is a development environment. What you care about is the production environment stability.

However, this hard protection can backfire easily on the production too.

Suppose you find a problem in the cross-service communication that you need to fix quickly, and it requires deploying a new IAM Role. In that case, this protection can prolong the issue resolution. Instead of deploying the Catalog service with a new resource and then deploying the Store service to use it, you need an additional step of updating the Store to stop using the old parameter first.

Inability to use CloudFormation outputs cross-region is a whole different kind of an issue potentially preventing you from using it to share parameters altogether.

Another way of sharing values is using the Systems Manager Parameter Store, better known as SSM. You create an SSM parameter in one stack and fetch it during the deployment of the second one.

You can achieve this in pure CloudFormation, as well as in SAM, Serverless Framework, and CDK. In fact, Serverless Framework and CDK handle SSM parameters on their own and pass the resolved values to the CloudFormation template to provide more flexibility.

An example with CloudFormation:

And the same with Serverless Framework:

Using SSM instead of CloudFormation Outputs has several advantages:

• No limitations on the SSM parameter name as with CloudFormation Outputs.
• No restrictions on modifying or deleting existing parameters.
• In Serverless Framework, the ability to reference SSM parameters from other regions (and maybe someday in CDK too).

Of course, you can see the lack of protection of the parameters used by other stacks as a drawback. Yes, it requires thinking about usage and consequences every time you want to change an existing parameter. But, on the other hand, it allows you to move forward quicker, without the restrictions being more of an obstruction than help.

Fallback to default parameter

service: store

params:
  prod:
    targetEnv: prod
  staging:
    targetEnv: staging
  default:
    targetEnv: test

custom:
  catalogApiEndpoint: ${ssm:/catalog/${sls:stage}/apiEndpoint, ssm:/catalog/${param:targetEnv}/apiEndpoint}

provider:
  name: aws
  runtime: nodejs14.x

functions:
  myLambda:
    handler: index.handler
	environment:
	  CATALOG_API_ENDPOINT: ${self:custom.catalogApiEndpoint}

import {ssmParameter} from 'aws-parameter-cache';

const catalogApiEndpointParam = ssmParameter({
    name: '/catalog/apiEndpoint',
    maxAge: 1000 * 60 * 5, // 5 minutes
});

export const handler = async () => {
    const catalogApiEndpoint = await catalogApiEndpointParam.value;
};

import {AssumeRoleCommand, STSClient} from '@aws-sdk/client-sts';
import {GetParameterCommand, SSMClient} from '@aws-sdk/client-ssm';
import {Credentials} from '@aws-sdk/types';

export const handler = async () => {
    const catalogSSMCredentials = await getCatalogSSMCredentials();
    const catalogApiEndpoint = await getCatalogApiEndpoint(catalogSSMCredentials);
};

const getCatalogSSMCredentials = async (): Promise<Credentials> => {
    const sts = new STSClient({});
    const response = await sts.send(new AssumeRoleCommand({
        DurationSeconds: 3600,
        RoleArn: process.env.CATALOG_SSM_ROLE_ARN,
        RoleSessionName: 'awssdk',
    }));

    if (!response.Credentials) {
        throw new Error('Cannot retrieve credentials');
    }

    return {  
        accessKeyId: response.Credentials.AccessKeyId,
        secretAccessKey: response.Credentials.SecretAccessKey,
        sessionToken: response.Credentials.SessionToken,
        expiration: response.Credentials.Expiration,
    };
};

const getCatalogApiEndpoint = async (credentials: Credentials): Promise<string> => {
    const catalogSSM = new SSMClient({
	    credentials: credentials,
	});

    const response = await catalogSSM.send(new GetParameterCommand({
	    Name: '/catalog/apiEndpoint',
    }));

    if (!response.Parameter?.Value) {
	    throw new Error(`Cannot retrieve parameter ${paramName}`);
    }

    return response.Parameter.Value;
};