Oct 26, 2020
In a series of blog posts we will focus on some of the best practices we use within Merapar to evolve our DevOps practices we have built around the AWS platform. Why? Because we think it’s fun to share knowledge and to learn from others in the industry!
Situation
In this blog post we focus on how we increase our (developers) cost awareness of the ongoing spend we have in our development AWS accounts.
It might be a stigma, but we developers are lazy by nature and not very interested in accounts and budgets. Actually this is something which Merapar promotes, don’t do repetitive tasks yourself but automate them wherever possible. We don’t look that often at the AWS spend we have in the multiple AWS accounts, and sometimes we overspend because we forget to delete a (set of) resource(s) which will cost the company unnecessary money.
To have this information pushed, we have created an automated service which sends a weekly spending report to a shared slack channel. Developers and budget owners are members of this channel to ensure complete visibility across the organisation and creating a culture where necessary action can be taken on a regular basis.
This is not a replacement for setting budget alarms in the AWS Budgets service, but rather provides additional insight in the spending in a frictionless way, even when the forecast budget is not reached.
Solution
Naturally we choose to run this service as a weekly task using AWS Lambda. In our AWS organisation where we use consolidated billing we don’t want to run this AWS Lambda in the root account. To comply with the best practices we have an AWS account for “internal services”. We run the AWS Lambda in this account and provide access to the billing information via cross account access.
The high level solution is depicted on the following diagram:
In the AWS Lambda, which is triggered by a CloudWatch event, we execute the following functionality:
- Configure the AWS javascript SDK to assume a role to the root account when requesting information.
- Setup a client to retrieve the AWS accounts in the AWS organisation.
- Get a list of all accounts (we need this later to map account IDs to account names).
- Setup a client to the AWS cost explorer.
- Ask the AWS cost explorer for all costs (while we exclude refunds, so we have the actual costs).
- Sort the accounts depending on the costs made, and log the result to slack. With the list of accounts we can map account IDs to account names to make the result easy to understand.
The following gist shows the code, this code accepts the following environment variables to configure :
- BILLING_ACCOUNT_ROLE : The role to assume to get right to AWS Organisation and AWS Costs Explorer in the root account
- SLACK_TOKEN : Token used to authenticate on slack
- SLACK_CHANNEL : The channel to post the messages to
Note that you only need a single service per AWS organisation, if you managed multiple AWS organisations you must deploy the service for each.
// Copyright 2020 Merapar
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Config must be provided as environment variables
// AWS Role to be assumed in billing account: BILLING_ACCOUNT_ROLE
const AwsBillingAccountRole = process.env.BILLING_ACCOUNT_ROLE;
// Slack token: SLACK_TOKEN
const SlackToken = process.env.SLACK_TOKEN;
// Slack channel: SLACK_CHANNEL
const SlackConversationId = process.env.SLACK_CHANNEL;
// AWS SDK
const AWS = require('aws-sdk');
// Moment for datetime operations
const moment = require('moment');
// Initialize Slack Client
const SlackWebClient = require('@slack/web-api').WebClient;
const slack = new SlackWebClient(SlackToken);
// Take AWS credentials from ~/.aws/config or environment variables
// AWS Billing is located in us-east-1
AWS.config.update({
region: 'us-east-1'
});
// Assume role in AWS Billing account
AWS.config.credentials = new AWS.ChainableTemporaryCredentials({
params: {
RoleArn: process.env.BILLING_ACCOUNT_ROLE
}
});
exports.handler = async (event) => {
try {
/* get all accounts in organization */
let AwsOrganizations = new AWS.Organizations();
let AwsListAccounts = await organizations.listAccounts({}).promise();
let AwsAccounts = listAccounts.Accounts;
/* Get costs for all accounts */
let AwsCostExplorer = new AWS.CostExplorer();
let AwsCostAndUsageParams = {
TimePeriod: {
End: moment().startOf('isoweek').format('YYYY-MM-DD'),
Start: moment().startOf('isoweek').subtract(1, 'week').format('YYYY-MM-DD')
},
Filter: {
Not: {
Dimensions: {
Key: 'RECORD_TYPE',
Values: ['Credit', 'Refund']
}
}
},
Granularity: 'MONTHLY',
GroupBy: [
{
Key: 'LINKED_ACCOUNT',
Type: 'DIMENSION'
}
],
Metrics: [
'UnblendedCost'
],
};
let AwsCostAndUsage = await AwsCostExplorer.getCostAndUsage(AwsCostAndUsageParams).promise();
// Maps and sort last weeks costs per AWS account
let AwsLastWeekCostsPerAccount = AwsCostAndUsage.ResultsByTime[0].Groups
.map(group => {
return {
accountId: group.Keys[0],
accountName: AwsAccounts.find(account => account.Id === group.Keys[0]).Name,
amount: Number(group.Metrics.UnblendedCost.Amount)
}
})
.sort((a, b) => {
// Sort on amount DESC
if ( a.amount < b.amount ){
return 1;
}
if ( a.amount > b.amount ){
return -1;
}
return 0;
});
let SlackPost = await slack.chat.postMessage({
text: `:dollar: AWS costs for week ${moment().isoWeek()}`,
channel: SlackConversationId,
attachments: AwsLastWeekCostsPerAccount.map(account => {
return {
text: `${account.accountName} _(${account.accountId})_: *\$${account.amount.toFixed(2)}*`,
color: "#123456",
mrkdwn_in: ["text"]
}
})
});
console.log(SlackPost);
return AwsLastWeekCostsPerAccount;
}
catch(e){
console.error(e);
}
};