Handling AWS Chargebacks for Enterprise Customers
AWS Architecture BlogHandling AWS Chargebacks for Enterprise Customersby Varad Ram | on23 JAN 2019| inAmazon DynamoDB,Amazon Simple Storage Service (S3),Architecture,AWS Lambda|Permalink|ShareAs AWS product portfolios and feature sets grow, as an enterprise customer, you are likely to migrate your existing workloads and innovate your new products on AWS. To help you keep your cloud charges simple, you can use consolidated billing. This can, however, create complexity for your internal chargebacks, especially if some of your resources and services are not tagged correctly. To help your individual teams and business units normalize and reduce their costs as your AWS implementation grows, you can implement chargebacks transparently and automate billing.This blog post includes a walkthrough of an end-to-end mechanism that you can use to automate your consolidated billing charges for either your existing AWS accounts, or for newly created accounts.WalkthroughPrerequisites for implementation:One account that is the payer account, which consolidates billing and links all other accounts (including admin accounts)An understanding of billing, Detailed Billing Report (DBR), Cost and Usage Report (CUR), and blended and unblended costsActivate propagation of necessary cost allocation tags to consolidated billingAccess to reservations across the linked accountsRead permission on the source bucket and write permission to the transformed bucketAn automated method (such as database access or an API) to verify the cost centers tagged to AWS resourcesPermissions to get access to the services described in this solution on the account targeted for this automationBefore you begin, it is important to understand the blended costs and unblended costs in consolidated billing. Blended costs are calculated based on the blended rate (the average rates for the reserved and on-demand instances that are used by your member accounts) for each service your accounts used, multiplied by the account usage of those services. Unblended costs are the charges for those services broken out for each linked account.Based on your organization’s strategy for savings (centralized or not), you could consider either the blended or unblended costs. The consolidated billing files that include the information for the chargeback are the Detailed Billing Report (DBR) and Cost and Usage Report (CUR). Both of these reports provide both the blended and unblended rates as separate columns.To help you create and maintain your AWS accounts, you can use AWS Account Vending Machine (AVM). You can launch AVM from either the AWS Landing Zone or with a custom solution. AVM keeps all your account information in a DynamoDB table (such as the account number, root mail ID, default cost center, name of the owner, etc.) and maintains reservation-related data (such as invoice ID, instance type, region, amount, cost center, etc.) in another table. To enable your account administrator to add invoice details for all your reservations, you can use a web page hosted on AWS Lambda, Amazon Simple Storage Service (Amazon S3), or a web server.To begin the process of billing transformation, you must add a trigger on an S3 bucket (which contains raw AWS billing files) that pushes messages (PutObject) into Amazon Simple Queue Service (SQS) and your billing transformation program (written in Python, Nodejs, Java, .net, etc. using AWS SDK) that runs on an Amazon Elastic Compute Cloud (Amazon EC2) instance, containers, or Lambda (if the bill can be processed within 15 minutes with file size restrictions).The billing transformation program must do the following:Cachethe Account details and reservation DynamoDB tablesVerifyif there are any messages in SQSIgnoreif the file is not a DBR or CUR file (process either of them, not both)Downloadthe file, unzip, and read row-by-row; for a DBR file, consider only the “LineItem” RecordTypeAddtwo new columns: Bill_CostCenter and Bill_NotesIf there is a valid value in the CostCenter tag (verified with internal automation processes), add the same value to the Bill_CostCenter column and any notes to the Bill_Notes columnIf the CostCenter is invalid, get the default Cost Center from the cached account details and add the information to the Bill_CostCenter and Bill_Notes columnsIf the row is a reservation invoice, the cost center information comes from the reservation table and is added to the correct columnCacheconsolidation of cost centers with the blended or unblended cost of each rowWriteeach of these processed line items into a new fileHandleexceptions by the normal organization practices (for example, email the owner of the cost center or the finance team)Pushthe new file into the transformed Amazon S3 bucketWritethe consolidated lines into a different file and upload to Transformed Amazon S3 bucketFigure 1 – Architecture of processing a billing chargebackFigure 2 – Validating the Cost Center processAfter you have the consolidated billing file aggregated by cost center, you can easily see and handle your internal chargebacks. To further simplify your chargeback model, you can get help from AWS Technical Account Managers and Billing Concierge, if your organization would like AWS to provide custom invoices from the consolidated billing file.Because the cost centers in your organization can expire over time, it’s important validate them frequently with automation, such as a Lambda program.ImprovementsIf your organization has a more complex chargeback structure, you can extend the logic described above to support deeper and broader chargeback codes, or implement hierarchical chargeback structure.You can also extend the transformation logic to support several chargeback codes (such as comma separated or with additional tags) if you have multiple teams or project that want to share a resource.SummaryAs enterprise organizations grow and consume more cloud services, the cost optimization process grows and evolves with them. Sophisticated chargeback models enable the teams and business units in the organization to be accountable and contribute to take the steps necessary to normalize the usage and costs of AWS services.TAGS:amazon dynamodb,Amazon EC2,amazon lambda,billing,chargebacksVarad RamVarad Ram likes to help customers adopt to cloud technologies and he is particularly interested in Artificial Intelligence. He believes Deep Learning will power future technology growth. In his spare time, his daughter and toddler son keep him busy biking and hiking.
As AWS product portfolios and feature sets grow, as an enterprise customer, you are likely to migrate your existing workloads and innovate your new products on AWS. To help you keep your cloud charges simple, you can use consolidated billing. This can, however, create complexity for your internal chargebacks, especially if some of your resources […]
Varad Ram
Amazon DynamoDB Amazon Simple Storage Service (S3) Architecture AWS Lambda