Building a Showback Dashboard for Cost Visibility with Serverless Architectures
AWS Architecture BlogBuilding a Showback Dashboard for Cost Visibility with Serverless Architecturesby Peter Chung, Renan Bertolazzi, and Natalie Rabinovich | on24 JUN 2021| inAmazon QuickSight,Amazon Simple Storage Service (S3),Architecture,AWS Cloud Financial Management,AWS Glue,Serverless|Permalink|ShareEnterprises with centralized IT organizations and multiple lines of businesses frequently useshowbackorchargebackmechanisms to hold their departments accountable for their technology usage and costs. Chargeback involves actually billing a department for the cost of their division’s usage. Showback focuses on visibility to make the department more cost conscientious and encourage operational efficiency.Building a showback mechanism can be potentially challenging for business and financial analysts of anAWS Organization. You may not have the scripting or data engineering skills needed to coordinate workflows and build reports at scale. Although you can useAWS Cost Exploreras starting point, you may want greater customizability, larger datasets beyond a one-year period, and more of a business intelligence (BI) experience.In this post, we discuss the benefits of building a showback dashboard using theAWS Cost and Usage Report (AWS CUR). You can track costs by cost center, business unit, or project using managed services. Using a showback strategy, you can consolidate and present costs to a business unit to show resource use over a set period of time for your entire AWS Organization. Building this solution with managed services allows you to spend time understanding your costs rather than maintaining the underlying infrastructure.This solution highlightsAWS Glue DataBrewto prepare your data into the appropriate format for dashboards. We recommend DataBrew because it provides a no-code environment for data transformation. It allows anyone to create dashboards similar to those built in theCloud Intelligence Dashboards Workshopfor your Organization.Figure 1. QuickSight showback dashboard using CUR data transformed by Glue DataBrew and leveraging QuickSight insightsTags for cost allocationThe success of your showback dashboard partially depends on your cost allocation tagging strategy. Typically, customers use business tags such as cost center, business unit, or project to associate AWS costs with traditional financial reporting dimensions within their organization.The CUR supports the ability to break down AWS costs by tag. For example, if a group of resources are labeled with the same tag, you’ll be able to see the total cost and usage of that group of resources. Read more aboutTagging Best Practicesto develop a tagging strategy for your organization.A serverless data workflow for showback dashboardsYou can build showback dashboards with managed services such asAmazon QuickSight, without the need to write any code or manage any servers.Figure 2. A serverless architecture representing data workflowAWS automatically delivers the data you need for showback dashboards through the CUR. Once this data arrives in anAmazon Simple Storage Service (S3)bucket, you can transform the data without the need to write any code by using DataBrew. You can also automatically identify the data schema, and catalog the data’s properties to run queries usingAmazon Athena. Lastly, you can visualize the results by publishing and sharing dashboards to key stakeholders within your organization using Amazon QuickSight.The key benefits of this approach are:Automatic data deliveryNo-code data transformationAutomatic cataloging and queryingServerless data visualizationLet’s take a look at each in more detail.Automatic data deliveryThe CUR is the source for historical cost and usage data. The CUR provides the most comprehensive set of cost and usage data available and will include your defined cost allocation tags for your entire Organization. You configure CUR to deliver your billing data to an Amazon S3 bucket at the payer account level. This will consolidate data for all linked accounts. After delivery starts, Amazon updates the CUR files at least once a day.No-code data transformationYou can use DataBrew to transform the data in the Amazon S3 bucket aggregating cost and usage according to your tagging strategy. DataBrew summarizes your data for discovery. You can also run transformations called “jobs” in DataBrew without writing any code, using over 250 built-in transforms. Figures 3 through 5 show several job examples.Figure 3. DataBrew recipe action: rename columnFigure 4. DataBrew recipe action: Create column from functionFigure 5. DataBrew recipe action: fill missing valuesFor a full list of columns available in CUR, review theCUR Data Dictionary. Following is a list of relevant columns for an executive summary showback dashboard:bill_billing_period_start_dateline_item_usage_account_idline_item_line_item_typeproduct_product_nameproduct_product_familyproduct_product_transfer_typesavings_plan_savings_plan_effective costreservation_effective_costline_item_unblended_costBased on data refresh and business requirements, DataBrew canrun a job on a recurring basis(for example, every 12 hours). This can be run at a particular time of day, or as defined by a valid CRON expression. This helps you automate your transformation workflows.Automatic cataloging and queryingYou can use a Gluecrawlerto automatically classify your data to determine the data’s format, schema, and associated properties. The crawlers write metadata to anAWS Glue Data Catalogto help data users find the data they need.With the results in Amazon S3, and the metadata in the Glue Data Catalog, you can run standard SQL to queries with Athena. This will help you make more informed business decisions by tracking financial metrics and optimizing costs. This is done directly in Amazon S3 without having to move around data. Using standard SQL, you can create views that aggregate cost and usage by your defined tags.Serverless data visualizationYou can use Amazon QuickSight to create and share dashboards with your teams for cost visibility. QuickSight provides native integration with Athena and S3, and lets you easily create and publish interactive BI dashboards that include ML-powered insights. When building a showback dashboard such as the example in Figure 1, QuickSight authors create visuals and publish interactive dashboards.Readers log in using your preferred authentication mechanism to view the shared dashboard. You can then filter data based on billing periods, account number, or cost allocation tags. You can also drill down to details using a web browser or mobile app.ConclusionIn this blog, we’ve discussed designing and building a data transformation process and a showback dashboard. This gives you highly granular cost visualization without having to provision and manage any servers. You can use managed services such as AWS Glue DataBrew, Amazon Athena, and Amazon QuickSight to crawl, catalog, analyze, and visualize your data.We recommend defining your organization tagging strategy to be able to view costs by tags. You can then get started by creatingCost and Usage Reports. With the data in Amazon S3, you can use the services described in this post to transform the data that works for your business. Additionally, you can get started today by experimenting with theCloud Intelligence Dashboards Workshop. This workshop provides examples of visualizations that you can build using native AWS services on top of your Cost and Usage Report. You will be able to get cost, usage, and operational insights about your AWS Cloud usage.TAGS:AWS Glue DataBrewPeter ChungPeter Chung is a Solutions Architect for AWS. He is passionate about helping customers uncover insights from their data. He has been building solutions to help organizations make data-driven decisions in both the public and private sectors. He holds all AWS certifications and two GCP certifications.Renan BertolazziRenan Bertolazzi is an Enterprise Solutions Architect helping customers to realize the potential of cloud computing on AWS. In this role, Renan is a technical leader advising executives and engineers on cloud solutions and strategies designed to innovate, simplify, and deliver results.Natalie RabinovichNatalie is an AWS Solutions Architect supporting enterprise customers with their journey to the cloud. She helps organizations design reliable and cost effective cloud solutions. Natalie has background in data center infrastructure, data storage, and big data and analytics.
Enterprises with centralized IT organizations and multiple lines of businesses frequently use showback or chargeback mechanisms to hold their departments accountable for their technology usage and costs. Chargeback involves actually billing a department for the cost of their division’s usage. Showback focuses on visibility to make the department more cost conscientious and encourage operational efficiency. […]
Peter Chung
Amazon QuickSight Amazon Simple Storage Service (S3) Architecture AWS Cloud Financial Management AWS Glue Serverless