Maintain visibility over the use of cloud architecture patterns
AWS Architecture BlogMaintain visibility over the use of cloud architecture patternsby Rostislav Markov and Ibrahim Bakayoko | on19 SEP 2022| inAmazon EventBridge,Amazon QuickSight,Architecture,AWS CloudTrail,AWS Service Catalog,Kinesis Data Firehose|Permalink|ShareCloud platform and enterprise architecture teams use architecture patterns to provide guidance for different use cases. Cloud architecture patterns are typically aggregates of multiple Amazon Web Services (AWS) resources, such asElastic Load BalancingwithAmazon Elastic Compute Cloud, orAmazon Relational Database ServicewithAmazon ElastiCache. In a large organization, cloud platform teams often have limitedgovernanceover cloud deployments, and, therefore, lack control or visibility over the actual cloud pattern adoption in their organization.While having decentralized responsibility for cloud deployments is essential to scale, a lack of visibility or controls leads to inefficiencies, such as proliferation of infrastructure templates, misconfigurations, and insufficient feedback loops to inform cloud platform roadmap.To address this, we present an integrated approach that allows cloud platform engineers to share and track use of cloud architecture patterns with:AWS Service Catalogto publish an IT service catalog of codified cloud architecture patterns that are pre-approved for use in the organization.Amazon QuickSightto track and visualize actual use of service catalog products across the organization.This solution enables cloud platform teams to maintain visibility into the adoption of cloud architecture patterns in their organization and build a release management process around them.Publish architectural patterns in your IT service catalogWe use AWS Service Catalog to create portfolios of pre-approved cloud architecture patterns and expose them as self-service to end users. This is accomplished in a shared services AWS account where cloud platform engineers manage the lifecycle of portfolios and publish new products (Figure 1). Cloud platform engineers can publish new versions of products within a portfolio and deprecate older versions, without affecting already-launched resources in end-user AWS accounts. We recommend usingorganizational sharingto share portfolios with multiple AWS accounts.Application engineers launch products by referencing the AWS Service Catalog API. Access can be via infrastructure code, likeAWS CloudFormationandTerraForm, or an IT service management tool, such as ServiceNow. We recommend using a multi-account setup for application deployments, with an application deployment account hosting the deployment toolchain: in our case, usingAWS developer tools.Although not explicitly depicted, the toolchain can be launched as an AWS Service Catalog product and include pre-populated infrastructure code to bootstrap initial product deployments, as described in the blog postAccelerate deployments on AWS with effective governance.Figure 1. Launching cloud architecture patterns as AWS Service Catalog productsTrack the adoption of cloud architecture patternsTrack the usage of AWS Service Catalog products by analyzing the corresponding AWS CloudTrail logs. The latter can be forwarded to anAmazon EventBridgerule with a filter on the following events: CreateProduct, UpdateProduct, DeleteProduct, ProvisionProduct and TerminateProvisionedProduct.The logs are generated no matter how you interact with the AWS Service Catalog API, such as through ServiceNow or TerraForm. Once in EventBridge, Amazon Kinesis Data Firehose delivers the events toAmazon Simple Storage Service (Amazon S3)from where QuickSight can access them. Figure 2 depicts the end-to-end flow.Figure 2. Tracking adoption of AWS Service Catalog products with Amazon QuickSightDepending on your AWSlanding zonesetup, CloudTrail logs from all relevant AWS accounts and regions need to be forwarded to a central S3 bucket in your shared services account or, otherwise, centralized logging account. Figure 3 provides an overview of this cross-account log aggregation.Figure 3. Aggregating AWS Service Catalog product logs across AWS accountsIf your landing zone allows, consider giving permissions to EventBridge in all accounts to write to a centralevent busin your shared services AWS account. This avoids having to set up Kinesis Data Firehose delivery streams in all participating AWS accounts and further simplifies the solution (Figure 4).Figure 4. Aggregating AWS Service Catalog product logs across AWS accounts to a central event busIf you are already using anorganization trail, you can useAmazon AthenaorAWS Lambdato discover the relevant logs in your QuickSight dashboard, without the need to integrate with EventBridge and Kinesis Data Firehose.Reporting on product adoption can be customized in QuickSight. The S3 bucket storing AWS Service Catalog logs can be defined in QuickSight asdatasets, for which you can create ananalysisand publish as a dashboard.In the past, we have reported on the top ten products used in the organization (if relevant, also filtered by product version or time period) and the top accounts in terms of product usage. The following figure offers an example dashboard visualizing product usage by product type and number of times they were provisioned. Note: the counts of provisioned and terminated products differ slightly, as logging was activated after the first products were created and provisioned for demonstration purposes.Figure 5. Example Amazon QuickSight dashboard tracking AWS Service Catalog product adoptionConclusionIn this blog, we described an integrated approach to track adoption of cloud architecture patterns using AWS Service Catalog and QuickSight. The solution has a number of benefits, including:Building an IT service catalog based on pre-approved architectural patternsMaintaining visibility into the actual use of patterns, including which patterns and versions were deployed in the organizational units’ AWS accountsCompliance with organizational standards, as architectural patterns are codified in the catalogIn our experience, the model may compromise on agility if you enforce a high level of standardization and only allow the use of a few patterns. However, there is the potential for proliferation of products, with many templates differing slightly without a centralgovernanceover the catalog. Ideally, cloud platform engineers assume responsibility for the roadmap of service catalog products, with formal intake mechanisms and feedback loops to account for builders’ localization requests.Rostislav MarkovRostislav Markov is principal architect with AWS Professional Services. As technical leader in Strategic Industries, he works with AWS customers and partners on their cloud transformation programs. Outside of work, he enjoys spending time with his family outdoors and exploring New York City culture.Ibrahim BakayokoIbrahim is a Cloud Infrastructure Architect with AWS Professional Services. He works with large healthcare companies providing technical guidance and assistance for cloud infrastructure, security, and DevOps. Outside of work, he enjoys spending time with family and reading.
Cloud platform and enterprise architecture teams use architecture patterns to provide guidance for different use cases. Cloud architecture patterns are typically aggregates of multiple Amazon Web Services (AWS) resources, such as Elastic Load Balancing with Amazon Elastic Compute Cloud, or Amazon Relational Database Service with Amazon ElastiCache. In a large organization, cloud platform teams often […]
Rostislav Markov
Amazon EventBridge Amazon QuickSight Architecture AWS CloudTrail AWS Service Catalog Kinesis Data Firehose