A modern approach to implementing the serverless Customer Data Platform
AWS Architecture BlogA modern approach to implementing the serverless Customer Data Platformby Larry Bell, Brian Maguire, and Lewis Steckler | on19 DEC 2022| inArchitecture,AWS Well-Architected,Customer Solutions,Marketing & Advertising,Serverless|Permalink|ShareWhen building a Customer Data Platform (CDP), advertising and marketing Independent Software Vendors (ISVs) face a unique set of challenges. The ISV can help organizations with the heavy lifting required to build, secure, and maintain near real-time, high volume CDPs. However, architecting CDPs using traditional on-premises technologies can introduce multiple complexities and can limit deployment options. One strategy that may address these complexities is to use serverless technologies.Serverless technologies feature automatic scaling, built-in high availability, and a pay-for-use billing model to increase agility, optimize costs, and reduce infrastructure management tasks such as capacity provisioning and patching. Using tools such as CloudFormation, each layer of the serverless CDP can be deployed on-demand in an independent manner to maximize portability and optimize performance.A Software as a Service (SaaS) CDP usually has significantly more data in a multi-tenant environment than a single instance of a CDP. Clients of a SaaS solution need to continually expand across different channels, and often across many AWS Regions. In some cases, an ISV might have an existing infrastructure that was built before some of these modern capabilities and techniques were mature. Today, an ISV can build or even modernize an existing CDP and gain huge benefits from a serverless implementation.This blog post explores how to use serverless technologies for the CDP. A modern, serverless CDP architecture can enable the ISV and the client companies to deliver in weeks instead of months, and provide a resilient infrastructure that supports agility and global deployment while maximizing operational efficiency and optimizing cost. This frees up technical resources to focus on differentiated product development instead of managing servers.Serverless implementation of a CDP on AWSA serverless architecture uses AWS services that don’t require the configuration of a server to provide an implementation. Serverless technology allows you to focus more time on rapidly building different components of the marketing CDP. The benefits of a CDP include the collection, aggregation, and organization of customer data sources. Implementing the CDP using serverless technology reduces the need to focus on managing infrastructure while reducing time to market, increasing agility, and resulting in cost optimization. Figure 1 is an architecture diagram that describes how various data sources can be prepared for consumption in the component based Customer Data Platform.Figure 1. Marketing CDP reference on AWSSource systems of customer data include customer interactions, clickstreams and call center logs.Data from customer touchpoints is ingested into the marketing customer data platform (CDP) data lake using Amazon Kinesis, Amazon AppFlow, Amazon EKS and an Amazon API Gateway.Ingested data is sent – in its original, immutable format – to an Amazon Simple Storage Service (Amazon S3) Raw Zone bucketRaw data is then transformed into efficient data formats – such as Parquet or Avro – and moved to a Clean Zone Amazon S3 bucket.CDP processing and pipeline orchestration is conducted using purpose-built data processing components and transformation libraries through AWS Step Functions and then Amazon Personalize, AWS Lambda, and AWS Glue.Data in the Amazon S3 Curated Zone is now ready for post-CDP-processing consumption and is organized by subject areas, segments, and profiles.The analytics layer uses Amazon Redshift, Amazon QuickSight, Amazon SageMaker and Amazon Athena to natively integrate with the Curated Zone for analytics, dashboards, ad hoc reporting, and ML purposes.Customer data is then aggregated across platforms and published using customer APIs for consumption using Amazon DynamoDB and an Amazon API Gateway.Amazon Pinpoint and Amazon Connect are used to activate multiple customer channels such as mobile push, voice, and email for targeted marketing communications.Using AWS Lake Formation, fine-grained access controls can be enforced on catalog tables, columns, and rows on the data lake.The resulting catalog in AWS Glue helps you manage both business and technical metadata, with versioning, at scale.Serverless implementation for ingestionThere are several methods of ingesting customer data, both internal to a customer and from external sources. Serverless options for ingestion could provide benefits to an ISV like cost or agility but it depends upon the use case. Examining serverless options for ingestion should be part of any modernization effort. If the CDP needs to stream data sources and ingest that data in near-real time, the ISV can useAmazon Kinesis. If you want a more traditional extract, transform, and load (ETL) tool,AWS Glueoffers a serverless option to generate code that can be customized.AWS Glue DataBrewoffers a visual data preparation tool. For more advanced governance and control, you can useAWS Lake Formation.To ingest sources using an API, theAmazon API Gatewayprovides a serverless approach. If you need more control over the ingestion, the use of customized scripts inAmazon AppFloworAmazon Managed Streaming for Apache Kafka(Amazon MSK) can provide a solution.Serverless storage implementationAmazon Simple Storage Service(Amazon S3) provides a serverless, cost-effective solution for virtually unbounded amounts of storage and read-write bandwidth. As per the reference architecture, there are three purpose-specific zones:A raw zone containing the original, immutable version of dataA trusted zone which can be used as a working area to combine, enhance and clean the dataA refined zone containing data ready for consumption by users and applicationsThis structure allows the improvement of customer data and profiles, and provides the ability to integrate various data sources and a structure that allows customer data to be recreated in a manner consistent with changing business rules.Serverless cataloging implementationThe cataloging services provide a grouping of the elements contained in structured and unstructured data sources that is intuitive and easy to understand, similar to a single relational database.AWS Glue Data Cataloggives logical structure to the data lake by allowing users to define tables and columns on top of Amazon S3 data sets. This serverless solution integrates with other analytics tools to enable data discovery and consistent usage. Fine-grained governance and access can also be enforced by AWS Lake Formation.Serverless processingThere are great choices for implementing processing, using serverless technologies. A CDP platform can package code and run on demand without servers usingAWS LambdaorAWS Step Functionsdepending up the complexity of the processing pipeline. These services can enable complex processing on customer data and profiles.Amazon SageMakeris a great serverless choice for incorporating artificial Intelligence / machine learning into your processing stream. For processing using big data techniquesAmazon EMR Serverlessis a good serverless option.Serverless implementation for consumptionAnalytics for the CDP provides several serverless technologies that enable different types of insights. For interactive SQL queries that integrate with our serverless AWS Glue Catalog, there isAmazon Athena. Athena provides SQL access to various data source, and can also use federated query functionality to connect to third-party sources, even if that data is sitting on another cloud or in a vendor’s environment. Athena can also work as an interface (middleware) to other reporting solutions.If performance is a concern,Amazon Redshiftis fast, petabyte-scale data warehouse solution that has a serverless option and fully integrates with these solutions. For a data visualization tool that can be embedded in your application or work as a standalone portal, examineAmazon QuickSight.To enable collaboration, many use cases can useAmazon API Gatewayto securely publish and expose API endpoints for consuming applications. This allows data to be shared from a single source of truth to consumers that use customer data for their processes. Most customers want to activate their customer data through marketing or advertising campaigns. To activate marketing communication over voice, email, text, or in-app messaging, you can use a serverless service calledAmazon Pinpoint. For an omnichannel contact center support, we recommendAmazon Connect, which uses AI/ML and the CDP data to analyze customer sentiment, implement chatbots, and authenticate voice callers.Serverless implementation for governanceAWS Lake Formation simplifies the process of configuring and securing access to the CDP. It can help orchestrate processing and ingestion, as well as enforcing fine-grained access controls on data catalogs. Other services such as AWS Glue DataBrew orAmazon Maciecan identify and help mitigate exposure of Personally identifiable information (PII).AWS Configenables you to assess, audit, and evaluate the configurations of your AWS resources to automate the evaluation of recorded configurations against desired configurations.ConclusionThis post described just some of the serverless solutions that are managed by AWS that allow you to build a modern, low-cost, data lake-centric CDP architecture in an accelerated manner. A decoupled, component-driven architecture lets you start small and quickly add new services to each independent component of the CDP. Use theData Analytics Lensfor guidance on designing, deploying, and architecting your analytics solution workloads in the AWS Cloud. Using this framework, you will learn the architectural best practices for designing and operating reliable, secure, efficient, and cost-effective systems in the cloud. Follow the links in this article to learn more about the services available in AWS that can help you build a serverless CDP.Further readingGuidance for Customer Data Platform on AWSAn overview and architecture of building a Customer Data Platform on AWSTAGS:Architecture,architecture best practices,C360,CDP,customer,Customer Solutions,ISV Solutions,serverless,well architectedLarry BellLarry Bell was a Solutions Architect at Amazon Web Services focusing on the advertising industry. Larry had a consulting background where he focused on data warehousing, analytics, and cloud architecture in previous roles. At AWS, he pursued his interests in Data and Analytics.Brian MaguireBrian Maguire is a Principal Solutions Architect at Amazon Web Services, where he is focused on helping customers build their ideas in the cloud. He is a technologist, writer, teacher, and student who loves learning. Brian is the co-author of the book Scalable Data Streaming with Amazon Kinesis.Lewis StecklerLewis Steckler is the Global Head of Marketing Technology at AWS. He advises AWS Martech customers and is the 1st-Party Data Solutions Owner for the Advertising and Marketing industry vertical. Lewis is formerly the Practice Lead of Advanced Digital Solutions at Epsilon. He has 15 years of experience in MarTech and AdTech and has delivered digital transformation, media, data and analytics solutions to over 100 Fortune 500 companies across every major industry vertical. He is a domain expert across consumer data management, identity, 1st- & 3rd-party data, segmentation, and omnichannel customer experience and analytics.
When building a Customer Data Platform (CDP), advertising and marketing Independent Software Vendors (ISVs) face a unique set of challenges. The ISV can help organizations with the heavy lifting required to build, secure, and maintain near real-time, high volume CDPs. However, architecting CDPs using traditional on-premises technologies can introduce multiple complexities and can limit deployment […]
Larry Bell
Architecture AWS Well-Architected Customer Solutions Marketing & Advertising Serverless