How Parametric Built Audit Surveillance using AWS Data Lake Architecture
AWS Architecture BlogHow Parametric Built Audit Surveillance using AWS Data Lake Architectureby Raghavarao Sodabathina, Amir Aliabadi, Anthony Galleno, and Johnny Hom | on28 OCT 2021| inAmazon Athena,Amazon EMR,Amazon QuickSight,Amazon Simple Storage Service (S3),Architecture,AWS Glue,AWS Key Management Service,AWS Lake Formation,AWS Step Functions,DevOps|Permalink|ShareParametric Portfolio Associates (Parametric), a wholly owned subsidiary of Morgan Stanley, is a registered investment adviser. Parametric provides investment advisory services to individual and institutional investors around the world. Parametric manages over 100,000 client portfolios with assets under management exceeding $400B (as of 9/30/21).As a registered investment adviser, Parametric is subject to numerous regulatory requirements. The Parametric Compliance team conducts regular reviews on the firm’s portfolio management activities. To accomplish this, the organization needs both active and archived audit data to be readily available.Parametric’s on-premises data lake solution was based on an MS-SQL server. They used an Apache Hadoop platform for their data storage, data management, and analytics. Significant gaps existed with the on-premises solution, which complicated audit processes. They were spending a large amount of effort on system maintenance, operational management, and software version upgrades. This required expensive consulting services and challenges with keeping the maintenance windows updated. This limited their agility, and also impacted their ability to derive more insights and value from their data. In an environment of rapid growth, adoption of more sophisticated analytics tools and processes has been slower to evolve.In this blog post, we will show how Parametric implemented their Audit Surveillance Data Lake on AWS withpurpose-built fully managed analytics services. With this solution, Parametric was able to respond to various audit requests within hours rather than days or weeks. This resulted in a system with acost savings of 5x, with no data growth. Additionally, this new system can seamlessly support a10x data growth.Audit surveillance platformThe Parametric data management office (DMO) was previously running their data workloads using an on-premises data lake, which ran on the Hortonworks data platform of Apache Hadoop. This platform wasn’t up to date, and Parametric’s hardware was reaching end-of-life. Parametric was faced with a decision to either reinvest in their on-premises infrastructure ormodernize their infrastructureusing a modern data analytics platform on AWS. After doing a detailed cost/benefit analysis, the DMO calculated a 5x cost savings by using AWS. They decided to move forward and modernize with AWS due to these cost benefits, in addition to elasticity and security features.The PPA compliance team asked the DMO to provide an enterprise data service to consume data from a data lake. This data was destined for downstream applications and ad-hoc data querying capabilities. It was accessed via standard JDBC tools and user-friendly business intelligence dashboards. The goal was to ensure that seven years of audit data would be readily available.The DMO team worked with AWS to conceptualize an audit surveillance data platform architecture and help accelerate the implementation. They attended a series of AWS Immersion Days focusing on AWS fundamentals, Data Lakes, Devops,Amazon EMR, and serverless architectures. They later were involved in a four-dayAWS Data Labwith AWS SMEs to create a data lake. The first use case in this Lab was creating the Audit Surveillance system on AWS.Audit surveillance architecture on AWSThe following diagram shows the Audit Surveillance data lake architecture on AWS by using AWS purpose-built analytics services.Figure 1. Audit Surveillance data lake architecture diagramArchitecture flowUser personas:As first step, the DMO team identified three user personas for the Audit Surveillance system on AWS.Data service compliance userswho would like to consume audit surveillance data from the data lake into their respective applications through an enterprise data service.Business userswho would like to create business intelligence dashboards using a BI tool to audit data for compliance needs.Complaince IT userswho would like to perform ad-hoc queries on the data lake to perform analytics using an interactive query tool.Data ingestion:Data is ingested intoAmazon Simple Storage Service (S3)from different on-premises data sources by usingAWS Lake Formation blueprints. AWS Lake Formation provides workflows that define the data source and schedule to import data into the data lake. It is a container for AWS Glue crawlers, jobs, and triggers that are used to orchestrate the process to load and update the data lake.Data storage:Parametric used Amazon S3 as a data storage to build an Audit Surveillance data lake, as it has unmatched 11 nines of durability and 99.99% availability. The existing Hadoop storage was replaced with Amazon S3. The DMO team created a drop zone (raw), an analytics zone (transformed), and curated (enriched) storage layers for their data lake on AWS.Data cataloging:AWS Glue Data Catalogwas the central catalog used to store and manage metadata for all datasets hosted in the Audit Surveillance data lake. The existing Hadoop metadata store was replaced with AWS Glue Data Catalog. AWS services such asAWS Glue, Amazon EMR, andAmazon Athena, natively integrate with AWS Glue Data Catalog.Data processing:Amazon EMR and AWS Glue process the raw data and places it into analytics zones (transformed) and curated zones (enriched) S3 buckets. Amazon EMR was used for big data processing and AWS Glue for standard ETL processes.AWS LambdaandAWS Step Functionswere used to initiate monitoring and ETL processes.Data consumption:After Audit Surveillance data was transformed and enriched, the data was consumed by various personas within the firm as follows:AWS Lambda andAmazon API Gatewaywere used to support consumption fordata service compliance users.Amazon QuickSightwas used to create business intelligence dashboards forcompliance business users.Amazon Athena was used to query transformed and enriched data forcompliance IT users.Security:AWS Key Management Service (KMS)customer managed keys were used for encryption at rest, and TLS for encryption at transition. Access to the encryption keys is controlled usingAWS Identity and Access Management (IAM)and is monitored through detailed audit trails inAWS CloudTrail.Amazon CloudWatchwas used for monitoring, and thresholds were created to determine when to send alerts.Governance:AWS IAM roles were attached to compliance users that permitted the administrator to grant access. This was only given to approved users or programs that went through authentication and authorization throughAWS SSO. Access is logged and permissions can be granted or denied by the administrator. AWS Lake Formation is used for fine-grained access controls to grant/revoke permissions at the database, table, or column-level access.ConclusionThe Parametric DMO team successfully replaced their on-premises Audit Surveillance Data Lake. They now have a modern, flexible, highly available, and scalable data platform on AWS, with purpose-built analytics services.This change resulted in a 5x cost savings, and provides for a 10x data growth. There are now fast responses to internal and external audit requests (hours rather than days or weeks). This migration has given the company access to a wider breadth of AWS analytics services, which offers greater flexibility and options.Maintaining the on-premises data lake would have required significant investment in both hardware upgrade costs and annual licensing and upgrade vendor consulting fees. Parametric’s decision to migrate their on-premises data lake has yielded proven cost benefits. And it has introduced new functions, service, and capabilities that were previously unavailable to Parametric DMO.You may also achieve similar efficiencies and increase scalability by migrating on-premises data platforms into AWS. Read more and get started on building Data Lakes on AWS.Data Lakes on AWSAWS purpose-built AnalyticsWhat is a Lake House approach?Harness the power of your data with AWS AnalyticsDerive Insights from AWS Lake HousewhitepaperRaghavarao SodabathinaRaghavarao Sodabathina is a Principal Solutions Architect at AWS, focusing on Data Analytics, AI/ML, and Serverless Platform. He engages with customers to create innovative solutions that address customer business problems and to accelerate the adoption of AWS services. In his spare time, Raghavarao enjoys spending time with his family, reading books, and watching movies.Amir AliabadiAmir Aliabadi is a hands on technologist with over 20 years of experience. He has been with Parametric Portfolio Associates, Inc. for 7 years. Amir's focus as Data Architect has been to bring new thought leadership on how to effectively centralize and manage the volumes of data used by the firm. The modernization of the data strategy started with an on-premises Data Lake with event driven data pipelines. With a firm-wide cloud adoption strategy in place, Amir is now focused on the migration of the data platform to AWS with a combination of lift-and-shift and net new development. Amir lives in the Pacific Northwest and in his spare time enjoys cycling, photography, and travel.Anthony GallenoAnthony Galleno is a Principal Solutions Architect at AWS, focusing on strategic customers in the financial services vertical. In that capacity, he has helped his customers design, implement, and deploy solutions on the AWS Cloud, to solve for some of their most challenging business problems. Outside of AWS, Anthony enjoys spending time with family, being outdoors, traveling, and cooking.Johnny HomJohnny Hom is the Head of the Data Management Office at Parametric Portfolio Associates, Inc. The focus of the Data Management Office has been to deliver high quality data in an efficient way to help the growth of its asset management business as it continues to scale. As part of this mission, he has directed his team to leverage the AWS platform vis-à-vis its Data Lake and Serverless architectures. Johnny is a husband and father and enjoys outdoor activities in the Pacific Northwest including road cycling, hiking, and skiing.
Parametric Portfolio Associates (Parametric), a wholly owned subsidiary of Morgan Stanley, is a registered investment adviser. Parametric provides investment advisory services to individual and institutional investors around the world. Parametric manages over 100,000 client portfolios with assets under management exceeding $400B (as of 9/30/21). As a registered investment adviser, Parametric is subject to numerous regulatory […]
Raghavarao Sodabathina
Amazon Athena Amazon EMR Amazon QuickSight Amazon Simple Storage Service (S3) Architecture AWS Glue AWS Key Management Service AWS Lake Formation AWS Step Functions DevOps