Zurich Spain: Managing Millions of Documents with AWS
AWS Architecture BlogZurich Spain: Managing Millions of Documents with AWSby Miguel Guillot and Jesus Bernal | on02 FEB 2021| inAmazon DocumentDB,Amazon Elastic Container Registry,Amazon Elastic Kubernetes Service,Architecture|Permalink|ShareThis post was cowritten with Oscar Gali, Head of Technology and Architecture for GI in Zurich, SpainAbout Zurich SpainZurich Spainis part ofZurich Insurance Group (Zurich), known for its financial soundness and solvency. With more than 135 years of history and over 2,000 employees, it is a leading company in the Spanish insurance market.IntroductionEnterprise Content Management (ECM) is a key capability for business operations in Insurance, due to the number of documents that must be managed every day. In our digital world, managing and storing business documents and images (such as policies or claims) in a secure, available, scalable, and performant platform is critical.Zurich Spain decided to use AWS to streamline management of their underlying infrastructure, in addition to the pay-as-you-go pricing model and advanced analytics services. All of these service features create a huge advantage for the company.The challengeZurich Spain was managing all documents for non-life insurance on an on-premises proprietary solution. This was based on an ECM market standard product and specific storage infrastructure. That solution over time had several pain points: cost, scalability, and flexibility. This platform has become obsolete and was an obstacle for covering future analytical needs.After considering different alternatives, Zurich Spain decided to base their new ECM platform on AWS, leveraging many of the managed services.AWS Managed Serviceshelps to reduce your operational overhead and risk. AWS Managed Services automates common activities, such as change requests, monitoring, patch management, security, and backup services. It provides full lifecycle services to provision, run, and support your infrastructure.Although the architecture design was clear, the challenge was huge. Zurich Spain had to integrate all the existing business applications with the new ECM platform. Concurrently, the company needed to migrate up to 150 million documents including metadata, in less than 6 months.The PlatformFunctionally, features provided by ECM are:ECM FeaturesAuthentication:every request must come from an authenticated user (OpenID Connect JWT).Authorization:on every request, appropriate user permissions are validated.Documentation Services:exposed API that allows interaction with documents (CRUD). For example:The ability toIngesta document either synchronously (attaching the document to the request) or asynchronously (providing a link to the requester that can be used to attach a document when required).Upload operation stores documents ontoAmazon Simple Storage Service (S3)and its metadata, which is saved usingAmazon DocumentDB.DocumentsRetrieve, similarly to the upload operation, can be obtained either synchronously or asynchronously. The latter provides a link to be used to download the document within a time range.ECM has been developed to give the users the ability to search among all the documents uploaded into it.Metadata:every document has technical and business metadata. This gives Zurich Spain the ability to enrich every single document with all the information that is relevant for their business, for example: Customers, Author, Date of creation.Record Management:policies to manage documents lifecycle.Audit:every transaction is logged into the system.Observability:capabilities to monitor and operate all services involved: logging, performance metrics and transactions traceability.The ArchitectureThe ECM platform uses AWS services such as Amazon S3 to store documents. In addition, it uses Amazon DocumentDB to store document metadata and audit trail.The rationale for choosing these services was:Amazon S3 delivers strong read-after-write consistency automatically for all applications, without changes to performance or availability. With strong consistency, Amazon S3 simplifies the migration of on-premises analytics workloads by removing the need to update applications. This reduces costs by removing the need for extra infrastructure to provide strong consistency.Amazon DocumentDB is a NoSQL document-oriented database where its schema flexibility accommodates the different metadata needs. It was key to design the index strategy in advance to ensure the right query performance, considering the volume of data.A microservices layer has been built on top to provide the right services for the business applications. These include access control, storing or retrieving documents, metadata, and more.These microservices are built using Thunder, the internal framework and technology stack for digital applications of Zurich Spain. Thunder leverages AWS and provides a K8s environment based onAmazon Elastic Kubernetes Service (Amazon EKS)for microservice deployment.Figure 2 – Zurich Spain ArchitectureZurich Spain usesAWS Direct Connectto connect from their data center to AWS. With AWS Direct Connect, Zurich Spain can connect to all their AWS resources in an AWS Region. They can transfer their business-critical data directly from their data center into and from AWS. This enables them to bypass their internet service provider and remove network congestion.Amazon EKS gives Zurich Spain the flexibility to start, run, and scale Kubernetes applications in the AWS Cloud or on-premises. Amazon EKS is helping Zurich Spain to provide highly available and secure clusters while automating key tasks such as patching, node provisioning, and updates. Zurich Spain is also usingAmazon Elastic Container Registry (Amazon ECR)to store, manage, share, and deploy container images and artifacts across their environment.Some interesting metrics of the migration and platform:Volume: 150+ millions (25 TB) of documents migratedDuration: migration took 4 months due to the limited extraction throughput of the old platformActivity: 50,000+ documents are ingested and 25,000+ retrieved dailyAverage response time:550 ms to upload a document300 ms for retrieving a document hosted in the platformConclusionZurich Spain successfully replaced a market standard ECM product with a new flexible, highly available, and scalable ECM. This resulted in a 65% run cost reduction, improved performance, and enablement of AWS analytical services.In addition, Zurich Spain has taken advantage of many benefits that AWS brings to their customers. They’ve demonstrated that Thunder, the new internal framework developed using AWS technology, provides fast application development with secure and frequent deployments.TAGS:Architecture,ECM,InsuranceMiguel GuillotMiguel Guillot Garcia is a Solutions Architect at AWS Spain. Miguel is based in Barcelona, where he works with Financial Services customers to help them design, deploy, and scale applications to achieve their business goals. In his spare time, Miguel loves to spend time with his family and friends where you will probably find him cooking a BBQ or a delicious paella. Miguel is also a passionate about the NBA.Jesus BernalJesús Bernal is a Solutions Architect at Amazon Web Services. Jesús is based in Madrid, Spain, where he works helping startups achieve their business goals while building secure, highly available and scalable solutions while keeping go-to-market times to a minimum. Outside work, you can find him watching football, training for the next half-marathon or reading the latest drama best seller.
This post was cowritten with Oscar Gali, Head of Technology and Architecture for GI in Zurich, Spain About Zurich Spain Zurich Spain is part of Zurich Insurance Group (Zurich), known for its financial soundness and solvency. With more than 135 years of history and over 2,000 employees, it is a leading company in the Spanish […]
Miguel Guillot
Amazon DocumentDB Amazon Elastic Container Registry Amazon Elastic Kubernetes Service Architecture