Content Repository for Unstructured Data with Multilingual Semantic Search: Part 1
AWS Architecture BlogContent Repository for Unstructured Data with Multilingual Semantic Search: Part 1by Patrik Nagel and Sid Singh | on01 MAR 2023| inAmazon API Gateway,Amazon Cognito,Architecture,AWS Identity and Access Management (IAM),Technical How-to|Permalink|ShareUnstructured data can make up to 80 percent of data in the day-to-day business of financial organizations. For example, these organizations typically store and read PDFs and images for claim processing, underwriting, and know your customer (KYC). Organizations need to make this ingested data accessible and searchable across different entities while logically separating data access according to role requirements.In this two-part series, we use AWS services to build an end-to-end content repository for storing and processing unstructured data with the following features:Dynamic access control-based logic over unstructured dataMultilingual semantic search capabilitiesIn part 1, we build the architectural foundation for the content repository, including the resource access control logic and a web UI to upload and list documents.Solution overviewThe content repository includes four building blocks:Frontend and interaction: For this function, we useAWS Amplify, which is a set of purpose-built tools and features to help frontend web and mobile developers quickly build full-stack applications on AWS. TheReactapplication uses theAWS Amplify authenticationfeature to quickly set up a complete authentication flow integrated intoAmazon Cognito. Amplify also hosts the frontend application.Authentication and authorization: Implementing dynamic resource access control with a combination of roles and attributes is fundamental to your content repository security. Amazon Cognito provides a managed, scalable user directory, user sign-up and sign-in flows, and federation capabilities through third-party identity providers. We useAmazon Cognito user poolsas the source of user identity for the content repository. You can work with user pool groups to represent different types of user collection, and you can manage their permissions using a group-associatedAWS Identity and Access Management (IAM)role.Users authenticate against the Amazon Cognito user pool. The web app will exchange the user pool tokens for AWS credentials through an Amazon Cognito identity pool in the content repository. You can complement the IAM role-based authorization model by mapping your relevantattributesto principal tags that will be evaluated as part of IAM permission policies. This allows a dynamic and flexible authorization strategy. For use cases that need federation with third-party identity providers, you can base your user collection on existing user group attributes, such as Active Directory group membership.Backend and business logic: Authenticated users are redirected to theAmazon API Gateway. API Gateway provides managed publishing for application programming interfaces (APIs) that act as the repository’s “front door.” API Gateway also interacts with the repository’s backend through RESTful APIs. This makes the business logic of the content repository extensible for future use cases, such as transcription and translation. We useAWS Lambdaas a serverless, event-driven compute service to run specific business logic code, such as uploading a document to the content repository.Content storage:Amazon Simple Storage Service(Amazon S3) provides virtually unlimited scalability and high durability. With Amazon S3, you can cost-effectively store unstructured documents in their native formats and make it accessible in a secure and scalable way. Enriching the uploaded documents with tags simplifies data governance with fine-grained access control.Technical architectureThe technical architecture of the content repository with these four components can be found in Figure 1.Figure 1. Technical architecture of the content repositoryLet’s explore the architecture step by step.The frontend uses theAmplify JS libraryto add the authentication UI component to your React app, allowing authenticated users to sign in.Once the user provides their sign-in credentials, they are redirected to Amazon Cognito user pools to be authenticated.Once the authentication is successful, Amazon Cognito invokes apre-token generation Lambda function. This function customizes the identity (ID) token with a new claim calleddepartment. This new claim is the Amazon Cognito group name from thecognito:preferred_roleclaim.Amazon Cognito returns the identity, access, and refresh token in JSON format to the frontend.The Amplify client library stores the tokens and handles refreshes using the refresh token while the React frontend application calls the API Gateway with the ID token.Note:Usually, you would use the access token to grant access to authorized resources. For this architecture, we use the ID token because we have enriched it with the custom claim during step 3.API Gateway uses its native integration with Amazon Cognito and validates the ID token’s signature and expiration using Amazon Cognito user pool authorizer. For more complex authorization scenarios, you can useAPI Gateway Lambda authorizerwith theAWS JSON Web Token (JWT) Verifylibrary for verifying JWTs signed by Amazon Cognito.After successful validation, API Gateway passes the ID token to the backend Lambda function, which can verify and authorize upon it for access control.Upon document upload action, the backend Lambda function calls the Amazon Cognito identity pool to exchange the ID token for the temporary AWS credentials associated with thecognito:preferred_roleclaim.The document upload Lambda function returns a pre-signed URL with the customdepartmentclaim in the Amazon S3 path prefix as well as the object tag. TheAmazon S3 pre-signed URLis used for the document upload from the frontend application directly to Amazon S3.Upon document list action, similar to step 8, the backend Lambda function exchanges the ID token for the temporary AWS credentials. The Lambda function returns only the documents based on the user’s preferred group and associated customdepartmentclaim.PrerequisitesYou must have the following prerequisites for this solution:An AWS account or sign up tocreate and activate one.The following software installed on your development machine, or use anAWS Cloud9environment:Install theAWS Command Line Interface(AWS CLI) andconfigure it to point to your AWS account.InstallTypeScriptand use a package manager such asnpm.Install theAWS Cloud Development Kit(AWS CDK).AppropriateAWS credentialsfor interacting with resources in your AWS account.WalkthroughSetupThe following steps will deploy two AWS CDK stacks into your AWS account:content-repo-stack(blog-content-repo-stack.ts) creates the environment detailed in Figure 1.demo-data-stack(userpool-demo-data-stack.ts) deploys sample users, groups, and role mappings.To continue setup, use the following commands:Clone the project git repository:git clone https://github.com/aws-samples/content-repository-with-dynamic-access-control content-repositoryInstall the necessary dependencies:cd content-repository/backend-cdk
npm installConfigure environment variables:export CDK_DEFAULT_ACCOUNT=$(aws sts get-caller-identity --query 'Account' --output text)
export CDK_DEFAULT_REGION=$(aws configure get region)Bootstrap your account for AWS CDK usage:cdk bootstrap aws://$CDK_DEFAULT_ACCOUNT/$CDK_DEFAULT_REGIONDeploy the code to your AWS account:cdk deploy --allUsing the repositoryOnce you deploy the CDK stacks in your AWS account, follow these steps:1. Access the frontend application:a. Copy theamplifyHostedAppUrlvalue shown in the AWS CDK output from the content-repo-stack.b. Use the URL with your web browser to access the frontend application.c. A temporary page displays until the automated build and deployment of the React application completes after 4-5 minutes.2. Application sign-in and role-based access control (RBAC):a. The React webpage prompts you to sign in and then change the temporary password.b. The content repository provides two demo users with credentials as part of the demo-data-stack in the AWS CDK output. In this walkthrough, we use thesales-useruser, which belongs to thesalesdepartment group to validate RBAC.3. Upload a document to the content repository:a. Authenticate assales-user.b. Selectuploadto upload your first document to the content repository.c. The repository provides sample documents in theassetssub-folder.4. List your uploaded document:a. Select list to show the uploaded sales content.b. To verify the dynamic access control, repeat steps 2 and 3 for themarketing-useruser, which belongs to themarketingdepartment group.c. Sign-in to the AWS Management Console and navigate to the Amazon S3 bucket with the prefixcontent-repo-stack-s3sourcebucketto confirm that all the uploaded content exists.Implementation notesFrontend deployment and cross-origin accessThe content-repo-stack contains anAwsCustomResourceconstruct. This construct uses the Amplify API to start the release job of the Amplify hosted frontend application. The preBuild step of the Amplify application build specification dynamically configures its backend for the Amazon Cognito-based authentication. The required Amazon Cognito configuration parameters are retrieved from theAWS Systems Manager Parameter Storeduring build time. Similarly, the Amplify application postBuild step updates the Amazon S3cross-origin resource sharing(CORS) rule for the Amazon S3 bucket to only allow cross-origin access from the Amplify-hosted URL of the frontend application.Application sign-in and access controlThe Amazon Cognito identity pool configuration is set toChoose role from tokenfor authenticated users, as in Figure 2. This setup permits authenticated users topass the roles in the ID tokenthat the Amazon Cognito user pool assigned. Backend Lambda functions use the roles that appear in thecognito:rolesandcognito:preferred_roleclaims in the ID token for RBAC.Figure 2. Amazon Cognito identity pool configuration – using tokens to assign roles to authenticated usersIn theattributes for access controlsection, we configured a custom mapping from the augmented department token claim to a tag key, as in Figure 3. The backend logic uses the tag key to match thePrincipalTagcondition in IAM policies to control access to AWS resources.Figure 3. Amazon Cognito identity pool configuration – custom mapping from claim names to tag keysDocument uploadThepresigned_url.pyLambda function generates a pre-signed Amazon S3 URL using thedepartmenttoken claim as the key. This function automatically organizes the uploaded document into a logical structure in the Amazon S3 source bucket. Accordingly, thecognito:preferred_roleused for the Amazon S3 client credentials in the Lambda function has a permission policy using thePrincipalTagdepartment to dynamically limit access to the Amazon S3 key, as in Figure 4.Figure 4. Permission policy using PrincipalTag to upload documents to Amazon S3Document listingThe list functionality only shows the uploaded content belonging to the preferred group of authenticated Amazon Cognito user pool user. To only list the files that a specific user (for example,sales-user) has access to, use thePrincipalTag s3:prefixcondition, as in Figure 5.Figure 5. Permission policy using s3:prefix condition with session tags to list documentsCleanupIn thebackend-cdksubdirectory, delete the deployed resources:cdk destroy --allConclusionIn this blog, we demonstrated how to build a content repository with an easy-to-use web application for unstructured data that ingests documents while maintaining dynamic access control for users within departments. These steps provide a foundation to build your own content repository to store and process documents. As next steps, based on your organization’s security requirements, you can implement more complex access control use cases by balancing IAM role and principal tags. For example, you can use Amazon Cognito user pool custom attributes for additional dimensions such as document “clearance” with optional modification in the pre-token generation Lambda.In the next part of this blog series, we will enrich the content repository with multi-lingual semantic search features while maintaining the access control fundamentals we’ve already implemented. For additional information on how you can build a solution to search for information across multiple scanned documents, PDFs, and images with compliance capabilities, please refer ourDocument Understanding Solution from AWS Solutions Library.Patrik NagelPatrik is a Principal Solutions Architect at AWS helping Global Financial Services customers to innovate and transform through modern software and practices. He has over 15 years of industry experience ranging from small startups to large enterprises covering a wide range of technologies. Outside of work, you find Patrik on the ice rink as an avid hockey player.Sid SinghSid is a Solutions Architect with Amazon Web Services. He works with the global financial services customers and has more than 10 years of industry experience covering a wide range of technologies. Outside of work, he loves traveling, is an avid foodie, and also a Bavarian beer enthusiast.
Unstructured data can make up to 80 percent of data in the day-to-day business of financial organizations. For example, these organizations typically store and read PDFs and images for claim processing, underwriting, and know your customer (KYC). Organizations need to make this ingested data accessible and searchable across different entities while logically separating data access […]
Patrik Nagel
Amazon API Gateway Amazon Cognito Architecture AWS Identity and Access Management (IAM) Technical How-to