Content Repository for Unstructured Data with Multilingual Semantic Search: Part 2
AWS Architecture BlogContent Repository for Unstructured Data with Multilingual Semantic Search: Part 2by Patrik Nagel and Sid Singh | on26 JUL 2023| inAmazon OpenSearch Service,Amazon SageMaker,Amazon Simple Storage Service (S3),Architecture,AWS Lambda,Technical How-to|Permalink|Comments|ShareLeveraging vast unstructured data poses challenges, particularly for global businesses needing cross-language data search. InPart 1of this blog series, we built the architectural foundation for the content repository. The key component of Part 1 was the dynamic access control-based logic with a web UI to upload documents.In Part 2, we extend the content repository with multilingual semantic search capabilities while maintaining the access control logic from Part 1. This allows users to ingest documents in content repository across multiple languages and then run search queries to get reference to semantically similar documents.Solution overviewBuilding on the architectural foundation from Part 1, we introduce four new building blocks to extend the search functionality.Optical character recognition (OCR) workflow:To automatically identify, understand, and extract text from ingested documents, we useAmazon Textractand a sample review dataset of.pngformat documents (Figure 1). We use Amazon Textract synchronous application programming interfaces (APIs) to capture key-value pairs for thereviewidandreviewBodyattributes. Based on your specific requirements, you can choose to capture either the complete extracted text or parts the text.Figure 1. Sample document for ingestionEmbedding generation:To capture the semantic relationship between the text, we use a machine learning (ML) model that maps words and sentences to high-dimensional vector embeddings. You can useAmazon SageMaker, a fully-managed ML service, to build, train, and deploy your ML models to production-ready hosted environments. You can also deploy ready-to-use pre-trained models from multiple avenues such asSageMaker JumpStart. For this blog post, we use the open-source pre-traineduniversal-sentence-encoder-multilingual modelfrom TensorFlow Hub. The model inference endpoint deployed to a SageMaker endpoint generates embeddings for the document text and the search query. Figure 2 is an example of n-dimensional vector that is generated as the output of thereviewBodyattribute text provided to the embeddings model.Figure 2. Sample embedding representation of the value ofreviewBodyEmbedding ingestion:To make the embeddings searchable for the content repository users, you can use thek-Nearest Neighbor(k-NN) search feature ofAmazon OpenSearch Service. The OpenSearch k-NN plugin providesdifferent methods. For this blog post, we use theApproximate k-NNsearch approach, based on theHierarchical Navigable Small World(HNSW) algorithm. HNSW uses a hierarchical set of proximity graphs in multiple layers to improve performance when searching large datasets to find the “nearest neighbors” for the search query text embeddings.Semantic search:We make the search service accessible as an additional backend logic onAmazon API Gateway. Authenticated content repository users send their search query using the frontend to receive the matching documents. The solution maintains end-to-end access control logic by using the user’s enrichedAmazon Cognitoprovided identity (ID) token claim with thedepartmentattribute to compare it with the ingested documents.Technical architectureThe technical architecture includes two parts:Implementing multilingual semantic search functionality:Describes the processing workflow for the document that the user uploads; makes the document searchable.Running input search query:Covers the search workflow for the input query; finds and returns the nearest neighbors of the input text query to the user.Part 1. Implementing multilingual semantic search functionalityOur previous blog post discussed blocks A through D (Figure 3), including user authentication, ID token enrichment,Amazon Simple Storage Service(Amazon S3) object tags for dynamic access control, and document upload to the source S3 bucket. In the following section, we cover blocks E through H. The overall workflow describes how an unstructured document is ingested in the content repository, run through the backend OCR and embeddings generation process and finally the resulting vector embedding are stored in OpenSearch service.Figure 3. Technical architecture for implementing multilingual semantic search functionalityThe OCR workflow extracts text from your uploaded documents.The source S3 bucket sends an event notification toAmazon Simple Queue Service(Amazon SQS).The document transformationAWS Lambdafunction subscribed to the Amazon SQS queue invokes an Amazon Textract API call to extract the text.The document transformation Lambda function makes aninference requestto the encoder model hosted on SageMaker. In this example, the Lambda function submits thereviewBodyattribute to the encoder model to generate the embedding.The document transformation Lambda function writes an output file in the transformed S3 bucket. The text file consists of:ThereviewidandreviewBodyattributes extracted from Step 1An additionalreviewBody_embeddingsattribute from Step 2Note:The workflow tags the output file with the same S3 object tags as the source document for downstream access control.The transformed S3 bucket sends an event notification to invoke the indexing Lambda function.The indexing Lambda function reads the text file content. Then indexing Lambda function makes an OpenSearch index API call along with source document tag as one of the indexing attributes for access control.Part 2. Running user-initiated search queryNext, we describe how the user’s request produces query results (Figure 4).Figure 4. Search query lifecycleThe user enters a search string in the web UI to retrieve relevant documents.Based on the active sign-in session, the UI passes the user’s ID token to the search endpoint of the API Gateway.The API Gateway uses Amazon Cognito integration to authorize the search API request.Once validated, the search API endpoint request invokes the search document Lambda function.The search document function sends the search query string as the inference request to the encoder model to receive the embedding as the inference response.The search document function uses the embedding response to build an OpenSearch k-NN search query. The HNSW algorithm is configured with theLucene engine and its filter optionto maintain the access control logic based on the customdepartmentclaim from the user’s ID token. The OpenSearch query returns the following to the query embeddings:Top three Approximate k-NNOther attributes, such asreviewidandreviewBodyThe workflow sends the relevant query result attributes back to the UI.PrerequisitesYou must have the following prerequisites for this solution:An AWS account. Sign up tocreate and activate one.The following software installed on your development machine, or use anAWS Cloud9environment:AWS Command Line Interface(AWS CLI);configure it to point to your AWS accountTypeScript;use a package manager, such asnpmAWS Cloud Development Kit(AWS CDK)Docker; ensure it’s runningAppropriateAWS credentialsfor interacting with resources in your AWS account.WalkthroughSetupThe following steps deploy two AWS CDK stacks into your AWS account:content-repo-search-stack(blog-content-repo-search-stack.ts) creates the environment detailed in Figure 3, except for the SageMaker endpoint, which you create in a spearate step.demo-data-stack(userpool-demo-data-stack.ts) deploys sample users, groups, and role mappings.To continue setup, use the following commands:Clone the project Git repository:git clone https://github.com/aws-samples/content-repository-with-multilingual-search content-repositoryInstall the necessary dependencies:cd content-repository/backend-cdk 
npm installConfigure environment variables:export CDK_DEFAULT_ACCOUNT=$(aws sts get-caller-identity --query 'Account' --output text)
export CDK_DEFAULT_REGION=$(aws configure get region)Bootstrap your account for AWS CDK usage:cdk bootstrap aws://$CDK_DEFAULT_ACCOUNT/$CDK_DEFAULT_REGIONDeploy the code to your AWS account:cdk deploy --allThe complete stack set-up may take up to 20 minutes.Creation of SageMaker endpointFollow below steps to create the SageMaker endpoint in the same AWS Region where you deployed the AWS CDK stack.Sign in to theSageMaker console.In the navigation menu, selectNotebook, thenNotebookinstances.ChooseCreate notebook instance.Under theNotebook instance settings, entercontent-repo-notebookas the notebook instance name, and leave other defaults as-is.Under thePermissions and encryptionsection (Figure 5), you need to set the IAM role section to the role with the prefixcontent-repo-search-stack. In case you don’t see this role automatically populated, select it from the drop-down. Leave the rest of the defaults, and chooseCreate notebook instance.Figure 5. Notebook permissionsThe notebook creation status changes toPendingbefore it’s available for use within 3-4 minutes.Once the notebook is in theAvailablestatus, chooseOpen Jupyter.Choose theUploadbutton and upload thecreate-sagemaker-endpoint.ipynbfile in thebackend-cdkfolder of the root of the blog repository.Open thecreate-sagemaker-endpoint.ipynbnotebook. Select the optionRun Allfrom theCellmenu (Figure 6). This might take up to 10 minutes.Figure 6. Runcreate-sagemaker-endpointnotebook cellsAfter all the cells have successfully run, verify that theAWS Systems Managerparametersagemaker-endpointis updated with the value of the SageMaker endpoint name. An example of value as the output of the cell is in Figure 7. In case you don’t see the output, check if the preceding steps were run correctly.Figure 7. SSM parameter updated with SageMaker endpointVerify in the SageMaker console that the inference endpoint with the prefixtensorflow-inferencehas been deployed and is set to statusInService.Upload sample data to the content repository:Update theS3_BUCKET_NAMEvariable in theupload_documents_to_S3.shscript in the root folder of the blog repository with thes3SourceBucketNamefrom the AWS CDK output of thecontent-repo-search-stack.Runupload_documents_to_S3.sh scriptto upload 150 sample documents to the content repository. This takes 5-6 minutes. During this process, the uploaded document triggers the workflow described in theImplementing multilingual semantic search functionality.Using the search serviceAt this stage, you have deployed all the building blocks for the content repository in your AWS account. Next, as part of the upload sample data to the content repository, you pushed a limited corpus of 150 sample documents (.png format). Each document is in one of the four different languages – English, German, Spanish and French. With the added multilingual search capability, you can query in one language and receive semantically similar results across different languages while maintaining the access control logic.Access the frontend application:Copy theamplifyHostedAppUrlvalue of the AWS CDK output from thecontent-repo-search-stackshown in the terminal.Enter the URL in your web browser to access the frontend application.A temporary page displays until the automated build and deployment of the React application completes after 4-5 minutes.Sign into the application:The content repository provides two demo users with credentials as part of thedemo-data-stackin the AWS CDK output. Copy the password from the terminal associated with thesales-user, which belongs to thesalesdepartment.Follow the prompts from the React webpage to sign in with the sales-user and change the temporary password.Enter search queries and verify results. The search action invokes the workflow described inRunning input search query. For example:Enterworks wellas the search query. Note the multilingual output and the semantically similar results (Figure 8).Figure 8. Positive sentiment multilingual search result for the sales-userEnterbad qualityas the search query. Note the multilingual output and the semantically similar results (Figure 9).Figure 9. Negative sentiment multi-lingual search result for the sales-userSign out as thesales-userwith theLog Outbutton on the webpage.Sign in using themarketing-usercredentials to verify access control:Follow the sign in procedure in step 2 but with themarketing-user.This time withworks wellas search query, you find different output. This is because the access control only allowsmarketing-userto search for the documents that belong to themarketingdepartment (Figure 10).Figure 10. Positive sentiment multilingual search result for the marketing-userCleanupIn thebackend-cdksubdirectory of the cloned repository, delete the deployed resources:cdk destroy --all.Additionally, you need to access the Amazon SageMaker console todelete the SageMaker endpoint and notebook instancecreated as part of theWalkthroughsetup section.ConclusionIn this blog, we enriched the content repository with multi-lingual semantic search features while maintaining the access control fundamentals that we implemented inPart 1. The building blocks of the semantic search for unstructured documents—Amazon Textract, Amazon SageMaker, and Amazon OpenSearch Service—set a foundation for you to customize and enhance the search capabilities for your specific use case. For example, you can leverage the fast developments inLarge Language Models (LLM) to enhance the semantic search experience. You can replace the encoder model with an LLM capable of generating multilingual embeddings while still maintaining the OpenSearch service to store and index data and perform vector search.Patrik NagelPatrik is a Principal Solutions Architect at AWS helping Global Financial Services customers to innovate and transform through modern software and practices. He has over 15 years of industry experience ranging from small startups to large enterprises covering a wide range of technologies. Outside of work, you find Patrik on the ice rink as an avid hockey player.Sid SinghSid is a Solutions Architect with Amazon Web Services. He works with the global financial services customers and has more than 10 years of industry experience covering a wide range of technologies. Outside of work, he loves traveling, is an avid foodie, and also a Bavarian beer enthusiast.CommentsView Comments
Leveraging vast unstructured data poses challenges, particularly for global businesses needing cross-language data search. In Part 1 of this blog series, we built the architectural foundation for the content repository. The key component of Part 1 was the dynamic access control-based logic with a web UI to upload documents. In Part 2, we extend the […]
Patrik Nagel
Amazon OpenSearch Service Amazon SageMaker Amazon Simple Storage Service (S3) Architecture AWS Lambda Technical How-to