Scale Up Language Detection with Amazon Comprehend and S3 Batch Operations
AWS Architecture BlogScale Up Language Detection with Amazon Comprehend and S3 Batch Operationsby Ameer Hakme and Jennifer Moran | on16 NOV 2021| inAdvanced (300),Amazon Comprehend,Amazon Simple Storage Service (S3),Architecture,AWS Lambda|Permalink|ShareOrganizations have been collecting text data for years. Text data can help you intelligently address a range of challenges, from customer experience to analytics. These mixed language, unstructured datasets can contain a wealth of information within business documents, emails, and webpages. If you’re able to process and interpret it, this information can provide insight that can help guide your business decisions.Amazon Comprehendis a natural language processing (NLP) service that extracts insights from text datasets. Amazon Comprehendasynchronous batch operationsprovides organizations with the ability to detect dominant languages from text documents stored inAmazon Simple Storage Service (S3)buckets. The asynchronous operations support a maximum document size of 1 MB for language detection. They can process up to one million documents per batch, for a total size of 5 GB.But what if your organization has millions, or even billions of documents stored in an S3 bucket waiting for language detection processing? What if your language detection process requires customization to let you organize your documents based on language? What if you need to create a search index that can help you quickly audit your text document repositories?In this blog post, we walk through a solution usingAmazon S3 Batch Operationsto initiate language detection jobs withAWS Lambdaand Amazon Comprehend.Real world language detection solution architectureIn our example, we have tens of millions of text objects stored in a single S3 bucket. These need to be processed to detect the dominant language. To create a language detection job, we must supply the S3 Batch Operations with a manifest file that lists all text objects. We can use anAmazon S3 Inventoryreport as an input to the manifest file to create S3 bucket object lists.One of the supported S3 Batch Operations is invoking an AWS Lambda function. The S3 Batch Operations job usesLambdaInvoketo run a Lambda function on every object listed in a manifest. Lambda jobs are subject to overall Lambdaconcurrency limitsfor the account and each Lambda invocation will have a defined runtime. Organizations can request a service quota increase if necessary. Lambda functions in a single AWS account and in one Region share the concurrency limit. You can set reserved capacity for Lambda functions to ensure that they can be invoked even when overall capacity has been exhausted.The Lambda function can be customized to take further actions based on the output received from Amazon Comprehend. The following diagram shows an architecture for language detection with S3 Batch Operations and Amazon Comprehend.Figure 1. Language detection with S3 Batch Operations and Amazon ComprehendHere is the architecture flow, as shown in Figure 1:S3 Batch Operations will pull the manifest file from the source S3 bucket.The S3 Batch Operations job will invoke the language detection Lambda function for each object listed in the manifest file. Lambda function code will perform a preliminary scan to check the file size, file extension, or any other requirements before callingAmazon Comprehend API. The Lambda function will then read the text object from S3 and then call the Amazon Comprehend API to detect the dominant language.The Language Detection API automatically identifies text written in over 100 languages. The API response contains the dominant language with a confidence score supporting the interpretation. An example API response would be: {‘LanguageCode’: ‘fr’, ‘Score’: 0.9888556003570557}. Once the Lambda function receives the API response, Lambda will return a message back to S3 Batch Operations with a result code.The Lambda function will then publish a message to anAmazon Simple Notification Service (SNS)topic.AnAmazon Simple Queue Service (SQS)queue subscribed to the SNS topic will receive the message with all required information related to each processed text object.The SQS queue will invoke a Lambda function to process the message.The Lambda function will move the targeted S3 object to a destination S3 bucket.S3 Batch Operations will generate a completion report and will store it in an S3 bucket. Thecompletion reportwill contain additional information for each task, including the object key name and version, status, error codes, and error descriptions.Leverage SNS fanout pattern for more complex use casesThis blog post describes the basic building blocks for the solution, but it can be extended for more complex use cases, as illustrated in Figure 2. Using an SNS fanout application integration pattern would enable many SQS queues to subscribe to the same SNS topic. These SQS queues would receive identical notifications for the processed text objects, and you could implement downstream services for additional evaluation. For example, you can store text object metadata in anAmazon DynamoDBtable. You can further analyze the number of processed text objects, dominant languages, object size, word count, and more.Your source S3 bucket may have objects being uploaded in real time in addition to the existing batch processes. In this case, you could process these objects in a new batch job, or process them individually during upload by using S3 event triggers and Lambda.Figure 2. Extending the solutionConclusionYou can implement a language detection job in a number of ways. All the Amazon Comprehend single document and synchronous API batch operations can be used for real-time analysis.Asynchronous batch operationscan analyze large documents and large collections of documents. However, by using S3 Batch Operations, you can scale language detection batch operations to billions of text objects stored in S3. This solution has the flexibility to add customized functionality. This may be useful for more complex jobs, or when you want to capture different data points from your S3 objects.For further reading:Using Lambda with Amazon S3 batch operationsManaging Tens to Billions of Objects at Scale with S3 Batch OperationsAmeer HakmeAmeer Hakme is an AWS Solutions Architect based in Pennsylvania. He works with Independent software vendors in the Northeast to help them design and build scalable and modern platforms on the AWS Cloud. In his spare time, he enjoys riding his motorcycle and spend time with his family.Jennifer MoranJennifer Moran is an AWS Solutions Architect based out of New York City. She has a diverse background having worked in many technical disciplines including Software Development, Agile Leadership and DevOps. She enjoys helping customers design creative solutions for technical challenges. Traveling and being outdoors with her family are some of her favorite pastimes.
Organizations have been collecting text data for years. Text data can help you intelligently address a range of challenges, from customer experience to analytics. These mixed language, unstructured datasets can contain a wealth of information within business documents, emails, and webpages. If you’re able to process and interpret it, this information can provide insight that […]
Ameer Hakme
Advanced (300) Amazon Comprehend Amazon Simple Storage Service (S3) Architecture AWS Lambda