Field Notes: How to Prepare Large Text Files for Processing with Amazon Translate and Amazon Comprehend
AWS Architecture BlogField Notes: How to Prepare Large Text Files for Processing with Amazon Translate and Amazon Comprehendby Veeresh Shringari and Randal Goomer | on27 SEP 2021| inAmazon Comprehend,Amazon Translate,Architecture,Technical How-to|Permalink|ShareBiopharmaceutical manufacturing is a highly regulated industry where deviation documents are used to optimize manufacturing processes. Deviation documents in biopharmaceutical manufacturing processes are geographically diverse, spanning multiple countries and languages. The document corpus is complex, with additional requirements for complete encryption. Therefore, to reduce downtime and increase process efficiency, it is critical to automate the ingestion and understanding of deviation documents. For this workflow, a large biopharma customer needed to translate and classify documents at their manufacturing site.The customer’s challenge included translation and classification of paragraph-sized text documents into statement types. First, the tokenizer previously used was failing for certain languages. Second, post-tokenization, big paragraphs were needed to be sliced into sizes smaller than 5,000 bytes to facilitate consumption intoAmazon TranslateandAmazon Comprehend. Because each sentence and paragraphs were of differing sizes, the customer needed to slice them so that each sentence and paragraph did not lose their context and meaning.This blog post describes a solution to tokenize text documents into appropriate-sized chunks for easy consumption by Amazon Translate and Amazon Comprehend.Overview of solutionThe solution is divided into the following steps. Text data coming from theAWS Glueoutput is transformed and stored inAmazon Simple Storage Service(Amazon S3) in a .txt file. This transformed data is passed into the sentence tokenizer with slicing and encryption usingAWS Key Management Service(AWS KMS). This data is now ready to be fed into Amazon Translate and Amazon Comprehend, and then to a Bidirectional Encoder Representations from Transformers (BERT) model for clustering. All of the models are developed and managed inAmazon SageMaker.PrerequisitesFor this walkthrough, you should have the following prerequisites:An AWS accountSageMakerAmazon S3Amazon ComprehendAmazon TranslateAmazon CloudWatchenabledThe architecture in Figure 1 shows a complete document classification and clustering workflow running the sentence tokenizer solution (step 4) as an input to Amazon Translate and Amazon Comprehend. The complete architecture also uses AWS Glue crawlers,Amazon Athena, Amazon S3 , AWS KMS, and SageMaker.Figure 1. Higher level architecture describing use of the tokenizer in the systemSolution stepsIngest the streaming data from the daily pharma supply chain incidents from theAWS Glue crawlersand Athena-based view tables.AWS Glueis used for ETL (extract, transform, and load), while Athena helps to analyze the data in Amazon S3 for its integrity.Ingest the streaming data into Amazon S3, which isAWS KMSencrypted. This limits any unauthorized access to the secured files, as required for the healthcare domain.Enable the CloudWatch logs. CloudWatch logs help to store, monitor, and access error messages logged by SageMaker.Open the SageMaker notebook using AWS console, and navigate to the integrated development environment (IDE) with Python notebook.Solution descriptionInitialize the Amazon S3 client, and enable the get_execution role.Figure 2. Code sample to initialize Amazon S3 Client execution rolesFigure 3 shows the code for tokenizing large paragraphs into sentences. This helps to feed a sentence of 5,000 byte chunks to Amazon Translate and Amazon Comprehend. Additionally, in the regulated environment, data at rest and in transition, is encrypted using AWS KMS (using S3 IO object) before chunking into 5,000-byte size files using last-in-first-out (LIFO) process.Figure 3. Code sample with file chunking function and AWS KMS encryptionFigure 4 shows the function for writing the file chunks to objects in Amazon S3, and objects are AWS KMS encrypted.Figure 4. Code sample for writing chunked 5,000-byte sized data to Amazon S3Code sampleThisexample code details the tokenizer and chunking toolwhich we subsequently run through SageMaker.Cleaning upTo avoid incurring future charges, delete the resources (like S3 objects) used for the practice files after you have completed implementation of the solution.ConclusionIn this blog post, we presented a solution which incorporates sentence-level tokenization with rules governing expected sentence size. The solution includes automation scripts to reduce bigger files into smaller chunked sizes of 5,000 bytes to facilitate Amazon Translate and Amazon Comprehend. The solution is effective for tokenizing and chunking complex environments with multi-language files. Furthermore, the solution uses file exchange security by using AWS KMS, as required by regulated industries.Field Notes provides hands-on technical guidance from AWS Solutions Architects, consultants, and technical account managers, based on their experiences in the field solving real-world business problems for customers.TAGS:Field NotesVeeresh ShringariVeeresh Shringari is a Senior Consultant AI/ML, with 20 years of experience in healthcare, industrial automation, and aerospace business. His expertise is in AI/ML, specifically deep learning.Randal GoomerRandal Goomer, PhD, is a Global Advisory in HealthCare and Life Sciences at AWS. He earned his AB and PhD from University of California, Berkeley, and Texas A&M University, respectively, and has extensive experience in AI/ML application across the healthcare and life sciences value chain.
Biopharmaceutical manufacturing is a highly regulated industry where deviation documents are used to optimize manufacturing processes. Deviation documents in biopharmaceutical manufacturing processes are geographically diverse, spanning multiple countries and languages. The document corpus is complex, with additional requirements for complete encryption. Therefore, to reduce downtime and increase process efficiency, it is critical to automate the […]
Veeresh Shringari
Amazon Comprehend Amazon Translate Architecture Technical How-to