Automate your Data Extraction for Oil Well Data with Amazon Textract
AWS Architecture BlogAutomate your Data Extraction for Oil Well Data with Amazon Textractby Ashutosh Pateriya, Ashutosh Dubey, Sudharsan Chinnappan, Thameem Syed, and Venkatesh Muthusami | on24 FEB 2022| inAmazon Comprehend,Amazon Simple Notification Service (SNS),Amazon Simple Queue Service (SQS),Amazon Textract,Architecture,AWS Lambda,Energy (Oil & Gas)|Permalink|ShareTraditionally, many businesses archive physical formats of their business documents. These can be invoices, sales memos, purchase orders, vendor-related documents, and inventory documents. As more and more businesses are moving towards digitizing their business processes, it is becoming challenging to effectively manage these documents and perform business analytics on them. For example, in the Oil and Gas (O&G) industry, companies have numerous documents that are generated through the exploration and production lifecycle of an oil well. These documents can provide many insights that can help inform business decisions.As documents are usually stored in a paper format, information retrieval can be time consuming and cumbersome. Even those available in a digital format may not have adequate metadata associated to efficiently perform search and build insights.In this post, you will learn how to build a text extraction solution usingAmazon Textractservice. This will automatically extract text and data from scanned documents and upload intoAmazon Simple Storage Service (S3). We will show you how to find insights and relationships in the extracted text usingAmazon Comprehend. This data is indexed and populated intoAmazon OpenSearch Serviceto search and visualize it in aKibanadashboard.Figure 1 illustrates a solution built with AWS, which extracts O&G well data information from PDF documents. This solution is serverless and built using AWS Managed Services. This will help you to decrease system maintenance overhead while making your solution scalable and reliable.Figure 1. Automated form data extraction architectureFollowing are the high-level steps:Upload an image file or PDF document to Amazon S3 for analysis. Amazon S3 is a durable document storage used for central document management.Amazon S3 event initiates theAWS Lambdafunction Fn-A. AWS Lambda has functional logic to call the Amazon Textract and Comprehend services and processing.AWS Lambda function Fn-A invokes Amazon Textract to extract text as key-value pairs from image or PDF. Amazon Textract automatically extracts data from the scanned documents.Amazon Textract sends the extracted keys from image/PDF toAmazon SNS.Amazon SNS notifiesAmazon SQSwhen text extraction is complete by sending the extracted keys to Amazon SQS.Amazon SQS initiates AWS Lambda function Fn-B with the extracted keys.AWS Lambda function Fn-B invokes Amazon Comprehend for the custom entity recognition. Comprehend uses custom-trained machine learning (ML) to find discrepancies in key names from Amazon Textract.The data is indexed and loaded into Amazon OpenSearch, which indexes and visualizes the data.Kibana processes the indexed data.User accesses Kibana to search documents.Steps illustrated with more detail:1. User uploads the document for analysis to Amazon S3.Uploaded document can be an image file or a PDF. Here we are using the S3 console fordocument upload. Figure 2 shows the sample file used for this demo.Figure 2. Sample input form2. Amazon S3 upload event initiates AWS Lambda function Fn-A.Refer to theAWS tutorialto learn about S3 Lambda configuration. ViewSample code for Lambda FunctionA.3. AWS Lambda function Fn-A invokes Amazon Textract.Amazon Textract uses artificial intelligence (AI) to read as a human would, by extracting text, layouts, tables, forms, and structured data with context and without configuration, training, or custom code.4. Amazon Textract starts processing the file as it is uploaded.This process takes few minutes since the file is a multipage document.5. Amazon SNS notifies Amazon Textract of completion.Amazon Textract processing works asynchronously, as we decouple our architecture using Amazon SQS. To configure Amazon SNS to send data to Amazon SQS:Create anSNS topic. ‘AmazonTextract-SNS’ is the SNS topic that we created for this demo.Then create anSQS queue. ‘AmazonTextract-SQS’ is the queue that we created for this demo.To receive messages published to a topic, you mustsubscribe an endpointto the topic. When you subscribe an endpoint to a topic, the endpoint begins to receive messages published to the associated topic. Figure 3 shows the SNS topic ‘AmazonTextract-SNS’ subscribed to Amazon SQS queue.Figure 3. Amazon SNS configurationFigure 4. Amazon SQS configuration6. Configure SQS queue to initiate the AWS Lambda function Fn-B.This should happen upon receiving extracted data via SNS topic. Refer to thisSQS tutorialto learn about SQS Lambda configuration. SeeSample code for Lambda FunctionB.7. AWS Lambda function Fn-B invokes Amazon Comprehend for the custom entity recognition.Figure 5. Lambda FunctionB configuration in Amazon ComprehendConfigure Amazon Comprehend to create a custom entity recognition (text-job2) for the entities. These can be API Number, Lease_Number, Water_Depth, Well_Number, and can use the model created in previous step (well_no, well#, well num). For instructions on labeling your data, see Developing NER models with Amazon SageMaker Ground Truth and Amazon Comprehend.Figure 6. Comprehend jobNow create an endpoint for the custom entity recognition for the Lambda function, to send the data to Amazon Comprehend service, as shown in Figure 7 and 8.Figure 7. Comprehend endpoint creationCopy the Amazon Comprehend endpoint ARN to include it in the Lambda function as an environment variable (see Figure 5).Figure 8. Comprehend endpoint created successfully8. Launch an Amazon OpenSearch domain.SeeCreating and managing Amazon OpenSearch Service domains.The data is indexed and populated into Amazon OpenSearch. The Amazon OpenSearch domain name is configured at Lambda FnB as an environment variable to push the extracted data to OpenSearch.9. Kibana processes the indexed data from Amazon OpenSearch.Amazon OpenSearch data is populated on Kibana, shown in Figure 9.Figure 9. Kibana dashboard showing Amazon OpenSearch data10. Access Kibana for document search.The selected fields can be viewed as a table using filters, see Figure 10.Figure 10. Kibana dashboard table view for selected fieldsYou can s­earch the LEASE_NUMBER = OCS-031, as shown in Figure 11.Figure 11. Kibana dashboard search on Lease NumberOR you can search all the information for the WATER_DEPTH = 60, see Figure 12.Figure 12. Kibana dashboard search on Water DepthCleanupShut down OpenSearch domainDelete the Comprehend endpointClear objects from S3 bucketConclusionData is growing at an enormous pace in all industries. As we have shown, you can build an ML-based text extraction solution to uncover the unstructured data from PDFs or images. You can derive intelligence from diverse data sources by incorporating a data extraction and optimization function. You can gain insights into the undiscovered data, by leveraging managed ML services, Amazon Textract, and Amazon Comprehend.The extracted data from PDFs or images is indexed and populated into Amazon OpenSearch. You can use Kibana to search and visualize the data. By implementing this solution, customers can reduce the costs of physical document storage, in addition to labor costs for manually identifying relevant information.This solution will drive decision-making efficiency. We discussed the oil and gas industry vertical as an example for this blog. But this solution can be applied to any industry that has physical/scanned documents such as legal documents, purchase receipts, inventory reports, invoices, and purchase orders.For further reading:Blog:Extracting custom entities from documents with Amazon Textract and Amazon ComprehendBlog:Extract entities from insurance documents using Amazon Comprehend named entity recognitionAshutosh PateriyaAshutosh Pateriya is a Solutions Architect with Amazon Web Services. He is passionate about Networking, Security and helping customers leverage the power of AWS cloud.Ashutosh DubeyAshutosh is a Global Technical leader and Solutions Architect at Amazon Web Services based out of New Jersey, USA. He has extensive experience specializing in the Data, Analytics, and Machine Learning field, and has helped Fortune 500 companies in their cloud journey to AWS.Sudharsan ChinnappanSudharsan Chinnappan is a Technology Analyst with Infosys, building serverless web applications using AWS services. He is an Alexa enthusiast who addresses business needs and improves user experience through Alexa Skills. He has implemented IoT solutions using AWS IoT services.Thameem SyedThameem is Technical lead working with Infosys, assisting clients on application migration and modernization using AWS. He is part of the Infosys Cloud Practice team, and has keen interest in the latest cloud trends. He specializes in using managed AWS containerized services, Python, Devops on AWS, AI, data analytics, and machine learning solutions. He is a Certified AWS Cloud Solution Architect Associate.Venkatesh MuthusamiVenkatesh is a Principal Consultant with Infosys, assisting customers with AWS platform builds, application migration, and modernization to the cloud. He is a member of the Infosys cloud practice team and leads various AWS initiatives. Venkatesh has a keen interest in industry and cloud trends.
Traditionally, many businesses archive physical formats of their business documents. These can be invoices, sales memos, purchase orders, vendor-related documents, and inventory documents. As more and more businesses are moving towards digitizing their business processes, it is becoming challenging to effectively manage these documents and perform business analytics on them. For example, in the Oil […]
Ashutosh Pateriya
Amazon Comprehend Amazon Simple Notification Service (SNS) Amazon Simple Queue Service (SQS) Amazon Textract Architecture AWS Lambda Energy (Oil & Gas)