Text analytics on AWS: implementing a data lake architecture with OpenSearch
AWS Architecture BlogText analytics on AWS: implementing a data lake architecture with OpenSearchby Francisco Losada | on20 JAN 2023| inAmazon DynamoDB,Amazon EventBridge,Amazon OpenSearch Service,Architecture,AWS Glue,AWS Lambda|Permalink|ShareText data is a common type of unstructured data found in analytics. It is often stored without a predefined format and can be hard to obtain and process.For example, web pages contain text data that data analysts collect through web scraping and pre-process using lowercasing, stemming, and lemmatization. After pre-processing, the cleaned text is analyzed by data scientists and analysts to extract relevant insights.This blog post covers how to effectively handle text data using a data lake architecture on Amazon Web Services (AWS). We explain how data teams can independently extract insights from text documents using OpenSearch as the central search and analytics service. We also discuss how to index and update text data inOpenSearchand evolve the architecture towards automation.Architecture overviewThis architecture outlines the use of AWS services to create an end-to-end text analytics solution, starting from the data collection and ingestion up to the data consumption in OpenSearch (Figure 1).Figure 1. Data lake architecture with OpenSearchCollect data from various sources, such as SaaS applications, edge devices, logs, streaming media, and social networks.Use tools likeAWS Database Migration Service (AWS DMS),AWS DataSync,Amazon Kinesis,Amazon Managed Streaming for Apache Kafka (Amazon MSK),AWS IoT Core, andAmazon AppFlowto ingest the data into the AWS data lake, depending on the data source type.Store the ingested data in the raw zone of theAmazon Simple Storage Service (Amazon S3)data lake—a temporary area where data is kept in its original form.Validate, clean, normalize, transform, and enrich the data through a series of pre-processing steps usingAWS GlueorAmazon EMR.Place the data that is ready to be indexed in the indexing zone.Use AWS Lambda to index the documents into OpenSearch and store them back in the data lake with a unique identifier.Use the clean zone as the source of truth for teams to consume the data and calculate additional metrics.Develop, train, and generate new metrics using machine learning (ML) models withAmazon SageMakeror artificial intelligence (AI) services likeAmazon Comprehend.Store the new metrics in the enriching zone along with the identifier of the OpenSearch document.Use the identifier column from the initial indexing phase to identify the correct documents and update them in OpenSearch with the newly calculated metrics using AWS Lambda.Use OpenSearch to search through the documents and visualize them with metrics using OpenSearch Dashboards.ConsiderationsData lake orchestration among teamsThis architecture allows data teams to work independently on text documents at different stages of their lifecycles. The data engineering team manages the raw and indexing zones, who also handle data ingestion and preprocessing for indexing in OpenSearch.The cleaned data is stored in the clean zone, where data analysts and data scientists generate insights and calculate new metrics. These metrics are stored in the enrich zone and indexed as new fields in the OpenSearch documents by the data engineering team (Figure 2).Figure 2. Data lake orchestration among teamsLet’s explore an example. Consider a company that periodically retrieves blog site comments and performs sentiment analysis using Amazon Comprehend. In this case:The comments are ingested into the raw zone of the data lake.The data engineering team processes the comments and stores them in the indexing zone.A Lambda function indexes the comments into OpenSearch, enriches the comments with the OpenSearch document ID, and saves it in the clean zone.The data science team consumes the comments and performs sentiment analysis using Amazon Comprehend.The sentiment analysis metrics are stored in the metrics zone of the data lake. A second Lambda function updates the comments in OpenSearch with the new metrics.If the raw data does not require any preprocessing steps, the indexing and clean zones can be combined. You can explore this specific example, along with code implementation, in theAWS samples repository.Schema evolutionAs your data progresses through data lake stages, the schema changes and gets enriched accordingly. Continuing with our previous example, Figure 3 explains how the schema evolves.Figure 3. Schema evolution through the data lake stagesIn the raw zone, there is a raw text field received directly from the ingestion phase. It’s best practice to keep a raw version of the data as a backup, or in case the processing steps need to be repeated later.In the indexing zone, the clean text field replaces the raw text field after being processed.In the clean zone, we add a new ID field that is generated during indexing and identifies the OpenSearch document of the text field.In the enrich zone, the ID field is required. Other fields with metric names are optional and represent new metrics calculated by other teams that will be added to OpenSearch.Consumption layer with OpenSearchIn OpenSearch, data is organized into indices, which can be thought of as tables in a relational database. Each index consists of documents—similar to table rows—and multiple fields, similar to table columns. You can add documents to an index by indexing and updating them using various client APIs for popular programming languages.Now, let’s explore how our architecture integrates with OpenSearch in the indexing and updating stage.Indexing and updating documents using PythonThe index document API operation allows you to index a document with a custom ID, or assigns one if none is provided. To speed up indexing, we can use the bulk index API to index multiple documents in one call.We need to store the IDs back from the index operation to later identify the documents we’ll update with new metrics. Let’s explore two ways of doing this:Use the requests library to call the REST Bulk Index API (preferred):the response returns the auto-generated IDs we need.Use thePython Low-Level Client for OpenSearch:The IDs are not returned and need to be pre-assigned to later store them. We can use an atomic counter inAmazon DynamoDBto do so. This allows multiple Lambda functions to index documents in parallel without ID collisions.As in Figure 4, the Lambda function:Increases the atomic counter by the number of documents that will index into OpenSearch.Gets the value of the counter back from the API call.Indexes the documents using the range that goes between[current counter value, current counter value – number of documents].Figure 4. Storing the IDs back from the bulk index operation using the Python Low-Level Client for OpenSearchData flow automationAs architectures evolve towards automation, the data flow between data lake stages becomes event-driven. Following our previous example, we can automate the processing steps of the data when moving from the raw to the indexing zone (Figure 5).Figure 5. Event-driven automation for data flowWithAmazon EventBridgeandAWS Step Functions, we can automatically trigger our pre-processingAWS Gluejobs so our data gets pre-processed without manual intervention.The same approach can be applied to the other data lake stages to achieve a fully automated architecture. Explore this implementation for anautomated language use case.ConclusionIn this blog post, we covered designing an architecture to effectively handle text data using a data lake on AWS. We explained how different data teams can work independently to extract insights from text documents at different lifecycle stages using OpenSearch as the search and analytics service.TAGS:Architecture,aws solutions,data lakesFrancisco LosadaFrancisco Losada is an Analytics Specialist Solutions Architect based out of Madrid, Spain. He works with customers across EMEA to architect, implement and evolve analytics solutions at AWS. He advocates for OpenSearch, the open source search and analytics suite, and supports the community by sharing code samples, writing content and speaking at conferences. In his spare time, Francisco enjoys playing tennis and running.
Text data is a common type of unstructured data found in analytics. It is often stored without a predefined format and can be hard to obtain and process. For example, web pages contain text data that data analysts collect through web scraping and pre-process using lowercasing, stemming, and lemmatization. After pre-processing, the cleaned text is […]
Francisco Losada
Amazon DynamoDB Amazon EventBridge Amazon OpenSearch Service Architecture AWS Glue AWS Lambda