Building event-driven architectures with IoT sensor data
AWS Architecture BlogBuilding event-driven architectures with IoT sensor databy Raghavarao Sodabathina, Jun Shan, and Moises Ulloa | on21 NOV 2022| inAmazon Kinesis,Amazon SageMaker,Architecture,AWS IoT Core,AWS IoT Greengrass,AWS Lambda|Permalink|ShareThe Internet of Things (IoT) brings sensors, cloud computing, analytics, and people together to improve productivity and efficiency. It empowers customers with the intelligence they need to build new services and business models, improve products and services over time, understand their customers’ needs to provide better services, and improve customer experiences. Business operations become more efficient by making intelligent decisions more quickly and over time develop a data-driven discipline leading to revenue growth and greater operational efficiency.In this post, we showcase how to build an event-driven architecture by usingAWS IoT servicesandAWS purpose-built data services. We also discuss key considerations and best practices while building event-driven application architectures with IoT sensor data.Deriving insights from IoT sensor dataOrganizations create value by making decisions from their IoT sensor data in near real time. Some common use cases and solutions that fit under event-driven architecture using IoT sensor data include:Medical device data collection for personalized patient health monitoring, adverse event prediction, and avoidance.Industrial IoT use casesto monitor equipment quality and determine actions like adjusting machine settings, using different sources of raw materials, or performing additional worker training to improve the quality of the factory output.Connected vehicle use cases, such as voice interaction, navigation, location-based services, remote vehicle diagnostics, predictive maintenance, media streaming, and vehicle safety, that are based on in-vehicle computing and near real-time predictive analytics in the cloud.Sustainability and waste reduction solutions, which provide access to dashboards, monitoring systems, data collection, and summarization tools that use machine learning (ML) algorithms to meet sustainability goals. Meeting sustainability goals is paramount for customers in the travel and hospitality industries.Event-driven reference architecture with IoT sensor dataFigure 1 illustrates how to architect an event-driven architecture with IoT sensor data for near real-time predictive analytics and recommendations.Figure 1. Building event-driven architecture with IoT sensor dataArchitecture flow:Data originates in IoT devices such as medical devices, car sensors, industrial IoT sensors.This telemetry data is collected usingAWS IoT Greengrass, an open-source IoT edge runtime and cloud service that helps your devices collect and analyze data closer to where the data is generated.When an event arrives, AWS IoT Greengrass reacts autonomously to local events, filters and aggregates device data, then communicates securely with the cloud and other local devices in your network to send the data.Event data is ingested into the cloud using edge-to-cloud interface services such asAWS IoT Core, a managed cloud platform that connects, manages, and scales devices easily and securely.AWS IoT Core interacts with cloud applications and other devices. You can also useAWS IoT SiteWise, a managed service that helps you collect, model, analyze, and visualize data from industrial equipment at scale.AWS IoT Core can directly stream ingested data intoAmazon Kinesis Data Streams. The ingested data gets transformed and analyzed in near real time usingAmazon Kinesis Data Analyticswith Apache Flink and Apache Beam frameworks.Stream data can further be enriched using lookup data hosted in a data warehouse such as Amazon Redshift. Amazon Kinesis Data Analytics can persist SQL results to Amazon Redshift after the customer’s integration and stream aggregation (for example, one minute or five minutes).The results in Amazon Redshift can be used for further downstream business intelligence (BI) reporting services, such asAmazon QuickSight.Amazon Kinesis Data Analytics can also write to an AWS Lambda function, which can invokeAmazon SageMaker models. Amazon SageMaker is a the most complete, end-to-end service for machine learning.Once the ML model is trained and deployed in SageMaker, inferences are invoked in a micro batch usingAWS Lambda. Inferenced data is sent toAmazon OpenSearch Serviceto create personalized monitoring dashboards usingAmazon OpenSearch Service dashboards.The transformed IoT sensor data can be stored inAmazon DynamoDB. Customers can useAWS AppSyncto provide near real-time data queries to API services for downstream applications. These enterprise applications can be mobile apps or business applications to track and monitor the IoT sensor data in near real-time.Amazon Kinesis Data Analytics can write to anAmazon Kinesis Data Firehosestream, which is a fully managed service for delivering near real-time streaming data to destinations like Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon OpenSearch Service, Splunk, and any custom HTTP endpoints or endpoints owned by supported third-party service providers, including Datadog, Dynatrace, LogicMonitor, MongoDB, New Relic, and Sumo Logic.In this example, data from Amazon Kinesis Data Analytics is written to Amazon Kinesis Data Firehose, which micro-batch streams data into an Amazon S3 data lake. The Amazon S3 data lake stores telemetry data for future batch analytics.Key considerations and best practicesKeep the following best practices in mind:Define the business value from IoT sensor data through interactive discovery sessions with various stakeholders within your organization.Identify the type of IoT sensor data you want to collect and analyze for predictive analytics.Choose the right tools for the job, depending upon your business use case and your data consumers. Please refer to step 5 earlier in this post, where different purpose-built data services were used based on user personas.Consider the event-driven architecture as three key components: event producers, event routers, and event consumers. A producer publishes an event to the router, which filters and pushes the events to consumers. Producer and consumer services are decoupled, which allows them to be scaled, updated, and deployed independently.In this architecture, IoT sensors are event producers. Amazon IoT Greengrass, Amazon IoT Core, Amazon Kinesis Data Streams, and Amazon Kinesis Data Analytics work together as the router from which multiple consumers can consume IoT sensor-generated data. These consumers include Amazon S3 data lakes for telemetry data analysis, Amazon OpenSearch Service for personalized dashboards, and Amazon DynamoDB or AWS AppSync for the downstream enterprise application’s consumption.ConclusionIn this post, we demonstrated how to build an event-driven architecture with IoT sensor data using AWS IoT services and AWS purpose-built data services. You can now build your own event-driven applications using this post with your IoT sensor data and integrate with your business applications as needed.Further readingAWS IoTservicesAnalytics on AWSAmazon SageMakerand otherartificial intelligence servicesfor data predictionSeven patterns for IoT data ingestion and visualizationAWS IoT solutionsAWS IoT blog postsRaghavarao SodabathinaRaghavarao Sodabathina is a Principal Solutions Architect at AWS, focusing on Data Analytics, AI/ML, and Serverless Platform. He engages with customers to create innovative solutions that address customer business problems and to accelerate the adoption of AWS services. In his spare time, Raghavarao enjoys spending time with his family, reading books, and watching movies.Jun ShanJun Shan is a Senior Solutions Architect at AWS. He works with customers to leverage the power of AWS to release their potentials to the maximum degree, enabling them to be more efficient, secure, reliable, and optimized for cost. Jun also teaches SQL and Relational Database in several New York City colleges and is the author of "SQL for Data Analytics, 3rd Edition".Moises UlloaMoises Ulloa is a Senior Solutions Architect at Amazon Web Services based out of New York City. He works with AWS enterprise customers helping them in their cloud journey solving complex business problems by making effective use of AWS services. Outside of work, he likes to spend time with family and traveling.
The Internet of Things (IoT) brings sensors, cloud computing, analytics, and people together to improve productivity and efficiency. It empowers customers with the intelligence they need to build new services and business models, improve products and services over time, understand their customers’ needs to provide better services, and improve customer experiences. Business operations become more […]
Raghavarao Sodabathina
Amazon Kinesis Amazon SageMaker Architecture AWS IoT Core AWS IoT Greengrass AWS Lambda