How Net at Work built an email threat report system on AWS
AWS Architecture BlogHow Net at Work built an email threat report system on AWSby Florian Mair and Tim Lenzen | on31 MAR 2022| inAmazon API Gateway,Amazon Athena,Amazon DynamoDB,Amazon Kinesis,Architecture|Permalink|ShareEmails are often used as an entry point for malicious software like trojan horses, rootkits, or encryption-based ransomware. TheNoSpamProxyoffering developed byNet at Worktackles this threat, providing secure and confidential email communication.A subservice of NoSpamProxy called 32guards is responsible for threat reports of inbound and outbound emails. With the increasing number of NoSpamProxy customers, 32guards was found to have several limitations. 32guards was previously built on a relational database. But with the growth in traffic, this database was not able to keep up with storage demands and expected query performance. Further, the relational database schema was limiting the possibilities of complex pattern detections, due to performance limitations. The NoSpamProxy team decided to rearchitect the service based on theLake House approach.The goal was to move away from a one-size-fits-all approach for data analytics and integrate a data lake with purpose-built data stores, unified governance, and smooth data movement.This post shows how Net at Work modernized their 32guards service, from a relational database to a fully serverless analytics solution. With adoption of theWell-Architected Analytics Lensbest practices and the use of fully managed services, the 32guards team was able to build a production-ready application within six weeks.Architecture for email threat reports and analyticsThis section gives a walkthrough of the solution’s architecture, as illustrated in Figure 1.Figure 1. 32guards threat reports architecture1. The entry point is anAmazon API Gateway, which receives email metadata in JSON format from the NoSpamProxy fleet. The message contains information about the email in general, email attachments, and URLs in the email. As an example, a subset of the data is presented in JSON as follows:{..."Attachments": [{"Sha256Hash": "69FB43BD7CCFD79E162B638596402AD1144DD5D762DEC7433111FC88EDD483FE","Classification": 0,"Filename": "test.ods.tar.gz","DetectedMimeType": "application/tar+gzip","Size": 5895}],"Urls": [{"Url": "http://www.aarhhie.work/","Classification": 0,},        {"Url": "http://www.netatwork.de/","Classification": 0,},{"Url": "http://aws.amazon.com/","Classification": 0,}]}2. This JSON message is forwarded to anAWS Lambdafunction (called “frontend”), which takes care of the further downstream processing. There are two activities the Lambda function initiates:Forwarding the record for real-time analysis/storageGenerating a threat report based on the information derived from the data stored in the indicators of compromises (IOCs)Amazon DynamoDBtableIOCs are patterns within the email metadata that are used to determine if emails are safe or not. For example, this could be for a suspicious file attachment or domain.Threat report for suspicious emailsIn the preceding JSON message, the attachments and URLs have been classified with “0” by the email service itself, which indicates that none of them look suspicious. ThefrontendLambda function uses the vast number of IOCs stored in the DynamoDB table and heuristics to determine any potential threats within the email. The use of DynamoDB enables fast lookup times to generate a threat report. For the example, the response to the API Gateway in step 2 looks like this:{"ReportedOnUtc": "2021-10-14T14:33:34.5070945Z","Reason": "realtimeSuspiciousOrganisationalDomain","Identifier": "aarhhie.work",...}This threat report shows that the top-level domain “aarhiie.work” has been detected as suspicious. The report is used to determine further actions for the email, such as blocking.Real-time data processing3. In the real-time analytics flow, the frontend Lambda function ingests email metadata into a data stream usingAmazon Kinesis Data Streams. This is a massively scalable, serverless, and durable real-time data streaming service. Compared to a queue, streaming storage permits more than one consumer of the same data.4. The first consumer is anApache Flinkapplication running inAmazon Kinesis Data Analytics. This application generates statistical metrics (for example, occurrences of the top-level domain “.work”). The output is stored inApache Parquetformat onAmazon S3. Parquet is a columnar storage format for row-based files like csv.The second consumer of the streaming data isAmazon Kinesis Data Firehose. Kinesis Data Firehose is a fully managed solution to reliably load streaming data into data lakes, data stores, and analytics services. Within the 32guards service, Kinesis Data Firehose is used to store all email metadata into Amazon S3. The data is stored in Apache Parquet format,which makes queries more time and cost efficient.IOC detectionNow that we have shown how data is ingested and threat reports are generated to respond quickly to requests, let’s look at how the IOCs are updated. These IOCs are used for generating the threat report within the “frontend” Lambda function. As attack vectors are changing over time, quickly analyzing the data for new threats, is crucial to provide high-quality reports to the NoSpamProxy service.The incoming email metadata is stored every few minutes in Amazon S3 by Kinesis Data Firehose. To query data directly in Amazon S3,Amazon Athenais used. Athena is a serverless query service that analyzes data stored in Amazon S3, by using standard SQL syntax.5. To be able to query data in S3, Amazon Athena uses theAWS Glue Data Catalog, which contains the structure of the email metadata stored in the data lake. The data structure is derived from the data itself usingAWS Glue Crawlers. Other external downstream processing services like business intelligence applications, also use Amazon Athena to consume the data.6. Athena queries are initiated on a predefined schedule to update or generate new IOCs. The results of these queries are stored in the DynamoDB table to enable fast lookup times for the “frontend” Lambda.ConclusionIn this blog post, we showed how Net at Work modernized their 32guards service within their NoSpamProxy product. The previous architecture used a relational database to ingest and store email metadata. This database was running into performance and storage issues, and must be redesigned into a more performant and scalable architecture.Amazon S3 is used as the storage layer, which can scale up to exabytes of data. With Amazon Athena as the query engine, there is no need to operate a high-performance database cluster, as compute and storage is separated. By using Amazon Kinesis Data Streams and Amazon Kinesis Data Analytics, valuable insight can be generated in real time, and acted upon more quickly.As a serverless, fully managed solution, the 32guards service has a lower-cost footprint of as much as 50% and requires less maintenance. By moving away from a relational database model, the query runtimes decrease significantly. You can now conduct analyses that have not been feasible before.Interested in the NoSpamProxy?Read more about NoSpamProxyorsign up for a free trial.Looking for more architecture content?AWS Architecture Centerprovides reference architecture diagrams, vetted architecture solutions, Well-Architected best practices, patterns, icons, and more!Florian MairFlorian Mair is a Solutions Architect and data streaming expert at AWS. He is a technologist that helps customers in Germany succeed and innovate by solving business challenges using AWS Cloud services. Besides working as a Solutions Architect, Florian is a passionate mountaineer, and has climbed some of the highest mountains across Europe.Tim LenzenTim Lenzen works as a data scientist for the NoSpamProxy team and leads the development of the 32guards service. With a PhD in theoretical chemistry, he now uses his software engineering and data science knowledge to analyze email traffic rather than chemical reactions. Tim strongly believes in data driven decision making.
Emails are often used as an entry point for malicious software like trojan horses, rootkits, or encryption-based ransomware. The NoSpamProxy offering developed by Net at Work tackles this threat, providing secure and confidential email communication. A subservice of NoSpamProxy called 32guards is responsible for threat reports of inbound and outbound emails. With the increasing number […]
Florian Mair
Amazon API Gateway Amazon Athena Amazon DynamoDB Amazon Kinesis Architecture