Using Amazon Macie to Validate S3 Bucket Data Classification
AWS Architecture BlogUsing Amazon Macie to Validate S3 Bucket Data Classificationby Bill Magee, Lijan Kuniyil, and Mikhail Vaynshteyn | on16 JUL 2021| inAmazon DynamoDB,Amazon EventBridge,Amazon Macie,Amazon Simple Notification Service (SNS),Architecture,Intermediate (200)|Permalink|ShareSecuring sensitive information is a high priority for organizations for many reasons. At the same time, organizations are looking for ways to empower development teams to stay agile and innovative. Centralized security teams strive to create systems that align to the needs of the development teams, rather than mandating how those teams must operate.Security teams who create automation for the discovery of sensitive data have some issues to consider. If development teams are able to self-provision data storage, how does the security team protect that data? If teams have a business need to store sensitive data, they must consider how, where, and with what safeguards that data is stored.Let’s look at how we can set upAmazon Macieto validate data classifications provided by decentralized software development teams. Macie is a fully managed service that uses machine learning (ML) to discover sensitive data in AWS. If you are not familiar with Macie, readNew – Enhanced Amazon Macie Now Available with Substantially Reduced Pricing.Data classificationis part of the security pillar of a Well-Architected application. Following the guidelines provided in theAWS Well-Architected Framework, we can develop a resource-tagging scheme that fits our needs.Overview of decentralized data validation systemIn our example, we have multiple levels of data classification that represent different levels of risk associated with each classification. When a software development team creates a newAmazon Simple Storage Service (S3)bucket, they are responsible for labeling that bucket with a tag. This tag represents the classification of data stored in that bucket. The security team must maintain a system to validate that the data in those buckets meets the classification specified by the development teams.This separation of roles and responsibilities for development and security teams who work independently requires a validation system that’s decoupled from S3 bucket creation. It should automatically detect new buckets or data in the existing buckets, and validate the data against the assigned classification tags. It should also notify the appropriate development teams of misclassified or unclassified buckets in a timely manner. These notifications can be through standard notification channels, such as email or Slack channel notifications.Validation and alerts with AWS servicesFigure 1. Validation system for data classificationWe assume that teams are permitted to create S3 buckets and we will useAWS Configto enforce the followingrequired tags:DataClassificationandSupportSNSTopic. TheDataClassificationtag indicates what type of data is allowed in the bucket. TheSupportSNSTopictag indicates anAmazon Simple Notification Service (SNS)topic. If there are issues found with the data in the bucket, a message is published to the topic, and Amazon SNS will deliver an alert. For example, if there is personally identifiable information (PII) data in a bucket that is classified as non-sensitive, the system will alert the owners of the bucket.Macie is configured to scan all S3 buckets on a scheduled basis. This configuration ensures that any new bucket and data placed in the buckets is analyzed the next time the Macie job runs.Macie provides severalmanaged data identifiersfor discovering and classifying the data. These include bank account numbers, credit card information, authentication credentials, PII, and more. You can also create custom identifiers (or rules) to gather information not covered by the managed identifiers.Macie integrates withAmazon EventBridgeto allow us to capture data classification events and route them to one or more destinations for reporting and alerting needs. In our configuration, the event initiates anAWS Lambda. The Lambda function is used to validate the data classification inferred by Macie against the classification specified in theDataClassificationtag using custom business logic. If a data classification violation is found, the Lambda then sends a message to the Amazon SNS topic specified in theSupportSNSTopictag.The Lambda function also creates custom metrics and sends those toAmazon CloudWatch. The metrics are organized by engineering team and severity. This allows the security team to create a dashboard of metrics based on the Macie findings. The findings can also be filtered per engineering team and severity to determine which teams need to be contacted to ensure remediation.ConclusionThis solution provides a centralized security team with the tools it needs. The team can validate the data classification of an Amazon S3 bucket that is self-provisioned by a development team. New Amazon S3 buckets are automatically included in the Macie jobs and alerts. These are only sent out if the data in the bucket does not conform to the classification specified by the development team. The data auditing process is loosely coupled with the Amazon S3 Bucket creation process, enabling self-service capabilities for development teams, while ensuring proper data classification. Your teams can stay agile and innovative, while maintaining a strong security posture.Learn more aboutAmazon MacieandData Classification.TAGS:piiBill MageeBill is a Solutions Architect at AWS supporting customers in the Education vertical. He loves all things Serverless and Devops. Both allow software engineers to focus on delivering value for their customers. In his free time, he likes traveling and trying new restaurants.Lijan KuniyilLijan Kuniyil is a Solutions Architect at AWS. Lijan enjoys working with customers and helping them deliver solutions in AWS in the area of Serverless Architecture. Lijan has more than 20 years of experience in developing solutions for financial and consulting companies.Mikhail VaynshteynMikhail is a Solutions Architect with Amazon Web Services. He works with Health Care Life Sciences customers and specializes in Data Analytics Services. Mikhail has more than 20 years of industry experience covering a wide range of technologies and sectors.
Securing sensitive information is a high priority for organizations for many reasons. At the same time, organizations are looking for ways to empower development teams to stay agile and innovative. Centralized security teams strive to create systems that align to the needs of the development teams, rather than mandating how those teams must operate. Security […]
Bill Magee
Amazon DynamoDB Amazon EventBridge Amazon Macie Amazon Simple Notification Service (SNS) Architecture Intermediate (200)