Field Notes: Creating Custom Analytics Dashboards with FireEye Helix and Amazon QuickSight
AWS Architecture BlogField Notes: Creating Custom Analytics Dashboards with FireEye Helix and Amazon QuickSightby Karish Chowdhury, Ashok Srirama, and Shaun Wood | on16 AUG 2021| inAmazon EventBridge,Amazon QuickSight,Architecture,AWS CloudFormation,AWS Lambda|Permalink|ShareFireEye Helixis a security operations platform that allows organizations to take control of any incident from detection to response. FireEye Helix detects security incidents by correlating logs and configuration settings from sources likeVPC Flow Logs,AWS CloudTrail, andSecurity groups.In this blog post, we will discuss an architecture that allows you to create custom analytics dashboards withAmazon QuickSight. These dashboards are based on the threat detection logs collected by FireEye Helix. We automate this process so that data can be pulled and ingested based on a provided schedule. This approach usesAWS Lambda, andAmazon Simple Storage Service (Amazon S3)in addition to QuickSight.Architecture OverviewThe solution outlines how to ingest the security log data from FireEye Helix to Amazon S3 and create QuickSight visualizations from the log data. With this approach, you needAmazon EventBridgeto invoke a Lambda function to connect to the FireEye Helix API. There are two steps to this process:Download the log data from FireEye Helix and store it in Amazon S3.Create a visualization Dashboard in QuickSight.The architecture shown in Figure 1 represents the process we will walk through in this blog post. To implement this solution, you will need the following AWS services and features involved:Amazon EventBridgeAmazon QuickSightAmazon S3AWS LambdaAWS Secrets ManagerAWS Identity and Access Management (IAM)AWS CloudFormationFigure 1: Solution architecturePrerequisites to implement the solution:The following items are required to get your environment set up for this walkthrough.AWS account.FireEye Helix search alerts API endpoint. This is available under the API documentation in the FireEye Helix console.FireEye Helix API key. ThisFireEye communitypage explains how to generate an API key with appropriate permissions (always follow least privilege principles). This key is used by the Lambda function to periodically fetch alerts.AWS Secrets Managersecret (to store the FireEye Helix API key). To set it up, follow the steps outlined in theCreating a secret.Extract the data from FireEye Helix and load it into Amazon S3You will use the following high-level steps to retrieve the necessary security log data from FireEye Helix and store it on Amazon S3 to make it available for QuickSight.Establish anAWS Identity and Access Management (IAM)role for the Lambda function. It must have permissions to access Secrets Manager and Amazon S3 so they can retrieve the FireEye Helix API key and store the extracted data, respectively.Create anAmazon S3 bucketto store the FireEye Helix security log data.Create aLambda functionthat uses the API key from Secrets Manager, calls theFireEye Helix search alerts APIto extract FireEye Helix’s threat detection logs, and stores the data in the S3 bucket.Establish aCloudWatch EventBridge ruleto invoke the Lambda function on an automated schedule.To simplify this deployment, we have developed aCloudFormationtemplateto automate creating the preceding requirements. Follow the below steps to deploy the template:Download the source code from theGitHub repositoryNavigate to CloudFormationconsoleand selectCreate StackSelect “Upload a template file” radio button, click on “Choose file” button, and select “helix-dashboard.yaml” file in the downloaded Github repository. Click “Next” to proceed.On “Specify stack details” screen enter the parameters shown in Figure 2.Figure 2: CloudFormation stack creation with initial parametersThe parameters in Figure 2 include:HelixAPISecretName– Enter the Secrets Manager secret name where the FireEye Helix API key is stored.HelixEndpointUrl– Enter the Helix search API endpoint URL.Amazon S3 bucket– Enter the bucket prefix (a random suffix will be added to make it unique).Schedule– Choose the default option that pulls logs once a day, or enter the CloudWatch event schedule expression.Select the check box next to “I acknowledge that AWS CloudFormation might create IAM resources.” and then press theCreate Stackbutton. After the CloudFormation stack completes, you will have a fully functional process that will retrieve the FireEye Helix security log data and store it on the S3 bucket.You can also select the Lambda function from the CloudFormation stack outputs to navigate to the Lambda console. Review the following default code, and add any additional transformation logic according to your needs after fetching the results (line 32).import boto3
from datetime import datetime
import requests
import os

region_name = os.environ['AWS_REGION']
secret_name = os.environ['APIKEY_SECRET_NAME']
bucket_name = os.environ['S3_BUCKET_NAME']
helix_api_url = os.environ['HELIX_ENDPOINT_URL']

def lambda_handler(event, context):

    now = datetime.now()
    # Create a Secrets Manager client to fetch API Key from Secrets Manager
    session = boto3.session.Session()
    client = session.client(
        service_name='secretsmanager',
        region_name=region_name
    )
    apikey = client.get_secret_value(
            SecretId=secret_name
        )['SecretString']
    
    datestr = now.strftime("%B %d, %Y")
    apiheader = {'x-fireeye-api-key': apikey}
    
    try:
        # Call Helix Rest API to fetch the Alerts
        helixalerts = requests.get(
            f'{helix_api_url}?format=csv&query=start:"{datestr}" end:"{datestr}" class=alerts', headers=apiheader)
    
        # Optionally transform the content according to your needs..
        
        # Create a S3 client to upload the CSV file
        s3 = boto3.client('s3')
        path = now.strftime("%Y/%m/%d")+'/alerts-'+now.strftime("%H-%M-%S")+'.csv'
        response = s3.put_object(
            Body=helixalerts.content,
            Bucket=bucket_name,
            Key=path,
        )
        print('S3 upload response:', response)
    except Exception as e:
        print('error while fetching the alerts', e)
        raise e
        
    return {
        'statusCode': 200,
        'body': f'Successfully fetched alerts from Helix and uploaded to {path}'
    }Creating a visualization in QuickSightOnce the FireEye data is ingested into an S3 bucket, you can start creating custom reports and dashboards using QuickSight. Following is a walkthrough on how to create a visualization in QuickSight based on the data that was ingested from FireEye Helix.Step 1– When placing the FireEye data into Amazon S3, ensure you have a clean directory structure so you can partition your data. By partitioning your data, you can restrict the amount of data scanned by each query, thus improving performance and reducing cost. The following is a sample directory structure you could use.ssw-fireeye-logs20210405The following is an example of what the data will look like after it is ingested from FireEye Helix into your Amazon S3 bucket. In this blog post, we will use theAlert_Desccolumn to report on the types of common attacks.Step 2– Next, you must create amanifestfile that will instruct QuickSight how to read the FireEye log files on Amazon S3. The preceding example is a manifest file that instructs QuickSight to recursively search for files in the ssw-fireeye-logs bucket, and can be seen in the URIPrefixes section. The GlobalUploadSettings section informs QuickSight the type and format of files it will read."fileLocations": [
	{
		"URIPrefixes": [
			"s3://ssw-fireeye-logs/"
		]
	},
	],
	"globalUploadSettings": {
		"format": "CSV",
		"delimiter": ",",
		"textqalifier": "'",
		"containsHeader": "true"
	}
}Step 3– Open Amazon QuickSight. Use theAWS Management Consoleand search for QuickSight.Step 4– Below the QuickSight logo, find and selectDatasets.Step 5– Push the blueNew datasetbutton.Step 6– Now you are onCreate a Datasetpage which enables you to select a data source you would like QuickSight to ingest. Because we have stored the FireEye Helix data on S3, you should choose the S3 data source.Step 7– A pop-up box will appear calledNew S3 data source. Type a data source name, and upload the manifest file you created. Next, push theConnectbutton.Step 8– You are now directed to the Visualize screen. For this exercise let’s choose a Pie chart, you can find this in the Visual types section by hovering over each icon and reading each tool tip that comes up. Look for the tool tip that says Pie chart. After selecting the Pie Chart visual type, two entries in the Field wells section at the top of the screen will show up called Group/Color and Value. Click the drop down in Group/Color and select the Alert_Desc column. Now click the drop down in Value and also select Alter_Desc column but choose count as an aggregate. This will create an informative visualization of the most common attacks based on the sample data shown previously in Step 1.Figure 3: Visualization screen in QuickSightClean upIf you created any resources in AWS for this solution, consider removing them and any example resources you deployed. This includes the FireEye Helix API keys, S3 buckets, Lambda functions, and QuickSight visualizations. This helps ensure that there are no unwanted recurring charges. For more information, review how todelete the CloudFormation stack.ConclusionThis blog post showed you how to ingest security log data from FireEye Helix and store that data in Amazon S3. We also showed you how to create your own visualizations in QuickSight to better understand the overall security posture of your AWS environment. With this solution, you can perform deep analysis and share these insights among different audiences in your organization (such as, operations, security, and executive management).Field Notes provides hands-on technical guidance from AWS Solutions Architects, consultants, and technical account managers, based on their experiences in the field solving real-world business problems for customers.TAGS:Field NotesKarish ChowdhuryKarish is an Enterprise Solutions Architect at Amazon Web Services. He is a cloud enthusiast and very passionate about finding solutions for customers that are simple and addresses their business outcomes.Ashok SriramaAshok is a Senior Solutions Architect at Amazon Web Services, based in Washington Crossing, PA. He specializes in serverless applications, containers, and architecting distributed systems. When he’s not spending time with his family, he enjoys watching cricket, and driving his bimmer.Shaun WoodShaun is an Enterprise Solutions Architect at Amazon Web Services, and he is based in Boston, MA. He loves working with cloud technologies providing simple, scalable solutions that drive positive business outcomes.
FireEye Helix is a security operations platform that allows organizations to take control of any incident from detection to response. FireEye Helix detects security incidents by correlating logs and configuration settings from sources like VPC Flow Logs, AWS CloudTrail, and Security groups. In this blog post, we will discuss an architecture that allows you to […]
Karish Chowdhury
Amazon EventBridge Amazon QuickSight Architecture AWS CloudFormation AWS Lambda