How to Run Massively Scalable ADAS Simulation Workloads on CAEdge
AWS Architecture BlogHow to Run Massively Scalable ADAS Simulation Workloads on CAEdgeby Hendrik Schoeneberg, Junjie Tang, Rumeshkrishan Mohan, and The An Binh Nguyen | on09 FEB 2022| inAmazon EC2,Amazon Elastic Container Registry,Amazon Managed Workflows for Apache Airflow (Amazon MWAA),Architecture,Automotive,AWS Batch,AWS Lake Formation|Permalink|ShareThis post was co-written by Hendrik Schoeneberg, Sr. Global Big Data Architect, The An Binh Nguyen, Product Owner for Cloud Simulation at Continental, Autonomous Mobility – Engineering Platform, Rumeshkrishnan Mohan, Global Big Data Architect, and Junjie Tang, Principal Consultant at AWS Professional Services.AV/ADAS simulations processing large-scale field sensor data such as radar, lidar, and high-resolution video come with many challenges. Typically, the simulation workloads are spiky with occasional, but high compute demands, so the platform must scale up or down elastically to match the compute requirements. The platform must be flexible enough to integrate specialized ADAS simulation software, use distributed computing or HPC frameworks, and leverage GPU accelerated compute resources where needed.Continentalcreated the Continental Automotive Edge (CAEdge) Framework to address these challenges. It is a modular multi-tenant hardware and software framework that connects the vehicle to the cloud. You can learn more about this inDeveloping a Platform for Software-defined Vehicles with Continental Automotive Edge (CAEdge)andDeveloping a platform for software-defined vehicles(re:Invent session-AUT304).In this blog post, we’ll illustrate how the CAEdge Framework orchestrates ADAS simulation workloads withAmazon Managed Workflows for Apache Airflow (MWAA). We’ll show how it delegates the high-performance workloads toAWS Batchfor elastic, highly scalable, and customizable compute needs. We’ll showcase the “bring your own software-in-the-loop” (BYO-SIL) pattern, detailing how to leverage specialized and proprietary simulation software in your workflow. We’ll also demonstrate how to integrate the simulation platform with tenant data in the CAEdge framework. Thisorchestrateanddelegatepattern has previously been introduced inField Notes: Deploying Autonomous Driving and ADAS Workloads at Scale with Amazon Managed Workflows for Apache Airflowand theReference Architecture for Autonomous Driving Data Lake.Solution overview for autonomous driving simulationThe following diagram shows a high-level overview of this solution.Figure 1. Architecture diagram for autonomous driving simulationIt can be broken down into five major parts, illustrated in Figure 1.Simulation API:Amazon API Gateway (label 1) provides a REST API for authenticated users to schedule or monitor simulation. It runs on Amazon Managed Workflows for Apache Airflow (MWAA) usingAWS Lambda(label 2).Simulation control plane:The simulation control plane (label 3) built on MWAA enables users to design and initiate workflows and integrate with AWS services.Scalable compute backend:We leverage the parallelization and elastic scalability capabilities of AWS Batch to distribute the workload (label 4). Additionally, we want to be able to run proprietary software components as part of the simulation workflows and use highly customizableAmazon EC2compute environments. For example, we can use GPU acceleration orGraviton-based instances for workloads that must run on ARM, instead of x86 architectures.Autonomous Driving Data Lake (ADDL) integration:The simulations’ input and output data will be stored in a data lake (label 5) onAmazon S3. To provide efficient read and write access, data gets copied before each simulation run using RAID0 bundled ephemeral instance storage drives. After the simulation, results are written back to the data lake and are ready for reporting and analytics. We useAWS Lake Formationfor metadata storage, data cataloging, and permission handling.Automated deployment:The solution architecture’s deployment is fully automated using a CI/CD pipeline inAWS CodePipelineandAWS CodeBuild(label 6). We can then perform automated testing and deployment to multiple target environments.In this blog we will focus on the simulation control plane, scalable compute backend, and ADDL integration.Simulation control planeIn a typical simulation, there are tasks like data movement (copying input data and persisting the results). These steps can require high levels of parallelization or GPU support. In addition, they can involve third-party or proprietary software, specialized runtime environments, or architectures like ARM. To facilitate all task requirements, we will delegate task initiation to the AWS services that best match the requirements. Correct sequence of tasks is achieved by an orchestration layer. Decoupling the simulation orchestration from task initiation follows the key design pattern to separate concerns. This enables the architecture to be adapted to specific requirements.For the high-level task orchestration, we’ll introduce a simulation control plane built on MWAA. Modeling all simulation tasks in a directed acyclic graph (DAG), MWAA performs scheduling and task initiation in the correct sequence. It also handles the integration with AWS’ services. You can intuitively monitor and debug the progress of a simulation run across different services and networks. For the workflow within CAEdge, we’ll use AWS Batch to run simulations that are packaged as Docker containers.Parameterizable workflows:Airflow supports parameterization of DAGs byproviding a JSON object when triggering a DAG runthat can be accessed at runtime. This enables us to create a simulation execution framework in which we model the simulation steps. It lets you specify the simulations’ parameters such as which Docker container to execute, runtime configuration, or input and output data locations, see Figures 2 and 3.Figure 2. Specify simulations’ parameters in JSONFigure 3. Read simulations’ parameters from JSONScalable compute backendPublishing the Docker image:To run a simulation, you must provide a Docker container inAmazon Elastic Container Registry (ECR).Manually push Docker imagesto ECR using the command line interface (CLI) or using CI/CD pipeline automation.Choosing the compute option:Containers at AWSdescribes the services options for developers. Various factors can contribute to your decision-making. For the CAEdge platform, we want to run thousands of containers with fine-grained control over the underlying compute instance’s configuration. We also want to run containers in privileged mode, so AWS Batch on EC2 is a great match. Figure 4 outlines the compute backend’s architecture.Figure 4. Compute backend architecture diagramTo run a Docker container on AWS Batch, we need three components: Ajob definition, acompute environment, and ajob queue. The AWS Batch job definition specifies which job to run and how to run it. For example, we can define the Docker image to use, and specify the vCPU and memory configuration. The job definition will be submitted to a job queue, which in turn is linked to one or more compute environments. The compute environment specifies the compute configuration used to run a containerized workload, in addition to instance types and storage configurations.Creating a compute environmentdescribes this more in detail. In the section ‘ADDL integration,’ we’ll describe how to select and configure the EC2 container instances to maximize network and disk I/O.TheAWS Batch array jobfeature can spin up thousands of independent, but similarly configured jobs. Instead of submitting a single job for each input file from the simulation control plane, we can instead submit a single array job. This provides the collection of input files asinput. AWS Batch can spin up child jobs to process each entry of the collection in parallel, reducing operational overhead drastically.With these components, we can now submit a job definition to a job queue from the simulation control plane, which will initiate on the corresponding compute environment.ADDL integrationIn the CAEdge platform, all simulation recordings and their metadata are stored and cataloged in the data lake. On average, single recordings are around 100 GB in size with simulation requests containing 100–300 recordings. A simulation request can therefore require 10–30 TB of data movement from S3 to the containers before the simulation starts. The containers’ performance will directly depend on I/O performance, as the input data is read and processed during execution. To provide the highest performance of data transfer and simulation workload, we need data storage options that maximize network and disk I/O throughput.Choosing the storage option:For CAEdge’s simulation workloads, the storage solution acts as a temporary scratch space for the simulation containers’ input data. It should maximize I/O throughput. These requirements are met in the most cost-efficient way by choosingEC2 instances of the M5d familyand their attached instance storage.The M5d are general-purpose instances with NVMe-based SSDs, which are physically connected to the host server and provide block-level storage coupled to the lifetime of the instance. As described in the previous section, we configured AWS Batch to create compute environments using EC2’sM5d instances. While other storage options likeAmazon Elastic File System (EFS)andAmazon Elastic Block Storage (EBS)can scale to match even demanding throughput scenarios, the simulation benefits from a high-performance, temporary storage location that is directly attached to the host.Bundling the volumes:M5d instances can have multiple NVMe drives attached, depending on their size and storage configuration. To provide a single stable storage location for the simulation containers, bundle all attached NVMe SSDs into a single RAID0 volume during the instance’s launch, using a modified user data script. With this, we can provide a fixed storage location that can be accessed from the simulation containers. Additionally, we are distributing I/O operations evenly across all disks in the RAID0 configuration, which improves the read and write performance.KPIs used for measurementThe SIL factor tells you how long the simulation takes compared to the duration of the recorded data.SIL factor example:For a recording with 60 minutes of data and a simulation duration of 120 minutes, the resulting SIL factor is 120 / 60 = 2.The simulation runs twice as long as real time.For a recording with 60 minutes of data and a simulation duration of 60 minutes, the resulting SIL factor is 60 / 60 = 1.The simulation runs in real time.Similarly, the aggregated SIL factor tells you how long the simulation for multiple recordings takes compared to the duration of the recorded data. It also factors in the horizontal scaling capabilities.Aggregated SIL factor example:For 3 recordings with 60 minutes of data and an overall simulation duration of 120 minutes, the resulting SIL factor is 120 / 3 * 60 = 0.67.The overall simulation is a third faster than real time.Performance resultsThe storage optimizations described in the ADDL integration KPI section preceding, led to an improvement of the overall simulation duration of 50%. To benchmark the overall simulation platform’s performance, we created a simulation run with 15 recordings. The simulation run completed successfully with anaggregated SIL factor of 0.363, or, alternatively put, the simulation was roughlythree times faster than real time. During production use, the platform will handle simulation runs with an average of 100-300 recordings. For these runs, the aggregated SIL factor is expected to be even smaller, as more simulations can be processed in parallel.ConclusionIn this post, we showed how to design a platform for ADAS simulation workloads using the Bring Your Own Software-In-the-Loop (BYO-SIL) concept. We covered the key components including simulation control plane, scalable compute backend, and ADDL integration based on Continental Automotive Edge (CAEdge). We discussed the key performance benefits due to horizontal scalability and how to choose an optimized storage integration pattern. This led to an improvement of the overall simulation duration of50%.Hendrik SchoenebergHendrik is a Senior Big Data Architect at AWS ProServe and helps customers with Data Platforms, ML operationalization and workflow orchestration. He is passionate about Big Data and Data Analytics and loves his job for its challenges and the opportunity to work with inspiring customers and colleagues.Junjie TangJunjie is Principal Consultant at AWS Professional Services. As a global technical lead, he heads a big data community of AWS Professional Services Consultants to develop data strategies and build accumulated expertise in data analytics across verticals.Rumeshkrishan MohanRumeshkrishnan Mohan is a Big Data Consultant with Amazon Web Services. He works with Global Customers in building their data lakes.The An Binh NguyenBinh is Product Owner for Cloud Simulation at Continental, Autonomous Mobility - Engineering Platform. He is passionate about technologies around driver-assistance and automated-driving systems. In his current role, he is responsible for utilizing cloud-based systems to boost up simulation at scale, thereby facilitating the development and validation process.
This post was co-written by Hendrik Schoeneberg, Sr. Global Big Data Architect, The An Binh Nguyen, Product Owner for Cloud Simulation at Continental, Autonomous Mobility – Engineering Platform, Rumeshkrishnan Mohan, Global Big Data Architect, and Junjie Tang, Principal Consultant at AWS Professional Services. AV/ADAS simulations processing large-scale field sensor data such as radar, lidar, and […]
Hendrik Schoeneberg
Amazon EC2 Amazon Elastic Container Registry Amazon Managed Workflows for Apache Airflow (Amazon MWAA) Architecture Automotive AWS Batch AWS Lake Formation