Migrating petabytes of data from on-premises file systems to Amazon FSx for Lustre
AWS Architecture BlogMigrating petabytes of data from on-premises file systems to Amazon FSx for Lustreby Vimala Pydi, Sanjukta Mukherjee, and Sarat Para | on18 MAR 2022| inAmazon CloudWatch,Amazon FSx for Lustre,Architecture,AWS DataSync|Permalink|ShareForInternational Women’s Dayand Women’s History Month, we’re featuring more than a week’s worth of posts that highlight female builders and leaders. We’re showcasing women in the industry who are building, creating, and, above all, inspiring, empowering, and encouraging everyone—especially women and girls—in tech.Many organizations use the Lustre filesystem for Linux-based applications that require petabytes of data and high-performance storage. Lustre file systems are used in machine learning (ML), high performance computing (HPC), big data, and financial analytics. Many such high-performance workloads are being migrated toAmazon Web Services (AWS)to take advantage of the scalability, elasticity, and agility that AWS offers.Amazon FSx for Lustreis a fully managed service that provides cost-effective, high-performance, and scalable storage for Lustre file systems on AWS.AWS DataSyncis an AWS managed service for copying data to and from Amazon FSx for Lustre. It provides high-speed transfer through its use of compression and parallel transfer mechanism and integrates withAmazon CloudWatchfor observability.This blog will show you how to migrate petabytes of data files from on-premises to Amazon FSx for Lustre using AWS DataSync. It will provide an overview of Amazon CloudWatch metrics and logs to help you monitor your data transfer using AWS DataSync and metrics from Amazon FSx for Lustre.Solution overview for file storage data migrationThe high-level architecture diagram in Figure 1 depicts file storage data migration from on-premises data center to Amazon FSx for Lustre using AWS DataSync.Following are the steps for the migration:Create an Amazon FSx file system.Install AWS DataSync agent on premises to connect to AWS DataSync service over secured TLS connection.Configure source and target locations to create an AWS DataSync task.Configure and start the AWS DataSync task to migrate the data from on-premises to Amazon FSx for Lustre.Figure 1. Architecture diagram for transferring files on-premises to Amazon FSx for Lustre using AWS DataSyncPrerequisitesOn-premiseshypervisororvirtual machineThe necessary network communications between the AWS DataSync agent and AWS as detailed in AWS DataSyncnetwork requirementsAWS Management Consoleaccess to AWS DataSync, Amazon FSx for Lustre, and Amazon CloudWatchSteps for migration1. Create an Amazon FSx file systemTo start the migration, create a Lustre file system inAmazon FSxservice and follow the step-by-step guidance provided inGetting started with Amazon FSx for Lustre.For this blog, a target of ‘Persistent 2’ deployment type FSx for Lustre is selected with a storage capacity of 1.2 TB (Figure 2.)Figure 2. FSx for Lustre target file system2. Install AWS DataSync agent on-premisesFollow steps in the article:Getting started with AWS DataSyncto get started with the AWS DataSync service. Configure the source system to migrate the file system data using the following steps:Deploy an AWS DataSync agent on-premises on a supported virtual machine or hypervisor (Figure 3.)Configure the AWS DataSync agent from AWS Management Console.Activate the AWS DataSync agent configured from the preceding step.Figure 3. Create AWS DataSync agent3. Configure source and destination locationsA DataSync task consists of a pair of locations between which data is transferred. Thesourcelocation defines the storage system that you want to read from. Thedestinationlocation defines the storage service that you want to write data to. Here the source location is an on-premises Lustre system and the destination location is the Amazon FSx for Lustre service (Figure 4.)Figure 4. Configure source and destination location for AWS DataSync task4. Configure and start taskA task is a set of two locations (source and destination) and a set of options that you use to control the behavior of the task. Create a task with the source and destination locations and chooseStartfrom theActionsmenu (Figure 5.)Figure 5. Start taskWait for the task status to change toRunning(Figure 6.)Figure 6. Task statusTo check the details of the task completion, select the task and click on theHistorytab (Figure 7.) The status should showSuccessonce the task successfully completes the migration.Figure 7. Task historyMonitoring the file transferAmazon CloudWatch is the AWS native observability service. It collects and processes raw data from AWS services such as Amazon FSx for Lustre and AWS DataSync into readable, near real-time metrics. It provides metrics that you can use to get more visibility into the data transfer. For a full list of CloudWatch metrics for AWS DataSync and Amazon FSx for Lustre, readMonitoring AWS DataSyncandMonitoring Amazon FSx for Lustre.Amazon FSx for Lustre can also provide various metrics, for example, the number of read or write operations usingDataReadOperationsandDataWriteOperations. To find the total storage available you can check the metricFreeDataStorageCapacity(Figure 8.)Figure 8. CloudWatch metrics for Amazon FSx for LustreAWS DataSync metrics such asFilesTransferred, gives the actual number of files or metadata that transferred over the network.BytesTransferredprovides the total number of bytes that transferred over the network when the agent reads from the source location to the destination location.A robust monitoring system can be built by setting up an automated notification process for any errors or issues in the data transfer task. Integrate Amazon CloudWatch in combination with theAmazon Simple Notification Service (SNS). Figure 9 depicts the AWS DataSync logs in Amazon CloudWatch.Figure 9. AWS DataSync logs in Amazon CloudWatchYou can also gather insights from the logs of the data transfer metrics usingCloudWatch Logs Insights. CloudWatch Log Insights enables you to quickly search and query your log data (Figure 10.) You can set a filter metric for error codes and alert the appropriate team.Figure 10. Amazon CloudWatch Logs Insights for querying logsCleanupIf you are no longer using the resources discussed in this blog, remove the unneeded AWS resources to avoid incurring charges. After finishing the file transfer, clean up resources by deleting the Amazon FSx file system and AWS DataSync objects (DataSync agent, task, source location, and destination location.)ConclusionIn this post, we demonstrated how we can accelerate migration of Lustre files from on-premises into Amazon FSx for Lustre using AWS DataSync. As a fully managed service, AWS DataSync securely and seamlessly connects to your Amazon FSx for Lustre file system. This makes it possible for you to move millions of files and petabytes of data without the need for deploying or managing infrastructure in the cloud. We walked through different observability metrics with Amazon CloudWatch integration to provide performance metrics, logging, and events. This can further help to speed up critical hybrid cloud storage workflows in industries that must move active files into AWS quickly. This capability is available in Regions where AWS DataSync and Amazon FSx for Lustre are available. For further details on using this cost-effective service, seeAmazon FSx for Lustre pricingandAWS DataSync pricing.For further reading:Migrating existing files to FSx for Windows File Server using AWS DataSyncAmazon FSx for Lustre CustomersAmazon FSx for Lustre ResourcesAWS DataSync customersMore posts for Women’s History Month!Celebrate International Women’s Day all week with the Architecture BlogDeploying service-mesh-based architectures using AWS App Mesh and Amazon ECSfromKesha Williams, anAWS Heroand award-winning software engineer.A collection of several blog posts written and co-authored by womenCurated contentfrom theLet’s Architect!team and alive Twitter chatWomen at AWS – Diverse backgrounds make great solutions architectsExtend SQL Server DR using log shipping for SQL Server FCI with Amazon FSx for Windows configurationBuilding your brand as a Solutions ArchitectMainframe offloading and modernization: Using mainframe data to build cloud native services with AWSMessage to the next generation of women disruptors in technologyOther ways to participateShare Your 2022 #SheCoded StoryInternational Women’s Day: Improve Equality for Women in TechAWS She Builds CloudUp programTAGS:International Women's DayVimala PydiVimala Pydi is a Sr. Solutions Architect based out of Orlando, FL. She works with the World-Wide Public-Sector team as a migrations specialist. She is passionate about helping customers with their cloud migration and modernization journey. Prior to AWS, she was helping customers in various industries in their digital transformations and has over 20 years of experience in IT industry.Sanjukta MukherjeeSanjukta Mukherjee is a Solutions Architect at AWS based out of New York. She works with customers from various industries to help them accelerate their adoption of AWS Cloud. She is responsible for helping customers to innovate and modernize their applications using AWS.Sarat ParaSarat Para is a Sr. Solutions Architect at AWS based out of Atlanta, GA. He enjoys learning new technologies and helping customers in their cloud adoption journey. He brings to his role over 20 years of technology experience in software development, architecture, and IT management.
For International Women’s Day and Women’s History Month, we’re featuring more than a week’s worth of posts that highlight female builders and leaders. We’re showcasing women in the industry who are building, creating, and, above all, inspiring, empowering, and encouraging everyone—especially women and girls—in tech. Many organizations use the Lustre filesystem for Linux-based applications that […]
Vimala Pydi
Amazon CloudWatch Amazon FSx for Lustre Architecture AWS DataSync