Managed File Transfer using AWS Transfer Family and Amazon S3
AWS Architecture BlogManaged File Transfer using AWS Transfer Family and Amazon S3by Dathu Patil | on04 JUN 2021| inAmazon Cognito,Amazon Route 53,Amazon Simple Storage Service (S3),Architecture,AWS Batch,AWS Lambda,AWS Transfer Family|Permalink|ShareFinancial, healthcare, retail and other companies exchange many different types of data. This can include stock information, healthcare claims, and sharing product data files with their partners. These companies need amanaged file transfersolution that supports data transformation, and exchange of data overFile Transfer Protocol over SSL (FTPS), andSecure File Transfer Protocol (SFTP). This post describes how you can build a managed file transfer solution onAmazon Web Services (AWS). This managed file transfer solution provides features to support data transformation, inbound, and outbound file transfers over FTP protocols.Managed file transfer using AWS Transfer FamilyFor this solution, you will use AWS services to build a managed file transfer solution that supports inbound and outbound transfers over FTP protocols. You will useAWS Transfer Familyto host endpoints, which support protocols such as SFTP, FTPS, and File Transfer Protocol (FTP).  FTP is not encrypted and we suggest using SFTP or FTPS when possible. The AWS Transfer Family provides fully managed support for file transfers directly into and out ofAmazon Simple Storage Service (S3)orAmazon EFS. The AWS Transfer Family integration withAmazon Route 53can be used for DNS routing. With your data in Amazon S3, you can use AWS services for processing, analytics, machine learning, archiving, home directories, and developer tools.AWS Transfer Family integration with Amazon S3 can be used for storing file data. You can take advantage of industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and exchange any amount of data. Note that there is a maximum size per individual file. AWS Transfer Family supports a custom API for user authentication. This enables you to use existing credentials or authentication providers, such asAmazon Cognito, Okta, LDAP, and more. You can build a serverless authentication API usingAmazon API GatewayandAWS Lambda. Another benefit to using AWS serverless services, is that you do not need to manage and maintain servers.AWS Batchcan run jobs with your custom code for transforming data files and processing outbound transfers. To store application data including entitlements and transactions data, you can use a managed relational database service such asAmazon Aurora. Based on your application needs, you can also use a NoSQL database service such asAmazon DynamoDB.Managed File Transfer Process FlowFigure 1. Managed File Transfer Process FlowThis architecture uses the Amazon Aurora service for storing application data and Amazon Cognito for identity management where user credentials are stored. The solution provides a process for inbound transfer and outbound transfer. In the inbound transfer flow, customers can interact with the SFTP endpoint to download or upload data files. In the outbound transfer flow, the system sends data file to the customer SFTP location based on their entitlements.Inbound Transfer Flow:When you request the SFTP domain URL, the request goes to Amazon Route 53 for DNS resolution. Route 53 resolves the domain name and provides the URL of the AWS Transfer Family endpoint.The user request is sent to the AWS Transfer Family endpoint along with user credentials (user name and password).AWS Transfer Family invokes a custom authentication API hosted in API Gateway, which reviews and validates the user credentials.API Gateway and Lambda integration invoke an AuthLogic Lambda function that authenticates the user credentials by calling the Amazon Cognito API. The AuthLogic Lambda function implements custom authentication logic, and complex business rules around entitlements.Once the user is authenticated, the AuthLogic Lambda function queries the Aurora database to get user entitlements.Based on the user entitlements, a dynamicAWS Identity and Access Management (IAM)policy and logical directory mapping is returned to AWS Transfer Family.AWS Transfer Family uses the logical directory-mapping feature to provide user access to logical directories that map to data stored in an S3 bucket. S3 applies the provided IAM policy, which validates and approves user access to data.Outbound Transfer Flow:This component can be an optional based on your application needs. You may have analytics or Artificial Intelligence/Machine Learning (AI/ML) applications that use data stored in Amazon S3.When a customer uploads a file to S3 through an SFTP endpoint, an S3 event notification is created which invokes a Lambda functionusing Lambda with S3.The Lambda function initiates code to get subscriptions data from the Aurora database for the given file group. It then creates a transform job for business use cases where data transformation is required. An example of transformation would be converting data from JSON to CSV format, or healthcare data conversion to comply withHL7 Standards. Since the data transformation send jobs can be long-running, they can be run on AWS Batch.Once data transformation is complete, the transform job creates a send job for the transformed file.The send job sends the given file to the subscribed customers’ SFTP location.For business use cases where data transformation is not required, only the send job is created.ConclusionIn this blog post, we showed how you can use AWS Transfer Family, Amazon S3, and other AWS services to build a managed file transfer application for your business. Using the AWS Managed Services for your managed file transfer application, you can take full benefit of AWS and achieve agility, elasticity, and cost savings. You can also leverage this architecture to migrate an existing home grown or proprietary vendor-managed file transfer application. To get started on inbound transfers, seeBuilding a Simple Data Distribution Serviceblog for guidance.TAGS:Managed File Transfer (MFT)Dathu PatilDathu is a Solutions Architect based out of Boston, MA. He helps customers architect scalable, highly available applications that leverage AWS services. He works as a technical leader alongside customer business, development and infrastructure teams providing deep software knowledge with respect to cloud architecture, design patterns and programming.
Financial, healthcare, retail and other companies exchange many different types of data. This can include stock information, healthcare claims, and sharing product data files with their partners. These companies need a managed file transfer solution that supports data transformation, and exchange of data over File Transfer Protocol over SSL (FTPS), and Secure File Transfer Protocol (SFTP). […]
Dathu Patil
Amazon Cognito Amazon Route 53 Amazon Simple Storage Service (S3) Architecture AWS Batch AWS Lambda AWS Transfer Family