title: Run HDFS cluster on AWS without EMR
body: <p>I want to run an HDFS cluster on AWS where I can store the data that needs to be processed using my custom application running on EC2 instances. <a href="https://aws.amazon.com/emr/" rel="nofollow noreferrer">AWS EMR</a> is the only way I could find to create an HDFS cluster on AWS. There are <a href="https://www.shubhamdipt.com/blog/how-to-setup-hadoop-in-aws-ec2-instances/" rel="nofollow noreferrer">tutorials</a> available on the web to create HDFS cluster using EC2 instances. But, if I use EC2 instances, I run the risk of losing the data when I shut down the instances.</p>
  
  <p>What I need is:<br>
   1. An HDFS cluster that can be shut down when not in use.<br>
   2. When shut down, data should remain persisted.</p>
  
  <p>There is a solution that says I can keep my data in S3 bucket and load it everytime I start the EMR cluster. However, this is repetitive and a huge overhead specially if the data is huge.</p>
  
  <p>In GCP, I used DataProc cluster which satisfied the above two criteria. Shutting down the cluster at least saved the cost of VMs and I only paid for storage when not using the HDFS cluster. I am wondering if there is some similar way in AWS.</p>