title: Load multiple .npy files (size > 10GB) in pytorch
body: <p>Im looking for a optimized solution to load multiple huge .npy files using pytorch data loader. 
  I'm currently using the following method which creates a new dataloader for each file in each epoch. </p>
  
  <p>My data loader is something like:</p>
  
  <pre><code>class GetData(torch.utils.data.Dataset):
  
      def __init__(self, data_path, target_path, transform=None):
          with open(data_path, 'rb') as train_pkl_file:
              data = pickle.load(train_pkl_file)
              self.data = torch.from_numpy(data).float()
          with open(target_path, 'rb') as target_pkl_file:
              targets = pickle.load(target_pkl_file)
              self.targets = torch.from_numpy(targets).float()
  
      def __getitem__(self, index):
          x = self.data[index]
          y = self.targets[index]
          return index, x, y
  
      def __len__(self):
          num_images = self.data.shape[0]
          return num_images
  </code></pre>
  
  <p>I have a npy list of files:</p>
  
  <pre><code>list1 = ['d1.npy', 'd2.npy','d3.npy']
  list1 = ['s1.npy', 's2.npy','s3.npy']
  </code></pre>
  
  <p>I have created a dataloader which gives the filenames</p>
  
  <pre><code>class MyDataset(torch.utils.data.Dataset):
      def __init__(self,flist):
          self.npy_list1 = flist1
          self.npy_list2 = flist2
  
      def __getitem__(self, idx):
          filename1 = self.npy_list1[idx]
          filename2 = self.npy_list2[idx]
          return filename1,filename2
  
      def __len__(self):
          return len(self.npy_list1)
  </code></pre>
  
  <p>And I itreate through them as follows:</p>
  
  <pre><code>for epoch in range(500):
      print('Epoch #%s' % epoch)
      model.train()
      loss_, elbo_, recon_ = [[] for _ in range(3)]
      running_loss = 0
  
      # FOR EVERY SMALL FILE
      print("Training: ")
  
      # TRAIN HERE
      my_dataset = MyDataset(npyList)
      for idx, (dynamic_file, static_file) in tqdm(enumerate(my_dataset)): 
           ...Do stuff ....
  </code></pre>
  
  <p>The above method works but i'm looking for more memory efficient solution. Note: I have huge amount of data > 200 GB so concatenating the numpy arrays into 1 file may not be the solution (due to RAM limitations). 
  Thanks in advance</p>