title: How to divide the data set into train, test and validation purpose
body: <p>My data frame consist of 44.2 billions rows. I want to split it into 3 sets (train,test and validation). So that no points are overlapped.</p>
  
  <p>I have done (1st process) - </p>
  
  <pre><code>train, valid, test = np.split(df.sample(frac=1), [int(.8*len(df)), int(.95*len(df))])
  </code></pre>
  
  <p>Checking any value is present or not - </p>
  
  <pre><code>len(valid[valid.id.isin(test.id)])
  len(train[train.id.isin(test.id)])
  </code></pre>
  
  <p>2nd process -</p>
  
  <pre><code>train = df[(np.random.rand(len(df)) &lt; 0.8)]
  valid = df[(np.random.rand(len(df)) &gt; 0.8) &amp; (np.random.rand(len(df)) &lt; 0.95)]
  test = df[(np.random.rand(len(df)) &gt; 0.95) &amp; (np.random.rand(len(df)) &lt; 1)]
  </code></pre>
  
  <p>But as per my understanding the above two methods are not perfect. Can anybody help me</p>