title: How to split the Cora dataset to train a GCN model only on training part?
body: <p>I am training a GCN (Graph Convolutional Network) on Cora dataset.</p>
  <p>The Cora dataset has the following attributes:</p>
  <pre><code>Number of graphs: 1
  Number of features: 1433
  Number of classes: 7
  Number of nodes: 2708
  Number of edges: 10556
  Number of training nodes: 140
  Training node label rate: 0.05
  Is undirected: True
  
  Data(edge_index=[2, 10556], test_mask=[2708], train_mask=[2708], val_mask=[2708], x=[2708, 1433], y=[2708])
  </code></pre>
  <p>Since my code is very long, I only put the relevent parts of my code here. Firstly, I split the Cora dataset as follows:</p>
  <pre><code>def to_mask(index, size):
      mask = torch.zeros(size, dtype=torch.bool)
      mask[index] = 1
      return mask
  
  def cora_splits(data, num_classes):
      indices = []
  
      for i in range(num_classes):
          # returns all indices of the elements = i from data.y tensor
          index = (data.y == i).nonzero().view(-1)
  
          # returns a random permutation of integers from 0 to index.size(0).
          index = index[torch.randperm(index.size(0))]
  
          # indices is a list of tensors and it has a length of 7
          indices.append(index)
  
      # select 20 nodes from each class for training
      train_index = torch.cat([i[:20] for i in indices], dim=0)
  
      rest_index = torch.cat([i[20:] for i in indices], dim=0)
      rest_index = rest_index[torch.randperm(len(rest_index))]
  
      data.train_mask = to_mask(train_index, size=data.num_nodes)
      data.val_mask = to_mask(rest_index[:500], size=data.num_nodes)
      data.test_mask = to_mask(rest_index[500:], size=data.num_nodes)
  
      return data
  </code></pre>
  <p>The <code>train</code> is as follows (taken from <a href="https://github.com/tkipf/pygcn/blob/master/pygcn/train.py" rel="nofollow noreferrer">here</a> with few modifications):</p>
  <pre><code>
  def train(model, optimizer, data, epoch):
      t = time.time()
      model.train()
      optimizer.zero_grad()
      output = model(data)
      loss_train = F.nll_loss(output[data.train_mask], data.y[data.train_mask])
      acc_train = accuracy(output[data.train_mask], data.y[data.train_mask])
      loss_train.backward()
      optimizer.step()
  
      loss_val = F.nll_loss(output[data.val_mask], data.y[data.val_mask])
      acc_val = accuracy(output[data.val_mask], data.y[data.val_mask])
  
  def accuracy(output, labels):
      preds = output.max(1)[1].type_as(labels)
      correct = preds.eq(labels).double()
      correct = correct.sum()
      return correct / len(labels)
  </code></pre>
  <p>When I ran my code with 200 epochs in 10 runs I gained:</p>
  <pre><code>tensor([0.7690, 0.8030, 0.8530, 0.8760, 0.8600, 0.8550, 0.8850, 0.8580, 0.8940, 0.8830])
  
  Val Loss: 0.5974, Test Accuracy: 0.854 ± 0.039
  </code></pre>
  <p>where each value in the tensor belongs to the model accurracy of each run and the mean accuracy of all those 10 runs is 0.854 with std ± 0.039.</p>
  <p>As it can be observed, the accuracy from the first run to the 10th one is increasing substantially. Therefore, I think the model is overfitting. One reason of overfitting is that in the code, the test data has been seen by the model in the training time since in the <code>train</code> function, there is a line <code>output = model(data)</code> so the model is trained over the whole data. What I intend to do is to train my model only on a part of the data (something similar to <code>data[data.train_mask]</code>) but the problem is I cannot pass <code>data[data.train_mask]</code>, due to the <code>forward</code> function of the <code>GCN</code> model (from <a href="https://github.com/rusty1s/pytorch_geometric/blob/master/benchmark/kernel/gcn.py" rel="nofollow noreferrer">this repository</a>):</p>
  <pre><code>def forward(self, data):
          x, edge_index = data.x, data.edge_index
          x = F.relu(self.conv1(x, edge_index))
          for conv in self.convs:
              x = F.relu(conv(x, edge_index))
          x = F.relu(self.lin1(x))
          x = F.dropout(x, p=0.5, training=self.training)
          x = self.lin2(x)
          return F.log_softmax(x, dim=-1)
  </code></pre>
  <p>If I pass <code>data[data.train_mask]</code> to the GCN model, then in the above <code>forward</code> function in line <code>x, edge_index = data.x, data.edge_index</code>, <code>x</code> and <code>edge_index</code> cannot be retrieved from <code>data[data.train_mask]</code>. Therefore, I need to find a way to split the Cora dataset in a way that I can pass a specefic part of it with the nodes, edge-index and other attributes to the model. My question is how to do it?</p>
  <p>Also, any suggestion about k-fold cross validation is much appreciated.</p>