<h3>TL;DR</h3>
<p>Your input data is not normalized.</p>
<ol>
<li>use <code>x_data = (x_data - x_data.mean()) / x_data.std() </code></li>
<li>increase the learning rate <code>optimizer = torch.optim.Adam(model.parameters(), lr=0.01)</code></li>
</ol>
<p>You'll get<br />
<a href="https://i.stack.imgur.com/vt4VK.png" rel="noreferrer"><img src="https://i.stack.imgur.com/vt4VK.png" alt="enter image description here" /></a></p>
<p>convergence in only 1000 iterations.</p>
<h3>More details</h3>
<p>The key difference between the two examples you have is that the data <code>x</code> in the first example is centered around (0, 0) and has very low variance.<br />
On the other hand, the data in the second example is centered around 92 and has relatively large variance.</p>
<p>This initial bias in the data is not taken into account when you randomly <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/linear.html#Linear" rel="noreferrer">initialize the weights</a> which is done based on the assumption that the inputs are  roughly normally distributed around <em>zero</em>.<br />
It is almost impossible for the optimization process to compensate for this gross deviation - thus the model gets stuck in a sub-optimal solution.</p>
<p>Once you normalize the inputs, by subtracting the mean and dividing by the std, the optimization process becomes stable again and rapidly converges to a good solution.</p>
<p>For more details about input normalization and weights initialization, you can read section 2.2 in <em>He et al</em> <a href="https://arxiv.org/abs/1502.01852" rel="noreferrer"><strong>Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification</strong></a> (ICCV 2015).</p>
<h3>What if I cannot normalize the data?</h3>
<p>If, for some reason, you cannot compute mean and std data in advance, you can still use <a href="https://pytorch.org/docs/stable/nn.html#batchnorm1d" rel="noreferrer"><code>nn.BatchNorm1d</code></a> to estimate and normalize the data as part of the training process.  For example</p>
<pre class="lang-py prettyprint-override"><code>class Model(nn.Module):
    def __init__(self, input_size, H1, output_size):
        super().__init__()
        self.bn = nn.BatchNorm1d(input_size)  # adding batchnorm
        self.linear = nn.Linear(input_size, H1)
        self.linear2 = nn.Linear(H1, output_size)
    
    def forward(self, x):
        x = torch.sigmoid(self.linear(self.bn(x)))  # batchnorm the input x
        x = torch.sigmoid(self.linear2(x))
        return x
</code></pre>
<p>This modification <em>without</em> any change to the input data, yields similar convergance after only 1000 epochs:<br />
<a href="https://i.stack.imgur.com/Q9XRw.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Q9XRw.png" alt="enter image description here" /></a></p>
<h3>A minor comment</h3>
<p>For numerical stability, it is better to use <a href="https://pytorch.org/docs/stable/nn.html#bcewithlogitsloss" rel="noreferrer"><code>nn.BCEWithLogitsLoss</code></a> instead of <a href="https://pytorch.org/docs/stable/nn.html#bceloss" rel="noreferrer"><code>nn.BCELoss</code></a>. For this end, you need to remove the <code>torch.sigmoid</code> from the <code>forward()</code> output, the <code>sigmoid</code> will be computed inside the loss.<br />
See, for example, <a href="https://stackoverflow.com/q/57253841/1714410">this thread</a> regarding the related sigmoid + cross entropy loss for binary predictions.</p>
PyTorch Binary Classification - same network structure, 'simpler' data, but worse performance?
<p>To get to grips with PyTorch (and deep learning in general) I started by working through some basic classification examples. One such example was classifying a non-linear dataset created using sklearn (full code available as notebook <a href="https://github.com/philipobrien/colab-notebooks/blob/master/Non_Linear_Data_Classification.ipynb" rel="noreferrer">here</a>)</p>

<pre><code>n_pts = 500
X, y = datasets.make_circles(n_samples=n_pts, random_state=123, noise=0.1, factor=0.2)
x_data = torch.FloatTensor(X)
y_data = torch.FloatTensor(y.reshape(500, 1))
</code></pre>

<p><a href="https://i.stack.imgur.com/w20Tb.png" rel="noreferrer"><img src="https://i.stack.imgur.com/w20Tb.png" alt="enter image description here"></a></p>

<p>This is then accurately classified using a pretty basic neural net</p>

<pre><code>class Model(nn.Module):
    def __init__(self, input_size, H1, output_size):
        super().__init__()
        self.linear = nn.Linear(input_size, H1)
        self.linear2 = nn.Linear(H1, output_size)

    def forward(self, x):
        x = torch.sigmoid(self.linear(x))
        x = torch.sigmoid(self.linear2(x))
        return x

    def predict(self, x):
        pred = self.forward(x)
        if pred &gt;= 0.5:
            return 1
        else:
            return 0
</code></pre>

<p>As I have an interest in health data I then decided to try and use the same network structure to classify some a basic real-world dataset. I took heart rate data for one patient from <a href="https://physionet.org/physiobank/database/bidmc/" rel="noreferrer">here</a>, and altered it so all values > 91 would be labelled as anomalies (e.g. a <code>1</code> and everything &lt;= 91 labelled a <code>0</code>). This is completely arbitrary, but I just wanted to see how the classification would work. The complete notebook for this example is <a href="https://github.com/philipobrien/colab-notebooks/blob/master/Basic_Heart_rate_Classification.ipynb" rel="noreferrer">here</a>.</p>

<p><a href="https://i.stack.imgur.com/6izzI.png" rel="noreferrer"><img src="https://i.stack.imgur.com/6izzI.png" alt="enter image description here"></a></p>

<p>What is not intuitive to me is why the first example reaches <strong>a loss of 0.0016 after 1,000 epochs</strong>, whereas the second example only reaches <strong>a loss of 0.4296 after 10,000 epochs</strong></p>

<p><a href="https://i.stack.imgur.com/rDag4.png" rel="noreferrer"><img src="https://i.stack.imgur.com/rDag4.png" alt="Training Loss for Example 1"></a></p>

<p><a href="https://i.stack.imgur.com/PMubT.png" rel="noreferrer"><img src="https://i.stack.imgur.com/PMubT.png" alt="Training Loss for Heart Rate Example"></a></p>

<p>Perhaps I am being naive in thinking that the heart rate example would be much easier to classify. Any insights to help me understand why this is not what I am seeing would be great!</p>