<jupyter_start><jupyter_text>Monte Carlo DropoutBased on ***Dropout as a Bayesian Approximation:Representing Model Uncertainty in Deep Learning*** (2015) by <NAME> and <NAME> (https://arxiv.org/pdf/1506.02142.pdf)  Data<jupyter_code># Create data
X = np.arange(0, 1000, .5)
y = X**2 + np.random.randn(X.shape[0]) * .1 * (1 + X)**2
X.shape, y.shape
plt.plot(X, y)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.2)
# Scale X
scaler = StandardScaler()

X_train = scaler.fit_transform(X_train.reshape(-1, 1))
X_test = scaler.transform(X_test.reshape(-1, 1))<jupyter_output><empty_output><jupyter_text>Model<jupyter_code># Define the model
model = keras.Sequential([
    keras.layers.Dense(64, input_shape=(1,), activation='selu', kernel_initializer='lecun_normal'),
    keras.layers.Dropout(.2),
    keras.layers.Dense(128, activation='selu', kernel_initializer='lecun_normal'),
    keras.layers.Dropout(.2),
    keras.layers.Dense(64, activation='selu', kernel_initializer='lecun_normal'),
    keras.layers.Dropout(.2),
    keras.layers.Dense(16, activation='selu', kernel_initializer='lecun_normal'),
    keras.layers.Dense(1),    
])
model.summary()
model.compile(optimizer='adam', loss='mean_squared_error', metrics=['mse'])
history = model.fit(X_train, y_train, 
                    epochs=300,
                    validation_data=(X_test, y_test), 
                    verbose=0)
plt.plot(history.history['loss'], label='loss')
plt.plot(history.history['val_loss'], label='val_loss')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Predict  Point prediction<jupyter_code>preds = model.predict(X_test)
plt.scatter(X_test, y_test, label='actuals', alpha=.5)
plt.scatter(X_test, preds, label='preds', alpha=.5)
plt.legend()
plt.title('Point prediction')
plt.show()<jupyter_output><empty_output><jupyter_text>MC droupout prediction<jupyter_code># Set n MC samples
N_SAMPLES = 1000

# predict
mc_preds_ = np.stack([model(X_test, training=True) for i in range(N_SAMPLES)])
mc_preds = mc_preds_.mean(axis=0)
mc_preds_.shape
# Plot MC samples in yellow
for sample in mc_preds_:
    plt.scatter(X_test, sample, alpha=.01, color='yellow')
    
# Plot actuals
plt.scatter(X_test, y_test, label='actuals', alpha=.5)

# Plot mean of MC samples
plt.scatter(X_test, mc_preds, label='preds', alpha=.5)
plt.legend()
plt.title('MC dropout prediction')
plt.show()<jupyter_output><empty_output>