<jupyter_start><jupyter_text><jupyter_code>!pip install optuna
!pip install tf-nightly-2.0-preview
import numpy as np
import tensorflow as tf
import optuna
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output>Using TensorFlow backend.<jupyter_text>Simple Practice objective function$$f(x) = 2x^4 - 5x^2 + x$$We will find the minimum value of $f(x^*)$ and $x^*$ using optuna which is provided Prefferd Networks.  graph of $f(x)$<jupyter_code>x = np.linspace(-2, 2, 1000)
plt.plot(x, 2*x**4 - 5*x**2 + x)<jupyter_output><empty_output><jupyter_text>optimizationWe will minimize $f(x)$ where $x \in [-2, 2]$.<jupyter_code>def objective(trial):
    x = trial.suggest_uniform('x', -2, 2)
    return 2*x**4 - 5*x**2 + x
study = optuna.create_study()
study.optimize(objective, n_trials=30)<jupyter_output>[I 2019-01-16 11:56:08,652] Finished a trial resulted in value: -0.14883524366617035. Current best value is -0.14883524366617035 with parameters: {'x': 0.3082780234526088}.
[I 2019-01-16 11:56:08,656] Finished a trial resulted in value: -2.003942800211039. Current best value is -2.003942800211039 with parameters: {'x': 1.0040586437270083}.
[I 2019-01-16 11:56:08,661] Finished a trial resulted in value: -1.2372743017197956. Current best value is -2.003942800211039 with parameters: {'x': 1.0040586437270083}.
[I 2019-01-16 11:56:08,665] Finished a trial resulted in value: 0.6234074742607496. Current best value is -2.003942800211039 with parameters: {'x': 1.0040586437270083}.
[I 2019-01-16 11:56:08,669] Finished a trial resulted in value: 2.173214593940769. Current best value is -2.003942800211039 with parameters: {'x': 1.0040586437270083}.
[I 2019-01-16 11:56:08,673] Finished a trial resulted in value: -0.1639477716829485. Current best value is -2.003942800211039 with parameters: {'x': 1.[...]<jupyter_text>result<jupyter_code>print("minimum f(x) : ", study.best_value)
print("argmin f(x) : ", study.best_params)

plt.plot(x, 2*x**4 - 5*x**2 + x, "b")
plt.plot(study.best_params['x'], study.best_value, "ro")
plt.legend(["objective function f(x)", "minimized solution"])<jupyter_output>minimum f(x) :  -4.162215448144136
argmin f(x) :  {'x': -1.0645092747749707}<jupyter_text>optuna for TensorFlow Eager  Create modelIn this section, create_model funtion have an arg `trial` which is represente search spaces of 'num of layers' and 'dropout_rate'.<jupyter_code>def create_model(trial):
    # num of hidden layer
    n_layers = trial.suggest_int('n_layers', 1, 3)
    
    # dropout_rate
    dropout_rate = trial.suggest_uniform('dropout_rate', 0.0, 0.5)
    
    layers = []
    for i in range(n_layers):
        layers.append(
            tf.keras.layers.Conv2D(filters=(i+1)*16, 
                                   kernel_size=3, 
                                   padding="same",
                                   activation="relu")
        )
        layers.append(
            tf.keras.layers.MaxPool2D(pool_size=2, padding="same")
        )
        layers.append(
            tf.keras.layers.Dropout(rate=dropout_rate)
        )
    
    layers.append(tf.keras.layers.Flatten())
    layers.append(tf.keras.layers.Dense(128, activation="relu"))
    layers.append(tf.keras.layers.Dense(10, activation="softmax"))
    
    return tf.keras.Sequential(layers)<jupyter_output><empty_output><jupyter_text>create optimizerThen, we create optimzier.<jupyter_code>def create_optimizer(trial):
    # Loguniform parameter
    learning_rate = trial.suggest_loguniform('learning_rate', 1e-5, 1e-2)
    
    optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)
    
    return optimizer<jupyter_output><empty_output><jupyter_text>create trainerIn this section we will create trainer which is simply training loop. We will use keras training API.<jupyter_code>def trainer(trial, x_train, y_train):
    
    batch_size=trial.suggest_categorical('batch_size', [256, 512, 1024])
    
    model = create_model(trial)
    optimizer = create_optimizer(trial)
    
    model.compile(optimizer=optimizer,
                  loss='categorical_crossentropy',
                  metrics=["accuracy"])

    model.fit(x=x_train,
              y=y_train,
              batch_size=batch_size,
              epochs=5,
              )
    
    model.fit = tf.function(model.fit)

    return model<jupyter_output><empty_output><jupyter_text>objectiveWe create objective function. Note that optuna minimize objective function, so when we create objective function, this is defined "better equal to smaller". Therefore, we define objective function "`1 - accuracy`".<jupyter_code>def objective(trial):
    
    (x_train, y_train),(x_test, y_test) = tf.keras.datasets.mnist.load_data()
    x_train, x_test = x_train / 255.0, x_test / 255.0
    x_train = x_train.reshape(-1, 28, 28, 1)
    x_test = x_test.reshape(-1, 28, 28, 1)

    x_train_ = tf.convert_to_tensor(x_train, dtype=tf.float32)
    y_train_ = tf.reshape(tf.one_hot(y_train, 10), (-1, 10))
    x_test_ = tf.convert_to_tensor(x_test, dtype=tf.float32)
    y_test_ = tf.reshape(tf.one_hot(y_test, 10), (-1, 10))


    model = trainer(trial, x_train_, y_train_)
    
    evaluate = model.evaluate(x=x_test_, y=y_test_)
    return 1 - evaluate[1]
study = optuna.create_study()
study.optimize(objective, n_trials=10)
study.trials
print("best params: ", study.best_params)
print("best test accuracy: ", 1 - study.best_value)<jupyter_output><empty_output>