<h2>Suggested Solution</h2>

<p>Reusing the code from the <a href="https://github.com/r0nn13/conditional-dcgan-keras" rel="noreferrer">repository you shared</a>, here are some suggested modifications to train a classifier along your generator and discriminator (their architectures and other losses are left untouched):</p>

<pre class="lang-python prettyprint-override"><code>from keras import backend as K
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation, Flatten
from keras.layers.convolutional import Convolution2D, MaxPooling2D

def lenet_classifier_model(nb_classes):
    # Snipped by Fabien Tanc - https://www.kaggle.com/ftence/keras-cnn-inspired-by-lenet-5
    # Replace with your favorite classifier...
    model = Sequential()
    model.add(Convolution2D(12, 5, 5, activation='relu', input_shape=in_shape, init='he_normal'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Convolution2D(25, 5, 5, activation='relu', init='he_normal'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Flatten())
    model.add(Dense(180, activation='relu', init='he_normal'))
    model.add(Dropout(0.5))
    model.add(Dense(100, activation='relu', init='he_normal'))
    model.add(Dropout(0.5))
    model.add(Dense(nb_classes, activation='softmax', init='he_normal'))

def generator_containing_discriminator_and_classifier(generator, discriminator, classifier):
    inputs = Input((IN_CH, img_cols, img_rows))
    x_generator = generator(inputs)

    merged = merge([inputs, x_generator], mode='concat', concat_axis=1)
    discriminator.trainable = False
    x_discriminator = discriminator(merged)

    classifier.trainable = False
    x_classifier = classifier(x_generator)

    model = Model(input=inputs, output=[x_generator, x_discriminator, x_classifier])

    return model


def train(BATCH_SIZE):
    (X_train, Y_train, LABEL_train) = get_data('train')  # replace with your data here
    X_train = (X_train.astype(np.float32) - 127.5) / 127.5
    Y_train = (Y_train.astype(np.float32) - 127.5) / 127.5
    discriminator = discriminator_model()
    generator = generator_model()
    classifier = lenet_classifier_model(6)
    generator.summary()
    discriminator_and_classifier_on_generator = generator_containing_discriminator_and_classifier(
        generator, discriminator, classifier)
    d_optim = Adagrad(lr=0.005)
    g_optim = Adagrad(lr=0.005)
    generator.compile(loss='mse', optimizer="rmsprop")
    discriminator_and_classifier_on_generator.compile(
        loss=[generator_l1_loss, discriminator_on_generator_loss, "categorical_crossentropy"],
        optimizer="rmsprop")
    discriminator.trainable = True
    discriminator.compile(loss=discriminator_loss, optimizer="rmsprop")
    classifier.trainable = True
    classifier.compile(loss="categorical_crossentropy", optimizer="rmsprop")

    for epoch in range(100):
        print("Epoch is", epoch)
        print("Number of batches", int(X_train.shape[0] / BATCH_SIZE))
        for index in range(int(X_train.shape[0] / BATCH_SIZE)):
            image_batch = Y_train[index * BATCH_SIZE:(index + 1) * BATCH_SIZE]
            label_batch = LABEL_train[index * BATCH_SIZE:(index + 1) * BATCH_SIZE]  # replace with your data here

            generated_images = generator.predict(X_train[index * BATCH_SIZE:(index + 1) * BATCH_SIZE])
            if index % 20 == 0:
                image = combine_images(generated_images)
                image = image * 127.5 + 127.5
                image = np.swapaxes(image, 0, 2)
                cv2.imwrite(str(epoch) + "_" + str(index) + ".png", image)
                # Image.fromarray(image.astype(np.uint8)).save(str(epoch)+"_"+str(index)+".png")

            # Training D:
            real_pairs = np.concatenate((X_train[index * BATCH_SIZE:(index + 1) * BATCH_SIZE, :, :, :], image_batch),
                                        axis=1)
            fake_pairs = np.concatenate(
                (X_train[index * BATCH_SIZE:(index + 1) * BATCH_SIZE, :, :, :], generated_images), axis=1)
            X = np.concatenate((real_pairs, fake_pairs))
            y = np.zeros((20, 1, 64, 64))  # [1] * BATCH_SIZE + [0] * BATCH_SIZE
            d_loss = discriminator.train_on_batch(X, y)
            print("batch %d d_loss : %f" % (index, d_loss))
            discriminator.trainable = False

            # Training C:
            c_loss = classifier.train_on_batch(image_batch, label_batch)
            print("batch %d c_loss : %f" % (index, c_loss))
            classifier.trainable = False

            # Train G:
            g_loss = discriminator_and_classifier_on_generator.train_on_batch(
                X_train[index * BATCH_SIZE:(index + 1) * BATCH_SIZE, :, :, :], 
                [image_batch, np.ones((10, 1, 64, 64)), label_batch])
            discriminator.trainable = True
            classifier.trainable = True
            print("batch %d g_loss : %f" % (index, g_loss[1]))
            if index % 20 == 0:
                generator.save_weights('generator', True)
                discriminator.save_weights('discriminator', True)
</code></pre>

<hr>

<h2>Theoretical Details</h2>

<p>I believe there are some misunderstandings regarding how conditional GANs work and what is the discriminators role in such schemes.</p>

<h3>Role of the Discriminator</h3>

<p>In the min-max game which is GAN training [4], the discriminator <code>D</code> is playing against the generator <code>G</code> (the network you actually care about) so that under <code>D</code>'s scrutiny, <code>G</code> becomes better at outputting realistic results.</p>

<p>For that, <code>D</code> is trained to tell apart real samples from samples from <code>G</code> ; while <code>G</code> is trained to fool <code>D</code> by generating realistic results / results following the target distribution. </p>

<blockquote>
  <p><em>Note: in the case of conditional GANs, i.e. GANs mapping an input sample from one domain <code>A</code> (e.g. real picture) to another domain <code>B</code>
  (e.g. sketch), <code>D</code> is usually fed with the pairs of samples stacked
  together and has to discriminate "real" pairs (input sample from <code>A</code> +
  corresponding target sample from <code>B</code>) and "fake" pairs (input sample
  from <code>A</code> + corresponding output from <code>G</code>)</em> [1, 2]</p>
</blockquote>

<p>Training a conditional generator against <code>D</code> (as opposed to simply training <code>G</code> alone, with a L1/L2 loss only e.g. DAE) improves the sampling capability of <code>G</code>, forcing it to output crisp, realistic results instead of trying to average the distribution.</p>

<p>Even though discriminators can have multiple sub-networks to cover other tasks (see next paragraphs), <code>D</code> should keep at least one sub-network/output to cover its main task: <strong>telling real samples from generated ones apart</strong>. Asking <code>D</code> to regress further semantic information (e.g. classes) alongside may interfere with this main purpose.</p>

<blockquote>
  <p><em>Note: <code>D</code> output is often not a simple scalar / boolean. It is common to have a discriminator (e.g. PatchGAN [1, 2]) returning a matrix of
  probabilities, evaluating how realistic patches made from its input
  are.</em></p>
</blockquote>

<hr>

<h3>Conditional GANs</h3>

<p>Traditional GANs are trained in an unsupervised manner to generate realistic data (e.g. images) from a random noise vector as input. [4]</p>

<p>As previously mentioned, conditional GANs have further input <em>conditions</em>. Along/instead of the noise vector, they take for input a sample from a domain <code>A</code> and return a corresponding sample from a domain <code>B</code>. <code>A</code> can be a completely different modality, e.g. <code>B = sketch image</code> while <code>A = discrete label</code> ; <code>B = volumetric data</code> while <code>A = RGB image</code>, etc. [3]</p>

<p>Such GANs can also be conditioned by multiples inputs, e.g. <code>A = real image + discrete label</code> while <code>B = sketch image</code>. A famous work introducing such methods is <strong>InfoGAN</strong> [5]. It presents how to condition GANs on multiple continuous or discrete inputs (e.g. <code>A = digit class + writing type</code>, <code>B = handwritten digit image</code>), <strong>using a more advanced discriminator which has for 2nd task to force <code>G</code> to maximize the mutual-information between its conditioning inputs and its corresponding outputs</strong>.</p>

<hr>

<h3>Maximizing the Mutual Information for cGANs</h3>

<p>InfoGAN discriminator has 2 heads/sub-networks to cover its 2 tasks [5]:</p>

<ul>
<li>One head <code>D1</code> does the traditional real/generated discrimination -- <code>G</code> has to minimize this result, i.e. it has to fool <code>D1</code> so that it can't tell apart real form generated data;</li>
<li>Another head <code>D2</code> (also named <code>Q</code> network) tries to regress the input <code>A</code> information -- <code>G</code> has to maximize this result, i.e. it has to output data which "show" the requested semantic information (c.f. mutual-information maximization between <code>G</code> conditional inputs and its outputs).</li>
</ul>

<p>You can find a Keras implementation here for instance: <a href="https://github.com/eriklindernoren/Keras-GAN/tree/master/infogan" rel="noreferrer">https://github.com/eriklindernoren/Keras-GAN/tree/master/infogan</a>.</p>

<p>Several works are using similar schemes to improve control over what a GAN is generating, by using provided labels and maximizing the mutual information between these inputs and <code>G</code> outputs [6, 7]. The basic idea is always the same though:</p>

<ul>
<li>Train <code>G</code> to generate elements of domain <code>B</code>, given some inputs of domain(s) <code>A</code>;</li>
<li>Train <code>D</code> to discriminate "real"/"fake" results -- <code>G</code> has to minimize this;</li>
<li>Train <code>Q</code> (e.g. a classifier ; can share layers with <code>D</code>) to estimate the original <code>A</code> inputs from <code>B</code> samples -- <code>G</code> has to maximize this).</li>
</ul>

<hr>

<h3>Wrapping Up</h3>

<p>In your case, it seems you have the following training data:</p>

<ul>
<li>real images <code>Ia</code></li>
<li>corresponding sketch images <code>Ib</code></li>
<li>corresponding class labels <code>c</code></li>
</ul>

<p>And you want to train a generator <code>G</code> so that given an image <code>Ia</code> and its class label <code>c</code>, it outputs a proper sketch image <code>Ib'</code>. </p>

<p>All in all, that's a lot of information you have, and you can supervise your training both on the conditioned images and the conditioned labels...
Inspired from the aforementioned methods [1, 2, 5, 6, 7], here is a possible way of using all this information to train your conditional <code>G</code>:</p>

<strong>Network <code>G</code>:</strong>

<ul>
<li>Inputs: <code>Ia</code> + <code>c</code></li>
<li>Output: <code>Ib'</code></li>
<li>Architecture: up-to-you (e.g. U-Net, ResNet, ...)</li>
<li>Losses: L1/L2 loss between <code>Ib'</code> &amp; <code>Ib</code>, <code>-D</code> loss, <code>Q</code> loss</li>
</ul>

<strong>Network <code>D</code>:</strong>

<ul>
<li>Inputs: <code>Ia</code> + <code>Ib</code> (real pair), <code>Ia</code> + <code>Ib'</code> (fake pair)</li>
<li>Output: "fakeness" scalar/matrix</li>
<li>Architecture: up-to-you (e.g. PatchGAN)</li>
<li>Loss: cross-entropy on the "fakeness" estimation</li>
</ul>

<strong>Network <code>Q</code>:</strong>

<ul>
<li>Inputs: <code>Ib</code> (real sample, for training <code>Q</code>), <code>Ib'</code> (fake sample, when back-propagating through <code>G</code>)</li>
<li>Output: <code>c'</code> (estimated class)</li>
<li>Architecture: up-to-you (e.g. LeNet, ResNet, VGG, ...)</li>
<li>Loss: cross-entropy between <code>c</code> and <code>c'</code></li>
</ul>

<strong>Training Phase:</strong>

<ol>
<li>Train <code>D</code> on a batch of real pairs <code>Ia</code> + <code>Ib</code> then on a batch of fake pairs <code>Ia</code> + <code>Ib'</code>;</li>
<li>Train <code>Q</code> on a batch of real samples <code>Ib</code>;</li>
<li>Fix <code>D</code> and <code>Q</code> weights;</li>
<li>Train <code>G</code>, passing its generated outputs <code>Ib'</code> to <code>D</code> and <code>Q</code> to back-propagate through them.</li>
</ol>

<blockquote>
  <p><em>Note: this is a really rough architecture description. I'd recommend going through the literature ([1, 5, 6, 7] as a good start) to get
  more details and maybe a more elaborate solution.</em></p>
</blockquote>

<hr>

<h3>References</h3>

<ol>
<li>Isola, Phillip, et al. "Image-to-image translation with conditional adversarial networks." arXiv preprint (2017). <a href="http://openaccess.thecvf.com/content_cvpr_2017/papers/Isola_Image-To-Image_Translation_With_CVPR_2017_paper.pdf" rel="noreferrer">http://openaccess.thecvf.com/content_cvpr_2017/papers/Isola_Image-To-Image_Translation_With_CVPR_2017_paper.pdf</a></li>
<li>Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." arXiv preprint arXiv:1703.10593 (2017). <a href="http://openaccess.thecvf.com/content_ICCV_2017/papers/Zhu_Unpaired_Image-To-Image_Translation_ICCV_2017_paper.pdf" rel="noreferrer">http://openaccess.thecvf.com/content_ICCV_2017/papers/Zhu_Unpaired_Image-To-Image_Translation_ICCV_2017_paper.pdf</a></li>
<li>Mirza, Mehdi, and Simon Osindero. "Conditional generative adversarial nets." arXiv preprint arXiv:1411.1784 (2014). <a href="https://arxiv.org/pdf/1411.1784" rel="noreferrer">https://arxiv.org/pdf/1411.1784</a></li>
<li>Goodfellow, Ian, et al. "Generative adversarial nets." Advances in neural information processing systems. 2014. <a href="http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf" rel="noreferrer">http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf</a></li>
<li>Chen, Xi, et al. "Infogan: Interpretable representation learning by information maximizing generative adversarial nets." Advances in Neural Information Processing Systems. 2016. <a href="http://papers.nips.cc/paper/6399-infogan-interpretable-representation-learning-by-information-maximizing-generative-adversarial-nets.pdf" rel="noreferrer">http://papers.nips.cc/paper/6399-infogan-interpretable-representation-learning-by-information-maximizing-generative-adversarial-nets.pdf</a></li>
<li>Lee, Minhyeok, and Junhee Seok. "Controllable Generative Adversarial Network." arXiv preprint arXiv:1708.00598 (2017). <a href="https://arxiv.org/pdf/1708.00598.pdf" rel="noreferrer">https://arxiv.org/pdf/1708.00598.pdf</a></li>
<li>Odena, Augustus, Christopher Olah, and Jonathon Shlens. "Conditional image synthesis with auxiliary classifier gans." arXiv preprint arXiv:1610.09585 (2016). <a href="http://proceedings.mlr.press/v70/odena17a/odena17a.pdf" rel="noreferrer">http://proceedings.mlr.press/v70/odena17a/odena17a.pdf</a></li>
</ol>
Add class information to keras network
<p>I am trying to figure out how I will use the label information of my dataset with Generative Adversarial Networks. I am trying to use the following implementation of conditional GANs that <a href="https://github.com/r0nn13/conditional-dcgan-keras" rel="nofollow noreferrer">can be found here</a>. My dataset contains two different image domains (real objects and sketches) with common class information (chair, tree, orange etc). I opted for this implementation which only considers the two different domains as different "classes" for the correspondence (train samples <code>X</code> correspond to the real images while target samples <code>y</code> correspond to the sketch images).</p>

<p>Is there a way to modify my code and take into account the class information (chair, tree, etc.) in my whole architecture? I want actually my discriminator to predict whether or not my generated images from the generator belong to a specific class and not only whether they are real or not. As it is, with the current architecture, the system learns to create similar sketches in all cases. </p>

<p><strong>Update:</strong> The discriminator returns a tensor of size <code>1x7x7</code> then both <code>y_true</code> and <code>y_pred</code> are passed through a flatten layer before calculating the loss:</p>

<pre><code>def discriminator_loss(y_true, y_pred):
     BATCH_SIZE=100
     return K.mean(K.binary_crossentropy(K.flatten(y_pred), K.concatenate([K.ones_like(K.flatten(y_pred[:BATCH_SIZE,:,:,:])),K.zeros_like(K.flatten(y_pred[:BATCH_SIZE,:,:,:])) ]) ), axis=-1)
</code></pre>

<p>and the loss function of the discriminator over the generator:</p>

<pre><code>def discriminator_on_generator_loss(y_true,y_pred):
     BATCH_SIZE=100
     return K.mean(K.binary_crossentropy(K.flatten(y_pred), K.ones_like(K.flatten(y_pred))), axis=-1)
</code></pre>

<p>Furthremore, my modification of the discriminator model for output 1 layer:</p>

<pre><code>model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))
#model.add(Activation('sigmoid'))
</code></pre>

<p>Now the discriminator outputs 1 layer. How can I modify the above-mentioned loss functions correspondingly? Should I have 7 instead of 1, for the <code>n_classes = 6</code> + one class for predicting real and fake pairs?</p>