{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/5985","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/5985/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/5985/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/5985/events","html_url":"https://github.com/tensorflow/tensorflow/issues/5985","id":192612459,"number":5985,"title":"None gradient from GRU","user":{"login":"petrbel","id":461519,"avatar_url":"https://avatars.githubusercontent.com/u/461519?v=3","gravatar_id":"","url":"https://api.github.com/users/petrbel","html_url":"https://github.com/petrbel","followers_url":"https://api.github.com/users/petrbel/followers","following_url":"https://api.github.com/users/petrbel/following{/other_user}","gists_url":"https://api.github.com/users/petrbel/gists{/gist_id}","starred_url":"https://api.github.com/users/petrbel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petrbel/subscriptions","organizations_url":"https://api.github.com/users/petrbel/orgs","repos_url":"https://api.github.com/users/petrbel/repos","events_url":"https://api.github.com/users/petrbel/events{/privacy}","received_events_url":"https://api.github.com/users/petrbel/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"ebrevdo","id":1794715,"avatar_url":"https://avatars.githubusercontent.com/u/1794715?v=3","gravatar_id":"","url":"https://api.github.com/users/ebrevdo","html_url":"https://github.com/ebrevdo","followers_url":"https://api.github.com/users/ebrevdo/followers","following_url":"https://api.github.com/users/ebrevdo/following{/other_user}","gists_url":"https://api.github.com/users/ebrevdo/gists{/gist_id}","starred_url":"https://api.github.com/users/ebrevdo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ebrevdo/subscriptions","organizations_url":"https://api.github.com/users/ebrevdo/orgs","repos_url":"https://api.github.com/users/ebrevdo/repos","events_url":"https://api.github.com/users/ebrevdo/events{/privacy}","received_events_url":"https://api.github.com/users/ebrevdo/received_events","type":"User","site_admin":false},"assignees":[{"login":"ebrevdo","id":1794715,"avatar_url":"https://avatars.githubusercontent.com/u/1794715?v=3","gravatar_id":"","url":"https://api.github.com/users/ebrevdo","html_url":"https://github.com/ebrevdo","followers_url":"https://api.github.com/users/ebrevdo/followers","following_url":"https://api.github.com/users/ebrevdo/following{/other_user}","gists_url":"https://api.github.com/users/ebrevdo/gists{/gist_id}","starred_url":"https://api.github.com/users/ebrevdo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ebrevdo/subscriptions","organizations_url":"https://api.github.com/users/ebrevdo/orgs","repos_url":"https://api.github.com/users/ebrevdo/repos","events_url":"https://api.github.com/users/ebrevdo/events{/privacy}","received_events_url":"https://api.github.com/users/ebrevdo/received_events","type":"User","site_admin":false}],"milestone":null,"comments":6,"created_at":"2016-11-30T16:28:13Z","updated_at":"2016-12-15T14:01:01Z","closed_at":null,"body":"Hello everybody,\r\n\r\n**Introduction:**\r\nI am trying to reproduce the work of Szegedy et al: [Intriguing properties of neural networks](https://arxiv.org/abs/1312.6199) and Dezfooli et al: [Universal adversarial perturbations](https://arxiv.org/abs/1610.08401). What they do, briefly, is that they take the gradient `g` of the network with respect to the input image `x`, then slightly modify the input image `x` \"against\" the obtained gradient, i.e. `x-0.001g`. The modified images are consequently considered as another network input, hence new gradients `g'` might be computed and obtained. The result gradients are manually set to the average of those two set obtains gradients, i.e. `g/2 + g'/2`. I hope I explained that clearly.\r\n\r\nI successfully implemented it for images in TF 0.10 (working on 0.12RC as well). Now I want to do something similar with recurrent networks (sentiment analysis task). So I have a input sequence of tokens (numbers), a variable `embeddings` which is used by `tf.nn.embedding_lookup`. After the lookup, the sequence of proper embeddings is traversed by GRU (dynamically). It's last state is considered as the representation of the sequence. Then a MLP with a single hidden layer and ReLU activation is applied, resulting in 2 neurons representing the positive and negative logits respectively. The loss is traditional categorical cross entropy.\r\n\r\n**Bug**\r\nSimilarly to the papers mentioned above, I obtain gradients of the embedding, and modify the current embeddings slightly \"against\" the obtained gradient. Than I compute the new loss and _new gradients (here is the problem!)_ and update the network by their mean.\r\n\r\nThe problem is that the returned _new gradients_ are all `None`s. I originally worked with TF 0.10, however, I got an error explaining that second order derivatives are impossible to obtain from scan. I upgraded to recently released TF 0.12RC which doesn't throw this error but returns `None`s instead.\r\n\r\n**Related Issues and SOs**\r\n#783 is somewhat similar, but not the same. [This question](http://stackoverflow.com/questions/36874522/tensorflow-gradients) deals with very the same problem  but the solution isn't sufficient as replacing the `None` value with zero is really worthless in my case since I need to modify the variable against the gradient.\r\n\r\n**Minimal Not-Working Example**\r\nI am sorry the code is so long. It is partially caused by the fact that the problem is non-trivial and requires few lines and partially by the amount of comments I tried to write in order to make the code more readable. All platform/version information is stored at beginning of the file.\r\n\r\nThe `main()` function constructs two `SimpleSentiment`s as the models that are trained. Each one is trained for few epochs and batches. First instance is without the adversarial, hence we perform no trick regarding modifying the gradients (for a sanity check). The second uses the complicated `if` branch in the constructor and obtains `None`s as described above.\r\n\r\nI believe this is a bug and not my mistake, however, it can't be ruled out as I am not as experienced with TF as I'd like to be.\r\n\r\n```python\r\n# #!/usr/bin/env python3\r\n\r\n# python 3.5 (anaconda)\r\n# TF 0.12 RC from https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.12.0rc0-cp35-cp35m-linux_x86_64.whl\r\n# $ python -c \"import tensorflow; print(tensorflow.__version__)\"\r\n#  0.12.0-rc0\r\n\r\n# OS: RedHat 7.2\r\n# CPU version only, no GPU, no CUDA, no CUDNN\r\n\r\nimport numpy as np\r\nimport tensorflow as tf\r\nfrom tensorflow.python.ops.rnn_cell import GRUCell\r\n\r\n\r\nBATCH = 5  # batch size\r\nMAX_LEN = 10  # max length of the sequence\r\nMLP_HIDDEN_DIM = 128  # number of hidden neurons in the MLP\r\nEMBEDDING_DIM = 300  # embedding dimension\r\nVOCAB_SIZE = 8  # vocabulary size\r\n\r\nTHREADS = 4  # number of threads to be used\r\nSTD=0.001  # standard deviation of ariable initializers\r\n\r\n\r\nclass SimpleSentiment:\r\n    def __init__(self, adversarial=False, device='/cpu:0'):\r\n\r\n        self.embeddings = tf.get_variable('word_embeddings',\r\n                                          initializer=tf.random_uniform([VOCAB_SIZE, EMBEDDING_DIM], -1.0, 1.0))\r\n\r\n        with tf.variable_scope('sentiment') as scope:\r\n            with tf.device(device):\r\n                # Inputs\r\n                self.text = tf.placeholder(tf.int32, [BATCH, MAX_LEN])\r\n                self.text_len = tf.placeholder(tf.int32, [BATCH])\r\n                self.sentiment = tf.placeholder(tf.int32, [BATCH])\r\n\r\n                # Normal loss\r\n                loss_normal = self._loss(self.text, self.text_len, self.sentiment)\r\n\r\n                # Define the optimizer\r\n                # Note: I've tried multiple of optimizers and none helped\r\n                optimizer = tf.train.AdamOptimizer(learning_rate=0.0005)\r\n\r\n                if adversarial:  # Define adversarial loss\r\n                    # Let's acces already defined variables\r\n                    scope.reuse_variables()\r\n\r\n                    # Gradients of all variable (according to normal loss)\r\n                    gradients = optimizer.compute_gradients(loss_normal)\r\n                    print(len(gradients), gradients)\r\n\r\n                    # gradients of the embeddings\r\n                    emb_gradient = optimizer.compute_gradients(loss_normal, [self.embeddings])[0][0]\r\n\r\n                    # this how much we want to shift the embeddings, i.e. going \"against\" the gradient\r\n                    delta = 0.001*tf.sign(emb_gradient)\r\n\r\n                    # let's compute the loss once again but this time we add the delta to the embeddings\r\n                    loss_adversarial = self._loss(self.text, self.text_len, self.sentiment, delta)\r\n\r\n                    # new gradient of the whole computational graph\r\n                    adversarial_gradients = optimizer.compute_gradients(loss_adversarial)\r\n                    print(len(adversarial_gradients), adversarial_gradients)  # everything is None!\r\n\r\n                    # Now we compute an average of old and new gradients\r\n                    new_gradients = [((g + ag)/2, vg) for ((g, vg), (ag, avg)) in zip(gradients, adversarial_gradients)]\r\n\r\n                    # and apply them\r\n                    self.training = optimizer.apply_gradients(new_gradients)\r\n\r\n                    # Btw this doesn't work either\r\n                    # self.training = optimizer.apply_gradients(adversarial_gradients)\r\n\r\n                    self.loss_final = (loss_normal + loss_adversarial) / 2\r\n\r\n                else:  # Normal loss\r\n                    # simply minimize according to the gradients\r\n                    self.loss_final = loss_normal\r\n                    self.training = optimizer.minimize(loss_normal)\r\n\r\n                # Create the session\r\n                self.session = tf.Session(config=tf.ConfigProto(inter_op_parallelism_threads=THREADS,\r\n                                                                intra_op_parallelism_threads=THREADS,\r\n                                                                allow_soft_placement=True))\r\n\r\n                # init everything (still deprecated way)\r\n                self.session.run(tf.initialize_all_variables())\r\n\r\n    def _loss(self, text, text_len, sentiment, emb_delta=0):\r\n        # use embedding\r\n        # note that emb_delta is zero as long as adversarial=False\r\n        # if adversarial=False then each embedding is shifted by appropriate emb_delta\r\n        text = tf.nn.embedding_lookup(self.embeddings + emb_delta, text)\r\n\r\n        # run gru\r\n        gru_cell = GRUCell(MLP_HIDDEN_DIM)\r\n        outputs, state = tf.nn.dynamic_rnn(cell=gru_cell,\r\n                                           inputs=text,\r\n                                           sequence_length=text_len,\r\n                                           dtype=tf.float32)\r\n\r\n        # define MLP\r\n        W1 = tf.get_variable(name='MLP_W1',\r\n                             shape=[state.get_shape()[1], MLP_HIDDEN_DIM],\r\n                             initializer=tf.random_normal_initializer(mean=0, stddev=STD))\r\n        W2 = tf.get_variable(name='MLP_W2',\r\n                             shape=[MLP_HIDDEN_DIM, 2],\r\n                             initializer=tf.random_normal_initializer(mean=0, stddev=STD))\r\n        h1 = tf.get_variable(name='MLP_h1',\r\n                             shape=[MLP_HIDDEN_DIM],\r\n                             initializer=tf.random_normal_initializer(mean=0, stddev=STD))\r\n        h2 = tf.get_variable(name='MLP_h2',\r\n                             shape=[2],\r\n                             initializer=tf.random_normal_initializer(mean=0, stddev=STD))\r\n\r\n        # apply MLP of the last GRU state\r\n        after_first_layer = tf.nn.relu(tf.matmul(state, W1) + h1)\r\n        logits = tf.matmul(after_first_layer, W2) + h2\r\n\r\n        # compute loss via categorial cross entropy\r\n        loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits, sentiment))\r\n        return loss\r\n\r\n\r\ndef main():\r\n    for adversarial in [False, True]:\r\n        print('\\n=================================')\r\n        if adversarial:\r\n            print('Using the adversarial loss')\r\n        else:\r\n            print('Using the standard loss')\r\n\r\n        net = SimpleSentiment(adversarial=adversarial)\r\n\r\n        for epoch in range(5):\r\n            print('Epoch {}'.format(epoch))\r\n\r\n            for batch in range(3):\r\n                _, loss_final = net.session.run([net.training, net.loss_final],\r\n                                                {net.text: np.array([[3, 1, 1, 2, 1, 0, 0, 0, 0, 0],\r\n                                                                     [3, 4, 1, 2, 1, 4, 4, 0, 0, 0],\r\n                                                                     [1, 1, 1, 2, 0, 0, 0, 0, 0, 0],\r\n                                                                     [3, 3, 3, 2, 1, 7, 0, 0, 0, 0],\r\n                                                                     [7, 1, 5, 2, 4, 2, 2, 2, 1, 7]], dtype='int32'),\r\n                                                 net.text_len: np.array([5, 7, 4, 6, 10], dtype='int32'),\r\n                                                 net.sentiment: np.array([0, 0, 1, 1, 0], dtype='int32')})\r\n\r\n                print('\\tBatch {}: {}'.format(batch, loss_final))\r\n\r\nif __name__ == '__main__':\r\n    main()\r\n```\r\n\r\nThanks for help\r\nPetr"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/267333975","html_url":"https://github.com/tensorflow/tensorflow/issues/5985#issuecomment-267333975","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/5985","id":267333975,"user":{"login":"petrbel","id":461519,"avatar_url":"https://avatars.githubusercontent.com/u/461519?v=3","gravatar_id":"","url":"https://api.github.com/users/petrbel","html_url":"https://github.com/petrbel","followers_url":"https://api.github.com/users/petrbel/followers","following_url":"https://api.github.com/users/petrbel/following{/other_user}","gists_url":"https://api.github.com/users/petrbel/gists{/gist_id}","starred_url":"https://api.github.com/users/petrbel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petrbel/subscriptions","organizations_url":"https://api.github.com/users/petrbel/orgs","repos_url":"https://api.github.com/users/petrbel/repos","events_url":"https://api.github.com/users/petrbel/events{/privacy}","received_events_url":"https://api.github.com/users/petrbel/received_events","type":"User","site_admin":false},"created_at":"2016-12-15T14:01:01Z","updated_at":"2016-12-15T14:01:01Z","body":"@vladfi1 It is as you said, it runs fine with 0.10. However I still think it is an issue as it doesn't work 0.12.\r\nPetr"}}
{"actor":{"display_login":"petrbel"}}