{"action":"created","issue":{"url":"https://api.github.com/repos/DeepRNN/image_captioning/issues/28","repository_url":"https://api.github.com/repos/DeepRNN/image_captioning","labels_url":"https://api.github.com/repos/DeepRNN/image_captioning/issues/28/labels{/name}","comments_url":"https://api.github.com/repos/DeepRNN/image_captioning/issues/28/comments","events_url":"https://api.github.com/repos/DeepRNN/image_captioning/issues/28/events","html_url":"https://github.com/DeepRNN/image_captioning/issues/28","id":322660470,"node_id":"MDU6SXNzdWUzMjI2NjA0NzA=","number":28,"title":"Attention Formula dismatch with the implementation","user":{"login":"HrsPythonix","id":16586180,"node_id":"MDQ6VXNlcjE2NTg2MTgw","avatar_url":"https://avatars2.githubusercontent.com/u/16586180?v=4","gravatar_id":"","url":"https://api.github.com/users/HrsPythonix","html_url":"https://github.com/HrsPythonix","followers_url":"https://api.github.com/users/HrsPythonix/followers","following_url":"https://api.github.com/users/HrsPythonix/following{/other_user}","gists_url":"https://api.github.com/users/HrsPythonix/gists{/gist_id}","starred_url":"https://api.github.com/users/HrsPythonix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HrsPythonix/subscriptions","organizations_url":"https://api.github.com/users/HrsPythonix/orgs","repos_url":"https://api.github.com/users/HrsPythonix/repos","events_url":"https://api.github.com/users/HrsPythonix/events{/privacy}","received_events_url":"https://api.github.com/users/HrsPythonix/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-05-14T03:21:36Z","updated_at":"2018-07-30T15:14:48Z","closed_at":null,"author_association":"NONE","body":"I found that in the paper， the formula of MLP attention is usually desribed as below：\r\n\r\n![image](https://user-images.githubusercontent.com/16586180/39976766-fd23c30e-5767-11e8-9a16-9d0238512c82.png)\r\n\r\nwhere vi is i-th feature map，ht is the output of lstm.\r\n\r\nBut in the code, the implementation goes like this:\r\n\r\n```\r\n    def attend(self, contexts, output):\r\n        \"\"\" Attention Mechanism. \"\"\"\r\n        config = self.config\r\n        reshaped_contexts = tf.reshape(contexts, [-1, self.dim_ctx])\r\n        reshaped_contexts = self.nn.dropout(reshaped_contexts)\r\n        output = self.nn.dropout(output)\r\n        if config.num_attend_layers == 1:\r\n            # use 1 fc layer to attend\r\n            logits1 = self.nn.dense(reshaped_contexts,\r\n                                    units = 1,\r\n                                    activation = None,\r\n                                    use_bias = False,\r\n                                    name = 'fc_a')\r\n            logits1 = tf.reshape(logits1, [-1, self.num_ctx])\r\n            logits2 = self.nn.dense(output,\r\n                                    units = self.num_ctx,\r\n                                    activation = None,\r\n                                    use_bias = False,\r\n                                    name = 'fc_b')\r\n            logits = logits1 + logits2\r\n        else:\r\n            # use 2 fc layers to attend\r\n            temp1 = self.nn.dense(reshaped_contexts,\r\n                                  units = config.dim_attend_layer,\r\n                                  activation = tf.tanh,\r\n                                  name = 'fc_1a')\r\n            temp2 = self.nn.dense(output,\r\n                                  units = config.dim_attend_layer,\r\n                                  activation = tf.tanh,\r\n                                  name = 'fc_1b')\r\n            temp2 = tf.tile(tf.expand_dims(temp2, 1), [1, self.num_ctx, 1])\r\n            temp2 = tf.reshape(temp2, [-1, config.dim_attend_layer])\r\n            temp = temp1 + temp2\r\n            temp = self.nn.dropout(temp)\r\n            logits = self.nn.dense(temp,\r\n                                   units = 1,\r\n                                   activation = None,\r\n                                   use_bias = False,\r\n                                   name = 'fc_2')\r\n            logits = tf.reshape(logits, [-1, self.num_ctx])\r\n        alpha = tf.nn.softmax(logits)\r\n        return alpha\r\n```\r\nHere I only consider the 2-fc branch.\r\nI think the fomula of the code is : wa(tanh(Wva vi) + tanh(Wha ht)), which is slightly different with the paper. But tanh(A) + tanh(B) != tanh(A+B)\r\n\r\nSo I wonder if there could be some problems that this difference may cause. Anyone can help?\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/DeepRNN/image_captioning/issues/comments/408899554","html_url":"https://github.com/DeepRNN/image_captioning/issues/28#issuecomment-408899554","issue_url":"https://api.github.com/repos/DeepRNN/image_captioning/issues/28","id":408899554,"node_id":"MDEyOklzc3VlQ29tbWVudDQwODg5OTU1NA==","user":{"login":"Yuli555","id":19490927,"node_id":"MDQ6VXNlcjE5NDkwOTI3","avatar_url":"https://avatars3.githubusercontent.com/u/19490927?v=4","gravatar_id":"","url":"https://api.github.com/users/Yuli555","html_url":"https://github.com/Yuli555","followers_url":"https://api.github.com/users/Yuli555/followers","following_url":"https://api.github.com/users/Yuli555/following{/other_user}","gists_url":"https://api.github.com/users/Yuli555/gists{/gist_id}","starred_url":"https://api.github.com/users/Yuli555/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yuli555/subscriptions","organizations_url":"https://api.github.com/users/Yuli555/orgs","repos_url":"https://api.github.com/users/Yuli555/repos","events_url":"https://api.github.com/users/Yuli555/events{/privacy}","received_events_url":"https://api.github.com/users/Yuli555/received_events","type":"User","site_admin":false},"created_at":"2018-07-30T15:14:48Z","updated_at":"2018-07-30T15:14:48Z","author_association":"NONE","body":"@HrsPythonix Hi! Have you solved the problem?"}}
{"actor":{"display_login":"Yuli555"}}