{"action":"opened","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3797","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3797/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3797/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3797/events","html_url":"https://github.com/tensorflow/tensorflow/issues/3797","id":171022138,"number":3797,"title":"RMSProp errors with embedding_lookup","user":{"login":"kuza55","id":387986,"avatar_url":"https://avatars.githubusercontent.com/u/387986?v=3","gravatar_id":"","url":"https://api.github.com/users/kuza55","html_url":"https://github.com/kuza55","followers_url":"https://api.github.com/users/kuza55/followers","following_url":"https://api.github.com/users/kuza55/following{/other_user}","gists_url":"https://api.github.com/users/kuza55/gists{/gist_id}","starred_url":"https://api.github.com/users/kuza55/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kuza55/subscriptions","organizations_url":"https://api.github.com/users/kuza55/orgs","repos_url":"https://api.github.com/users/kuza55/repos","events_url":"https://api.github.com/users/kuza55/events{/privacy}","received_events_url":"https://api.github.com/users/kuza55/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-08-13T18:50:35Z","updated_at":"2016-08-13T18:50:35Z","closed_at":null,"body":"I've been trying to train an LSTM with a word embedding and got some errors that seem related to https://github.com/tensorflow/tensorflow/issues/1117\r\n\r\n### Environment info\r\nOperating System:\r\nIf installed from binary pip package, provide:\r\n\r\nLinux CPU Only\r\n0.10.0rc0\r\n\r\n### Steps to reproduce\r\nThis is the code I've been running, which should repro the issue:\r\n\r\n```\r\nimport tensorflow as tf\r\nimport numpy as np\r\nimport pandas\r\nimport tensorflow as tf\r\nfrom tensorflow.contrib import learn\r\n#from preprocessing import VocabularyProcessor\r\nVocabularyProcessor = learn.preprocessing.VocabularyProcessor\r\n\r\ndef partition_length(x_train, y_train):\r\n  #Partition the training data by length of sequences\r\n  x_train_dict = {}\r\n  y_train_dict = {}\r\n  for i in range(x_train.shape[0]):\r\n    x = x_train[i]\r\n    y = y_train[i]\r\n    \r\n    l = len(x)\r\n    if l not in x_train_dict:\r\n      x_train_dict[l] = []\r\n      y_train_dict[l] = []\r\n    x_train_dict[l].append(x)\r\n    y_train_dict[l].append(y)\r\n\r\n\r\n  x_train_np = {}\r\n  y_train_np = {}\r\n  for l in x_train_dict:\r\n    x_train_np[l] = np.asarray(x_train_dict[l])\r\n    y_train_np[l] = np.asarray(y_train_dict[l])\r\n\r\n  return (x_train_np, y_train_np)\r\n\r\nn_words = 0\r\nMAX_DOCUMENT_LENGTH = 10\r\nEMBEDDING_SIZE = 50\r\n\r\n# Prepare training and testing data\r\ndbpedia = learn.datasets.load_dataset('dbpedia')\r\n\r\nx_train = pandas.DataFrame(dbpedia.train.data)[1]\r\ny_train = dbpedia.train.target\r\nx_test = pandas.DataFrame(dbpedia.test.data)[1]\r\ny_test = dbpedia.test.target\r\n\r\nprint x_train.shape, y_train.shape\r\nprint x_test.shape, y_test.shape\r\n\r\n# Process vocabulary\r\nvocab_processor = VocabularyProcessor(10)\r\nx_train = np.array(list(vocab_processor.fit_transform(x_train)))\r\nx_test = np.array(list(vocab_processor.transform(x_test)))\r\nn_words = len(vocab_processor.vocabulary_)\r\nprint('Total words: %d' % n_words)\r\n\r\nprint x_train.shape, y_train.shape\r\n\r\n(x_train_part, y_train_part) = partition_length(x_train, y_train)\r\n\r\ndata = tf.placeholder(tf.int32, [None, None], name='data')\r\nword_vectors = learn.ops.categorical_variable(data, n_classes=n_words,\r\n  embedding_size=EMBEDDING_SIZE, name='words')\r\n\r\nprint('data: ', data.get_shape())\r\nprint('word_vectors: ', word_vectors.get_shape())\r\nprint('word_vectors: ', tf.shape(word_vectors))\r\n\r\ntarget = tf.placeholder(tf.int32, [None], name='target')\r\none_hot = tf.one_hot(target, 15, 1.0, 0.0, dtype=tf.float32)\r\n\r\nprint('target: ', target.get_shape())\r\nprint('one_hot: ', one_hot.get_shape())\r\n\r\nnum_hidden = 250\r\n\r\n_, state = tf.nn.dynamic_rnn(\r\n    tf.nn.rnn_cell.GRUCell(num_hidden),\r\n    word_vectors,\r\n    dtype=tf.float32,\r\n)\r\n\r\nprint('state: ', state.get_shape())\r\n\r\nin_size, out_size = (num_hidden, 15)\r\n\r\nweight = tf.Variable(tf.truncated_normal([in_size, out_size], stddev=0.01))\r\nbias = tf.Variable(tf.constant(0.1, shape=[out_size]))\r\n\r\nprediction = tf.nn.softmax(tf.matmul(state, weight) + bias)\r\n#prediction = tf.contrib.losses.softmax_cross_entropy(logits, onehot_labels, weight=1.0, label_smoothing=0, scope=None)\r\nprediction_idx = tf.argmax(prediction, 1)\r\none_hot_idx = tf.argmax(one_hot, 1)\r\n\r\nprint('prediction: ', prediction.get_shape())\r\nprint('prediction_idx: ', prediction_idx.get_shape())\r\nprint('one_hot_idx: ', one_hot_idx.get_shape())\r\n\r\nmistakes = tf.not_equal(one_hot_idx, prediction_idx)\r\nerror =  tf.reduce_mean(tf.cast(mistakes, tf.float32))\r\n\r\ncross_entropy = -tf.reduce_sum(one_hot * tf.log(prediction))\r\n\r\nlearning_rate = 0.003\r\noptimizer = tf.train.RMSPropOptimizer(learning_rate)\r\noptimize = optimizer.minimize(cross_entropy)\r\n\r\nsess = tf.Session()\r\nsess.run(tf.initialize_all_variables())\r\n\r\n\r\n\r\nbatch_size = 5\r\nfor length in x_train_part:\r\n  x_train = x_train_part[length]\r\n  y_train = y_train_part[length]\r\n\r\n  print \"Training on length \", length, \" # elements: \", x_train.shape[0]\r\n\r\n  error_pct = sess.run(error, {data: x_test, target: y_test})\r\n  print('Epoch {:2d} error {:3.1f}%'.format(0, 100 * error_pct))\r\n\r\n  for epoch in range(10):\r\n    print('Epoch {:2d}'.format(epoch+1))\r\n    for i in range(x_train.shape[0]/batch_size):\r\n      sess.run(optimize, {data: x_train[i:i+batch_size], target: y_train[i:i+batch_size]})\r\n      \r\n    error_pct = sess.run(error, {data: x_test, target: y_test})\r\n    print('Epoch {:2d} error {:3.1f}%'.format(epoch + 1, 100 * error_pct))\r\n\r\n```\r\n\r\n### What have you tried?\r\nUsing any non-RMSProp Optimizer works\r\n\r\n### Logs or other output that would be helpful\r\nThis is the exception I get:\r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \"lstm.py\", line 131, in <module>\r\n    sess.run(optimize, {data: x_train[i:i+batch_size], target: y_train[i:i+batch_size]})\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py\", line 710, in run\r\n    run_metadata_ptr)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py\", line 908, in _run\r\n    feed_dict_string, options, run_metadata)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py\", line 958, in _do_run\r\n    target_list, options, run_metadata)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py\", line 978, in _do_call\r\n    raise type(e)(node_def, op, message)\r\ntensorflow.python.framework.errors.InvalidArgumentError: var and grad do not have the same shape[7664,50] [50,50]\r\n         [[Node: RMSProp/update_words/words_embeddings/SparseApplyRMSProp = SparseApplyRMSProp[T=DT_FLOAT, Tindices=DT_INT32, _class=[\"loc:@words/words_embeddings\"], use_locking=false, _device=\"/job:localhost/replica:0/task:0/cpu:0\"](words/words_embeddings, words/words_embeddings/RMSProp, words/words_embeddings/RMSProp_1, RMSProp/learning_rate, RMSProp/decay, RMSProp/momentum, RMSProp/epsilon, gradients/words/embedding_lookup/embedding_lookup_grad/Reshape, gradients/words/embedding_lookup/embedding_lookup_grad/Reshape_1)]]\r\nCaused by op u'RMSProp/update_words/words_embeddings/SparseApplyRMSProp', defined at:\r\n  File \"lstm.py\", line 111, in <module>\r\n    optimize = optimizer.minimize(cross_entropy)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/optimizer.py\", line 198, in minimize\r\n    name=name)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/optimizer.py\", line 313, in apply_gradients\r\n    update_ops.append(self._apply_sparse(grad, var))\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/rmsprop.py\", line 122, in _apply_sparse\r\n    use_locking=self._use_locking)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/gen_training_ops.py\", line 664, in sparse_apply_rms_prop\r\n    use_locking=use_locking, name=name)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py\", line 703, in apply_op\r\n    op_def=op_def)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py\", line 2317, in create_op\r\n    original_op=self._default_original_op, op_def=op_def)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py\", line 1239, in __init__\r\n    self._traceback = _extract_stack()\r\n```"}}
{"actor":{"display_login":"kuza55"}}