{"action":"opened","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4022","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4022/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4022/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/4022/events","html_url":"https://github.com/tensorflow/tensorflow/issues/4022","id":173031205,"number":4022,"title":"train.batch with dynamic_pad=True and input as list of tensors not working as expected","user":{"login":"alexbeloi","id":9807648,"avatar_url":"https://avatars.githubusercontent.com/u/9807648?v=3","gravatar_id":"","url":"https://api.github.com/users/alexbeloi","html_url":"https://github.com/alexbeloi","followers_url":"https://api.github.com/users/alexbeloi/followers","following_url":"https://api.github.com/users/alexbeloi/following{/other_user}","gists_url":"https://api.github.com/users/alexbeloi/gists{/gist_id}","starred_url":"https://api.github.com/users/alexbeloi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexbeloi/subscriptions","organizations_url":"https://api.github.com/users/alexbeloi/orgs","repos_url":"https://api.github.com/users/alexbeloi/repos","events_url":"https://api.github.com/users/alexbeloi/events{/privacy}","received_events_url":"https://api.github.com/users/alexbeloi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-08-24T18:58:18Z","updated_at":"2016-08-24T18:58:18Z","closed_at":null,"body":"### Environment info\r\nOperating System: \r\nUbuntu 14.04.4 LTS (running in Virtual Box 5.0.22 r108108)\r\n\r\nInstalled version of CUDA and cuDNN: \r\nNone\r\n\r\nIf installed from binary pip package, provide:\r\n1. Which pip package you installed.\r\npip 8.1.2\r\n2. The output from `python -c \"import tensorflow; print(tensorflow.__version__)\"`.\r\n0.10.0rc0\r\n\r\n### Preface\r\nThis issue arose from attempting to expand on an example using dynamic padding as written in: http://www.wildml.com/2016/08/rnns-in-tensorflow-a-practical-guide-and-undocumented-features/\r\nthe necessary code is copied below.\r\n\r\n### Steps to reproduce\r\nRun the following minimal example:\r\n```python\r\nimport tensorflow as tf\r\n\r\n# [0, 1, 2, 3, 4 ,...]\r\nx = tf.range(1, 10, name=\"x\")\r\n \r\n# A queue that outputs 0,1,2,3,...\r\nrange_q = tf.train.range_input_producer(limit=5, shuffle=False)\r\nslice_end = range_q.dequeue()\r\n \r\n# Slice x to variable length, i.e. [0], [0, 1], [0, 1, 2], ....\r\ny = tf.slice(x, [0], [slice_end], name=\"y\")\r\n\r\nbatched_data = tf.train.batch(\r\n    tensors=[y],\r\n    batch_size=5,\r\n    dynamic_pad=True,\r\n    name=\"y_batch\"\r\n)\r\n \r\n# Run the graph\r\n# tf.contrib.learn takes care of starting the queues for us\r\nres = tf.contrib.learn.run_n({\"y\": batched_data}, n=1, feed_dict=None)\r\n \r\n# Print the result\r\nprint(\"Batch shape: {}\".format(res[0][\"y\"].shape))\r\nprint(res[0][\"y\"])\r\n```\r\nOutput (correct behavior):\r\n```\r\nBatch shape: (5, 4)\r\n[[0 0 0 0]\r\n [1 0 0 0]\r\n [1 2 0 0]\r\n [1 2 3 0]\r\n [1 2 3 4]]\r\n```\r\n\r\nWhen attempted with different input (list of tensors of different lengths)\r\n```python\r\nimport tensorflow as tf\r\n\r\ny = [tf.constant(range(n)) for n in range(1,10)]\r\n\r\nbatched_data = tf.train.batch(\r\n    tensors=[y],\r\n    batch_size=5,\r\n    dynamic_pad=True,\r\n    name=\"y_batch\"\r\n)\r\n \r\n# Run the graph\r\n# tf.contrib.learn takes care of starting the queues for us\r\nres = tf.contrib.learn.run_n({\"y\": batched_data}, n=1, feed_dict=None)\r\n \r\n# Print the result\r\nprint(\"Batch shape: {}\".format(res[0][\"y\"].shape))\r\nprint(res[0][\"y\"])\r\n```\r\nOutput:\r\n```\r\n---------------------------------------------------------------------------\r\nValueError                                Traceback (most recent call last)\r\n<ipython-input-12-8b4d52a4df68> in <module>()\r\n     19     batch_size=5,\r\n     20     dynamic_pad=True,\r\n---> 21     name=\"y_batch\"\r\n     22 )\r\n     23 \r\n\r\n/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.pyc in batch(tensors, batch_size, num_threads, capacity, enqueue_many, shapes, dynamic_pad, allow_smaller_final_batch, shared_name, name)\r\n    577   tensor_list = _as_tensor_list(tensors)\r\n    578   with ops.op_scope(tensor_list, name, \"batch\") as name:\r\n--> 579     tensor_list = _validate(tensor_list)\r\n    580     (tensor_list, sparse_info) = _serialize_sparse_tensors(\r\n    581         tensor_list, enqueue_many)\r\n\r\n/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.pyc in _validate(tensor_list)\r\n    411 \r\n    412 def _validate(tensor_list):\r\n--> 413   tensor_list = ops.convert_n_to_tensor_or_indexed_slices(tensor_list)\r\n    414   if not tensor_list:\r\n    415     raise ValueError(\"Expected at least one tensor in batch().\")\r\n\r\n/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.pyc in convert_n_to_tensor_or_indexed_slices(values, dtype, name, as_ref)\r\n    735       ret.append(\r\n    736           convert_to_tensor_or_indexed_slices(value, dtype=dtype, name=n,\r\n--> 737                                               as_ref=as_ref))\r\n    738   return ret\r\n    739 \r\n\r\n/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.pyc in convert_to_tensor_or_indexed_slices(value, dtype, name, as_ref)\r\n    696     return value\r\n    697   else:\r\n--> 698     return convert_to_tensor(value, dtype=dtype, name=name, as_ref=as_ref)\r\n    699 \r\n    700 \r\n\r\n/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.pyc in convert_to_tensor(value, dtype, name, as_ref)\r\n    619     for base_type, conversion_func in funcs_at_priority:\r\n    620       if isinstance(value, base_type):\r\n--> 621         ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)\r\n    622         if ret is NotImplemented:\r\n    623           continue\r\n\r\n/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.pyc in _autopacking_conversion_function(v, dtype, name, as_ref)\r\n    628   if dtype is not None and dtype != inferred_dtype:\r\n    629     return NotImplemented\r\n--> 630   return _autopacking_helper(v, inferred_dtype, name or \"packed\")\r\n    631 # pylint: enable=invalid-name\r\n    632 \r\n\r\n/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.pyc in _autopacking_helper(list_or_tuple, dtype, name)\r\n    591           elems_as_tensors.append(\r\n    592               constant_op.constant(elem, dtype=dtype, name=str(i)))\r\n--> 593       return gen_array_ops._pack(elems_as_tensors, name=scope)\r\n    594     else:\r\n    595       return converted_elems\r\n\r\n/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.pyc in _pack(values, axis, name)\r\n   1452     A `Tensor`. Has the same type as `values`. The packed tensor.\r\n   1453   \"\"\"\r\n-> 1454   result = _op_def_lib.apply_op(\"Pack\", values=values, axis=axis, name=name)\r\n   1455   return result\r\n   1456 \r\n\r\n/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.pyc in apply_op(self, op_type_name, name, **keywords)\r\n    701           op = g.create_op(op_type_name, inputs, output_types, name=scope,\r\n    702                            input_types=input_types, attrs=attr_protos,\r\n--> 703                            op_def=op_def)\r\n    704           outputs = op.outputs\r\n    705           return _Restructure(ops.convert_n_to_tensor(outputs),\r\n\r\n/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.pyc in create_op(self, op_type, inputs, dtypes, input_types, name, attrs, op_def, compute_shapes, compute_device)\r\n   2310                     original_op=self._default_original_op, op_def=op_def)\r\n   2311     if compute_shapes:\r\n-> 2312       set_shapes_for_outputs(ret)\r\n   2313     self._add_op(ret)\r\n   2314     self._record_op_seen_by_control_dependencies(ret)\r\n\r\n/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.pyc in set_shapes_for_outputs(op)\r\n   1702       raise RuntimeError(\"No shape function registered for standard op: %s\"\r\n   1703                          % op.type)\r\n-> 1704   shapes = shape_func(op)\r\n   1705   if shapes is None:\r\n   1706     raise RuntimeError(\r\n\r\n/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.pyc in _PackShape(op)\r\n    767 \r\n    768   for inp in op.inputs[1:]:\r\n--> 769     input_shape = input_shape.merge_with(inp.get_shape())\r\n    770 \r\n    771   input_shape = input_shape.as_list()\r\n\r\n/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_shape.pyc in merge_with(self, other)\r\n    568       except ValueError:\r\n    569         raise ValueError(\"Shapes %s and %s are not compatible\" %\r\n--> 570                          (self, other))\r\n    571 \r\n    572   def concatenate(self, other):\r\n\r\nValueError: Shapes (1,) and (2,) are not compatible\r\n```\r\n\r\nExpected same result as previous script.\r\n\r\n### What have you tried?\r\n* Attempted transforming `y` into a tensor but have not found a way. Given the first example, it appears possible to construct tensors with variable size in some dimension, but given a dataset in the form of List of List of primitive (with inner lists being of varying lengths), I don't know how to transform this into a tensor without padding the entire dataset.\r\n```python\r\nimport tensorflow as tf\r\n\r\n# [0, 1, 2, 3, 4 ,...]\r\nx = tf.range(1, 10, name=\"x\")\r\n \r\n# A queue that outputs 0,1,2,3,...\r\nrange_q = tf.train.range_input_producer(limit=5, shuffle=False)\r\nslice_end = range_q.dequeue()\r\n \r\n# Slice x to variable length, i.e. [0], [0, 1], [0, 1, 2], ....\r\ny = tf.slice(x, [0], [slice_end], name=\"y\")\r\n\r\nprint 'Dynamic shape of y:', tf.shape(y)\r\nprint 'Static shape of y:', y.get_shape()\r\n```\r\nOutput:\r\n```\r\nDynamic shape of y: Tensor(\"Shape_16:0\", shape=(1,), dtype=int32)\r\nStatic shape of y: (?,)\r\n```\r\n\r\n* Attempted replacing `tf.train.batch` line with the `tf.PaddingFIFOQueue` as follows\r\n```python\r\n# Creating a new queue\r\npadding_q = tf.PaddingFIFOQueue(\r\n    capacity=10,\r\n    dtypes=tf.int32,\r\n    shapes=[[None]])\r\n \r\n# Enqueue the examples\r\nenqueue_op = padding_q.enqueue([y])\r\n \r\n# Add the queue runner to the graph\r\nqr = tf.train.QueueRunner(padding_q, [enqueue_op])\r\ntf.train.add_queue_runner(qr)\r\n \r\n# Dequeue padded data\r\nbatched_data = padding_q.dequeue_many(5)\r\n```\r\nOutput: identical to above\r\n\r\n* Using placeholders: tried replacing `y` with a placeholder of shape `[None]` and feeding it the data, cannot seem to get this to work either.\r\n\r\n* Padding entire dataset: this works, but defeats the purpose of dynamic padding\r\n\r\n* Todo: am going to try passing dataset through `tf.train.SequenceExample()`, constructing an example for each sequence, but I would rather this not be necessary.\r\n"}}
{"actor":{"display_login":"alexbeloi"}}