{"comment":{"url":"https://api.github.com/repos/rwth-i6/returnn/comments/27399108","html_url":"https://github.com/rwth-i6/returnn/commit/9e16d81bf55d682c906099dd6128addf3bba7ec3#commitcomment-27399108","id":27399108,"user":{"login":"mennetob","id":14050679,"avatar_url":"https://avatars1.githubusercontent.com/u/14050679?v=4","gravatar_id":"","url":"https://api.github.com/users/mennetob","html_url":"https://github.com/mennetob","followers_url":"https://api.github.com/users/mennetob/followers","following_url":"https://api.github.com/users/mennetob/following{/other_user}","gists_url":"https://api.github.com/users/mennetob/gists{/gist_id}","starred_url":"https://api.github.com/users/mennetob/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mennetob/subscriptions","organizations_url":"https://api.github.com/users/mennetob/orgs","repos_url":"https://api.github.com/users/mennetob/repos","events_url":"https://api.github.com/users/mennetob/events{/privacy}","received_events_url":"https://api.github.com/users/mennetob/received_events","type":"User","site_admin":false},"position":null,"line":null,"path":null,"commit_id":"9e16d81bf55d682c906099dd6128addf3bba7ec3","created_at":"2018-02-07T14:41:35Z","updated_at":"2018-02-07T14:41:35Z","author_association":"COLLABORATOR","body":"It only occures in training. Basic forwarding still works fine, so the problem should be with the gradient.\r\n\r\nIt's hard for me to tell, why the problem occures. I have not yet come around to separate the specific inputs and outputs.\r\nBut I am not using safe_log in the computation, which gives the error, but only in the \"standard\" parts of returnn (I think it's due to CrossEntropy)\r\n\r\nThe only reason I found that this is the problem, is by narrowing it down through checking which commit breaks the setup. There is no hint (as far as I can see) in the stacktrace, that points towards the safe_log. This is also why it's quite hard to construct a \"simple\" test_case (I'll try to have a closer look at it though).\r\n\r\n```python\r\nEXCEPTION\r\nTraceback (most recent call last):\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFEngine.py\", line 438, in run\r\n    line: fetches_results = sess.run(fetches_dict, feed_dict=feed_dict)  # type: dict[str,numpy.ndarray|str]\r\n    locals:\r\n      fetches_results = <local> {'loss': 3404.6704, 'size:classes:0': array([50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50,\r\n                                       50, 50, 50, 50, 50, 50, 29,  4, 50, 50, 50, 50, 50, 50, 50, 50, 50,\r\n                                       50, 50, 50, 50, 50, 50], dtype=int32), 'optim_op': None, 'error:output': 744.0, 'size:data:0': array([..., len = 7\r\n      sess = <local> <tensorflow.python.client.session.Session object at 0x7f57f45ad390>\r\n      sess.run = <local> <bound method Session.run of <tensorflow.python.client.session.Session object at 0x7f57f45ad390>>\r\n      fetches_dict = <local> {'loss': <tf.Tensor 'objective/add:0' shape=() dtype=float32>, 'summary': <tf.Tensor 'Merge/MergeSummary:0' shape=() dtype=string>, 'size:classes:0': <tf.Tensor 'extern_data/placeholders/classes/classes_dim0_size:0' shape=(?,) dtype=int32>, 'optim_op': <tf.Operation 'optimize/optim_and_step_incr'..., len = 76,  926, ...,  566,  566,  449],\r\n                                 [1500, 1500, 1500, ...,  413,  404,  404],\r\n                                 [ 440,  464, 1442, ...,  354,  354,  354],\r\n                                 ..., \r\n                                 [1491, 1491, 1491, ...,  746,  746,  746],\r\n                          ...\r\n  File \"/u/menne/py-envs/py2.7tfUpdate/local/lib/python2.7/site-packages/tensorflow/python/client/session.py\", line 889, in run\r\n    line: result = self._run(None, fetches, feed_dict, options_ptr,\r\n                             run_metadata_ptr)\r\n    locals:\r\n      result = <not found>\r\n      self = <local> <tensorflow.python.client.session.Session object at 0x7f57f45ad390>\r\n      self._run = <local> <bound method Session._run of <tensorflow.python.client.session.Session object at 0x7f57f45ad390>>\r\n      fetches = <local> {'loss': <tf.Tensor 'objective/add:0' shape=() dtype=float32>, 'summary': <tf.Tensor 'Merge/MergeSummary:0' shape=() dtype=string>, 'size:classes:0': <tf.Tensor 'extern_data/placeholders/classes/classes_dim0_size:0' shape=(?,) dtype=int32>, 'optim_op': <tf.Operation 'optimize/optim_and_step_incr'..., len = 7,  926,  926, ...,  566,  566,  449],\r\n                                 [1500, 1500, 1500, ...,  413,  404,  404],\r\n                                 [ 440,  464, 1442, ...,  354,  354,  354],\r\n                                 ..., \r\n                                 [1491, 1491, 1491, ...,  746,  746,  746],\r\n                          ...\r\n      options_ptr = <local> None\r\n      run_metadata_ptr = <local> None\r\n  File \"/u/menne/py-envs/py2.7tfUpdate/local/lib/python2.7/site-packages/tensorflow/python/client/session.py\", line 1120, in _run\r\n    line: results = self._do_run(handle, final_targets, final_fetches,\r\n                                 feed_dict_tensor, options, run_metadata)\r\n    locals:\r\n      results = <not found>\r\n      self = <local> <tensorflow.python.client.session.Session object at 0x7f57f45ad390>\r\n      self._do_run = <local> <bound method Session._do_run of <tensorflow.python.client.session.Session object at 0x7f57f45ad390>>\r\n      handle = <local> None\r\n      final_targets = <local> [<tf.Operation 'optimize/optim_and_step_incr' type=NoOp>]\r\n      final_fetches = <local> [<tf.Tensor 'objective/add:0' shape=() dtype=float32>, <tf.Tensor 'Merge/MergeSummary:0' shape=() dtype=string>, <tf.Tensor 'objective/loss/output/loss/loss_ce/Sum:0' shape=() dtype=float32>, <tf.Tensor 'objective/loss/output/error/loss_frame_error/Sum:0' shape=() dtype=float32>] ?) dtype=int32>: array([[ 926,  926,  926, ...,  566,  566,  449],\r\n                                        [1500, 1500, 1500, ...,  413,  404,  404],\r\n                                        [ 440,  464, 1442, ...,  354,  354,  354],\r\n                                        ..., \r\n                                        [1491, 1491, 1491, ...,  746,  746,  746],\r\n                                 ...\r\n      options = <local> None\r\n      run_metadata = <local> None\r\nFile \"/u/menne/py-envs/py2.7tfUpdate/local/lib/python2.7/site-packages/tensorflow/python/client/session.py\", line 1317, in _do_run\r\n    line: return self._do_call(_run_fn, self._session, feeds, fetches, targets,\r\n                               options, run_metadata)\r\n    locals:\r\n      self = <local> <tensorflow.python.client.session.Session object at 0x7f57f45ad390>\r\n      self._do_call = <local> <bound method Session._do_call of <tensorflow.python.client.session.Session object at 0x7f57f45ad390>>\r\n      _run_fn = <local> <function _run_fn at 0x7f578a501a28>\r\n      self._session = <local> <Swig Object of type 'TF_DeprecatedSession *' at 0x7f57f05434b0>\r\n      feeds = <local> {'extern_data/placeholders/data/data:0': array([[[ -5.88383436e+00,   5.80307096e-06,   4.50901318e+00, ...,\r\n                                 7.26330982e-05,   7.87791912e-04,  -2.32080879e-06],\r\n                              [ -2.37389231e+00,   2.32923776e-06,   1.98482239e+00, ...,\r\n                                 5.05884353e-04,  -6.00978034e-04,  -1.1804809...\r\n      fetches = <local> ['objective/add:0', 'Merge/MergeSummary:0', 'objective/loss/output/loss/loss_ce/Sum:0', 'objective/loss/output/error/loss_frame_error/Sum:0'], _[0]: {len = 15}\r\n      targets = <local> ['optimize/optim_and_step_incr'], _[0]: {len = 28}\r\n      options = <local> None\r\n      run_metadata = <local> None\r\n  File \"/u/menne/py-envs/py2.7tfUpdate/local/lib/python2.7/site-packages/tensorflow/python/client/session.py\", line 1336, in _do_call\r\n    line: raise type(e)(node_def, op, message)\r\n    locals:\r\n      type = <builtin> <type 'type'>\r\n      e = <local> InvalidArgumentError()\r\n      node_def = <local> name: \"beamformingLayer/TfGevBeamformer._getFilterMatrix/MatrixDeterminant\"\r\n                         op: \"MatrixDeterminant\"\r\n                         input: \"beamformingLayer/TfGevBeamformer._getFilterMatrix/TfMaskBasedGevBeamformer._getSpatialCorrelationByMask/div\"\r\n                         attr {\r\n                           key: \"T\"\r\n                           value {\r\n                             type: DT_COMPLEX64\r\n                           }\r\n                         }\r\n\r\n      op = <local> <tf.Operation 'beamformingLayer/TfGevBeamformer._getFilterMatrix/MatrixDeterminant' type=MatrixDeterminant>\r\n      message = <local> u'The determinant is not finite.\\n\\t [[Node: beamformingLayer/TfGevBeamformer._getFilterMatrix/MatrixDeterminant = MatrixDeterminant[T=DT_COMPLEX64, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](beamformingLayer/TfGevBeamformer._getFilterMatrix/TfMaskBasedGevBeamformer._getSpatialCorrel..., len = 720\r\nInvalidArgumentError: The determinant is not finite.\r\n[[Node: beamformingLayer/TfGevBeamformer._getFilterMatrix/MatrixDeterminant = MatrixDeterminant[T=DT_COMPLEX64, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](beamformingLayer/TfGevBeamformer._getFilterMatrix/TfMaskBasedGevBeamformer._getSpatialCorrelationByMask/div)]]\r\n         [[Node: optimize/gradients/medianMasks/TopKV2_grad/Reshape_1/_1125 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/device:CPU:0\", send_device=\"/job:localhost/replica:0/task:0/device:GPU:0\", send_device_incarnation=1, tensor_name=\"edge_8777_optimize/gradients/medianMasks/TopKV2_grad/Reshape_1\", tensor_type=DT_INT32, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"]()]]\r\n\r\nCaused by op u'beamformingLayer/TfGevBeamformer._getFilterMatrix/MatrixDeterminant', defined at:\r\n  File \"/work/asr3/menne/work/experiments/chime4/20171208_jointBfAmTraining/tuneLearningrate/jointTrainings/jointTraining_withoutBan//step007_jointTraining/dependencies/pythonModules/returnn/rnn.py\", line 552, in <module>\r\n    main(sys.argv)\r\n  File \"/work/asr3/menne/work/experiments/chime4/20171208_jointBfAmTraining/tuneLearningrate/jointTrainings/jointTraining_withoutBan//step007_jointTraining/dependencies/pythonModules/returnn/rnn.py\", line 540, in main\r\n    executeMainTask()\r\n  File \"/work/asr3/menne/work/experiments/chime4/20171208_jointBfAmTraining/tuneLearningrate/jointTrainings/jointTraining_withoutBan//step007_jointTraining/dependencies/pythonModules/returnn/rnn.py\", line 382, in executeMainTask\r\n    engine.init_train_from_config(config, train_data, dev_data, eval_data)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFEngine.py\", line 705, in init_train_from_config\r\n    self.init_network_from_config(config)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFEngine.py\", line 736, in init_network_from_config\r\n    self._init_network(net_desc=net_dict, epoch=self.epoch)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFEngine.py\", line 829, in _init_network\r\n    network.construct_from_dict(net_desc)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetwork.py\", line 313, in construct_from_dict\r\n    self.construct_layer(net_dict, name)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetwork.py\", line 377, in construct_layer\r\n    layer_class.transform_config_dict(layer_desc, network=self, get_layer=get_layer)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetworkLayer.py\", line 288, in transform_config_dict\r\n    if not src_name == \"none\"]\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetwork.py\", line 368, in get_layer\r\n    return self.construct_layer(net_dict=net_dict, name=src_name)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetwork.py\", line 377, in construct_layer\r\n    layer_class.transform_config_dict(layer_desc, network=self, get_layer=get_layer)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetworkLayer.py\", line 288, in transform_config_dict\r\n    if not src_name == \"none\"]\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetwork.py\", line 368, in get_layer\r\n    return self.construct_layer(net_dict=net_dict, name=src_name)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetwork.py\", line 377, in construct_layer\r\n    layer_class.transform_config_dict(layer_desc, network=self, get_layer=get_layer)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetworkLayer.py\", line 288, in transform_config_dict\r\n    if not src_name == \"none\"]\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetwork.py\", line 368, in get_layer\r\n    return self.construct_layer(net_dict=net_dict, name=src_name)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetwork.py\", line 377, in construct_layer\r\n    layer_class.transform_config_dict(layer_desc, network=self, get_layer=get_layer)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetworkLayer.py\", line 288, in transform_config_dict\r\n    if not src_name == \"none\"]\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetwork.py\", line 368, in get_layer\r\n    return self.construct_layer(net_dict=net_dict, name=src_name)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetwork.py\", line 377, in construct_layer\r\n    layer_class.transform_config_dict(layer_desc, network=self, get_layer=get_layer)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetworkLayer.py\", line 288, in transform_config_dict\r\n    if not src_name == \"none\"]\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetwork.py\", line 368, in get_layer\r\n    return self.construct_layer(net_dict=net_dict, name=src_name)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetwork.py\", line 380, in construct_layer\r\n    return add_layer(name=name, layer_class=layer_class, **layer_desc)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetwork.py\", line 459, in add_layer\r\nlayer = self._create_layer(name=name, layer_class=layer_class, **layer_desc)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetwork.py\", line 423, in _create_layer\r\n    layer = layer_class(**layer_desc)\r\n  File \"/work/asr2/menne/programs/returnn/head/returnn/TFNetworkSigProcLayer.py\", line 137, in __init__\r\n    gevBf = TfMaskBasedGevBeamformer(flag_inputHasBatch=1, tfFreqDomInput=complexSpectrogram, tfNoiseMask=noiseMasks, tfSpeechMask=speechMasks, postFilterId=postfilter_id)\r\n  File \"/u/menne/backbone/python/packages/tfSi6Proc/audioProcessing/enhancement/beamforming.py\", line 336, in __init__\r\n    super(TfMaskBasedGevBeamformer, self).__init__(**kwargs)\r\n  File \"/u/menne/backbone/python/packages/tfSi6Proc/audioProcessing/enhancement/beamforming.py\", line 168, in __init__\r\n    super(TfGevBeamformer, self).__init__(**kwargs)\r\n  File \"/u/menne/backbone/python/packages/tfSi6Proc/audioProcessing/enhancement/beamforming.py\", line 52, in __init__\r\n    self._buildGraph()\r\n  File \"/u/menne/backbone/python/packages/tfSi6Proc/audioProcessing/enhancement/beamforming.py\", line 69, in _buildGraph\r\n    filterMatrix = self._getFilterMatrix()\r\n  File \"/u/menne/backbone/python/packages/tfSi6Proc/audioProcessing/enhancement/beamforming.py\", line 195, in _getFilterMatrix\r\n    corMat = tf.matmul(safeMatrixInversion(tfNoiseCorrelationMatrix), tfSpeechCorrelationMatrix)\r\n  File \"/u/menne/backbone/python/packages/tfSi6Proc/basics/math/functions.py\", line 71, in safeMatrixInversion\r\n    tfDeterminant = tf.matrix_determinant(tfMatrix)\r\n  File \"/u/menne/py-envs/py2.7tfUpdate/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_linalg_ops.py\", line 548, in matrix_determinant\r\n    \"MatrixDeterminant\", input=input, name=name)\r\n  File \"/u/menne/py-envs/py2.7tfUpdate/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py\", line 787, in _apply_op_helper\r\n    op_def=op_def)\r\n  File \"/u/menne/py-envs/py2.7tfUpdate/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py\", line 2956, in create_op\r\n    op_def=op_def)\r\n  File \"/u/menne/py-envs/py2.7tfUpdate/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py\", line 1470, in __init__\r\n    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access\r\n\r\nInvalidArgumentError (see above for traceback): The determinant is not finite.\r\n         [[Node: beamformingLayer/TfGevBeamformer._getFilterMatrix/MatrixDeterminant = MatrixDeterminant[T=DT_COMPLEX64, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](beamformingLayer/TfGevBeamformer._getFilterMatrix/TfMaskBasedGevBeamformer._getSpatialCorrelationByMask/div)]]\r\n         [[Node: optimize/gradients/medianMasks/TopKV2_grad/Reshape_1/_1125 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/device:CPU:0\", send_device=\"/job:localhost/replica:0/task:0/device:GPU:0\", send_device_incarnation=1, tensor_name=\"edge_8777_optimize/gradients/medianMasks/TopKV2_grad/Reshape_1\", tensor_type=DT_INT32, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"]()]]\r\n```\r\n"}}
{"actor":{"display_login":"mennetob"}}