{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16115","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16115/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16115/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16115/events","html_url":"https://github.com/tensorflow/tensorflow/issues/16115","id":288448052,"number":16115,"title":"Gradients w.r.t. eigenvalues/eigenvectors ","user":{"login":"mingu6","id":25823392,"avatar_url":"https://avatars1.githubusercontent.com/u/25823392?v=4","gravatar_id":"","url":"https://api.github.com/users/mingu6","html_url":"https://github.com/mingu6","followers_url":"https://api.github.com/users/mingu6/followers","following_url":"https://api.github.com/users/mingu6/following{/other_user}","gists_url":"https://api.github.com/users/mingu6/gists{/gist_id}","starred_url":"https://api.github.com/users/mingu6/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mingu6/subscriptions","organizations_url":"https://api.github.com/users/mingu6/orgs","repos_url":"https://api.github.com/users/mingu6/repos","events_url":"https://api.github.com/users/mingu6/events{/privacy}","received_events_url":"https://api.github.com/users/mingu6/received_events","type":"User","site_admin":false},"labels":[{"id":404586594,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20tensorflower","name":"stat:awaiting tensorflower","color":"b60205","default":false},{"id":473172988,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/type:bug/performance","name":"type:bug/performance","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-01-14T23:08:53Z","updated_at":"2018-02-06T07:36:01Z","closed_at":null,"author_association":"NONE","body":"Please go to Stack Overflow for help and support:\r\n\r\nhttps://stackoverflow.com/questions/tagged/tensorflow\r\n\r\nIf you open a GitHub issue, here is our policy:\r\n\r\n1. It must be a bug or a feature request.\r\n2. The form below must be filled out.\r\n3. It shouldn't be a TensorBoard issue. Those go [here](https://github.com/tensorflow/tensorboard/issues).\r\n\r\n**Here's why we have that policy**: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.\r\n\r\n------------------------\r\n\r\n### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: YES\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: macOS High Sierra 10.13.1\r\n- **TensorFlow installed from (source or binary)**: pip install tensorflow\r\n- **TensorFlow version (use command below)**: v1.2.0-5-g435cdfc 1.2.1\r\n- **Python version**: 3.5.2\r\n- **Bazel version (if compiling from source)**: N/A\r\n- **GCC/Compiler version (if compiling from source)**: N/A\r\n- **CUDA/cuDNN version**: N/A\r\n- **GPU model and memory**: N/A\r\n- **Exact command to reproduce**: see code\r\n\r\nYou can collect some of this information using our environment capture script:\r\n\r\nhttps://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh\r\n\r\nYou can obtain the TensorFlow version with\r\n\r\npython -c \"import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)\"\r\n\r\n### Describe the problem\r\nDescribe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.\r\n\r\nHi,\r\n\r\nI'm having issues with evaluating the gradients of eigenvalues/eigenvectors with respect to the underlying matrix. Tensorflow evaluates the gradients, but these are not correct when compared to the true gradients.\r\n\r\nWe are using tf.self_adjoint_eig to evaluate the spectral decomposition for the tensorflow variable input. We have initialised this with a symmetric matrix to satisfy the self adjoint operator property. We wish to take the derivative of individual eigenvalues or eigenvector with respect to the original matrix (note for eigenvectors we take one element from one eigenvector, e.g. element 2 in eigenvector 1 to avoid the issue of gradient aggregation for now).\r\n\r\nThe methodology for evaluating true gradients of eigenvalues and vectors of a symmetric real matrix can be found in the paper \"On differentiating Eigenvalues and Eigenvectors\" by Magnus (1985), Theorem 1 eqn (6) + (7). We evaluated using our input matrix the gradients under this paper and compared it to tensorflow evaluated gradients and the gradients from finite difference approximation. For the eigenvalues, the gradients are similar (identical on diagonal entries, off by a factor of 2 on off-diagonal elements), however the eigenvectors are off by quite a bit outside the diagonal entries. To start, define a matrix A as (excuse the matrix output formatting from Python)\r\n\r\nA=[[-3 -2  4]\r\n      [-2  1  1]\r\n      [ 4  1  5]]\r\n\r\nusing np.linalg.eig and tf.self_adjoint_eig on A (I simply initialised a variable with A and computed the gradient for the tf implementation)\r\n\r\nTensorflow eigenvalues: [-5.43071561  1.76904987  6.66166575]\r\nPython eigenvalues: [-5.43071561  6.66166575  1.76904987]\r\n\r\nI now wish to evaluate the gradient of eigenvalue 1 (-5.43071561) w.r.t. A. \r\n\r\nAnalytical gradient:\r\n[[ 0.75896178  0.28555906 -0.31842553]\r\n [ 0.28555906  0.10744148 -0.11980748]\r\n [-0.31842553 -0.11980748  0.13359674]]\r\n\r\nTensorflow gradient:\r\n[[[ 0.75896178,  0.        ,  0.        ],\r\n   [ 0.57111812,  0.10744148,  0.        ],\r\n  [-0.63685107, -0.23961495,  0.13359674]]]\r\n\r\nThe diagonal entries are the same but the off-diagonal entries are clearly off by a factor of 2.  Now we try and evaluate gradients for the eigenvectors.\r\n\r\nTensorflow eigenvectors:\r\n[[-0.87118413 -0.31452619 -0.37697678]\r\n [-0.32778267  0.94426587 -0.0303395 ]\r\n [ 0.36550888  0.09713517 -0.92572567]]\r\nPython eigenvectors:\r\n[[ 0.87118413  0.37697678 -0.31452619]\r\n [ 0.32778267  0.0303395   0.94426587]\r\n [-0.36550888  0.92572567  0.09713517]]\r\n\r\nWe try and find the gradient of the eigenvector 1 element 2 (+/-0.32778267). We expect the tensorflow gradient to the equivalent to the analytical gradient (after taking into account the sign difference).\r\n\r\nAnalytical gradient:\r\n[[ 0.03511309 -0.10795607 -0.01312188]\r\n [ 0.01321128 -0.04061843 -0.0049371 ]\r\n [-0.01473184  0.04529341  0.00550534]]\r\n\r\nTensorflow gradient:\r\n[[[-0.03511309,  0.        ,  0.        ],\r\n   [ 0.09474478,  0.04061843,  0.        ],\r\n   [ 0.02785372, -0.04035631, -0.00550534]]]\r\n\r\nBesides the entries being different between the tensorflow evaluated and real gradients, one issue is that tensorflow only returns the lower triangle of the gradient. Despite A being symmetric, the gradient matrix is not symmetric (as per the true gradient) and so the upper right triangle shouldn't be zeros. I believe this to be a bug, however I understand there may be reasons for the differences. Thank you for reading!\r\n\r\n### Source code / logs\r\nInclude any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.\r\n\r\nHere is a script that reproduces the above.\r\n\r\n[eigen_decomp_examplev2.py.zip](https://github.com/tensorflow/tensorflow/files/1546631/eigen_decomp_examplev2.py.zip)\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/363335924","html_url":"https://github.com/tensorflow/tensorflow/issues/16115#issuecomment-363335924","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16115","id":363335924,"user":{"login":"tensorflowbutler","id":28546240,"avatar_url":"https://avatars2.githubusercontent.com/u/28546240?v=4","gravatar_id":"","url":"https://api.github.com/users/tensorflowbutler","html_url":"https://github.com/tensorflowbutler","followers_url":"https://api.github.com/users/tensorflowbutler/followers","following_url":"https://api.github.com/users/tensorflowbutler/following{/other_user}","gists_url":"https://api.github.com/users/tensorflowbutler/gists{/gist_id}","starred_url":"https://api.github.com/users/tensorflowbutler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tensorflowbutler/subscriptions","organizations_url":"https://api.github.com/users/tensorflowbutler/orgs","repos_url":"https://api.github.com/users/tensorflowbutler/repos","events_url":"https://api.github.com/users/tensorflowbutler/events{/privacy}","received_events_url":"https://api.github.com/users/tensorflowbutler/received_events","type":"User","site_admin":false},"created_at":"2018-02-06T07:36:01Z","updated_at":"2018-02-06T07:36:01Z","author_association":"MEMBER","body":"Nagging Awaiting TensorFlower: It has been 14 days with no activity and the `awaiting tensorflower` label was assigned. Please update the label and/or status accordingly."}}
{"actor":{"display_login":"tensorflowbutler"}}