{"action":"created","issue":{"url":"https://api.github.com/repos/dawidkski/bert-tf2/issues/1","repository_url":"https://api.github.com/repos/dawidkski/bert-tf2","labels_url":"https://api.github.com/repos/dawidkski/bert-tf2/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/dawidkski/bert-tf2/issues/1/comments","events_url":"https://api.github.com/repos/dawidkski/bert-tf2/issues/1/events","html_url":"https://github.com/dawidkski/bert-tf2/issues/1","id":452184446,"node_id":"MDU6SXNzdWU0NTIxODQ0NDY=","number":1,"title":"ValueError: No gradients provided for any variable","user":{"login":"dawidkski","id":16166575,"node_id":"MDQ6VXNlcjE2MTY2NTc1","avatar_url":"https://avatars0.githubusercontent.com/u/16166575?v=4","gravatar_id":"","url":"https://api.github.com/users/dawidkski","html_url":"https://github.com/dawidkski","followers_url":"https://api.github.com/users/dawidkski/followers","following_url":"https://api.github.com/users/dawidkski/following{/other_user}","gists_url":"https://api.github.com/users/dawidkski/gists{/gist_id}","starred_url":"https://api.github.com/users/dawidkski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dawidkski/subscriptions","organizations_url":"https://api.github.com/users/dawidkski/orgs","repos_url":"https://api.github.com/users/dawidkski/repos","events_url":"https://api.github.com/users/dawidkski/events{/privacy}","received_events_url":"https://api.github.com/users/dawidkski/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2019-06-04T20:09:39Z","updated_at":"2019-06-16T22:10:08Z","closed_at":"2019-06-16T22:10:08Z","author_association":"OWNER","body":"**I'm getting this error:**\r\n\r\n  `ValueError: No gradients provided for any variable: ['bert_text_classifier/bert_model/bert_embeddings/embedding/embeddings:0', ..., ], which says that theres no gradient for any of the models parameters.\r\n\r\n**train function that cause the error**\r\n```python\r\n@tf.function\r\ndef train(model, dataset, optimizer, dtype=tf.float32):\r\n    for batch_i, batch in enumerate(dataset):\r\n        input_tensors = [\r\n            batch['input_ids'],\r\n            batch['input_mask'],\r\n            batch['position_ids'],\r\n            batch['segment_ids'],\r\n        ]\r\n\r\n        with tf.GradientTape(persistent=True) as tape:\r\n            result = model(input_tensors)\r\n            loss = cross_entropy_loss(tf.argmax(result, axis=1), tf.cast(batch['label_ids'], dtype=dtype))\r\n\r\n        gradients = tape.gradient(loss, model.trainable_variables)\r\n        optimizer.apply_gradients(zip(gradients, model.trainable_variables))  # < -- this line raises error\r\n        print(f\"batch_i: {batch_i}, loss: {loss}\")\r\n```\r\n\r\n**Sanity checklist:**\r\n- model outputs values that kind of make sense to me\r\n- loss is not zero or None and changes with each new batch\r\n- tf-2.0 nightly preview tested on both cpu only version  and gpu version\r\n- do or do not wrap train function with @tf.function decorator\r\n\r\n**My guess**\r\nThere are suggestions on tensorflow issue page that this might be caused by some operations that are not wrapped with proper keras functons. This cuts the chain of calculations which prevents the gradient to flow.\r\n- [ ] look for any obsure tf1 operation used in the codebase\r\n- [ ] make sure that in trainng time graph is correctly constructed ( idk Dropouts or something)"},"comment":{"url":"https://api.github.com/repos/dawidkski/bert-tf2/issues/comments/502489996","html_url":"https://github.com/dawidkski/bert-tf2/issues/1#issuecomment-502489996","issue_url":"https://api.github.com/repos/dawidkski/bert-tf2/issues/1","id":502489996,"node_id":"MDEyOklzc3VlQ29tbWVudDUwMjQ4OTk5Ng==","user":{"login":"dawidkski","id":16166575,"node_id":"MDQ6VXNlcjE2MTY2NTc1","avatar_url":"https://avatars0.githubusercontent.com/u/16166575?v=4","gravatar_id":"","url":"https://api.github.com/users/dawidkski","html_url":"https://github.com/dawidkski","followers_url":"https://api.github.com/users/dawidkski/followers","following_url":"https://api.github.com/users/dawidkski/following{/other_user}","gists_url":"https://api.github.com/users/dawidkski/gists{/gist_id}","starred_url":"https://api.github.com/users/dawidkski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dawidkski/subscriptions","organizations_url":"https://api.github.com/users/dawidkski/orgs","repos_url":"https://api.github.com/users/dawidkski/repos","events_url":"https://api.github.com/users/dawidkski/events{/privacy}","received_events_url":"https://api.github.com/users/dawidkski/received_events","type":"User","site_admin":false},"created_at":"2019-06-16T22:10:08Z","updated_at":"2019-06-16T22:10:08Z","author_association":"OWNER","body":"for some reasons, I thought tf.argmax was differentiable, lmao"}}
{"actor":{"display_login":"dawidkski"}}