{"action":"created","issue":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/5553","repository_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j","labels_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/5553/labels{/name}","comments_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/5553/comments","events_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/5553/events","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/5553","id":331042347,"node_id":"MDU6SXNzdWUzMzEwNDIzNDc=","number":5553,"title":"DL4J: CuDNN batch norm fails with FP16","user":{"login":"AlexDBlack","id":2360237,"node_id":"MDQ6VXNlcjIzNjAyMzc=","avatar_url":"https://avatars0.githubusercontent.com/u/2360237?v=4","gravatar_id":"","url":"https://api.github.com/users/AlexDBlack","html_url":"https://github.com/AlexDBlack","followers_url":"https://api.github.com/users/AlexDBlack/followers","following_url":"https://api.github.com/users/AlexDBlack/following{/other_user}","gists_url":"https://api.github.com/users/AlexDBlack/gists{/gist_id}","starred_url":"https://api.github.com/users/AlexDBlack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlexDBlack/subscriptions","organizations_url":"https://api.github.com/users/AlexDBlack/orgs","repos_url":"https://api.github.com/users/AlexDBlack/repos","events_url":"https://api.github.com/users/AlexDBlack/events{/privacy}","received_events_url":"https://api.github.com/users/AlexDBlack/received_events","type":"User","site_admin":false},"labels":[{"id":67900999,"node_id":"MDU6TGFiZWw2NzkwMDk5OQ==","url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/labels/Bug","name":"Bug","color":"d93f0b","default":false},{"id":934488384,"node_id":"MDU6TGFiZWw5MzQ0ODgzODQ=","url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/labels/DL4J","name":"DL4J","color":"98f2d7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-06-11T04:32:48Z","updated_at":"2018-06-11T06:07:15Z","closed_at":null,"author_association":"MEMBER","body":"\r\nModel: RESNET50. OK on FP32, fails on FP16.\r\nCUDA 9.1, V100\r\n\r\n```\r\nException in thread \"main\" java.lang.RuntimeException: cuDNN status = 3: CUDNN_STATUS_BAD_PARAM\r\n        at org.deeplearning4j.nn.layers.BaseCudnnHelper.checkCudnn(BaseCudnnHelper.java:45)\r\n        at org.deeplearning4j.nn.layers.normalization.CudnnBatchNormalizationHelper.preOutput(CudnnBatchNormalizationHelper.java:233)\r\n        at org.deeplearning4j.nn.layers.normalization.BatchNormalization.preOutput(BatchNormalization.java:272)\r\n        at org.deeplearning4j.nn.layers.normalization.BatchNormalization.activate(BatchNormalization.java:235)\r\n        at org.deeplearning4j.nn.graph.vertex.impl.LayerVertex.doForward(LayerVertex.java:105)\r\n        at org.deeplearning4j.nn.graph.ComputationGraph.ffToLayerActivationsInWS(ComputationGraph.java:2055)\r\n        at org.deeplearning4j.nn.graph.ComputationGraph.computeGradientAndScore(ComputationGraph.java:1403)\r\n        at org.deeplearning4j.nn.graph.ComputationGraph.computeGradientAndScore(ComputationGraph.java:1372)\r\n        at org.deeplearning4j.optimize.solvers.BaseOptimizer.gradientAndScore(BaseOptimizer.java:162)\r\n        at org.deeplearning4j.optimize.solvers.StochasticGradientDescent.optimize(StochasticGradientDescent.java:53)\r\n        at org.deeplearning4j.optimize.Solver.optimize(Solver.java:54)\r\n        at org.deeplearning4j.nn.graph.ComputationGraph.fitHelper(ComputationGraph.java:1195)\r\n        at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1144)\r\n        at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1112)\r\n        at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1050)\r\n        at org.deeplearning4j.benchmarks.BaseBenchmark.benchmark(BaseBenchmark.java:103)\r\n        at org.deeplearning4j.benchmarks.BaseBenchmark$Benchmark.execute(BaseBenchmark.java:40)\r\n        at org.deeplearning4j.benchmarks.BenchmarkCnn.run(BenchmarkCnn.java:109)\r\n        at org.deeplearning4j.benchmarks.BenchmarkCnn.main(BenchmarkCnn.java:124)\r\n```"},"comment":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/comments/396131960","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/5553#issuecomment-396131960","issue_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/5553","id":396131960,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NjEzMTk2MA==","user":{"login":"saudet","id":3328023,"node_id":"MDQ6VXNlcjMzMjgwMjM=","avatar_url":"https://avatars1.githubusercontent.com/u/3328023?v=4","gravatar_id":"","url":"https://api.github.com/users/saudet","html_url":"https://github.com/saudet","followers_url":"https://api.github.com/users/saudet/followers","following_url":"https://api.github.com/users/saudet/following{/other_user}","gists_url":"https://api.github.com/users/saudet/gists{/gist_id}","starred_url":"https://api.github.com/users/saudet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saudet/subscriptions","organizations_url":"https://api.github.com/users/saudet/orgs","repos_url":"https://api.github.com/users/saudet/repos","events_url":"https://api.github.com/users/saudet/events{/privacy}","received_events_url":"https://api.github.com/users/saudet/received_events","type":"User","site_admin":false},"created_at":"2018-06-11T06:07:15Z","updated_at":"2018-06-11T06:07:15Z","author_association":"MEMBER","body":"There are functions for such data conversion scattered around, such as here:\r\nhttps://github.com/deeplearning4j/deeplearning4j/blob/master/nd4j/nd4j-backends/nd4j-backend-impls/nd4j-cuda/src/main/java/org/nd4j/linalg/jcublas/JCublasNDArrayFactory.java#L1393\r\n@raver119 would know best what to call though :)"}}
{"actor":{"display_login":"saudet"}}