{"action":"opened","issue":{"url":"https://api.github.com/repos/intel-analytics/BigDL/issues/2617","repository_url":"https://api.github.com/repos/intel-analytics/BigDL","labels_url":"https://api.github.com/repos/intel-analytics/BigDL/issues/2617/labels{/name}","comments_url":"https://api.github.com/repos/intel-analytics/BigDL/issues/2617/comments","events_url":"https://api.github.com/repos/intel-analytics/BigDL/issues/2617/events","html_url":"https://github.com/intel-analytics/BigDL/issues/2617","id":352768845,"node_id":"MDU6SXNzdWUzNTI3Njg4NDU=","number":2617,"title":"Error 'assertion failed: total weight must larger than 0' - occasional failures when using NLLCriterion with some unlabeled data","user":{"login":"griffinkardos","id":30094626,"node_id":"MDQ6VXNlcjMwMDk0NjI2","avatar_url":"https://avatars0.githubusercontent.com/u/30094626?v=4","gravatar_id":"","url":"https://api.github.com/users/griffinkardos","html_url":"https://github.com/griffinkardos","followers_url":"https://api.github.com/users/griffinkardos/followers","following_url":"https://api.github.com/users/griffinkardos/following{/other_user}","gists_url":"https://api.github.com/users/griffinkardos/gists{/gist_id}","starred_url":"https://api.github.com/users/griffinkardos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/griffinkardos/subscriptions","organizations_url":"https://api.github.com/users/griffinkardos/orgs","repos_url":"https://api.github.com/users/griffinkardos/repos","events_url":"https://api.github.com/users/griffinkardos/events{/privacy}","received_events_url":"https://api.github.com/users/griffinkardos/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-08-22T01:37:34Z","updated_at":"2018-08-22T01:37:34Z","closed_at":null,"author_association":"NONE","body":"I'm training on an dataset with some unlabeled data, using a parallel criterion, where one of the criteria is a reconstruction loss, and the other is a classification loss, which should be ignored for unlabeled data. When I call optimizer.optimize() with this criterion, it trains for some number of iterations (i.e, it doesn't fail on every iteration), before crashing with this error:\r\n\r\n```\r\n2018-08-21 23:56:02 ERROR ThreadPool$:115 - Error: java.lang.AssertionError: assertion failed: total weight must larger than 0\r\n\tat scala.Predef$.assert(Predef.scala:170)\r\n\tat com.intel.analytics.bigdl.nn.ClassNLLCriterion.updateGradInput(ClassNLLCriterion.scala:169)\r\n\tat com.intel.analytics.bigdl.nn.CrossEntropyCriterion.updateGradInput(CrossEntropyCriterion.scala:48)\r\n\tat com.intel.analytics.bigdl.nn.CrossEntropyCriterion.updateGradInput(CrossEntropyCriterion.scala:31)\r\n\tat com.intel.analytics.bigdl.nn.ParallelCriterion.updateGradInput(ParallelCriterion.scala:74)\r\n\tat com.intel.analytics.bigdl.nn.ParallelCriterion.updateGradInput(ParallelCriterion.scala:35)\r\n\tat com.intel.analytics.bigdl.nn.abstractnn.AbstractCriterion.backward(AbstractCriterion.scala:84)\r\n\tat com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$6$$anonfun$apply$2.apply$mcI$sp(DistriOptimizer.scala:258)\r\n\tat com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$6$$anonfun$apply$2.apply(DistriOptimizer.scala:249)\r\n\tat com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$4$$anonfun$6$$anonfun$apply$2.apply(DistriOptimizer.scala:249)\r\n\tat com.intel.analytics.bigdl.utils.ThreadPool$$anonfun$1$$anon$4.call(ThreadPool.scala:112)\r\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n\tat java.lang.Thread.run(Thread.java:748)\r\n```\r\n\r\nI've optimized the same model with the same criterion on the same data with the unlabeled samples filtered out just fine. But I'd like to make use of the unlabeled data too.\r\nAccording to the docs, \r\n> Note that if the target is -1, the training process will skip this sample. In other words, the forward process will return zero output and the backward process will also return zero gradInput.\r\n\r\nExperimentally however, this works out with just a single data point:\r\n```\r\ntest_criterion = ParallelCriterion()\r\ntest_criterion.add(MSECriterion()).add(CrossEntropyCriterion())\r\n\r\ntest_criterion = ParallelCriterion()\r\ntest_criterion.add(MSECriterion()).add(CrossEntropyCriterion())\r\n\r\npred = np.array([0.1,0.5,2,1,0.])\r\nrecon = np.array([1.,3.,4.])\r\ncombined = [recon, pred]\r\ntarget = [np.array([2.,2.,5.]),np.array([-1])]\r\nprint(test_criterion.forward(combined, target))\r\nprint(test_criterion.backward(combined, target))\r\n```\r\n\r\nIn this case forwards gives an appropriate loss (i.e it ignores the class scores), and backwards also gives an appropriate gradient.\r\n\r\nHowever if I make this into a minibatch and do the same:\r\n```\r\ntest_criterion = ParallelCriterion()\r\ntest_criterion.add(MSECriterion()).add(CrossEntropyCriterion())\r\n\r\npred = np.array([[0.1,0.5,2,1,0.]])\r\nrecon = np.array([[1.,3.,4.]])\r\ncombined = [recon, pred]\r\ntarget = [np.array([[2.,2.,5.]]),np.array([[-1]])]\r\nprint(test_criterion.forward(combined, target))\r\nprint(test_criterion.backward(combined, target))\r\n```\r\nI get the same error as above, that the total weight needs to be greater than 0. If I do a minibatch where one target is -1 and the other is a positive integer, it works fine too. \r\n\r\nLooking at the source for ClassNLLCriterion, it appears that, for a single sample, the default weight is 1 regardess of whether the target is -1, whereas with the batch case, the default value is 0 for -1 labels, and 1 otherwise, which makes sense so you don't include those samples in calculating the total weight of a batch, but also means it fails if all the samples are unlabeled, and it would be more desirable for me if it would still return a forwards loss of 0, and a backwards gradient of zero. Not sure what the desired behavior or optimal workaround is though."}}
{"actor":{"display_login":"griffinkardos"}}