{"action":"created","issue":{"url":"https://api.github.com/repos/cmusatyalab/openface/issues/127","repository_url":"https://api.github.com/repos/cmusatyalab/openface","labels_url":"https://api.github.com/repos/cmusatyalab/openface/issues/127/labels{/name}","comments_url":"https://api.github.com/repos/cmusatyalab/openface/issues/127/comments","events_url":"https://api.github.com/repos/cmusatyalab/openface/issues/127/events","html_url":"https://github.com/cmusatyalab/openface/issues/127","id":150609141,"number":127,"title":"Trained CUDA model unable to run on x86","user":{"login":"bamos","id":707462,"avatar_url":"https://avatars.githubusercontent.com/u/707462?v=3","gravatar_id":"","url":"https://api.github.com/users/bamos","html_url":"https://github.com/bamos","followers_url":"https://api.github.com/users/bamos/followers","following_url":"https://api.github.com/users/bamos/following{/other_user}","gists_url":"https://api.github.com/users/bamos/gists{/gist_id}","starred_url":"https://api.github.com/users/bamos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bamos/subscriptions","organizations_url":"https://api.github.com/users/bamos/orgs","repos_url":"https://api.github.com/users/bamos/repos","events_url":"https://api.github.com/users/bamos/events{/privacy}","received_events_url":"https://api.github.com/users/bamos/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2016-04-24T00:07:52Z","updated_at":"2016-06-13T23:44:07Z","closed_at":"2016-06-13T18:04:09Z","body":"Original Post by @neuraldomi: https://groups.google.com/forum/#!topic/cmu-openface/lhT5Cwjx8XQ\r\n\r\n> My current setup is that I train a DNN on a CUDA server (also classifiers I create with CUDA) and I would like to use the final model.t7 and classifier.pkl on a non CUDA server.\r\n> \r\n> However this does for some reason not work. I found out that classifiers or models created on CUDA machines only work later on machines supporting CUDA. Is there a way around this behavior or I am missing something?\r\n> \r\n> The issue I see is that the file reps.csv has only \"NaN\" values inside or that the classifier gives an error about \"NaN\".\r\n> \r\n> Nothing important but some observation I made.\r\n\r\n---\r\n\r\nThe training code used to support this,  I trained all of the released\r\nOpenFace models with CUDA and they work on x86 and CUDA.\r\nI think this might be from some recent changes to the training code\r\nin #106.\r\n\r\n\\cc @melgor"},"comment":{"url":"https://api.github.com/repos/cmusatyalab/openface/issues/comments/225740762","html_url":"https://github.com/cmusatyalab/openface/issues/127#issuecomment-225740762","issue_url":"https://api.github.com/repos/cmusatyalab/openface/issues/127","id":225740762,"user":{"login":"fmassa","id":9110200,"avatar_url":"https://avatars.githubusercontent.com/u/9110200?v=3","gravatar_id":"","url":"https://api.github.com/users/fmassa","html_url":"https://github.com/fmassa","followers_url":"https://api.github.com/users/fmassa/followers","following_url":"https://api.github.com/users/fmassa/following{/other_user}","gists_url":"https://api.github.com/users/fmassa/gists{/gist_id}","starred_url":"https://api.github.com/users/fmassa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fmassa/subscriptions","organizations_url":"https://api.github.com/users/fmassa/orgs","repos_url":"https://api.github.com/users/fmassa/repos","events_url":"https://api.github.com/users/fmassa/events{/privacy}","received_events_url":"https://api.github.com/users/fmassa/received_events","type":"User","site_admin":false},"created_at":"2016-06-13T23:44:07Z","updated_at":"2016-06-13T23:44:07Z","body":"Hi,\r\nJust a few thoughts on this.\r\n* maybe the issue we are seeing is due to inputting a non-initialized tensor to the network, and the output of the network after a few layers is all zero. This could lead to nans depending of the way the std is computed. Could you check if randomly initializing the input tensor with https://github.com/cmusatyalab/openface/pull/118/files#diff-274250189174f880cbc124a766174416R63 helps with the issue?\r\n* the nans in the weight/bias could happen because of https://github.com/soumith/cudnn.torch/pull/199 . Maybe we would need to feed a batch of size 2? \r\n\r\nI think one possible solution would be to reset all the `running_mean` and `running_var` to their initial values after optimizing the network, as we are feeding a random input they are not really contributing to the BatchNorm statistics. This could be done by manually copying the old statistics, or by adding a `freezeStats` method in BatchNorm that do not update the stats during train mode.\r\n\r\nAny thoughts on this ?"}}
{"actor":{"display_login":"fmassa"}}