{"action":"closed","issue":{"url":"https://api.github.com/repos/deeplearning4j/nd4j/issues/1405","repository_url":"https://api.github.com/repos/deeplearning4j/nd4j","labels_url":"https://api.github.com/repos/deeplearning4j/nd4j/issues/1405/labels{/name}","comments_url":"https://api.github.com/repos/deeplearning4j/nd4j/issues/1405/comments","events_url":"https://api.github.com/repos/deeplearning4j/nd4j/issues/1405/events","html_url":"https://github.com/deeplearning4j/nd4j/issues/1405","id":187751440,"number":1405,"title":"Wrong / inconsistent standard deviations in NormalizerStandardize","user":{"login":"EdeMeijer","id":5758565,"avatar_url":"https://avatars.githubusercontent.com/u/5758565?v=3","gravatar_id":"","url":"https://api.github.com/users/EdeMeijer","html_url":"https://github.com/EdeMeijer","followers_url":"https://api.github.com/users/EdeMeijer/followers","following_url":"https://api.github.com/users/EdeMeijer/following{/other_user}","gists_url":"https://api.github.com/users/EdeMeijer/gists{/gist_id}","starred_url":"https://api.github.com/users/EdeMeijer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/EdeMeijer/subscriptions","organizations_url":"https://api.github.com/users/EdeMeijer/orgs","repos_url":"https://api.github.com/users/EdeMeijer/repos","events_url":"https://api.github.com/users/EdeMeijer/events{/privacy}","received_events_url":"https://api.github.com/users/EdeMeijer/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2016-11-07T16:04:20Z","updated_at":"2016-11-08T17:58:59Z","closed_at":"2016-11-08T17:58:58Z","body":"Fitting a DataSet gives a bias corrected std while fitting an iterator gives a non bias corrected std. Furthermore, iterators that don't produce equally sized chunks, even for 2D cases, give totally whack stds.\r\n\r\n```\r\n     @Test\r\n    public void testDifferentBatchSizes() {\r\n        // Create 6x1 matrix of the numbers 1 through 6\r\n        INDArray values = Nd4j.linspace(1, 6, 6).transpose();\r\n\r\n        DataSet dataSet = new DataSet(values, values);\r\n        NormalizerStandardize norm1 = new NormalizerStandardize();\r\n        norm1.fit(dataSet);\r\n        assertEquals(3.5f, norm1.getMean().getFloat(0), 1e-6);\r\n        assertEquals(1.8708387f, norm1.getStd().getFloat(0), 1e-6); // <-- bias corrected\r\n\r\n        DataSetIterator testIter1 = new TestDataSetIterator(dataSet, 3); // Will yield 2 batches of 3 rows\r\n        NormalizerStandardize norm2 = new NormalizerStandardize();\r\n        norm2.fit(testIter1);\r\n        assertEquals(3.5f, norm2.getMean().getFloat(0), 1e-6);\r\n        assertEquals(1.7078355f, norm2.getStd().getFloat(0), 1e-6); // <-- not bias corrected\r\n\r\n        DataSetIterator testIter2 = new TestDataSetIterator(dataSet, 4); // Will yield batch of 4 and batch of 2 rows\r\n        NormalizerStandardize norm3 = new NormalizerStandardize();\r\n        norm3.fit(testIter2);\r\n        assertEquals(3.5f, norm3.getMean().getFloat(0), 1e-6);\r\n        assertEquals(1.5433291f, norm3.getStd().getFloat(0), 1e-6); // <-- What's this? Not even close.\r\n    }\r\n```"}}
{"actor":{"display_login":"eraly"}}