{"action":"created","issue":{"url":"https://api.github.com/repos/joblib/joblib/issues/260","labels_url":"https://api.github.com/repos/joblib/joblib/issues/260/labels{/name}","comments_url":"https://api.github.com/repos/joblib/joblib/issues/260/comments","events_url":"https://api.github.com/repos/joblib/joblib/issues/260/events","html_url":"https://github.com/joblib/joblib/pull/260","id":113053476,"number":260,"title":"[MRG] Numpy pickle to single file","user":{"login":"aabadie","id":1375137,"avatar_url":"https://avatars.githubusercontent.com/u/1375137?v=3","gravatar_id":"","url":"https://api.github.com/users/aabadie","html_url":"https://github.com/aabadie","followers_url":"https://api.github.com/users/aabadie/followers","following_url":"https://api.github.com/users/aabadie/following{/other_user}","gists_url":"https://api.github.com/users/aabadie/gists{/gist_id}","starred_url":"https://api.github.com/users/aabadie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aabadie/subscriptions","organizations_url":"https://api.github.com/users/aabadie/orgs","repos_url":"https://api.github.com/users/aabadie/repos","events_url":"https://api.github.com/users/aabadie/events{/privacy}","received_events_url":"https://api.github.com/users/aabadie/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":122,"created_at":"2015-10-23T16:22:17Z","updated_at":"2016-01-12T17:23:18Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/joblib/joblib/pulls/260","html_url":"https://github.com/joblib/joblib/pull/260","diff_url":"https://github.com/joblib/joblib/pull/260.diff","patch_url":"https://github.com/joblib/joblib/pull/260.patch"},"body":"I gave a try to gzip.GzipFile to dump numpy object. This PR is a POC of such usage and also contains a fairly large refactoring of numpy_pickle.\r\n\r\nIn terms of performance, I reused the bench script from #255. With Numpy 1.10, there's no memory copy but the read/write is slightly slower (31s dump, 5.5s read). It also makes the code more 'readable'.\r\nI had to comment out the test regarding the compatibility between pickles.\r\n\r\nWaiting for the CI's"},"comment":{"url":"https://api.github.com/repos/joblib/joblib/issues/comments/170981623","html_url":"https://github.com/joblib/joblib/pull/260#issuecomment-170981623","issue_url":"https://api.github.com/repos/joblib/joblib/issues/260","id":170981623,"user":{"login":"aabadie","id":1375137,"avatar_url":"https://avatars.githubusercontent.com/u/1375137?v=3","gravatar_id":"","url":"https://api.github.com/users/aabadie","html_url":"https://github.com/aabadie","followers_url":"https://api.github.com/users/aabadie/followers","following_url":"https://api.github.com/users/aabadie/following{/other_user}","gists_url":"https://api.github.com/users/aabadie/gists{/gist_id}","starred_url":"https://api.github.com/users/aabadie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aabadie/subscriptions","organizations_url":"https://api.github.com/users/aabadie/orgs","repos_url":"https://api.github.com/users/aabadie/repos","events_url":"https://api.github.com/users/aabadie/events{/privacy}","received_events_url":"https://api.github.com/users/aabadie/received_events","type":"User","site_admin":false},"created_at":"2016-01-12T17:23:18Z","updated_at":"2016-01-12T17:23:18Z","body":"> I tried it but didn't notice any sensible speed improvement.\r\n\r\nI spoke too fast. Your buffered writer is slightly faster than the one coming form the `io` module. I tried to implement an equivalent version for the reader one but couldn't reach good speed results (mine is twice slower than io.BufferedReader).\r\n\r\nI also included the benchmark script I used to test different strategies in the last commit (see the OrderedDict's at the bottom of the file https://github.com/aabadie/joblib/commit/4a1358796af8d2dd089b87b5c7e8549b2eeb6b60#diff-58ae84bc1a24dcb6a5ffdebf7be74fa4R205) :\r\n* multiple type of objects: dict, list, arrays\r\n* multiple picklers/unpicklers: pickle, cPickle, joblib\r\n* multiple buffer strategies: None, io.BytesIO, io.BufferedWriter/Reader, PicklerBufferedWriter/Reader (implemented in the bench script itself)\r\n* multiple compressors: None, zlib, joblib, gzip, bz2, lzma, xz.\r\n\r\nI won't comment on the compressors but I can on the buffering strategies as it appears they had an impact on the performances between the actual strategy in master (based on io.BytesIO) and the one in this PR.\r\nSo, comparing master and this PR, the number of system calls to write and read are the same but takes more cumulated time in the later, the time spent compressing/decompressing is the same in each case.\r\nThe conclusion is that the problem comes from the fact that the read/write methods of the compressor/decompressor file object are called a huge amount of time and are python code. In io.BytesIO, it's probably C code, that's why it's a bit faster.\r\n\r\nI didn't include the PickleBufferedWriter (it's still in the compression benchmark script). Maybe these optimization strategies could be part of another PR ?\r\n\r\nTo me, this PR is ready to be reviewed (again ;) ).\r\n"}}