{"action":"created","issue":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/17554","repository_url":"https://api.github.com/repos/JuliaLang/julia","labels_url":"https://api.github.com/repos/JuliaLang/julia/issues/17554/labels{/name}","comments_url":"https://api.github.com/repos/JuliaLang/julia/issues/17554/comments","events_url":"https://api.github.com/repos/JuliaLang/julia/issues/17554/events","html_url":"https://github.com/JuliaLang/julia/issues/17554","id":167014520,"number":17554,"title":"Julia slows down when reading thousands of JLD files","user":{"login":"gasagna","id":744738,"avatar_url":"https://avatars.githubusercontent.com/u/744738?v=3","gravatar_id":"","url":"https://api.github.com/users/gasagna","html_url":"https://github.com/gasagna","followers_url":"https://api.github.com/users/gasagna/followers","following_url":"https://api.github.com/users/gasagna/following{/other_user}","gists_url":"https://api.github.com/users/gasagna/gists{/gist_id}","starred_url":"https://api.github.com/users/gasagna/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gasagna/subscriptions","organizations_url":"https://api.github.com/users/gasagna/orgs","repos_url":"https://api.github.com/users/gasagna/repos","events_url":"https://api.github.com/users/gasagna/events{/privacy}","received_events_url":"https://api.github.com/users/gasagna/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2016-07-22T10:15:11Z","updated_at":"2016-07-22T23:06:40Z","closed_at":"2016-07-22T16:49:36Z","body":"I have some complex piece of code where I need to read thousands of JLD files, each containing some data that I need to process. I have noticed that as files are read, Julia becomes increasingly slower at reading these files. \r\n\r\nHere is minimal working example:\r\n\r\n````julia\r\nusing JLD\r\n\r\n# create `Nfiles`  JLD files in `dir`, each containing some small amount of data\r\nfunction bar(Nfiles, dir)\r\n    !ispath(dir) && mkdir(dir)\r\n    for i = 1:Nfiles\r\n        f = @sprintf \"%05d.jld\" i\r\n        JLD.save(joinpath(dir, f), \"X\", randn(5, 5))\r\n        print(\"\\r \", i)\r\n    end\r\nend\r\n\r\n# Read `Nfiles` in `Nbatch` batches and evaluate the time required to load each batch. \r\n# Return a vector of times\r\nfunction foo(Nfiles, Nbatch)\r\n   M = div(Nfiles, Nbatch)\r\n   s = 0.0\r\n   times = Float64[]\r\n   for j = 1:Nbatch\r\n       I = (j-1)*M + 1\r\n       t = @elapsed for i = I:(I+M-1)\r\n           f = @sprintf \"tmp/%05d.jld\" i\r\n           X = JLD.load(f, \"X\")\r\n           s += sum(X)\r\n       end\r\n       push!(times, t)\r\n       print(\"\\r $j $t\")\r\n   end\r\n   times\r\nend\r\n\r\n# make 50000 files in `tmp`\r\nbar(50000, \"tmp\")\r\n\r\n# now read files in batches of 1000\r\ntimes = foo(50000, 50);\r\n````\r\n\r\nLet's plot the `times` vector.\r\n\r\n![figure_1](https://cloud.githubusercontent.com/assets/744738/17054063/0ba6460a-4ffd-11e6-89e8-5df648a20b73.png)\r\n\r\nAs you can see, the time per batch increases, seemingly linearly with the batch number. If you run the above code in a REPL, then it also takes ages to close it with CTRL+D. Maybe I am just missing something, but this is some weird behaviour I have not seen before.\r\n\r\nI am pinging the JLD/HDF maintainers @timholy, @simonster as it might be related to those packages.\r\n\r\nEDIT:\r\n````julia\r\njulia> versioninfo()\r\nJulia Version 0.4.7-pre+1\r\nCommit 57d0834* (2016-06-19 17:17 UTC)\r\nPlatform Info:\r\n  System: Darwin (x86_64-apple-darwin15.5.0)\r\n  CPU: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz\r\n  WORD_SIZE: 64\r\n  BLAS: libgfortblas\r\n  LAPACK: liblapack\r\n  LIBM: libopenlibm\r\n  LLVM: libLLVM-3.3\r\n````"},"comment":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/comments/234678008","html_url":"https://github.com/JuliaLang/julia/issues/17554#issuecomment-234678008","issue_url":"https://api.github.com/repos/JuliaLang/julia/issues/17554","id":234678008,"user":{"login":"eschnett","id":213514,"avatar_url":"https://avatars.githubusercontent.com/u/213514?v=3","gravatar_id":"","url":"https://api.github.com/users/eschnett","html_url":"https://github.com/eschnett","followers_url":"https://api.github.com/users/eschnett/followers","following_url":"https://api.github.com/users/eschnett/following{/other_user}","gists_url":"https://api.github.com/users/eschnett/gists{/gist_id}","starred_url":"https://api.github.com/users/eschnett/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eschnett/subscriptions","organizations_url":"https://api.github.com/users/eschnett/orgs","repos_url":"https://api.github.com/users/eschnett/repos","events_url":"https://api.github.com/users/eschnett/events{/privacy}","received_events_url":"https://api.github.com/users/eschnett/received_events","type":"User","site_admin":false},"created_at":"2016-07-22T23:06:40Z","updated_at":"2016-07-22T23:06:40Z","body":"The new HDF5 release (version 1.10) is supposed to be much more efficient here. We should update the HDF5 package to use this version.\r\n\r\nBut maybe before that we should ask the original reporter what HDF5 version he/she is using, and what flags when the file is opened?"}}
{"actor":{"display_login":"eschnett"}}