{"action":"created","issue":{"url":"https://api.github.com/repos/duplicati/duplicati/issues/1745","repository_url":"https://api.github.com/repos/duplicati/duplicati","labels_url":"https://api.github.com/repos/duplicati/duplicati/issues/1745/labels{/name}","comments_url":"https://api.github.com/repos/duplicati/duplicati/issues/1745/comments","events_url":"https://api.github.com/repos/duplicati/duplicati/issues/1745/events","html_url":"https://github.com/duplicati/duplicati/issues/1745","id":152732136,"number":1745,"title":"Extremely heavy disk I/O resulting in unreasonable performance","user":{"login":"dsl","id":380103,"avatar_url":"https://avatars.githubusercontent.com/u/380103?v=3","gravatar_id":"","url":"https://api.github.com/users/dsl","html_url":"https://github.com/dsl","followers_url":"https://api.github.com/users/dsl/followers","following_url":"https://api.github.com/users/dsl/following{/other_user}","gists_url":"https://api.github.com/users/dsl/gists{/gist_id}","starred_url":"https://api.github.com/users/dsl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dsl/subscriptions","organizations_url":"https://api.github.com/users/dsl/orgs","repos_url":"https://api.github.com/users/dsl/repos","events_url":"https://api.github.com/users/dsl/events{/privacy}","received_events_url":"https://api.github.com/users/dsl/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-05-03T09:22:19Z","updated_at":"2016-05-05T16:22:51Z","closed_at":null,"body":"Hi,\r\n\r\nI recently updated a duplicati installation from 2.0.0.73 to 2.0.1.11.\r\n\r\nPrior to this, the backups (I run 9 separate backups sequentially) were completing within around 6 hours, but following the upgrade, and despite my efforts to improve disk I/O performance in the virtual machine, it's been running for over 3 days (the last of the 9 backups started 2 days ago so accounts for the bulk of this time).\r\n\r\nI noticed quite quickly using strace that the process was busily seeking heavily and reading/writing data between files.\r\n\r\nTurning on full debugging/profiling showed that sqlite was creating a temporary table.\r\n\r\nThe above information prompted me to improve disk I/O performance for the VM, which I did, in the hope that it would fix the problem, but although it made a difference, it's still a long way off from how it was prior to the update.\r\n\r\nI've attached a profiling log for the one that is still running, and can add further information as required as I'm going to let it keep going for now.\r\n\r\n[duplicati.log.txt](https://github.com/duplicati/duplicati/files/246567/duplicati.log.txt)\r\n\r\nFor this last backup location, it's 23Gb of data, 28k directories, 620k files. The sqlite database is 7.2Gb, which in itself seems excessive too.. :|\r\n\r\nAs mentioned, happy to help debug further etc, in order to improve performance.\r\n\r\nThanks."},"comment":{"url":"https://api.github.com/repos/duplicati/duplicati/issues/comments/217200575","html_url":"https://github.com/duplicati/duplicati/issues/1745#issuecomment-217200575","issue_url":"https://api.github.com/repos/duplicati/duplicati/issues/1745","id":217200575,"user":{"login":"FootStark","id":17236227,"avatar_url":"https://avatars.githubusercontent.com/u/17236227?v=3","gravatar_id":"","url":"https://api.github.com/users/FootStark","html_url":"https://github.com/FootStark","followers_url":"https://api.github.com/users/FootStark/followers","following_url":"https://api.github.com/users/FootStark/following{/other_user}","gists_url":"https://api.github.com/users/FootStark/gists{/gist_id}","starred_url":"https://api.github.com/users/FootStark/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/FootStark/subscriptions","organizations_url":"https://api.github.com/users/FootStark/orgs","repos_url":"https://api.github.com/users/FootStark/repos","events_url":"https://api.github.com/users/FootStark/events{/privacy}","received_events_url":"https://api.github.com/users/FootStark/received_events","type":"User","site_admin":false},"created_at":"2016-05-05T16:22:51Z","updated_at":"2016-05-05T16:22:51Z","body":"Pathes are currently stored plain text and for each fileset in the db. Additionally there is some log entries that can add up over time. So if your files have long pathes and you keep many filesets, the db can get quite large as in your case.\r\nFor the problem at hand, there is no direct solution. The query itself is ok and is processed with indexes as it should be. The long duration might be caused by thrashing because DB-engine runs out of page-buffer or a very slow I/O subsystem for the VM (maybe a combination of it). There could be some adjustments to solve your issue, but these might negatively affect other setups. Furthermore, as I am not sure of the cause, I cannot be sure if changes to the code will actually help you, as I do not have a setup to test it.\r\n@dsl: Are you able to run test-queries on your db to check what helps improving your situation?"}}