{"action":"created","issue":{"url":"https://api.github.com/repos/ganga-devs/ganga/issues/68","labels_url":"https://api.github.com/repos/ganga-devs/ganga/issues/68/labels{/name}","comments_url":"https://api.github.com/repos/ganga-devs/ganga/issues/68/comments","events_url":"https://api.github.com/repos/ganga-devs/ganga/issues/68/events","html_url":"https://github.com/ganga-devs/ganga/issues/68","id":122052879,"number":68,"title":"Extremely slow performance for large job submission (XML and Dataset related)","user":{"login":"rob-c","id":1102021,"avatar_url":"https://avatars.githubusercontent.com/u/1102021?v=3","gravatar_id":"","url":"https://api.github.com/users/rob-c","html_url":"https://github.com/rob-c","followers_url":"https://api.github.com/users/rob-c/followers","following_url":"https://api.github.com/users/rob-c/following{/other_user}","gists_url":"https://api.github.com/users/rob-c/gists{/gist_id}","starred_url":"https://api.github.com/users/rob-c/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rob-c/subscriptions","organizations_url":"https://api.github.com/users/rob-c/orgs","repos_url":"https://api.github.com/users/rob-c/repos","events_url":"https://api.github.com/users/rob-c/events{/privacy}","received_events_url":"https://api.github.com/users/rob-c/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ganga-devs/ganga/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/ganga-devs/ganga/labels/Core","name":"Core","color":"d4c5f9"},{"url":"https://api.github.com/repos/ganga-devs/ganga/labels/enhancement","name":"enhancement","color":"009800"},{"url":"https://api.github.com/repos/ganga-devs/ganga/labels/LHCb","name":"LHCb","color":"d4c5f9"}],"state":"open","locked":false,"assignee":{"login":"rob-c","id":1102021,"avatar_url":"https://avatars.githubusercontent.com/u/1102021?v=3","gravatar_id":"","url":"https://api.github.com/users/rob-c","html_url":"https://github.com/rob-c","followers_url":"https://api.github.com/users/rob-c/followers","following_url":"https://api.github.com/users/rob-c/following{/other_user}","gists_url":"https://api.github.com/users/rob-c/gists{/gist_id}","starred_url":"https://api.github.com/users/rob-c/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rob-c/subscriptions","organizations_url":"https://api.github.com/users/rob-c/orgs","repos_url":"https://api.github.com/users/rob-c/repos","events_url":"https://api.github.com/users/rob-c/events{/privacy}","received_events_url":"https://api.github.com/users/rob-c/received_events","type":"User","site_admin":false},"milestone":null,"comments":9,"created_at":"2015-12-14T14:26:11Z","updated_at":"2016-01-15T13:31:03Z","closed_at":null,"body":"After running some profiling and benchmarking I've discovered that a large source of performance problems is in parsing the XML for a job.\r\n\r\nIn order to reproduce this I was attempting to run over a 'standard' LHCb dataset of 8082 LFN with a standard LHCb job that effectively just counted the integrated luminosity for each lfn.\r\n\r\nThis job would stall for a few minutes at a time which didn't make much sense as I don't see this sort of performance bottleneck in simple profiling.\r\n\r\nThis turned up too late to be considered for 6.1.14, although I wanted to track down and identify what was causing this performance problem before 6.1.14 was released to know if this was a deal breaker.\r\n\r\nThe problem that I see is that constructing 8k DiracFile objects takes approximately 40s to perform and storing this on disk as a flat array of DiracFiles causes ganga to pause for the 2-3minutes required to process the xml file.\r\n\r\nI think for 6.1.15 I would like to rewrite the internals of LHCbDataset to store the LFN in a flat array of strings and act effectively as a generator producing the DiracFile on demand.\r\n\r\nGiven that there is now a generic GangaDataset I would like to add this generator type functionality there and have a GangaDataset class which is able to store 'flat' information and generate IGangaFile objects on demand which would be more sensible than simply generating 8k (or more) DiracFile objects and storing the full object on disk in a huge XML.\r\n\r\n(This could prove tricky with iterators and such but should also reduce the memory footprint which is a bit higher than I would like it to be atm ~200Mb per job when submitting which can expand when the dataset is copied/modified.)\r\n\r\nHopefully this would allow for the LHCbDataset to become a lose wrapper around the Core GangaDataset providing some minor additional LHCb functionality on top of this."},"comment":{"url":"https://api.github.com/repos/ganga-devs/ganga/issues/comments/171962413","html_url":"https://github.com/ganga-devs/ganga/issues/68#issuecomment-171962413","issue_url":"https://api.github.com/repos/ganga-devs/ganga/issues/68","id":171962413,"user":{"login":"rob-c","id":1102021,"avatar_url":"https://avatars.githubusercontent.com/u/1102021?v=3","gravatar_id":"","url":"https://api.github.com/users/rob-c","html_url":"https://github.com/rob-c","followers_url":"https://api.github.com/users/rob-c/followers","following_url":"https://api.github.com/users/rob-c/following{/other_user}","gists_url":"https://api.github.com/users/rob-c/gists{/gist_id}","starred_url":"https://api.github.com/users/rob-c/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rob-c/subscriptions","organizations_url":"https://api.github.com/users/rob-c/orgs","repos_url":"https://api.github.com/users/rob-c/repos","events_url":"https://api.github.com/users/rob-c/events{/privacy}","received_events_url":"https://api.github.com/users/rob-c/received_events","type":"User","site_admin":false},"created_at":"2016-01-15T13:31:03Z","updated_at":"2016-01-15T13:31:03Z","body":"It's there because the lists are being populated in parallel. Run the lists linearly and you see the same sort of performance.\r\nTake out the check and you have an object in an inconsistent state"}}