{"action":"created","issue":{"url":"https://api.github.com/repos/matz-e/lobster/issues/391","repository_url":"https://api.github.com/repos/matz-e/lobster","labels_url":"https://api.github.com/repos/matz-e/lobster/issues/391/labels{/name}","comments_url":"https://api.github.com/repos/matz-e/lobster/issues/391/comments","events_url":"https://api.github.com/repos/matz-e/lobster/issues/391/events","html_url":"https://github.com/matz-e/lobster/issues/391","id":159653818,"number":391,"title":"Optimize default task creation order.","user":{"login":"annawoodard","id":5114833,"avatar_url":"https://avatars.githubusercontent.com/u/5114833?v=3","gravatar_id":"","url":"https://api.github.com/users/annawoodard","html_url":"https://github.com/annawoodard","followers_url":"https://api.github.com/users/annawoodard/followers","following_url":"https://api.github.com/users/annawoodard/following{/other_user}","gists_url":"https://api.github.com/users/annawoodard/gists{/gist_id}","starred_url":"https://api.github.com/users/annawoodard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/annawoodard/subscriptions","organizations_url":"https://api.github.com/users/annawoodard/orgs","repos_url":"https://api.github.com/users/annawoodard/repos","events_url":"https://api.github.com/users/annawoodard/events{/privacy}","received_events_url":"https://api.github.com/users/annawoodard/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-06-10T14:56:18Z","updated_at":"2016-06-10T18:37:55Z","closed_at":null,"body":"When creating tasks, we currently ([here](https://github.com/matz-e/lobster/blob/master/lobster/core/config.py#L94) and [here](https://github.com/matz-e/lobster/blob/master/lobster/core/source.py#L365)) cycle through the categories so that the smallest task limit is processed first. If no task limit is specified, the default sort is on the category name. I think we should consider alternate default sorting methods. For example, we could sort by number of units available such that we keep the categories processed as evenly possible. This has two advantages: 1) it is likely that different categories have different bottlenecks, so maximizing the mixing might minimize the per-category bottlenecks, and 2) for chained workflows with different categories at various steps, this would mean that halfway through processing you'd have half of your last step processed, which might be useful if you want to start using output before the entire project is finished.\r\n\r\nSee, for example, my current situation, where I'm only running digi and reco tasks at a severe inefficiency because I have 513 fit workers but only 251 tasks running, but there are plenty of mAOD tasks that could be run.\r\n\r\nThis could also be partially addressed by changes on the factory side which I believe @btovar is considering, although I think the points above would still stand.\r\n\r\n```\r\n(.lobster) [earth] ~/lobster-spring-16-scale-test/ttW >work_queue_status -A earth.crc.nd.edu 9001\r\nCATEGORY        RUNNING    WAITING  FIT-WORKERS  MAX-CORES MAX-MEMORY   MAX-DISK\r\ndigi                247       1002            0         ~4      ~4400     ~22131\r\nreco                  4          0          513         ~5      ~4400      ~4504\r\n```"},"comment":{"url":"https://api.github.com/repos/matz-e/lobster/issues/comments/225262255","html_url":"https://github.com/matz-e/lobster/issues/391#issuecomment-225262255","issue_url":"https://api.github.com/repos/matz-e/lobster/issues/391","id":225262255,"user":{"login":"btovar","id":3081826,"avatar_url":"https://avatars.githubusercontent.com/u/3081826?v=3","gravatar_id":"","url":"https://api.github.com/users/btovar","html_url":"https://github.com/btovar","followers_url":"https://api.github.com/users/btovar/followers","following_url":"https://api.github.com/users/btovar/following{/other_user}","gists_url":"https://api.github.com/users/btovar/gists{/gist_id}","starred_url":"https://api.github.com/users/btovar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/btovar/subscriptions","organizations_url":"https://api.github.com/users/btovar/orgs","repos_url":"https://api.github.com/users/btovar/repos","events_url":"https://api.github.com/users/btovar/events{/privacy}","received_events_url":"https://api.github.com/users/btovar/received_events","type":"User","site_admin":false},"created_at":"2016-06-10T18:37:55Z","updated_at":"2016-06-10T18:37:55Z","body":"If there is at least one worker with some space to run a tasks (e.g., at least 1 core, 1 MB of memory, and 1 MB of disk), then wq will try to fit tasks to it. Only when all the workers are completely busy, waiting tasks are not considered.\r\n"}}
{"actor":{"display_login":"btovar"}}