{"action":"created","issue":{"url":"https://api.github.com/repos/artefactual/archivematica/issues/1105","repository_url":"https://api.github.com/repos/artefactual/archivematica","labels_url":"https://api.github.com/repos/artefactual/archivematica/issues/1105/labels{/name}","comments_url":"https://api.github.com/repos/artefactual/archivematica/issues/1105/comments","events_url":"https://api.github.com/repos/artefactual/archivematica/issues/1105/events","html_url":"https://github.com/artefactual/archivematica/pull/1105","id":325512193,"node_id":"MDExOlB1bGxSZXF1ZXN0MTg5ODMwMTk4","number":1105,"title":"Rework the MCP Server, MCP Client and MCP Client scripts to support batching tasks","user":{"login":"jambun","id":438290,"node_id":"MDQ6VXNlcjQzODI5MA==","avatar_url":"https://avatars0.githubusercontent.com/u/438290?v=4","gravatar_id":"","url":"https://api.github.com/users/jambun","html_url":"https://github.com/jambun","followers_url":"https://api.github.com/users/jambun/followers","following_url":"https://api.github.com/users/jambun/following{/other_user}","gists_url":"https://api.github.com/users/jambun/gists{/gist_id}","starred_url":"https://api.github.com/users/jambun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jambun/subscriptions","organizations_url":"https://api.github.com/users/jambun/orgs","repos_url":"https://api.github.com/users/jambun/repos","events_url":"https://api.github.com/users/jambun/events{/privacy}","received_events_url":"https://api.github.com/users/jambun/received_events","type":"User","site_admin":false},"labels":[{"id":941991635,"node_id":"MDU6TGFiZWw5NDE5OTE2MzU=","url":"https://api.github.com/repos/artefactual/archivematica/labels/Jisc%20RDSS","name":"Jisc RDSS","color":"14e278","default":false}],"state":"open","locked":false,"assignee":{"login":"jambun","id":438290,"node_id":"MDQ6VXNlcjQzODI5MA==","avatar_url":"https://avatars0.githubusercontent.com/u/438290?v=4","gravatar_id":"","url":"https://api.github.com/users/jambun","html_url":"https://github.com/jambun","followers_url":"https://api.github.com/users/jambun/followers","following_url":"https://api.github.com/users/jambun/following{/other_user}","gists_url":"https://api.github.com/users/jambun/gists{/gist_id}","starred_url":"https://api.github.com/users/jambun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jambun/subscriptions","organizations_url":"https://api.github.com/users/jambun/orgs","repos_url":"https://api.github.com/users/jambun/repos","events_url":"https://api.github.com/users/jambun/events{/privacy}","received_events_url":"https://api.github.com/users/jambun/received_events","type":"User","site_admin":false},"assignees":[{"login":"jambun","id":438290,"node_id":"MDQ6VXNlcjQzODI5MA==","avatar_url":"https://avatars0.githubusercontent.com/u/438290?v=4","gravatar_id":"","url":"https://api.github.com/users/jambun","html_url":"https://github.com/jambun","followers_url":"https://api.github.com/users/jambun/followers","following_url":"https://api.github.com/users/jambun/following{/other_user}","gists_url":"https://api.github.com/users/jambun/gists{/gist_id}","starred_url":"https://api.github.com/users/jambun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jambun/subscriptions","organizations_url":"https://api.github.com/users/jambun/orgs","repos_url":"https://api.github.com/users/jambun/repos","events_url":"https://api.github.com/users/jambun/events{/privacy}","received_events_url":"https://api.github.com/users/jambun/received_events","type":"User","site_admin":false}],"milestone":null,"comments":9,"created_at":"2018-05-23T00:55:29Z","updated_at":"2018-06-26T21:22:40Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/artefactual/archivematica/pulls/1105","html_url":"https://github.com/artefactual/archivematica/pull/1105","diff_url":"https://github.com/artefactual/archivematica/pull/1105.diff","patch_url":"https://github.com/artefactual/archivematica/pull/1105.patch"},"body":"  * The MCP Server now batches file-level tasks into fixed-size\r\n    groups, creating one Gearman task per batch, rather than one per\r\n    file.  It also uses fixed-size thread pools to limit contention\r\n    between threads.\r\n\r\n  * The MCP Client now operates in batches, processing one batch at a\r\n    time.  It also supports running tasks using a pool of\r\n    processes (improving throughput where tasks benefit from spanning\r\n    multiple CPUs.)\r\n\r\n  * The MCP Client scripts now accept a batch of jobs and process them\r\n    as a single unit.  There is a new `Job` API that provides a\r\n    standard interface for these client scripts, and all scripts have\r\n    been converted to use this.\r\n\r\nThe motivation for this work was to improve performance on transfer and ingest workflows, and to provide an improved interface for implementing client scripts.\r\n\r\nOur testing shows transfers and ingests taking approximately half the time they did without these changes.\r\n\r\nThese changes also permit further optimisation of client scripts, by taking advantage of processing files in batches rather than one at a time. We did some work on optimising a few of the client scripts, but there is likely more improvement to be gained by further optimisation.\r\n\r\n--- \r\n<Sevein> This is connected to #938."},"comment":{"url":"https://api.github.com/repos/artefactual/archivematica/issues/comments/400466857","html_url":"https://github.com/artefactual/archivematica/pull/1105#issuecomment-400466857","issue_url":"https://api.github.com/repos/artefactual/archivematica/issues/1105","id":400466857,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMDQ2Njg1Nw==","user":{"login":"marktriggs","id":149123,"node_id":"MDQ6VXNlcjE0OTEyMw==","avatar_url":"https://avatars1.githubusercontent.com/u/149123?v=4","gravatar_id":"","url":"https://api.github.com/users/marktriggs","html_url":"https://github.com/marktriggs","followers_url":"https://api.github.com/users/marktriggs/followers","following_url":"https://api.github.com/users/marktriggs/following{/other_user}","gists_url":"https://api.github.com/users/marktriggs/gists{/gist_id}","starred_url":"https://api.github.com/users/marktriggs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marktriggs/subscriptions","organizations_url":"https://api.github.com/users/marktriggs/orgs","repos_url":"https://api.github.com/users/marktriggs/repos","events_url":"https://api.github.com/users/marktriggs/events{/privacy}","received_events_url":"https://api.github.com/users/marktriggs/received_events","type":"User","site_admin":false},"created_at":"2018-06-26T21:22:40Z","updated_at":"2018-06-26T21:22:40Z","author_association":"CONTRIBUTOR","body":"Ah right.  James & Payten were both testing on Macs too, so I'll get them to run another test.  Otherwise, let me know if I can help troubleshoot :)"}}
{"actor":{"display_login":"marktriggs"}}