{"action":"closed","number":1498,"pull_request":{"url":"https://api.github.com/repos/zenoss/zenoss-prodbin/pulls/1498","id":59221891,"html_url":"https://github.com/zenoss/zenoss-prodbin/pull/1498","diff_url":"https://github.com/zenoss/zenoss-prodbin/pull/1498.diff","patch_url":"https://github.com/zenoss/zenoss-prodbin/pull/1498.patch","issue_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/issues/1498","number":1498,"state":"closed","locked":false,"title":"Improve speed of loading objects from XML","user":{"login":"cluther","id":86579,"avatar_url":"https://avatars.githubusercontent.com/u/86579?v=3","gravatar_id":"","url":"https://api.github.com/users/cluther","html_url":"https://github.com/cluther","followers_url":"https://api.github.com/users/cluther/followers","following_url":"https://api.github.com/users/cluther/following{/other_user}","gists_url":"https://api.github.com/users/cluther/gists{/gist_id}","starred_url":"https://api.github.com/users/cluther/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cluther/subscriptions","organizations_url":"https://api.github.com/users/cluther/orgs","repos_url":"https://api.github.com/users/cluther/repos","events_url":"https://api.github.com/users/cluther/events{/privacy}","received_events_url":"https://api.github.com/users/cluther/received_events","type":"User","site_admin":false},"body":"This commit makes multiple related changes.\n\n* Fix a bug that resulted in a commit-per-object for ZenPack upgrades\n\n  self.objectnumber in ImportRM would stay at 0. 0 modulo any chunk_size\n  evalutates to true. So a commit would happen for every object. There\n  was also a less significant case for ZenPack installations that did\n  add objects. In these cases self.objectnumber could remain modulu 0 to\n  chunk_size even if no further objects were being updated. To address\n  this, I added uncommittedObjects to track that number separately from\n  objectnumber.\n\n* Wrap all XML object loading in pausedAndOptimizedIndexing\n\n  The main delay in loading objects is round trips to CatalogService\n  when commits happen. pausedAndOptimizedIndexing batches all of these\n  calls together to reduce round trips.\n\n* Increase object commit chunk size from 100 to 500\n\n  500 objects will still keep transactions at a reasonable size. I\n  changed this value mainly because in testing I was seeing a many\n  situations where a single-template XML file would have more than 100\n  objects in it. So it would commit once when almost complete, and again\n  when complete because there's always one commit per file.\n\nI was primarily profiling the following scenario to come to these\nchanges. Installing the CiscoUCS 2.3.0 ZenPack over itself on a UCS-PM\n2 (Zenoss 5.1.0) instance. This means that CatalogService was in use. It\nalso means that 0 objects were actually being loaded from XML because\nthey all already existed in the database.\n\nZPLObject.load() timings (in seconds):\n\n* Baseline: 239.394\n* Bugfix only: 26.608\n* Bugfix + pausedAndOptimizedIndexing: 15.270\n\nThese changes make a big dent in ZEN-22000, but these optimizations only\nbecame apparent because it appears that serviced is adding latency to\nzencatalogservice queries, and it adds several minutes of startup time\nto the ZenPack install command.\n\nRefs ZEN-22000.","created_at":"2016-02-12T19:50:43Z","updated_at":"2016-02-12T20:35:45Z","closed_at":"2016-02-12T20:35:45Z","merged_at":"2016-02-12T20:35:45Z","merge_commit_sha":"1d4070429dc305d838e82f904b5e6d7c61bba71e","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/pulls/1498/commits","review_comments_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/pulls/1498/comments","review_comment_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/pulls/comments{/number}","comments_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/issues/1498/comments","statuses_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/statuses/f336561600dbdb357cbf2e07fd94d03f6ba9d8e8","head":{"label":"zenoss:feature/ZEN-22000","ref":"feature/ZEN-22000","sha":"f336561600dbdb357cbf2e07fd94d03f6ba9d8e8","user":{"login":"zenoss","id":243767,"avatar_url":"https://avatars.githubusercontent.com/u/243767?v=3","gravatar_id":"","url":"https://api.github.com/users/zenoss","html_url":"https://github.com/zenoss","followers_url":"https://api.github.com/users/zenoss/followers","following_url":"https://api.github.com/users/zenoss/following{/other_user}","gists_url":"https://api.github.com/users/zenoss/gists{/gist_id}","starred_url":"https://api.github.com/users/zenoss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zenoss/subscriptions","organizations_url":"https://api.github.com/users/zenoss/orgs","repos_url":"https://api.github.com/users/zenoss/repos","events_url":"https://api.github.com/users/zenoss/events{/privacy}","received_events_url":"https://api.github.com/users/zenoss/received_events","type":"Organization","site_admin":false},"repo":{"id":10389144,"name":"zenoss-prodbin","full_name":"zenoss/zenoss-prodbin","owner":{"login":"zenoss","id":243767,"avatar_url":"https://avatars.githubusercontent.com/u/243767?v=3","gravatar_id":"","url":"https://api.github.com/users/zenoss","html_url":"https://github.com/zenoss","followers_url":"https://api.github.com/users/zenoss/followers","following_url":"https://api.github.com/users/zenoss/following{/other_user}","gists_url":"https://api.github.com/users/zenoss/gists{/gist_id}","starred_url":"https://api.github.com/users/zenoss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zenoss/subscriptions","organizations_url":"https://api.github.com/users/zenoss/orgs","repos_url":"https://api.github.com/users/zenoss/repos","events_url":"https://api.github.com/users/zenoss/events{/privacy}","received_events_url":"https://api.github.com/users/zenoss/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/zenoss/zenoss-prodbin","description":"Zenoss - zenoss-prodbin module","fork":false,"url":"https://api.github.com/repos/zenoss/zenoss-prodbin","forks_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/forks","keys_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/teams","hooks_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/hooks","issue_events_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/issues/events{/number}","events_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/events","assignees_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/assignees{/user}","branches_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/branches{/branch}","tags_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/tags","blobs_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/git/refs{/sha}","trees_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/statuses/{sha}","languages_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/languages","stargazers_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/stargazers","contributors_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/contributors","subscribers_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/subscribers","subscription_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/subscription","commits_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/commits{/sha}","git_commits_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/git/commits{/sha}","comments_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/comments{/number}","issue_comment_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/issues/comments{/number}","contents_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/contents/{+path}","compare_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/merges","archive_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/downloads","issues_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/issues{/number}","pulls_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/pulls{/number}","milestones_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/milestones{/number}","notifications_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/labels{/name}","releases_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/releases{/id}","deployments_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/deployments","created_at":"2013-05-30T18:36:05Z","updated_at":"2016-01-25T14:45:58Z","pushed_at":"2016-02-12T20:35:45Z","git_url":"git://github.com/zenoss/zenoss-prodbin.git","ssh_url":"git@github.com:zenoss/zenoss-prodbin.git","clone_url":"https://github.com/zenoss/zenoss-prodbin.git","svn_url":"https://github.com/zenoss/zenoss-prodbin","homepage":"http://www.zenoss.com","size":529636,"stargazers_count":13,"watchers_count":13,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":10,"mirror_url":null,"open_issues_count":27,"forks":10,"open_issues":27,"watchers":13,"default_branch":"develop"}},"base":{"label":"zenoss:develop","ref":"develop","sha":"81c5e1d891f88c6a2af288ff6ff4f46ce24157ee","user":{"login":"zenoss","id":243767,"avatar_url":"https://avatars.githubusercontent.com/u/243767?v=3","gravatar_id":"","url":"https://api.github.com/users/zenoss","html_url":"https://github.com/zenoss","followers_url":"https://api.github.com/users/zenoss/followers","following_url":"https://api.github.com/users/zenoss/following{/other_user}","gists_url":"https://api.github.com/users/zenoss/gists{/gist_id}","starred_url":"https://api.github.com/users/zenoss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zenoss/subscriptions","organizations_url":"https://api.github.com/users/zenoss/orgs","repos_url":"https://api.github.com/users/zenoss/repos","events_url":"https://api.github.com/users/zenoss/events{/privacy}","received_events_url":"https://api.github.com/users/zenoss/received_events","type":"Organization","site_admin":false},"repo":{"id":10389144,"name":"zenoss-prodbin","full_name":"zenoss/zenoss-prodbin","owner":{"login":"zenoss","id":243767,"avatar_url":"https://avatars.githubusercontent.com/u/243767?v=3","gravatar_id":"","url":"https://api.github.com/users/zenoss","html_url":"https://github.com/zenoss","followers_url":"https://api.github.com/users/zenoss/followers","following_url":"https://api.github.com/users/zenoss/following{/other_user}","gists_url":"https://api.github.com/users/zenoss/gists{/gist_id}","starred_url":"https://api.github.com/users/zenoss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zenoss/subscriptions","organizations_url":"https://api.github.com/users/zenoss/orgs","repos_url":"https://api.github.com/users/zenoss/repos","events_url":"https://api.github.com/users/zenoss/events{/privacy}","received_events_url":"https://api.github.com/users/zenoss/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/zenoss/zenoss-prodbin","description":"Zenoss - zenoss-prodbin module","fork":false,"url":"https://api.github.com/repos/zenoss/zenoss-prodbin","forks_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/forks","keys_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/teams","hooks_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/hooks","issue_events_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/issues/events{/number}","events_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/events","assignees_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/assignees{/user}","branches_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/branches{/branch}","tags_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/tags","blobs_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/git/refs{/sha}","trees_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/statuses/{sha}","languages_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/languages","stargazers_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/stargazers","contributors_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/contributors","subscribers_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/subscribers","subscription_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/subscription","commits_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/commits{/sha}","git_commits_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/git/commits{/sha}","comments_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/comments{/number}","issue_comment_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/issues/comments{/number}","contents_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/contents/{+path}","compare_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/merges","archive_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/downloads","issues_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/issues{/number}","pulls_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/pulls{/number}","milestones_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/milestones{/number}","notifications_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/labels{/name}","releases_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/releases{/id}","deployments_url":"https://api.github.com/repos/zenoss/zenoss-prodbin/deployments","created_at":"2013-05-30T18:36:05Z","updated_at":"2016-01-25T14:45:58Z","pushed_at":"2016-02-12T20:35:45Z","git_url":"git://github.com/zenoss/zenoss-prodbin.git","ssh_url":"git@github.com:zenoss/zenoss-prodbin.git","clone_url":"https://github.com/zenoss/zenoss-prodbin.git","svn_url":"https://github.com/zenoss/zenoss-prodbin","homepage":"http://www.zenoss.com","size":529636,"stargazers_count":13,"watchers_count":13,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":10,"mirror_url":null,"open_issues_count":27,"forks":10,"open_issues":27,"watchers":13,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/zenoss/zenoss-prodbin/pulls/1498"},"html":{"href":"https://github.com/zenoss/zenoss-prodbin/pull/1498"},"issue":{"href":"https://api.github.com/repos/zenoss/zenoss-prodbin/issues/1498"},"comments":{"href":"https://api.github.com/repos/zenoss/zenoss-prodbin/issues/1498/comments"},"review_comments":{"href":"https://api.github.com/repos/zenoss/zenoss-prodbin/pulls/1498/comments"},"review_comment":{"href":"https://api.github.com/repos/zenoss/zenoss-prodbin/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/zenoss/zenoss-prodbin/pulls/1498/commits"},"statuses":{"href":"https://api.github.com/repos/zenoss/zenoss-prodbin/statuses/f336561600dbdb357cbf2e07fd94d03f6ba9d8e8"}},"merged":true,"mergeable":null,"mergeable_state":"unknown","merged_by":{"login":"jplouis","id":385623,"avatar_url":"https://avatars.githubusercontent.com/u/385623?v=3","gravatar_id":"","url":"https://api.github.com/users/jplouis","html_url":"https://github.com/jplouis","followers_url":"https://api.github.com/users/jplouis/followers","following_url":"https://api.github.com/users/jplouis/following{/other_user}","gists_url":"https://api.github.com/users/jplouis/gists{/gist_id}","starred_url":"https://api.github.com/users/jplouis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jplouis/subscriptions","organizations_url":"https://api.github.com/users/jplouis/orgs","repos_url":"https://api.github.com/users/jplouis/repos","events_url":"https://api.github.com/users/jplouis/events{/privacy}","received_events_url":"https://api.github.com/users/jplouis/received_events","type":"User","site_admin":false},"comments":0,"review_comments":0,"commits":1,"additions":16,"deletions":5,"changed_files":2}}