{"action":"created","issue":{"url":"https://api.github.com/repos/cms-sw/cmssw/issues/26232","repository_url":"https://api.github.com/repos/cms-sw/cmssw","labels_url":"https://api.github.com/repos/cms-sw/cmssw/issues/26232/labels{/name}","comments_url":"https://api.github.com/repos/cms-sw/cmssw/issues/26232/comments","events_url":"https://api.github.com/repos/cms-sw/cmssw/issues/26232/events","html_url":"https://github.com/cms-sw/cmssw/pull/26232","id":423858478,"node_id":"MDExOlB1bGxSZXF1ZXN0MjYzMzM3OTUw","number":26232,"title":"Save DQM output per lumisection for UL rereco","user":{"login":"schneiml","id":17005446,"node_id":"MDQ6VXNlcjE3MDA1NDQ2","avatar_url":"https://avatars1.githubusercontent.com/u/17005446?v=4","gravatar_id":"","url":"https://api.github.com/users/schneiml","html_url":"https://github.com/schneiml","followers_url":"https://api.github.com/users/schneiml/followers","following_url":"https://api.github.com/users/schneiml/following{/other_user}","gists_url":"https://api.github.com/users/schneiml/gists{/gist_id}","starred_url":"https://api.github.com/users/schneiml/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/schneiml/subscriptions","organizations_url":"https://api.github.com/users/schneiml/orgs","repos_url":"https://api.github.com/users/schneiml/repos","events_url":"https://api.github.com/users/schneiml/events{/privacy}","received_events_url":"https://api.github.com/users/schneiml/received_events","type":"User","site_admin":false},"labels":[{"id":673935047,"node_id":"MDU6TGFiZWw2NzM5MzUwNDc=","url":"https://api.github.com/repos/cms-sw/cmssw/labels/code-checks-approved","name":"code-checks-approved","color":"2cbe4e","default":false},{"id":88653795,"node_id":"MDU6TGFiZWw4ODY1Mzc5NQ==","url":"https://api.github.com/repos/cms-sw/cmssw/labels/comparison-pending","name":"comparison-pending","color":"fbca04","default":false},{"id":52820549,"node_id":"MDU6TGFiZWw1MjgyMDU0OQ==","url":"https://api.github.com/repos/cms-sw/cmssw/labels/dqm-pending","name":"dqm-pending","color":"fbca04","default":false},{"id":52881747,"node_id":"MDU6TGFiZWw1Mjg4MTc0Nw==","url":"https://api.github.com/repos/cms-sw/cmssw/labels/operations-pending","name":"operations-pending","color":"fbca04","default":false},{"id":56953091,"node_id":"MDU6TGFiZWw1Njk1MzA5MQ==","url":"https://api.github.com/repos/cms-sw/cmssw/labels/orp-pending","name":"orp-pending","color":"fbca04","default":false},{"id":53167770,"node_id":"MDU6TGFiZWw1MzE2Nzc3MA==","url":"https://api.github.com/repos/cms-sw/cmssw/labels/pending-signatures","name":"pending-signatures","color":"ff8000","default":false},{"id":56131195,"node_id":"MDU6TGFiZWw1NjEzMTE5NQ==","url":"https://api.github.com/repos/cms-sw/cmssw/labels/tests-approved","name":"tests-approved","color":"2cbe4e","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/cms-sw/cmssw/milestones/80","html_url":"https://github.com/cms-sw/cmssw/milestone/80","labels_url":"https://api.github.com/repos/cms-sw/cmssw/milestones/80/labels","id":4076143,"node_id":"MDk6TWlsZXN0b25lNDA3NjE0Mw==","number":80,"title":"CMSSW_10_6_X","description":null,"creator":{"login":"smuzaffar","id":4115138,"node_id":"MDQ6VXNlcjQxMTUxMzg=","avatar_url":"https://avatars3.githubusercontent.com/u/4115138?v=4","gravatar_id":"","url":"https://api.github.com/users/smuzaffar","html_url":"https://github.com/smuzaffar","followers_url":"https://api.github.com/users/smuzaffar/followers","following_url":"https://api.github.com/users/smuzaffar/following{/other_user}","gists_url":"https://api.github.com/users/smuzaffar/gists{/gist_id}","starred_url":"https://api.github.com/users/smuzaffar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smuzaffar/subscriptions","organizations_url":"https://api.github.com/users/smuzaffar/orgs","repos_url":"https://api.github.com/users/smuzaffar/repos","events_url":"https://api.github.com/users/smuzaffar/events{/privacy}","received_events_url":"https://api.github.com/users/smuzaffar/received_events","type":"User","site_admin":false},"open_issues":66,"closed_issues":322,"state":"open","created_at":"2019-02-21T13:47:27Z","updated_at":"2019-04-11T20:30:12Z","due_on":null,"closed_at":null},"comments":180,"created_at":"2019-03-21T17:39:30Z","updated_at":"2019-04-11T22:08:47Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/cms-sw/cmssw/pulls/26232","html_url":"https://github.com/cms-sw/cmssw/pull/26232","diff_url":"https://github.com/cms-sw/cmssw/pull/26232.diff","patch_url":"https://github.com/cms-sw/cmssw/pull/26232.patch"},"body":"#### PR description:\r\n\r\nThis PR enables per-lumisection saving of *all* DQM output for the UL rereco. This wish came up for machine learning studies targeting per-LS data certification or online monitoring.\r\n\r\nIn prompt reco, we have per-LS DQM output available due to how Tier0 processes only ~1 LS per job, and the per-job DQMIO output files are available. It would be very nice to have the same granularity of data preserved for the UL rereco, which in addition provides the full run2 dataset processed using the same software.\r\n\r\nThis is safe to do for most of DQM, since HARVESTING will re-generate per run/job statistics from the per-lumisection snapshots. It is not save for legacy modules that modify MEs at non-standard times or that do not support merging correctly; none of these should exist in RECO/DQM step since the threaded migration, but since there are still a few, I apply the change only to the non-legacy MEs.\r\n\r\n#### PR validation:\r\n\r\nValidated on a modified `136.85` that crosses a lumi boundary: `runTheMatrix.py -l 136.85 --command '--customise_commands \"if process.process == \\\"reHLT\\\": process.source.skipEvents=cms.untracked.uint32(3800)\\n\"' --ibeos -t 8`. Will need some special attention during relval since the DQMIO output file size will increase.\r\n\r\nKnown problems:\r\n- Various \"known unstable\" plots change behaviour. These change behaviour on trivial changes such as adding debug output, and might be caused by race conditions.\r\n- The behaviour of per-lumi plots in harvesting changes. This can lead to \"double counting\" in the existing offline per-lumi harvesting code, and cause subtly incorrect results. Since there is very little such code, I manually adapted the places I could find; however, due to the freedom that harvesting code has, I can not guarantee that these are all cases."},"comment":{"url":"https://api.github.com/repos/cms-sw/cmssw/issues/comments/482345077","html_url":"https://github.com/cms-sw/cmssw/pull/26232#issuecomment-482345077","issue_url":"https://api.github.com/repos/cms-sw/cmssw/issues/26232","id":482345077,"node_id":"MDEyOklzc3VlQ29tbWVudDQ4MjM0NTA3Nw==","user":{"login":"cmsbuild","id":4170616,"node_id":"MDQ6VXNlcjQxNzA2MTY=","avatar_url":"https://avatars1.githubusercontent.com/u/4170616?v=4","gravatar_id":"","url":"https://api.github.com/users/cmsbuild","html_url":"https://github.com/cmsbuild","followers_url":"https://api.github.com/users/cmsbuild/followers","following_url":"https://api.github.com/users/cmsbuild/following{/other_user}","gists_url":"https://api.github.com/users/cmsbuild/gists{/gist_id}","starred_url":"https://api.github.com/users/cmsbuild/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmsbuild/subscriptions","organizations_url":"https://api.github.com/users/cmsbuild/orgs","repos_url":"https://api.github.com/users/cmsbuild/repos","events_url":"https://api.github.com/users/cmsbuild/events{/privacy}","received_events_url":"https://api.github.com/users/cmsbuild/received_events","type":"User","site_admin":false},"created_at":"2019-04-11T22:08:47Z","updated_at":"2019-04-11T22:08:47Z","author_association":"CONTRIBUTOR","body":"Comparison is ready\nhttps://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26232/34163/summary.html\n\nComparison Summary:\n - No significant changes to the logs found\n - Reco comparison results: 0 differences found in the comparisons\n - DQMHistoTests: Total files compared: 32\n - DQMHistoTests: Total histograms compared: 3138479\n - DQMHistoTests: Total failures: 1643\n - DQMHistoTests: Total nulls: 0\n - DQMHistoTests: Total successes: 3136639\n - DQMHistoTests: Total skipped: 197\n - DQMHistoTests: Total Missing objects: 0\n - DQMHistoSizes: Histogram memory added: -601.622 KiB( 31 files compared)\n - DQMHistoSizes: changed ( 10024.0,... ): -42.973 KiB      CTPPS/TrackingStrip\n - Checked 133 log files, 14 edm output root files, 32 DQM output files\n"}}
{"actor":{"display_login":"cmsbuild"}}