{"action":"created","issue":{"url":"https://api.github.com/repos/dmwm/WMCore/issues/6895","repository_url":"https://api.github.com/repos/dmwm/WMCore","labels_url":"https://api.github.com/repos/dmwm/WMCore/issues/6895/labels{/name}","comments_url":"https://api.github.com/repos/dmwm/WMCore/issues/6895/comments","events_url":"https://api.github.com/repos/dmwm/WMCore/issues/6895/events","html_url":"https://github.com/dmwm/WMCore/pull/6895","id":157217468,"number":6895,"title":"switch logcollect stagein to use edmCopyUtil, fixes #6740","user":{"login":"hufnagel","id":1798355,"avatar_url":"https://avatars.githubusercontent.com/u/1798355?v=3","gravatar_id":"","url":"https://api.github.com/users/hufnagel","html_url":"https://github.com/hufnagel","followers_url":"https://api.github.com/users/hufnagel/followers","following_url":"https://api.github.com/users/hufnagel/following{/other_user}","gists_url":"https://api.github.com/users/hufnagel/gists{/gist_id}","starred_url":"https://api.github.com/users/hufnagel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hufnagel/subscriptions","organizations_url":"https://api.github.com/users/hufnagel/orgs","repos_url":"https://api.github.com/users/hufnagel/repos","events_url":"https://api.github.com/users/hufnagel/events{/privacy}","received_events_url":"https://api.github.com/users/hufnagel/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/dmwm/WMCore/labels/Do%20not%20merge%20yet","name":"Do not merge yet","color":"fbca04"}],"state":"open","locked":false,"assignee":{"login":"hufnagel","id":1798355,"avatar_url":"https://avatars.githubusercontent.com/u/1798355?v=3","gravatar_id":"","url":"https://api.github.com/users/hufnagel","html_url":"https://github.com/hufnagel","followers_url":"https://api.github.com/users/hufnagel/followers","following_url":"https://api.github.com/users/hufnagel/following{/other_user}","gists_url":"https://api.github.com/users/hufnagel/gists{/gist_id}","starred_url":"https://api.github.com/users/hufnagel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hufnagel/subscriptions","organizations_url":"https://api.github.com/users/hufnagel/orgs","repos_url":"https://api.github.com/users/hufnagel/repos","events_url":"https://api.github.com/users/hufnagel/events{/privacy}","received_events_url":"https://api.github.com/users/hufnagel/received_events","type":"User","site_admin":false},"milestone":null,"comments":20,"created_at":"2016-05-27T14:20:48Z","updated_at":"2016-05-31T03:24:28Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/dmwm/WMCore/pulls/6895","html_url":"https://github.com/dmwm/WMCore/pull/6895","diff_url":"https://github.com/dmwm/WMCore/pull/6895.diff","patch_url":"https://github.com/dmwm/WMCore/pull/6895.patch"},"body":"Most obvious change is to not use the stagin method defined in site-local-config to copy logArchives to the local node, but edmCopyUtil. This requires setting up a Scram environment and calling edmCopyUtil via scram (and also some proxy forwarding). There is a hardcoded dependency on CMSSW_8_0_8, but since all we do here is copy logArchives to local disk, this should work as long as we don't retire CMSSW_8_0_8. Well, and as long as the chosen slc6_amd64_gcc493 architecture can run on all of our hardware.\r\n\r\nAt the moment edmCopyUtil is called for 10 files at a time, so if any of the 10 files can't be read, all 10 files are skipped.\r\n\r\nI also fixed a bunch of design flaws and bugs in the previous code:\r\n\r\nInput file deletion didn't work (at all) due to a wrong dictionary keyword, fixed that.\r\n\r\nInput file deletion should only be done after LogCollect stageout and after job is considered successful.\r\n\r\nThe skipped files were added to the job report without an LFN, leading to error messages in the JobAccountant.\r\n\r\nReporting LogCollect output was done in a weird way. We reported the output file that was staged to Castor, but with an EOS location and immediately after the mandatory Castor stageout succeeded (and even if the optional EOS stageout failed). Changed that to only report the EOS stageout, since that is optional, you can have a LogCollect job that succeeds but doesn't report any output since the EOS stageout failed. Which is correct IMO if you care about output actually present on EOS.\r\n"},"comment":{"url":"https://api.github.com/repos/dmwm/WMCore/issues/comments/222584460","html_url":"https://github.com/dmwm/WMCore/pull/6895#issuecomment-222584460","issue_url":"https://api.github.com/repos/dmwm/WMCore/issues/6895","id":222584460,"user":{"login":"cmsdmwmbot","id":15169205,"avatar_url":"https://avatars.githubusercontent.com/u/15169205?v=3","gravatar_id":"","url":"https://api.github.com/users/cmsdmwmbot","html_url":"https://github.com/cmsdmwmbot","followers_url":"https://api.github.com/users/cmsdmwmbot/followers","following_url":"https://api.github.com/users/cmsdmwmbot/following{/other_user}","gists_url":"https://api.github.com/users/cmsdmwmbot/gists{/gist_id}","starred_url":"https://api.github.com/users/cmsdmwmbot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmsdmwmbot/subscriptions","organizations_url":"https://api.github.com/users/cmsdmwmbot/orgs","repos_url":"https://api.github.com/users/cmsdmwmbot/repos","events_url":"https://api.github.com/users/cmsdmwmbot/events{/privacy}","received_events_url":"https://api.github.com/users/cmsdmwmbot/received_events","type":"User","site_admin":false},"created_at":"2016-05-31T03:24:28Z","updated_at":"2016-05-31T03:24:28Z","body":"Beginning pylint and unit tests.\nSee https://cmssdt.cern.ch/jenkins/job/DMWM-WMCore-PR-test/2334/ for details"}}
{"actor":{"display_login":"cmsdmwmbot"}}