{"action":"created","issue":{"url":"https://api.github.com/repos/dmwm/CRABServer/issues/5210","repository_url":"https://api.github.com/repos/dmwm/CRABServer","labels_url":"https://api.github.com/repos/dmwm/CRABServer/issues/5210/labels{/name}","comments_url":"https://api.github.com/repos/dmwm/CRABServer/issues/5210/comments","events_url":"https://api.github.com/repos/dmwm/CRABServer/issues/5210/events","html_url":"https://github.com/dmwm/CRABServer/issues/5210","id":151316957,"number":5210,"title":"Do not overflow jobs if last try failure is 8028","user":{"login":"juztas","id":646582,"avatar_url":"https://avatars.githubusercontent.com/u/646582?v=3","gravatar_id":"","url":"https://api.github.com/users/juztas","html_url":"https://github.com/juztas","followers_url":"https://api.github.com/users/juztas/followers","following_url":"https://api.github.com/users/juztas/following{/other_user}","gists_url":"https://api.github.com/users/juztas/gists{/gist_id}","starred_url":"https://api.github.com/users/juztas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juztas/subscriptions","organizations_url":"https://api.github.com/users/juztas/orgs","repos_url":"https://api.github.com/users/juztas/repos","events_url":"https://api.github.com/users/juztas/events{/privacy}","received_events_url":"https://api.github.com/users/juztas/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-04-27T07:28:16Z","updated_at":"2016-05-02T10:29:13Z","closed_at":null,"body":"~30%  of jobs which are overflowed to T2_US sites are failing from T1_US As you can see from the image, it has a lot of failures.\r\n\r\nOne dataset which is not accessible at all remotely (100% failure):\r\n![image](https://cloud.githubusercontent.com/assets/646582/14844486/03ace47e-0c58-11e6-9a6f-231d13f1b41f.png)\r\nData is on FNAL and job runs elsewhere (30% failure):\r\n![image](https://cloud.githubusercontent.com/assets/646582/14844562/81e1f91a-0c58-11e6-92b8-db07091358a6.png)\r\n\r\n\r\nFor example I took one task: http://dashb-cms-job.cern.ch/dashboard/templates/task-analysis/#user=default&refresh=0&table=Jobs&p=1&records=25&activemenu=2&status=&site=&tid=160425_215922%3Adburns_crab_bkg_Spring15_QCD_Pt_800to1000 and:\r\n```\r\n== CMSSW: 26-Apr-2016 22:37:02 PDT  Initiating request to open file /hadoop/cms/phedex/store/mc/RunIIFall15DR76/QCD_Pt_800to1000_TuneCUETP8M1_13TeV_pythia8/AODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/70000/F0B7651D-2B9F-E511-B727-00259073E504.root\r\n== CMSSW: 26-Apr-2016 22:37:02 PDT  Fallback request to file root://cmsxrootd.fnal.gov//store/mc/RunIIFall15DR76/QCD_Pt_800to1000_TuneCUETP8M1_13TeV_pythia8/AODSIM/PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/70000/F0B7651D-2B9F-E511-B727-00259073E504.root\r\n== CMSSW: [2016-04-26 22:37:03.976467 -0700][Error  ][Utility           ] Unable to resolve cmsdcadisk01:1095: Name or service not known\r\n== CMSSW: [2016-04-26 22:37:03.976591 -0700][Error  ][PostMaster        ] [cmsdcadisk01:1095 #0] Unable to resolve IP address for the host\r\n```\r\nThis for sure shows site issue, but in the end CRAB3 could be smarter in this respect. If previous job failed with 8028, do not allow to overflow it with CMS_ALLOW_OVERFLOW = False\r\nWhy it is good to be a bit smarter:\r\na) Some users do not think or contact for help. I see some jobs are resubmitted up to 10 times and they end up the same.\r\nb) Failed job runtime ~2minutes, which leads pilot to sleep ~18mins and this is not good as we discussed already. Success job runtime a bit more than 20mins.\r\n\r\nWhat else we can do to improve:\r\nSome improvements are coming to cms-gwmsmon to monitor efficiency and failures, same as runtime and memory usage. \r\nAnother thing, there is no ticket to site, is there anyone following usual failures, monitoring them and reporting, if it is site issue?"},"comment":{"url":"https://api.github.com/repos/dmwm/CRABServer/issues/comments/216192266","html_url":"https://github.com/dmwm/CRABServer/issues/5210#issuecomment-216192266","issue_url":"https://api.github.com/repos/dmwm/CRABServer/issues/5210","id":216192266,"user":{"login":"belforte","id":1837785,"avatar_url":"https://avatars.githubusercontent.com/u/1837785?v=3","gravatar_id":"","url":"https://api.github.com/users/belforte","html_url":"https://github.com/belforte","followers_url":"https://api.github.com/users/belforte/followers","following_url":"https://api.github.com/users/belforte/following{/other_user}","gists_url":"https://api.github.com/users/belforte/gists{/gist_id}","starred_url":"https://api.github.com/users/belforte/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/belforte/subscriptions","organizations_url":"https://api.github.com/users/belforte/orgs","repos_url":"https://api.github.com/users/belforte/repos","events_url":"https://api.github.com/users/belforte/events{/privacy}","received_events_url":"https://api.github.com/users/belforte/received_events","type":"User","site_admin":false},"created_at":"2016-05-02T10:29:13Z","updated_at":"2016-05-02T10:29:13Z","body":"This is an 'after the fact' solution. But is is not the way.\r\n\r\nIf we expect 8028 to be a glitch, we retry. If we are confident that xrootd failing means \"no-way\" we shoul call it fatal and not retry.\r\nIt has nothing to do with overflow. Those jobs could all have run at the same site, fail with 8028 and waste resources exactly the same way. That we run jobs bound to fail instead of keeping them in the idle queue is not a bug.\r\nAnd user resubmitting blindly for weeks is a more general issue for which we have no good idea yet.\r\n\r\nUnless I miss something, and you have much stronger arguments, let's close.\r\n"}}