{"action":"created","issue":{"url":"https://api.github.com/repos/cms-l1t-offline/cmssw/issues/423","repository_url":"https://api.github.com/repos/cms-l1t-offline/cmssw","labels_url":"https://api.github.com/repos/cms-l1t-offline/cmssw/issues/423/labels{/name}","comments_url":"https://api.github.com/repos/cms-l1t-offline/cmssw/issues/423/comments","events_url":"https://api.github.com/repos/cms-l1t-offline/cmssw/issues/423/events","html_url":"https://github.com/cms-l1t-offline/cmssw/issues/423","id":182268347,"number":423,"title":"Memory leaks - L1T code in CMSSW for Sept 2016 MC Production","user":{"login":"rekovic","id":4990487,"avatar_url":"https://avatars.githubusercontent.com/u/4990487?v=3","gravatar_id":"","url":"https://api.github.com/users/rekovic","html_url":"https://github.com/rekovic","followers_url":"https://api.github.com/users/rekovic/followers","following_url":"https://api.github.com/users/rekovic/following{/other_user}","gists_url":"https://api.github.com/users/rekovic/gists{/gist_id}","starred_url":"https://api.github.com/users/rekovic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rekovic/subscriptions","organizations_url":"https://api.github.com/users/rekovic/orgs","repos_url":"https://api.github.com/users/rekovic/repos","events_url":"https://api.github.com/users/rekovic/events{/privacy}","received_events_url":"https://api.github.com/users/rekovic/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2016-10-11T13:47:03Z","updated_at":"2016-10-18T09:50:33Z","closed_at":null,"body":"This issue is to track memory leaks from L1Trigger Offline software tagged l1t-tsg-v9.2\r\nafter it has been merged in CMSSW together with the big O2O PR.\r\nBoth the core L1T software and O2O L1T software have been merged by the IB CMSSW_8_0_X_2016-10-06-2300. \r\n\r\nSince then, a PR #16085 reduced the big memory blob allocated by the OMTF XMLConfigReader, by 80%, from about 500MB to 100MB.\r\n\r\nTo track the memory leaks used igprof. \r\n\r\n<pre>\r\ncmsrel CMSSW_8_0_X_2016-10-06-2300\r\ncd CMSSW_8_0_X_2016-10-06-2300/src/\r\ncmsenv\r\ngit cms-init\r\ngit checkout -b mydevel\r\ngit cms-addpkg L1Trigger/L1TCommon\r\ngit cms-merge-topic 16085\r\ngit cms-addpkg L1Trigger/L1TMuon\r\ngit cms-addpkg L1Trigger/L1Calorimeter\r\ngit cms-addpkg L1Trigger/L1TCalorimeter\r\ngit cms-addpkg L1Trigger/L1TMuonEndcap\r\ngit cms-addpkg L1Trigger/L1TMuonEndCap\r\ngit cms-addpkg L1Trigger/L1TGlobal\r\ngit clone https://github.com/cms-data/L1Trigger-L1TGlobal.git L1Trigger/L1TGlobal/data\r\ngit clone https://github.com/cms-data/L1Trigger-L1TMuon.git L1Trigger/L1TMuon/data\r\ngit clone https://github.com/cms-data/L1Trigger-L1TCalorimeter.git L1Trigger/L1TCalorimeter/data\r\nscram b -j7\r\ncmsDriver.py l1Ntuple -s RAW2DIGI\r\n--python_filename=l1Ntuple_RAW_wRECO.py -n 100 --no_output --no_exec\r\n--era=Run2_2016 --data --conditions=80X_dataRun2_Prompt_v11\r\n--filein=/store/express/Run2016G/ExpressPhysics/FEVT/Express-v1/000/280/384/00000/0AC379C2-7976-E611-A9A7-02163E0118D8.root\r\n--customise=L1Trigger/Configuration/customiseReEmul.L1TReEmulFromRAW\r\n--customise=L1Trigger/L1TNtuples/customiseL1Ntuple.L1NtupleAODRAWEMU\r\n</pre>\r\n\r\n**Igprof results  CMSSW_8_0_X_2016-10-06-2300 + PR #16085**: \r\nhttps://treis.web.cern.ch/treis/cgi-bin/igprof-navigator/igreport_l1ntuple_mp_live_CMSSW_8_0_X_2016-10-06-2300_PR16085/self showed remaining leaks:\r\n\r\n* OMTF XMLConfigReader has still a leak after the big one was fixed (const):\r\nhttps://treis.web.cern.ch/treis/cgi-bin/igprof-navigator/igreport_l1ntuple_mp_live_CMSSW_8_0_X_2016-10-06-2300_PR16085/68\r\n\r\n* TwinMux (10kB/event):\r\nhttps://treis.web.cern.ch/treis/cgi-bin/igprof-navigator/igreport_l1ntuple_mp_live_CMSSW_8_0_X_2016-10-06-2300_PR16085/472\r\nhttps://treis.web.cern.ch/treis/cgi-bin/igprof-navigator/igreport_l1ntuple_mp_live_CMSSW_8_0_X_2016-10-06-2300_PR16085/1285\r\n\r\n* BMTF unpacker (8kB/event):\r\nhttps://treis.web.cern.ch/treis/cgi-bin/igprof-navigator/igreport_l1ntuple_mp_live_CMSSW_8_0_X_2016-10-06-2300_PR16085/550\r\n\r\n* EMTF forest (const):\r\nhttps://treis.web.cern.ch/treis/cgi-bin/igprof-navigator/igreport_l1ntuple_mp_live_CMSSW_8_0_X_2016-10-06-2300_PR16085/1246\r\n\r\n\r\n**Igprof results TwinMux fix (+ above)**:\r\nhttps://rekovic.web.cern.ch/rekovic/cms/cgi-bin/igprof-navigator/igreport_live_memoryManage-TwinMux-only/self does not contain leaks from TwinMux any more.  Code in branch:  <pre> cms-l1t-offline/pr80x-l1t-fixed-memoryManage-TwinMux-only</pre>"},"comment":{"url":"https://api.github.com/repos/cms-l1t-offline/cmssw/issues/comments/254460465","html_url":"https://github.com/cms-l1t-offline/cmssw/issues/423#issuecomment-254460465","issue_url":"https://api.github.com/repos/cms-l1t-offline/cmssw/issues/423","id":254460465,"user":{"login":"kkotov","id":11047399,"avatar_url":"https://avatars.githubusercontent.com/u/11047399?v=3","gravatar_id":"","url":"https://api.github.com/users/kkotov","html_url":"https://github.com/kkotov","followers_url":"https://api.github.com/users/kkotov/followers","following_url":"https://api.github.com/users/kkotov/following{/other_user}","gists_url":"https://api.github.com/users/kkotov/gists{/gist_id}","starred_url":"https://api.github.com/users/kkotov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kkotov/subscriptions","organizations_url":"https://api.github.com/users/kkotov/orgs","repos_url":"https://api.github.com/users/kkotov/repos","events_url":"https://api.github.com/users/kkotov/events{/privacy}","received_events_url":"https://api.github.com/users/kkotov/received_events","type":"User","site_admin":false},"created_at":"2016-10-18T09:50:33Z","updated_at":"2016-10-18T09:50:33Z","body":"Hi @rekovic , the inaccurate use of the xercesc::XMLString::transcode on a massive scale can contribute to the rate at which the memory leaks. As documentation suggests:\r\nhttp://xerces.apache.org/xerces-c/apiDocs-3/classXMLString.html#a91b02742b80418fbc6c54b7dd353ad8c\r\none should use xercesc::XMLString::release(char *) call in conjunction with every xercesc::XMLString::transcode() call.\r\nThis was, for example, added in the following two blocks:\r\nhttps://github.com/cms-sw/cmssw/blob/CMSSW_8_1_X/L1Trigger/L1TCommon/src/XmlConfigReader.cc#L345-L347\r\nand\r\nhttps://github.com/cms-sw/cmssw/blob/CMSSW_8_1_X/L1Trigger/L1TCommon/src/XmlConfigReader.cc#L419-L421\r\nPerhaps, there is still some sort of garbage collection available on deleting the parser or calling XMLPlatformUtils::Terminate(), but I didn't read specs on xerces to such detail.\r\nUnless, the GC is available, one should think of avoiding using constructions like:\r\nhttps://github.com/cms-sw/cmssw/blob/CMSSW_8_1_X/L1Trigger/L1TMuonOverlap/src/XMLConfigReader.cc#L30-L32"}}
{"actor":{"display_login":"kkotov"}}