{"action":"created","issue":{"url":"https://api.github.com/repos/cms-sw/cmssw/issues/21717","repository_url":"https://api.github.com/repos/cms-sw/cmssw","labels_url":"https://api.github.com/repos/cms-sw/cmssw/issues/21717/labels{/name}","comments_url":"https://api.github.com/repos/cms-sw/cmssw/issues/21717/comments","events_url":"https://api.github.com/repos/cms-sw/cmssw/issues/21717/events","html_url":"https://github.com/cms-sw/cmssw/pull/21717","id":282127644,"node_id":"MDExOlB1bGxSZXF1ZXN0MTU4Mzc3ODQz","number":21717,"title":"LWTNN Deep Neural Networks for track classification","user":{"login":"hajohajo","id":7345076,"node_id":"MDQ6VXNlcjczNDUwNzY=","avatar_url":"https://avatars3.githubusercontent.com/u/7345076?v=4","gravatar_id":"","url":"https://api.github.com/users/hajohajo","html_url":"https://github.com/hajohajo","followers_url":"https://api.github.com/users/hajohajo/followers","following_url":"https://api.github.com/users/hajohajo/following{/other_user}","gists_url":"https://api.github.com/users/hajohajo/gists{/gist_id}","starred_url":"https://api.github.com/users/hajohajo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hajohajo/subscriptions","organizations_url":"https://api.github.com/users/hajohajo/orgs","repos_url":"https://api.github.com/users/hajohajo/repos","events_url":"https://api.github.com/users/hajohajo/events{/privacy}","received_events_url":"https://api.github.com/users/hajohajo/received_events","type":"User","site_admin":false},"labels":[{"id":673935047,"node_id":"MDU6TGFiZWw2NzM5MzUwNDc=","url":"https://api.github.com/repos/cms-sw/cmssw/labels/code-checks-approved","name":"code-checks-approved","color":"2cbe4e","default":false},{"id":88645479,"node_id":"MDU6TGFiZWw4ODY0NTQ3OQ==","url":"https://api.github.com/repos/cms-sw/cmssw/labels/comparison-available","name":"comparison-available","color":"2cbe4e","default":false},{"id":56953091,"node_id":"MDU6TGFiZWw1Njk1MzA5MQ==","url":"https://api.github.com/repos/cms-sw/cmssw/labels/orp-pending","name":"orp-pending","color":"fbca04","default":false},{"id":53167770,"node_id":"MDU6TGFiZWw1MzE2Nzc3MA==","url":"https://api.github.com/repos/cms-sw/cmssw/labels/pending-signatures","name":"pending-signatures","color":"ff8000","default":false},{"id":52820010,"node_id":"MDU6TGFiZWw1MjgyMDAxMA==","url":"https://api.github.com/repos/cms-sw/cmssw/labels/reconstruction-pending","name":"reconstruction-pending","color":"fbca04","default":false},{"id":541952670,"node_id":"MDU6TGFiZWw1NDE5NTI2NzA=","url":"https://api.github.com/repos/cms-sw/cmssw/labels/requires-external","name":"requires-external","color":"0000ff","default":false},{"id":56131195,"node_id":"MDU6TGFiZWw1NjEzMTE5NQ==","url":"https://api.github.com/repos/cms-sw/cmssw/labels/tests-approved","name":"tests-approved","color":"2cbe4e","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/cms-sw/cmssw/milestones/78","html_url":"https://github.com/cms-sw/cmssw/milestone/78","labels_url":"https://api.github.com/repos/cms-sw/cmssw/milestones/78/labels","id":3727036,"node_id":"MDk6TWlsZXN0b25lMzcyNzAzNg==","number":78,"title":"CMSSW_10_4_X","description":null,"creator":{"login":"smuzaffar","id":4115138,"node_id":"MDQ6VXNlcjQxMTUxMzg=","avatar_url":"https://avatars3.githubusercontent.com/u/4115138?v=4","gravatar_id":"","url":"https://api.github.com/users/smuzaffar","html_url":"https://github.com/smuzaffar","followers_url":"https://api.github.com/users/smuzaffar/followers","following_url":"https://api.github.com/users/smuzaffar/following{/other_user}","gists_url":"https://api.github.com/users/smuzaffar/gists{/gist_id}","starred_url":"https://api.github.com/users/smuzaffar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smuzaffar/subscriptions","organizations_url":"https://api.github.com/users/smuzaffar/orgs","repos_url":"https://api.github.com/users/smuzaffar/repos","events_url":"https://api.github.com/users/smuzaffar/events{/privacy}","received_events_url":"https://api.github.com/users/smuzaffar/received_events","type":"User","site_admin":false},"open_issues":66,"closed_issues":361,"state":"open","created_at":"2018-10-10T09:27:21Z","updated_at":"2018-12-04T23:01:51Z","due_on":null,"closed_at":null},"comments":265,"created_at":"2017-12-14T14:55:30Z","updated_at":"2018-12-04T23:47:03Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/cms-sw/cmssw/pulls/21717","html_url":"https://github.com/cms-sw/cmssw/pull/21717","diff_url":"https://github.com/cms-sw/cmssw/pull/21717.diff","patch_url":"https://github.com/cms-sw/cmssw/pull/21717.patch"},"body":"This PR changes the track classifiers in iterative track reconstruction for Phase1 to use a Deep Neural Network implemented in the LWTNN library instead of Boosted Decision Trees. Uses the model .json stored in  #https://github.com/cms-data/RecoTracker-FinalTrackSelectors/pull/1\r\n\r\nPre-PR tests ran on CMSSW_10_0_X_2017-12-13-2300\r\n\r\nMTV comparison plots for the current BDT and the new LWTNN implementations: http://cmsdoc.cern.ch/~jhavukai/plots_LWTNNvsBDT/\r\n\r\nRecent talks where this work has been presented:\r\n-CMS Fall17 Offline and Computing Week 15.11.2017\r\nhttps://indico.cern.ch/event/625239/#12-track-selection-with-tf\r\n\r\n-Tracking POG meeting 16.10.2017\r\nhttps://indico.cern.ch/event/672746/#4-updates-on-mva-selection\r\n\r\n@makortel @VinInn "},"comment":{"url":"https://api.github.com/repos/cms-sw/cmssw/issues/comments/444303344","html_url":"https://github.com/cms-sw/cmssw/pull/21717#issuecomment-444303344","issue_url":"https://api.github.com/repos/cms-sw/cmssw/issues/21717","id":444303344,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NDMwMzM0NA==","user":{"login":"slava77","id":4676718,"node_id":"MDQ6VXNlcjQ2NzY3MTg=","avatar_url":"https://avatars2.githubusercontent.com/u/4676718?v=4","gravatar_id":"","url":"https://api.github.com/users/slava77","html_url":"https://github.com/slava77","followers_url":"https://api.github.com/users/slava77/followers","following_url":"https://api.github.com/users/slava77/following{/other_user}","gists_url":"https://api.github.com/users/slava77/gists{/gist_id}","starred_url":"https://api.github.com/users/slava77/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slava77/subscriptions","organizations_url":"https://api.github.com/users/slava77/orgs","repos_url":"https://api.github.com/users/slava77/repos","events_url":"https://api.github.com/users/slava77/events{/privacy}","received_events_url":"https://api.github.com/users/slava77/received_events","type":"User","site_admin":false},"created_at":"2018-12-04T23:47:03Z","updated_at":"2018-12-04T23:47:03Z","author_association":"CONTRIBUTOR","body":"Looking at 2K events with flat-pt QCD with PU in 2018 setup (wf 11071), I think is the least affected by the real high-pt electons. The behavior appears to be somewhat similar to the no-PU plots posted a week ago.\r\n\r\n[red is with this PR; the beseline in black is 10_4_0_pre2]\r\n\r\nThere is some loss in efficiency and a very significant increase in fake rate for the high-purity tracks at high pt\r\n<img width=\"739\" alt=\"wf11071_hp_eff_pt\" src=\"https://user-images.githubusercontent.com/4676718/49479308-663b6d00-f7d7-11e8-964c-23cf02e2edc5.png\">\r\n<img width=\"739\" alt=\"wf11071_hp_fr_pt\" src=\"https://user-images.githubusercontent.com/4676718/49479335-781d1000-f7d7-11e8-8d4b-d56e2e5e8ae0.png\">\r\n\r\ngeneral tracks look more as expected: not much change in efficiency and an increase in the fake rate due to the new pass-through selections.\r\n<img width=\"738\" alt=\"wf11071_gen_eff_pt\" src=\"https://user-images.githubusercontent.com/4676718/49479364-9c78ec80-f7d7-11e8-9090-c081a0439725.png\">\r\n<img width=\"737\" alt=\"wf11071_gen_fr_pt\" src=\"https://user-images.githubusercontent.com/4676718/49479369-a13da080-f7d7-11e8-8f48-220f36c3720b.png\">\r\n\r\nb-tagging has some small loss in deepCSV tagger, but perhaps this can be mitigated by retraining\r\n<img width=\"739\" alt=\"wf11071_deepcsv_bvl\" src=\"https://user-images.githubusercontent.com/4676718/49479410-ce8a4e80-f7d7-11e8-8574-8876dd35a886.png\">\r\n\r\nThe effect for the old CSVv2 is a bit more significant [perhaps related: there is  \\~15% increase in the number of ```inclusiveSecondaryVertices``` which are likely fake and push up the mistag rate]\r\n<img width=\"740\" alt=\"wf11071_csv2_bvl\" src=\"https://user-images.githubusercontent.com/4676718/49479503-1ad58e80-f7d8-11e8-97e6-0f632cce9be7.png\">\r\n\r\nSo, in this sample, overall, I think that the most puzzling is the change in performance of the high-purity tracks at high pt (both the efficiency and the fake rate are going in the \"wrong\" direction).\r\n\r\nIn ZEE events with PU (wf 11025)\r\nthe loss in electron efficiency is fairly visible and will likely trigger invalidation of this change\r\n<img width=\"738\" alt=\"wf11025_ele_mc_eff_pt\" src=\"https://user-images.githubusercontent.com/4676718/49480076-3346a880-f7da-11e8-8e45-1b26852f0324.png\">\r\nThe effect is not really visible in the gsf MTV plot\r\n<img width=\"877\" alt=\"wf11025_gsf_eff_pt\" src=\"https://user-images.githubusercontent.com/4676718/49480151-7739ad80-f7da-11e8-972c-eafb51476063.png\">\r\n\r\nIn this sample the HP efficiency drops a lot, but this is \"expected\" for electrons-dominated higher pt range\r\n<img width=\"878\" alt=\"wf11025_hp_eff_pt\" src=\"https://user-images.githubusercontent.com/4676718/49480198-9afcf380-f7da-11e8-93e3-d00a143a7318.png\">\r\n\r\nMuons in ZMM with PU (wf 11042) look \\~OK\r\n<img width=\"738\" alt=\"wf11042_hp_eff_pt\" src=\"https://user-images.githubusercontent.com/4676718/49480258-dbf50800-f7da-11e8-9532-3f5aecf789b6.png\">\r\nThe increased fake rate in HP is also significantly increased here\r\n<img width=\"878\" alt=\"wf11042_hp_fr_pt\" src=\"https://user-images.githubusercontent.com/4676718/49480280-fcbd5d80-f7da-11e8-975d-fec703bb75af.png\">\r\n\r\n\r\n@hajohajo  IIUC, your training was done using flat-pt dijet sample with pileup. In your TRK POG slides I see\r\n```\r\nQCD Flat 15-3000 GeV with PU\r\nQCD 3000-3500 GeV no PU\r\n```\r\nPlease share the validation plots for the flat and high-pt dijet samples.\r\nI wanted to compare/confirm my observations.\r\n"}}
{"actor":{"display_login":"slava77"}}