{"action":"created","issue":{"url":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/1553","repository_url":"https://api.github.com/repos/cooperative-computing-lab/cctools","labels_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/1553/labels{/name}","comments_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/1553/comments","events_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/1553/events","html_url":"https://github.com/cooperative-computing-lab/cctools/issues/1553","id":191799498,"number":1553,"title":"Resource monitor, jemalloc 4.2 clash and are hanging","user":{"login":"matz-e","id":2151261,"avatar_url":"https://avatars.githubusercontent.com/u/2151261?v=3","gravatar_id":"","url":"https://api.github.com/users/matz-e","html_url":"https://github.com/matz-e","followers_url":"https://api.github.com/users/matz-e/followers","following_url":"https://api.github.com/users/matz-e/following{/other_user}","gists_url":"https://api.github.com/users/matz-e/gists{/gist_id}","starred_url":"https://api.github.com/users/matz-e/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matz-e/subscriptions","organizations_url":"https://api.github.com/users/matz-e/orgs","repos_url":"https://api.github.com/users/matz-e/repos","events_url":"https://api.github.com/users/matz-e/events{/privacy}","received_events_url":"https://api.github.com/users/matz-e/received_events","type":"User","site_admin":false},"labels":[{"id":31803945,"url":"https://api.github.com/repos/cooperative-computing-lab/cctools/labels/bug","name":"bug","color":"fc2929","default":true},{"id":57108892,"url":"https://api.github.com/repos/cooperative-computing-lab/cctools/labels/critical","name":"critical","color":"e11d21","default":false},{"id":42527980,"url":"https://api.github.com/repos/cooperative-computing-lab/cctools/labels/Resource%20Monitor","name":"Resource Monitor","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":{"login":"btovar","id":3081826,"avatar_url":"https://avatars.githubusercontent.com/u/3081826?v=3","gravatar_id":"","url":"https://api.github.com/users/btovar","html_url":"https://github.com/btovar","followers_url":"https://api.github.com/users/btovar/followers","following_url":"https://api.github.com/users/btovar/following{/other_user}","gists_url":"https://api.github.com/users/btovar/gists{/gist_id}","starred_url":"https://api.github.com/users/btovar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/btovar/subscriptions","organizations_url":"https://api.github.com/users/btovar/orgs","repos_url":"https://api.github.com/users/btovar/repos","events_url":"https://api.github.com/users/btovar/events{/privacy}","received_events_url":"https://api.github.com/users/btovar/received_events","type":"User","site_admin":false},"assignees":[{"login":"btovar","id":3081826,"avatar_url":"https://avatars.githubusercontent.com/u/3081826?v=3","gravatar_id":"","url":"https://api.github.com/users/btovar","html_url":"https://github.com/btovar","followers_url":"https://api.github.com/users/btovar/followers","following_url":"https://api.github.com/users/btovar/following{/other_user}","gists_url":"https://api.github.com/users/btovar/gists{/gist_id}","starred_url":"https://api.github.com/users/btovar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/btovar/subscriptions","organizations_url":"https://api.github.com/users/btovar/orgs","repos_url":"https://api.github.com/users/btovar/repos","events_url":"https://api.github.com/users/btovar/events{/privacy}","received_events_url":"https://api.github.com/users/btovar/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/cooperative-computing-lab/cctools/milestones/26","html_url":"https://github.com/cooperative-computing-lab/cctools/milestones/6.0","labels_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/milestones/26/labels","id":1255076,"number":26,"title":"6.0","description":"","creator":{"login":"btovar","id":3081826,"avatar_url":"https://avatars.githubusercontent.com/u/3081826?v=3","gravatar_id":"","url":"https://api.github.com/users/btovar","html_url":"https://github.com/btovar","followers_url":"https://api.github.com/users/btovar/followers","following_url":"https://api.github.com/users/btovar/following{/other_user}","gists_url":"https://api.github.com/users/btovar/gists{/gist_id}","starred_url":"https://api.github.com/users/btovar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/btovar/subscriptions","organizations_url":"https://api.github.com/users/btovar/orgs","repos_url":"https://api.github.com/users/btovar/repos","events_url":"https://api.github.com/users/btovar/events{/privacy}","received_events_url":"https://api.github.com/users/btovar/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":42,"state":"open","created_at":"2015-08-14T12:11:08Z","updated_at":"2016-11-28T16:16:58Z","due_on":null,"closed_at":null},"comments":11,"created_at":"2016-11-26T10:39:40Z","updated_at":"2016-11-28T17:34:14Z","closed_at":null,"body":"Some of our software stopped working when run under parrot with the resource monitor.\r\n\r\nI've been able to trace this back to the MWE below, which will work **if** the resource monitor is removed and parrot run directly. You should be able to run this example on `crcfe01` and `crcfe02`. I've also attached the last bit of the output of `parrot_run -d all …`.\r\n\r\n[Here are the release notes for the newer software release](https://github.com/cms-sw/cmssw/releases/tag/CMSSW_8_0_16). Note that the changes list the following update, which is the only thing that stands out to me:\r\n\r\n> * cms-sw/cmsdist#2423 from @iahmad-khan: updates jemalloc to 4.2.0\r\n\r\nFurther random notes/ramblings in matz-e/lobster#530.\r\n\r\nMWE:\r\n```shell\r\ndir=$(mktemp -d)\r\ncd $dir\r\n\r\nexport HTTP_PROXY=\"http://eddie.crc.nd.edu:3128\"\r\n# export RELEASE=8_0_16  # XXX BROKEN\r\nexport RELEASE=8_0_15  # WORKS\r\n\r\ncat <<EOB >script\r\nset -x\r\nsource /cvmfs/cms.cern.ch/cmsset_default.sh\r\nscramv1 project CMSSW CMSSW_$RELEASE\r\ncd CMSSW_$RELEASE/src\r\neval \\$(scramv1 runtime -sh)\r\ncmsDriver.py MinBias_13TeV_pythia8_TuneCUETP8M1_cfi \\\r\n\t--python_filename sim.py \\\r\n\t--conditions auto:run2_mc --fast -n 100 \\\r\n\t--era Run2_2016 --eventcontent FASTPU --relval 100000,1000 \\\r\n\t-s GEN,SIM,RECOBEFMIX --datatier GEN-SIM-RECO --beamspot Realistic50ns13TeVCollision\r\nEOB\r\n\r\nresource_monitor --no-pprint --sh \"parrot_run -d all bash script\"  # XXX Broken\r\n# resource_monitor --no-pprint --sh \"bash script\"  # XXX Also broken\r\n\r\necho Please remove $dir\r\n```\r\n\r\nParrot output:\r\n```\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> libcall: = 0 [pfs_mmap_delete]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: = 0 [munmap]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: set_tid_address\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: = 10745 [set_tid_address]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: set_robust_list\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: = 0 [set_robust_list]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: futex\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: = 0 [futex]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: futex\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: = -11 Resource temporarily unavailable [futex]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: rt_sigaction\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: = 0 [rt_sigaction]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: rt_sigaction\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: = 0 [rt_sigaction]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: rt_sigprocmask\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: = 0 [rt_sigprocmask]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: getrlimit\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: = 0 [getrlimit]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: statfs\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> libcall: statfs /selinux 0x7ffff6d64040\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> resolve: /selinux = /selinux,0\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> local: readlink /selinux 0x7ffff6d612d0 1023\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> local: = -1 Invalid argument [readlink]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> local: statfs /selinux 0x7ffff6d64040\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> local: = 0 [statfs]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> libcall: = 0 [pfs_statfs]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> debug: allocated channel `(null)':9fe000:4096\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> debug: divert_to_channel(10745, pread64, 0x7fffc6ba38d0, 120, 10477568)\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> debug: --> pread64(65535, 0x7fffc6ba38d0, 120, 10477568)\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> debug: channel read 120\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> debug: decreasing refcount to 0 for channel `(null)':9fe000:4096\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> debug: freeing channel `(null)':9fe000:4096\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: = 0 [statfs]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: futex\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: = 0 [futex]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: readlink\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> libcall: readlink /etc/malloc.conf 0x7ffff6d75520 1024\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> resolve: /etc/malloc.conf = /etc/malloc.conf,4\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> local: readlink /etc/malloc.conf 0x7ffff6d75520 1024\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> local: = -1 No such file or directory [readlink]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> libcall: = -1 No such file or directory [pfs_readlink]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: = -2 No such file or directory [readlink]\r\n2016/11/26 05:37:24.65 parrot_run[9980] <child:10745> syscall: futex\r\n```"},"comment":{"url":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/comments/263337630","html_url":"https://github.com/cooperative-computing-lab/cctools/issues/1553#issuecomment-263337630","issue_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/1553","id":263337630,"user":{"login":"btovar","id":3081826,"avatar_url":"https://avatars.githubusercontent.com/u/3081826?v=3","gravatar_id":"","url":"https://api.github.com/users/btovar","html_url":"https://github.com/btovar","followers_url":"https://api.github.com/users/btovar/followers","following_url":"https://api.github.com/users/btovar/following{/other_user}","gists_url":"https://api.github.com/users/btovar/gists{/gist_id}","starred_url":"https://api.github.com/users/btovar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/btovar/subscriptions","organizations_url":"https://api.github.com/users/btovar/orgs","repos_url":"https://api.github.com/users/btovar/repos","events_url":"https://api.github.com/users/btovar/events{/privacy}","received_events_url":"https://api.github.com/users/btovar/received_events","type":"User","site_admin":false},"created_at":"2016-11-28T17:34:14Z","updated_at":"2016-11-28T17:34:14Z","body":"Yes, it actually did not hang last time. My current try is quite past \"Starting cmsRun ...\".\r\n\r\nCan you do:\r\n\r\n/usr/bin/env -i HOME=$HOME sh\r\nexport \r\nCCTOOLS=/afs/crc.nd.edu/group/ccl/software/cctools-138-37409e5b-cvmfs-40cf5bba/x86_64/redhat6/bin\r\n(set other env vars)\r\n$CCTOOLS/resource_monitor  --no-pprint --sh \"$CCTOOLS/parrot_run ./script\" -d all -o DEBUG -O mon\r\n\r\n"}}
{"actor":{"display_login":"btovar"}}