{"action":"created","issue":{"url":"https://api.github.com/repos/dmwm/AsyncStageout/issues/4473","labels_url":"https://api.github.com/repos/dmwm/AsyncStageout/issues/4473/labels{/name}","comments_url":"https://api.github.com/repos/dmwm/AsyncStageout/issues/4473/comments","events_url":"https://api.github.com/repos/dmwm/AsyncStageout/issues/4473/events","html_url":"https://github.com/dmwm/AsyncStageout/issues/4473","id":129110272,"number":4473,"title":"Cleanup Monitor script not working ","user":{"login":"dciangot","id":4144326,"avatar_url":"https://avatars.githubusercontent.com/u/4144326?v=3","gravatar_id":"","url":"https://api.github.com/users/dciangot","html_url":"https://github.com/dciangot","followers_url":"https://api.github.com/users/dciangot/followers","following_url":"https://api.github.com/users/dciangot/following{/other_user}","gists_url":"https://api.github.com/users/dciangot/gists{/gist_id}","starred_url":"https://api.github.com/users/dciangot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dciangot/subscriptions","organizations_url":"https://api.github.com/users/dciangot/orgs","repos_url":"https://api.github.com/users/dciangot/repos","events_url":"https://api.github.com/users/dciangot/events{/privacy}","received_events_url":"https://api.github.com/users/dciangot/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-01-27T11:05:17Z","updated_at":"2016-02-02T12:29:07Z","closed_at":null,"body":"Looking at ASO@prod, I noticed that there are a lot (>1M) of old monitor json (since 9th Jan) and that still in the monitor folder. The fact that Monitor component doesn't remove those files is a known issue, worked around by putting a cron doing that.\r\nThat script seems not to work on prod due to a wrong path (don't know since when).\r\n \r\nI fixed it and tested in preprod, but I would wait for next deployment in order to cleanup prod."},"comment":{"url":"https://api.github.com/repos/dmwm/AsyncStageout/issues/comments/178550230","html_url":"https://github.com/dmwm/AsyncStageout/issues/4473#issuecomment-178550230","issue_url":"https://api.github.com/repos/dmwm/AsyncStageout/issues/4473","id":178550230,"user":{"login":"juztas","id":646582,"avatar_url":"https://avatars.githubusercontent.com/u/646582?v=3","gravatar_id":"","url":"https://api.github.com/users/juztas","html_url":"https://github.com/juztas","followers_url":"https://api.github.com/users/juztas/followers","following_url":"https://api.github.com/users/juztas/following{/other_user}","gists_url":"https://api.github.com/users/juztas/gists{/gist_id}","starred_url":"https://api.github.com/users/juztas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juztas/subscriptions","organizations_url":"https://api.github.com/users/juztas/orgs","repos_url":"https://api.github.com/users/juztas/repos","events_url":"https://api.github.com/users/juztas/events{/privacy}","received_events_url":"https://api.github.com/users/juztas/received_events","type":"User","site_admin":false},"created_at":"2016-02-02T12:29:07Z","updated_at":"2016-02-02T12:29:07Z","body":"new script is in prod and finished in 30m. Seems previous clean up script was really slow and was running in ages...\r\n```\r\nfrom threading import Thread\r\nimport os\r\nfrom Queue import Queue\r\nimport subprocess\r\nimport json\r\n\r\nmyList = Queue(maxsize=0)\r\nlockFile = 'Cleanup-locked'\r\nif os.path.isfile(lockFile):\r\n    print 'Clean up lock file exists. Exiting and not running on top.'\r\n    exit\r\n\r\n#Touch the file to let know other that lock is available.\r\nopen(lockFile, 'a').close()\r\n\r\ncounter = []\r\nif os.path.isfile(lastCounter):\r\n    with open(lastCounter) as fd:\r\n        counter = json.loads(fd.read())\r\n\r\nif len(counter) == 0:\r\n    counter.append(0)\r\n\r\ndef deleteFile(i, q):\r\n    while True:\r\n        success = False\r\n        fileName = q.get()\r\n        fileName = \"/data/srv/asyncstageout/current/install/asyncstageout/Monitor/\" + fileName\r\n        #print fileName\r\n        if os.path.isfile(fileName):\r\n            os.remove(fileName)\r\n            success = True\r\n            print '%s %s %s' % (i, fileName, success)\r\n        q.task_done()\r\n\r\nprint 'grep all in ended state'\r\nreturnCode = subprocess.call([\"grep 'ended in state' aso-monitor.log | awk '{print $4}' > cleanup_out1.txt\"], shell=True)\r\nprint 'grep all in ended state exit %s' % returnCode\r\nprint 'grep all JOBIDs'\r\nreturnCode = subprocess.call([\"grep 'JOBID' cleanup_out1.txt | awk -F= '{print $2}' > cleanup_out2.txt\"], shell=True)\r\nprint 'grep all JOBIDs exit %s' % returnCode\r\n\r\nwith open('cleanup_out2.txt') as fd:\r\n    lines = fd.readlines()\r\n\r\nsize = len(lines)\r\nprint 'Number of files in monitor log file: %s' % size\r\nprint 'Last time cleanup processed: %s' % counter[0]\r\nwhile size != counter[0]:\r\n    size -= 1\r\n    myList.put(\"work/Monitor.%s.json\" % lines[size].rstrip())\r\n\r\nfor i in range(24):\r\n    worker = Thread(target=deleteFile, args=(i, myList))\r\n    worker.setDaemon(True)\r\n    worker.start()\r\nprint 'Main wait'\r\nmyList.join()\r\nprint 'Main done'\r\n\r\n# Cleanup\r\nwith open(lastCounter, 'w') as fd:\r\n    json.dump(counter, fd)\r\nos.remove('cleanup_out1.txt')\r\nos.remove('cleanup_out2.txt')\r\nos.remove(lockFile)\r\n```\r\n\r\nOf course root problem should be fixed and not hacks made, but as we are short on time for prod, this is the way to go. First time run will take longer time as it does not know how many files there was previous run. Second run will proceed much faster."}}