{"action":"created","issue":{"url":"https://api.github.com/repos/dmwm/WMCore/issues/8522","repository_url":"https://api.github.com/repos/dmwm/WMCore","labels_url":"https://api.github.com/repos/dmwm/WMCore/issues/8522/labels{/name}","comments_url":"https://api.github.com/repos/dmwm/WMCore/issues/8522/comments","events_url":"https://api.github.com/repos/dmwm/WMCore/issues/8522/events","html_url":"https://github.com/dmwm/WMCore/issues/8522","id":306739213,"number":8522,"title":"Cleanup/alarm for logs in /store/logs/prod/recent/ ","user":{"login":"amaltaro","id":1798520,"avatar_url":"https://avatars0.githubusercontent.com/u/1798520?v=4","gravatar_id":"","url":"https://api.github.com/users/amaltaro","html_url":"https://github.com/amaltaro","followers_url":"https://api.github.com/users/amaltaro/followers","following_url":"https://api.github.com/users/amaltaro/following{/other_user}","gists_url":"https://api.github.com/users/amaltaro/gists{/gist_id}","starred_url":"https://api.github.com/users/amaltaro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amaltaro/subscriptions","organizations_url":"https://api.github.com/users/amaltaro/orgs","repos_url":"https://api.github.com/users/amaltaro/repos","events_url":"https://api.github.com/users/amaltaro/events{/privacy}","received_events_url":"https://api.github.com/users/amaltaro/received_events","type":"User","site_admin":false},"labels":[{"id":31815401,"url":"https://api.github.com/repos/dmwm/WMCore/labels/Medium%20Priority","name":"Medium Priority","color":"d7e102","default":false}],"state":"open","locked":false,"assignee":{"login":"amaltaro","id":1798520,"avatar_url":"https://avatars0.githubusercontent.com/u/1798520?v=4","gravatar_id":"","url":"https://api.github.com/users/amaltaro","html_url":"https://github.com/amaltaro","followers_url":"https://api.github.com/users/amaltaro/followers","following_url":"https://api.github.com/users/amaltaro/following{/other_user}","gists_url":"https://api.github.com/users/amaltaro/gists{/gist_id}","starred_url":"https://api.github.com/users/amaltaro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amaltaro/subscriptions","organizations_url":"https://api.github.com/users/amaltaro/orgs","repos_url":"https://api.github.com/users/amaltaro/repos","events_url":"https://api.github.com/users/amaltaro/events{/privacy}","received_events_url":"https://api.github.com/users/amaltaro/received_events","type":"User","site_admin":false},"assignees":[{"login":"amaltaro","id":1798520,"avatar_url":"https://avatars0.githubusercontent.com/u/1798520?v=4","gravatar_id":"","url":"https://api.github.com/users/amaltaro","html_url":"https://github.com/amaltaro","followers_url":"https://api.github.com/users/amaltaro/followers","following_url":"https://api.github.com/users/amaltaro/following{/other_user}","gists_url":"https://api.github.com/users/amaltaro/gists{/gist_id}","starred_url":"https://api.github.com/users/amaltaro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amaltaro/subscriptions","organizations_url":"https://api.github.com/users/amaltaro/orgs","repos_url":"https://api.github.com/users/amaltaro/repos","events_url":"https://api.github.com/users/amaltaro/events{/privacy}","received_events_url":"https://api.github.com/users/amaltaro/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/dmwm/WMCore/milestones/147","html_url":"https://github.com/dmwm/WMCore/milestone/147","labels_url":"https://api.github.com/repos/dmwm/WMCore/milestones/147/labels","id":2861247,"number":147,"title":"WMAgent1804","description":"","creator":{"login":"ticoann","id":1247993,"avatar_url":"https://avatars0.githubusercontent.com/u/1247993?v=4","gravatar_id":"","url":"https://api.github.com/users/ticoann","html_url":"https://github.com/ticoann","followers_url":"https://api.github.com/users/ticoann/followers","following_url":"https://api.github.com/users/ticoann/following{/other_user}","gists_url":"https://api.github.com/users/ticoann/gists{/gist_id}","starred_url":"https://api.github.com/users/ticoann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ticoann/subscriptions","organizations_url":"https://api.github.com/users/ticoann/orgs","repos_url":"https://api.github.com/users/ticoann/repos","events_url":"https://api.github.com/users/ticoann/events{/privacy}","received_events_url":"https://api.github.com/users/ticoann/received_events","type":"User","site_admin":false},"open_issues":23,"closed_issues":3,"state":"open","created_at":"2017-10-24T05:25:59Z","updated_at":"2018-03-23T15:59:01Z","due_on":"2018-04-30T07:00:00Z","closed_at":null},"comments":1,"created_at":"2018-03-20T06:02:28Z","updated_at":"2018-03-27T09:10:11Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Follow up of #8456\r\nIt's not a WMCore issue, but I think it's worthy having a ticket here and discuss how we're going to handle that log storage space. The allocation is 30TB and a maximum of 50M files, which can be seen with\r\n```\r\namaltaro@aiadm61:~ $ eoscms quota /eos/cms/store/logs/prod/recent\r\n\r\nBy group:\r\n┏━> Quota Node: /eos/cms/store/logs/prod/recent/\r\n┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┐\r\n│group     │used bytes│logi bytes│used files│aval bytes│aval logib│aval files│ filled[%]│vol-status│ino-status│\r\n└──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘\r\n zh           21.24 GB   10.62 GB     7.91 K   60.00 TB   30.00 TB    50.00 M     0.04 %         ok         ok \r\n┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛\r\n```\r\n\r\nAccording to dashboard, we're running between 500 and 600k jobs a day (based on the last week resource utilization). From Seangchan's recent tests, the tarballs are around 5MB each (though it can vary by quite a bit depending on the workflow and job configuration - and site stage out commands). In summary:\r\n* that gives us around 100 days worth of files\r\n* 10 to 12 days worth of storage space (6M tarballs only!)\r\n\r\nThus, I'd suggest\r\n* we need to get more space for these logs. In avg, we need 100TB of _usable space_ in order to keep 1 month worth of logs. (So I'd say we need 200TB of usable space) \r\n* for the cleanup, I think we can contact the transfer team and ask if they can setup some cleanup script. I know Sid is working on a script to perform the /store/unmerged cleanup\r\n* then for monitoring the quota and making sure we never reach the limit (including the limit of files!), we can ask the VOC to setup an alarm once we reach 80%(?) of the total space.\r\n\r\nAny thoughts?"},"comment":{"url":"https://api.github.com/repos/dmwm/WMCore/issues/comments/376452645","html_url":"https://github.com/dmwm/WMCore/issues/8522#issuecomment-376452645","issue_url":"https://api.github.com/repos/dmwm/WMCore/issues/8522","id":376452645,"user":{"login":"vlimant","id":4989875,"avatar_url":"https://avatars3.githubusercontent.com/u/4989875?v=4","gravatar_id":"","url":"https://api.github.com/users/vlimant","html_url":"https://github.com/vlimant","followers_url":"https://api.github.com/users/vlimant/followers","following_url":"https://api.github.com/users/vlimant/following{/other_user}","gists_url":"https://api.github.com/users/vlimant/gists{/gist_id}","starred_url":"https://api.github.com/users/vlimant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vlimant/subscriptions","organizations_url":"https://api.github.com/users/vlimant/orgs","repos_url":"https://api.github.com/users/vlimant/repos","events_url":"https://api.github.com/users/vlimant/events{/privacy}","received_events_url":"https://api.github.com/users/vlimant/received_events","type":"User","site_admin":false},"created_at":"2018-03-27T09:10:11Z","updated_at":"2018-03-27T09:10:11Z","author_association":"CONTRIBUTOR","body":"the availability of reports/logs from unified is set to \"keep everything you have for workflows that have not been announced more than 2 weeks ago\".\r\nthe cron is already set, I would just need to add /eos/cms/store/logs/prod/recent to the paths to clean and be given delete permission in it."}}
{"actor":{"display_login":"vlimant"}}