{"action":"created","issue":{"url":"https://api.github.com/repos/LanguageMachines/frog/issues/49","repository_url":"https://api.github.com/repos/LanguageMachines/frog","labels_url":"https://api.github.com/repos/LanguageMachines/frog/issues/49/labels{/name}","comments_url":"https://api.github.com/repos/LanguageMachines/frog/issues/49/comments","events_url":"https://api.github.com/repos/LanguageMachines/frog/issues/49/events","html_url":"https://github.com/LanguageMachines/frog/issues/49","id":322776145,"number":49,"title":"Frog gets progressively slower when running for hours, days","user":{"login":"gmjonker","id":6223710,"avatar_url":"https://avatars3.githubusercontent.com/u/6223710?v=4","gravatar_id":"","url":"https://api.github.com/users/gmjonker","html_url":"https://github.com/gmjonker","followers_url":"https://api.github.com/users/gmjonker/followers","following_url":"https://api.github.com/users/gmjonker/following{/other_user}","gists_url":"https://api.github.com/users/gmjonker/gists{/gist_id}","starred_url":"https://api.github.com/users/gmjonker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gmjonker/subscriptions","organizations_url":"https://api.github.com/users/gmjonker/orgs","repos_url":"https://api.github.com/users/gmjonker/repos","events_url":"https://api.github.com/users/gmjonker/events{/privacy}","received_events_url":"https://api.github.com/users/gmjonker/received_events","type":"User","site_admin":false},"labels":[{"id":104767460,"url":"https://api.github.com/repos/LanguageMachines/frog/labels/help%20wanted","name":"help wanted","color":"159818","default":true}],"state":"open","locked":false,"assignee":{"login":"proycon","id":75427,"avatar_url":"https://avatars3.githubusercontent.com/u/75427?v=4","gravatar_id":"","url":"https://api.github.com/users/proycon","html_url":"https://github.com/proycon","followers_url":"https://api.github.com/users/proycon/followers","following_url":"https://api.github.com/users/proycon/following{/other_user}","gists_url":"https://api.github.com/users/proycon/gists{/gist_id}","starred_url":"https://api.github.com/users/proycon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/proycon/subscriptions","organizations_url":"https://api.github.com/users/proycon/orgs","repos_url":"https://api.github.com/users/proycon/repos","events_url":"https://api.github.com/users/proycon/events{/privacy}","received_events_url":"https://api.github.com/users/proycon/received_events","type":"User","site_admin":false},"assignees":[{"login":"proycon","id":75427,"avatar_url":"https://avatars3.githubusercontent.com/u/75427?v=4","gravatar_id":"","url":"https://api.github.com/users/proycon","html_url":"https://github.com/proycon","followers_url":"https://api.github.com/users/proycon/followers","following_url":"https://api.github.com/users/proycon/following{/other_user}","gists_url":"https://api.github.com/users/proycon/gists{/gist_id}","starred_url":"https://api.github.com/users/proycon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/proycon/subscriptions","organizations_url":"https://api.github.com/users/proycon/orgs","repos_url":"https://api.github.com/users/proycon/repos","events_url":"https://api.github.com/users/proycon/events{/privacy}","received_events_url":"https://api.github.com/users/proycon/received_events","type":"User","site_admin":false},{"login":"kosloot","id":5870017,"avatar_url":"https://avatars1.githubusercontent.com/u/5870017?v=4","gravatar_id":"","url":"https://api.github.com/users/kosloot","html_url":"https://github.com/kosloot","followers_url":"https://api.github.com/users/kosloot/followers","following_url":"https://api.github.com/users/kosloot/following{/other_user}","gists_url":"https://api.github.com/users/kosloot/gists{/gist_id}","starred_url":"https://api.github.com/users/kosloot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kosloot/subscriptions","organizations_url":"https://api.github.com/users/kosloot/orgs","repos_url":"https://api.github.com/users/kosloot/repos","events_url":"https://api.github.com/users/kosloot/events{/privacy}","received_events_url":"https://api.github.com/users/kosloot/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2018-05-14T11:26:40Z","updated_at":"2018-05-14T12:49:39Z","closed_at":null,"author_association":"NONE","body":"When running frog for a long time, performance decreases significantly. \r\n\r\nFor instance, I'm processing a 2.8GB file. These are some `pv` outputs at several moments:\r\n\r\n    frog: 15.8MiB  0:05:10 [39.2KiB/s] [>                                 ]  0% ETA  4:20:44:08\r\n    frog:  545MiB  3:29:02 [62.2KiB/s] [>                                 ]  2% ETA  5:13:26:11     \r\n    frog:  858MiB  5:23:19 [30.6KiB/s] [=>                                ]  4% ETA  5:09:10:47\r\n    frog: 3.05GiB 73:28:54 [    0 B/s] [====>                             ] 14% ETA 17:22:57:01\r\n\r\nAt first, the expected time is 4 days and 20 hours. After having run for 3 days, the expected time has gone up to almost 18 days.\r\n\r\nThis is the script used:\r\n\r\n\t#!/usr/bin/env bash\r\n\tFILE=$1\r\n\tFILE_SIZE=`wc -c < \"$FILE\" | cut -f1`\r\n\tlet \"EXPECTED_FROG_SIZE = 8 * FILE_SIZE\"\r\n\tBODY=\"${FILE%.*}\"\r\n\t\r\n\techo \"Processing: $FILE\"\r\n\techo \"Size: $FILE_SIZE\"\r\n\techo \"Writing output to: ${BODY}_frog.txt\"\r\n\t\r\n\tpv -s ${FILE_SIZE} -cN in ${FILE} | frog --skip=acmnp 2> ${BODY}_frog.log | pv -s ${EXPECTED_FROG_SIZE} -cN frog > ${BODY}_frog.txt\r\n"},"comment":{"url":"https://api.github.com/repos/LanguageMachines/frog/issues/comments/388804838","html_url":"https://github.com/LanguageMachines/frog/issues/49#issuecomment-388804838","issue_url":"https://api.github.com/repos/LanguageMachines/frog/issues/49","id":388804838,"user":{"login":"kosloot","id":5870017,"avatar_url":"https://avatars1.githubusercontent.com/u/5870017?v=4","gravatar_id":"","url":"https://api.github.com/users/kosloot","html_url":"https://github.com/kosloot","followers_url":"https://api.github.com/users/kosloot/followers","following_url":"https://api.github.com/users/kosloot/following{/other_user}","gists_url":"https://api.github.com/users/kosloot/gists{/gist_id}","starred_url":"https://api.github.com/users/kosloot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kosloot/subscriptions","organizations_url":"https://api.github.com/users/kosloot/orgs","repos_url":"https://api.github.com/users/kosloot/repos","events_url":"https://api.github.com/users/kosloot/events{/privacy}","received_events_url":"https://api.github.com/users/kosloot/received_events","type":"User","site_admin":false},"created_at":"2018-05-14T12:49:39Z","updated_at":"2018-05-14T12:49:39Z","author_association":"COLLABORATOR","body":"Well, i am not sure what is happening here, but my first guess is that the process gets low on memory.\r\nDo you have enough RAM to run this kind of large input?\r\nfrog will construct a large FoLiA document, covering the whole input. I wouldn't be surprised if it needs 50G of RAM or more.\r\n\r\nThis is the result of an unfortunate design decision long ago.\r\nFor the time being, it might be a solution to split the input file into smaller parts.\r\n\r\nYou also might experiment with the -n option (one sentence per line) \r\nor the --threads option, setting it to 1\r\n\r\n"}}
{"actor":{"display_login":"kosloot"}}