{"action":"closed","issue":{"url":"https://api.github.com/repos/earonesty/ea-utils/issues/33","repository_url":"https://api.github.com/repos/earonesty/ea-utils","labels_url":"https://api.github.com/repos/earonesty/ea-utils/issues/33/labels{/name}","comments_url":"https://api.github.com/repos/earonesty/ea-utils/issues/33/comments","events_url":"https://api.github.com/repos/earonesty/ea-utils/issues/33/events","html_url":"https://github.com/earonesty/ea-utils/issues/33","id":67274313,"number":33,"title":"Giant FASTQ support in stats","user":{"login":"GoogleCodeExporter","id":9614759,"avatar_url":"https://avatars0.githubusercontent.com/u/9614759?v=4","gravatar_id":"","url":"https://api.github.com/users/GoogleCodeExporter","html_url":"https://github.com/GoogleCodeExporter","followers_url":"https://api.github.com/users/GoogleCodeExporter/followers","following_url":"https://api.github.com/users/GoogleCodeExporter/following{/other_user}","gists_url":"https://api.github.com/users/GoogleCodeExporter/gists{/gist_id}","starred_url":"https://api.github.com/users/GoogleCodeExporter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoogleCodeExporter/subscriptions","organizations_url":"https://api.github.com/users/GoogleCodeExporter/orgs","repos_url":"https://api.github.com/users/GoogleCodeExporter/repos","events_url":"https://api.github.com/users/GoogleCodeExporter/events{/privacy}","received_events_url":"https://api.github.com/users/GoogleCodeExporter/received_events","type":"User","site_admin":false},"labels":[{"id":197467823,"url":"https://api.github.com/repos/earonesty/ea-utils/labels/Priority-Medium","name":"Priority-Medium","color":"ededed","default":false},{"id":197467941,"url":"https://api.github.com/repos/earonesty/ea-utils/labels/Type-Enhancement","name":"Type-Enhancement","color":"ededed","default":false},{"id":197467822,"url":"https://api.github.com/repos/earonesty/ea-utils/labels/auto-migrated","name":"auto-migrated","color":"ededed","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2015-04-09T03:44:34Z","updated_at":"2018-02-12T14:11:35Z","closed_at":"2018-02-12T14:11:35Z","author_association":"NONE","body":"```\nSome stats programs have things like kmer (with -K) reports and probe-id \ncounting (with -D).\n\nThese programs can consume a lot of RAM (>10GB), even with the highly efficient \nsparsehash library on very large files (> 200 mil reads).\n\nThe use of a disk-backed key-value store, like levelDB could see decent \nperformance, like a hash, but would also allow growth past available RAM with \ndecent performance.   I'm thinking that the code should switch to a DB-backed \nstore at the 200 mil record level.   This would slow things down by about 3x \n(from 1 mil writes/sec to 300k writes/sec), but would also allow infinte \ngrowth.  Enabling a large LRU cache could it perform so similarly that the \nsparse hash can be abandoned, especially if the db remains an insigificant \nfraction of the stats collection process.   \n```\n\nOriginal issue reported on code.google.com by `earone...@gmail.com` on 9 Jul 2014 at 2:26\n"}}
{"actor":{"display_login":"earonesty"}}