{"action":"created","issue":{"url":"https://api.github.com/repos/feross/webtorrent/issues/599","repository_url":"https://api.github.com/repos/feross/webtorrent","labels_url":"https://api.github.com/repos/feross/webtorrent/issues/599/labels{/name}","comments_url":"https://api.github.com/repos/feross/webtorrent/issues/599/comments","events_url":"https://api.github.com/repos/feross/webtorrent/issues/599/events","html_url":"https://github.com/feross/webtorrent/issues/599","id":131901283,"number":599,"title":"Lazy file chunks store","user":{"login":"oleiba","id":16321515,"avatar_url":"https://avatars.githubusercontent.com/u/16321515?v=3","gravatar_id":"","url":"https://api.github.com/users/oleiba","html_url":"https://github.com/oleiba","followers_url":"https://api.github.com/users/oleiba/followers","following_url":"https://api.github.com/users/oleiba/following{/other_user}","gists_url":"https://api.github.com/users/oleiba/gists{/gist_id}","starred_url":"https://api.github.com/users/oleiba/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oleiba/subscriptions","organizations_url":"https://api.github.com/users/oleiba/orgs","repos_url":"https://api.github.com/users/oleiba/repos","events_url":"https://api.github.com/users/oleiba/events{/privacy}","received_events_url":"https://api.github.com/users/oleiba/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/feross/webtorrent/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/feross/webtorrent/labels/need%20more%20info","name":"need more info","color":"d4c5f9"},{"url":"https://api.github.com/repos/feross/webtorrent/labels/node","name":"node","color":"fbca04"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-02-06T21:24:40Z","updated_at":"2016-02-08T08:02:48Z","closed_at":null,"body":"Hi, I am currently using WebTorrent for seeding a large amount of files in server-side (node.js).\r\nOne of the problems I've noticed when trying to scale-up WebTorrent is the amount of opened files.\r\nWhen each torrent is created - it eventually reaches `Torrent._onMetadata` where it creates an instance of FSChunkStore (npm module fs-chunk-store).\r\nThis opens a file for tracking and validating pieces downloaded.\r\nIf one assumption is that most torrents will have very little and rare activity, files could be loaded on demand, rather than up-front.\r\nI want to suggest that the file's .open will occur lazily only when a peer is interested, or alternatively closing the file and re-open it upon request.\r\nI know this is an important feature in libtorrent's codebase, I believe this would allow WebTorrent to scale above limitations it currently has."},"comment":{"url":"https://api.github.com/repos/feross/webtorrent/issues/comments/181248623","html_url":"https://github.com/feross/webtorrent/issues/599#issuecomment-181248623","issue_url":"https://api.github.com/repos/feross/webtorrent/issues/599","id":181248623,"user":{"login":"oleib","id":14790931,"avatar_url":"https://avatars.githubusercontent.com/u/14790931?v=3","gravatar_id":"","url":"https://api.github.com/users/oleib","html_url":"https://github.com/oleib","followers_url":"https://api.github.com/users/oleib/followers","following_url":"https://api.github.com/users/oleib/following{/other_user}","gists_url":"https://api.github.com/users/oleib/gists{/gist_id}","starred_url":"https://api.github.com/users/oleib/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oleib/subscriptions","organizations_url":"https://api.github.com/users/oleib/orgs","repos_url":"https://api.github.com/users/oleib/repos","events_url":"https://api.github.com/users/oleib/events{/privacy}","received_events_url":"https://api.github.com/users/oleib/received_events","type":"User","site_admin":false},"created_at":"2016-02-08T08:02:48Z","updated_at":"2016-02-08T08:02:48Z","body":"Hi @feross, thank you so much for your reference.\r\n\r\n- I'm using webtorrent@0.72.2, and create-torrent@0.3.10\r\n- I'm seeding thousands of torrents (I plan to scale to millions in future).\r\n- Each torrent refer to just a single JSON file (no multi-file torrents), under 1KB each.\r\n- Performance issues: I came to the need to limit the number of torrents I'm seeding (about ~2,500) because otherwise the server will simply crash. \r\nUnfortunately, it crashed without a meaningful log - I suspect it is because of the amount of opened files: each torrent requires one open data-file to be stored locally in order to seed it; one .torrent file to create the magnet link from; one file for the chunk-store (I think I can handle closing the .torrent files from my side since they are only used to create the magnet link, and I'm still looking for a way where I can also close my data-file and re-open it only when the file is required from a connected peer).\r\nMoreover, I need to reannounce my files periodically to the DHT since they are naturally rare (I'm their initial seeder and downloading is rare) - when it comes to the hundreds mark, a DHT announce can take 10-20 minutes (from the moment a `dht.announce` was called until it has called back - see PR feross/torrent-discovery#16).\r\n- Error messages: I got the LRU error-messages described in #592 (I can update to use @mafintosh latest changes to chriso/lru#14 to see if resolved) and also a lot of `warning: possible EventEmitter memory leak detected. 11 error listeners added` ; `Trace: RPC.addListener` or `Trace: DHT.addListener`.\r\n\r\nOther than that, I'll be glad to get your insight. And thanks for this awesome project ! :+1: "}}