{"action":"created","issue":{"url":"https://api.github.com/repos/joehand/hypercloud/issues/20","repository_url":"https://api.github.com/repos/joehand/hypercloud","labels_url":"https://api.github.com/repos/joehand/hypercloud/issues/20/labels{/name}","comments_url":"https://api.github.com/repos/joehand/hypercloud/issues/20/comments","events_url":"https://api.github.com/repos/joehand/hypercloud/issues/20/events","html_url":"https://github.com/joehand/hypercloud/issues/20","id":198100382,"number":20,"title":"Architecture decisions - initial thoughts","user":{"login":"pfrazee","id":1270099,"avatar_url":"https://avatars.githubusercontent.com/u/1270099?v=3","gravatar_id":"","url":"https://api.github.com/users/pfrazee","html_url":"https://github.com/pfrazee","followers_url":"https://api.github.com/users/pfrazee/followers","following_url":"https://api.github.com/users/pfrazee/following{/other_user}","gists_url":"https://api.github.com/users/pfrazee/gists{/gist_id}","starred_url":"https://api.github.com/users/pfrazee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pfrazee/subscriptions","organizations_url":"https://api.github.com/users/pfrazee/orgs","repos_url":"https://api.github.com/users/pfrazee/repos","events_url":"https://api.github.com/users/pfrazee/events{/privacy}","received_events_url":"https://api.github.com/users/pfrazee/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-12-30T03:34:48Z","updated_at":"2016-12-30T17:29:24Z","closed_at":null,"body":"I think we should take a moment to talk out the architecture of the hypercloud system. It's tempting to build a simple core server around leveldb and then sharded archives servers - and maybe that's the right call - but I dont think that will scale well, and I get *really nervous* about using custom software at scale without a very experienced devops team.\r\n\r\nMy analysis here involves some guesswork. I may get some things wrong.\r\n\r\nWe should have these questions in mind:\r\n\r\n - What is our scale target?\r\n - How will we scale horizontally?\r\n - How will we monitor nodes, and the network?\r\n - How will we handle failover and redundancy?\r\n - When can we use off-the-shelf solutions?\r\n\r\n## Scale target\r\n\r\nThis determines almost every other decision.\r\n\r\nIf we don't use any off-the-shelf solutions (Postgres, Cassandra, Kafka, etc), I think we could handle 1k users with about 10k archives. If we expect that the archives will average ~5mb in size, that's 50GB of data.\r\n\r\nBut, 1k users is not that far off, and things get challenging pretty quickly. If we plan for it, and use some off-the-shelf software, I think we could reasonably target 10k-100k users / 100k-1mm archives (5TB of data), without having to significantly re-architect hypercloud.\r\n\r\nThe tradeoff is system complexity. A 1k target can probably be made into an easily-deployed bundle, whereas a 10-100k target requires some ops work. If we want hypercloud to be something that anybody can conveniently deploy (\"your own hypercloud is easy!\") then 1k is the right target. But, we'll have to rewrite a lot once we get to 1k.\r\n\r\nAs a quick reference, here's everything hypercloud will require, off the top of my head.\r\n\r\n- Load balancer\r\n- Caching server\r\n- HTTP server\r\n- User DB\r\n- Archive DB\r\n- Computed view DB\r\n- Changes feed\r\n- Indexer / Data-processing pipeline\r\n- Monitoring"},"comment":{"url":"https://api.github.com/repos/joehand/hypercloud/issues/comments/269798566","html_url":"https://github.com/joehand/hypercloud/issues/20#issuecomment-269798566","issue_url":"https://api.github.com/repos/joehand/hypercloud/issues/20","id":269798566,"user":{"login":"maxogden","id":39759,"avatar_url":"https://avatars.githubusercontent.com/u/39759?v=3","gravatar_id":"","url":"https://api.github.com/users/maxogden","html_url":"https://github.com/maxogden","followers_url":"https://api.github.com/users/maxogden/followers","following_url":"https://api.github.com/users/maxogden/following{/other_user}","gists_url":"https://api.github.com/users/maxogden/gists{/gist_id}","starred_url":"https://api.github.com/users/maxogden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maxogden/subscriptions","organizations_url":"https://api.github.com/users/maxogden/orgs","repos_url":"https://api.github.com/users/maxogden/repos","events_url":"https://api.github.com/users/maxogden/events{/privacy}","received_events_url":"https://api.github.com/users/maxogden/received_events","type":"User","site_admin":false},"created_at":"2016-12-30T17:29:24Z","updated_at":"2016-12-30T17:29:24Z","body":"I think a single node solution can handle these parts for a long time without too much fuss:\r\n\r\n- User accounts/authentication (township aka leveldb)\r\n- Metadata storage/stats API (leveldb or sql)\r\n- Web app serving (node http)\r\n\r\nReasoning here is that these parts don't involve much data/bandwidth, and are cacheable/optimizable.\r\n\r\nParts that should be on other nodes:\r\n\r\n- Hypercore storage/serving (e.g. hypercore-archiver-server)\r\n- Search (e.g. elasticsearch)\r\n- Dat network crawlers (for statistics)\r\n\r\nThese will be the services that incur most of the processing/storage/bandwidth overhead. The crawlers will be a series of processes that query all dat keys on some interval and investigate swarms to collect stats like how many peers are online, how many copies of the data exist.\r\n\r\nTo avoid running an expensive (in time and money) elasticsearch thing up front we wanted to make sure we had great static html rendering on the webapp so you can nicely google for things. Since the web app will be more akin to NPM than GitHub (e.g. like npm, pages for dats aren't that dynamic and don't change too often, vs github which has many more interactive web app features) we can also cache a lot of the html views.\r\n\r\nThe idea of the architecture overall is to allow the hypercore storage nodes to live on other machines from the central metadata/state machine. So if we figure out that pattern it solves our horizontal scalability issue for the most part.\r\n\r\nSo I'd say a sensible v1 would be everything simple/single node except the hypercore archiving part, and we can skip search on v1. The crawlers ideally can run on any machine and just use the metadata api (if they have permission to) to update repo stats.\r\n"}}
{"actor":{"display_login":"maxogden"}}