{"action":"created","issue":{"url":"https://api.github.com/repos/gavinmcdermott/pubsub_testnet/issues/4","repository_url":"https://api.github.com/repos/gavinmcdermott/pubsub_testnet","labels_url":"https://api.github.com/repos/gavinmcdermott/pubsub_testnet/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/gavinmcdermott/pubsub_testnet/issues/4/comments","events_url":"https://api.github.com/repos/gavinmcdermott/pubsub_testnet/issues/4/events","html_url":"https://github.com/gavinmcdermott/pubsub_testnet/issues/4","id":178962875,"number":4,"title":"Update 4: Tracking / Stats Tooling","user":{"login":"gavinmcdermott","id":302978,"avatar_url":"https://avatars.githubusercontent.com/u/302978?v=3","gravatar_id":"","url":"https://api.github.com/users/gavinmcdermott","html_url":"https://github.com/gavinmcdermott","followers_url":"https://api.github.com/users/gavinmcdermott/followers","following_url":"https://api.github.com/users/gavinmcdermott/following{/other_user}","gists_url":"https://api.github.com/users/gavinmcdermott/gists{/gist_id}","starred_url":"https://api.github.com/users/gavinmcdermott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gavinmcdermott/subscriptions","organizations_url":"https://api.github.com/users/gavinmcdermott/orgs","repos_url":"https://api.github.com/users/gavinmcdermott/repos","events_url":"https://api.github.com/users/gavinmcdermott/events{/privacy}","received_events_url":"https://api.github.com/users/gavinmcdermott/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-09-23T20:07:51Z","updated_at":"2016-09-26T22:10:38Z","closed_at":null,"body":"_cc: @diasdavid - these notes are based on our conversation around building out a stats/feedback tool for the testnet. Any feedback much appreciated!_\r\n\r\n_Thanks to @reidwilliams for the conversations and initial writeup._\r\n\r\n## Understanding testnet performance\r\n\r\nThere is a need for the testnet to provide a means of allowing developers a way to quickly and easily understand the performance of their messaging implementation based on a variety of factors (compare similar algos over time, compare different algos, etc.). **We're placing this as the highest priority next step in development—hopefully I'll have the basics completed in a few days.**\r\n\r\n### The `libp2p-datalogger`\r\n\r\n#### Overview\r\n\r\nAssumptions about how that the tracking/logging work:\r\n\r\n- Testing framework will be configured to instantiate a given number of IPFS nodes. Currently they are all within a single Node.js process. In the future they will likely be separate Node.js processes possibly running inside containers on separate machines.\r\n\r\n- Testing framework has a configurable node topology (partial mesh, ring, minimum spanning tree, etc). Each node instantiated with an experimental libp2p which implements a routing strategy.\r\n\r\n- Some kind of messaging / load test will be run.\r\n\r\n- Performance data will be collected and analyzed. This will likely include simple numerical stats and an evolving visual interface.\r\n\r\n#### Features\r\n\r\n`libp2p-datalogger` is a simple module to make it easy to log arbitrary data during a libp2p testnet messaging experiment. It should be useful for logging (_and viewing?_) any kind of data including status messages, event messages, and numerical performance data. Data should be loggable on a per-node and global network basis.\r\n\r\n- Each libp2p node in a test network can create and use a `datalogger`.\r\n\r\n- Log messages and numerical performance data are arbitrary JSON recorded on a key. For example, log(“currentQueueLength”, 20)\r\n\r\n- Logging automatically timestamps the data.\r\n\r\n- Logging more than one time on the same key creates a list of logged data for the key. This makes it easy to periodically log performance metrics over time.\r\n\r\n- Logged data is kept in memory until it is explicitly published via something like a `log.publish()` command. Publishing the log creates a set of data objects for the published log. At the end of an experiment, this makes it easy to fetch individual node performance data.\r\n\r\n- The datalogger does not define what data should be logged or provide statistical methods on the data. It just logs messages and publishes them (eventually using IPFS?). Statistical methods (E.g.: computing a max, min, and average queue length) should either be done in the node before logging or in processing log data after the experiment.\r\n\r\n\r\n### Questions\r\n\r\nSpecifically @diasdavid...\r\n\r\n- **Specific initial stats:** What are the most important things you want to know/learn about message propagation in these initial implementations? \r\n\r\n- **libp2p versions:** Because you'll be swapping versions of `libp2p` that have a specific messaging strategy built in, we'll need the ability to know which lib version and algo you're using (perhaps even historically). Any thoughts on how to track this (I know it'll be a branch, but the npm package has no knowledge of this...)?\r\n\r\n- **Entry points and Methods:** I was thinking of using the decorator style for implementing logging / tracking tools in `swarm` / `bitswap` / `libp2p`. Basically passing them in, overwriting the specific calls I want to track at certain points, building/logging the data we want to collect, and then passing the call on to the library. 1) Does this sound ok to you? 2) Based on the stats you want to collect, are there any modules I should pay closer attention to for instrumenting these decorators (E.g.: some useful methods in `bitswap`)?\r\n\r\n- **Information at Scales:** It’ll be useful to collect per node statistics. For tests with a large number of nodes (10k, 100k, 1M more?) it probably won’t be necessary to analyze data from all nodes, but it may be useful to analyze data from a random subset. Thoughts?\r\n\r\n- **For later:** When nodes are running in separate processes, how will we coordinate an experiment? For example, say we want to fire up 10k nodes with a particular libp2p, have 10% publish and subscribe heavily, run for 5 minutes, then shut down, how do we coordinate this experimental scenario across all the nodes? Maybe we need a simple remote procedure call like interface between a master node and test nodes, with some functions like: set configuration (libp2p location, network size, run time, node behavior, etc); start experiment; stop experiment; publish data.\r\n"},"comment":{"url":"https://api.github.com/repos/gavinmcdermott/pubsub_testnet/issues/comments/249712682","html_url":"https://github.com/gavinmcdermott/pubsub_testnet/issues/4#issuecomment-249712682","issue_url":"https://api.github.com/repos/gavinmcdermott/pubsub_testnet/issues/4","id":249712682,"user":{"login":"nicola","id":1424850,"avatar_url":"https://avatars.githubusercontent.com/u/1424850?v=3","gravatar_id":"","url":"https://api.github.com/users/nicola","html_url":"https://github.com/nicola","followers_url":"https://api.github.com/users/nicola/followers","following_url":"https://api.github.com/users/nicola/following{/other_user}","gists_url":"https://api.github.com/users/nicola/gists{/gist_id}","starred_url":"https://api.github.com/users/nicola/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nicola/subscriptions","organizations_url":"https://api.github.com/users/nicola/orgs","repos_url":"https://api.github.com/users/nicola/repos","events_url":"https://api.github.com/users/nicola/events{/privacy}","received_events_url":"https://api.github.com/users/nicola/received_events","type":"User","site_admin":false},"created_at":"2016-09-26T22:10:38Z","updated_at":"2016-09-26T22:10:38Z","body":"Perfect! I will have a read very soon (from mobile now)"}}
{"actor":{"display_login":"nicola"}}