{"action":"closed","issue":{"url":"https://api.github.com/repos/firehol/netdata/issues/604","repository_url":"https://api.github.com/repos/firehol/netdata","labels_url":"https://api.github.com/repos/firehol/netdata/issues/604/labels{/name}","comments_url":"https://api.github.com/repos/firehol/netdata/issues/604/comments","events_url":"https://api.github.com/repos/firehol/netdata/issues/604/events","html_url":"https://github.com/firehol/netdata/issues/604","id":161797321,"number":604,"title":"python plugins: threads","user":{"login":"ktsaou","id":2662304,"avatar_url":"https://avatars.githubusercontent.com/u/2662304?v=3","gravatar_id":"","url":"https://api.github.com/users/ktsaou","html_url":"https://github.com/ktsaou","followers_url":"https://api.github.com/users/ktsaou/followers","following_url":"https://api.github.com/users/ktsaou/following{/other_user}","gists_url":"https://api.github.com/users/ktsaou/gists{/gist_id}","starred_url":"https://api.github.com/users/ktsaou/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ktsaou/subscriptions","organizations_url":"https://api.github.com/users/ktsaou/orgs","repos_url":"https://api.github.com/users/ktsaou/repos","events_url":"https://api.github.com/users/ktsaou/events{/privacy}","received_events_url":"https://api.github.com/users/ktsaou/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/firehol/netdata/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-06-22T22:02:24Z","updated_at":"2016-06-23T17:54:18Z","closed_at":"2016-06-23T17:54:18Z","body":"Today, python modules are executed one after another. This has the problem that delays in data collection of one module increase the latency of another. If for example we have to collect data from 30 servers, per second, and each server responds in 50ms, we will not be able to collect all data within 1 second.\r\n\r\nTo solve this in `node.d.plugin` I have done the following:\r\n\r\n1. I utilized javascript async features to collect data concurrently. So all the tasks to collect data are executed in parallel (node.js is single threaded, but has async I/O allowing to behave like there are threads in place).\r\n\r\n  So, the main loop keeps track of which jobs has started and which have finished, to avoid collecting data for the same job more than once concurrently.\r\n\r\n2. To avoid multiplexing the output sent to netdata from multiple data collections, all modules generate data for netdata in a buffer and only when they finish one iteration, the whole data stream is send to netdata in a single print operation.\r\n\r\n@paulfantom any ideas?\r\n"}}
{"actor":{"display_login":"ktsaou"}}