{"action":"created","issue":{"url":"https://api.github.com/repos/meetecho/janus-gateway/issues/536","repository_url":"https://api.github.com/repos/meetecho/janus-gateway","labels_url":"https://api.github.com/repos/meetecho/janus-gateway/issues/536/labels{/name}","comments_url":"https://api.github.com/repos/meetecho/janus-gateway/issues/536/comments","events_url":"https://api.github.com/repos/meetecho/janus-gateway/issues/536/events","html_url":"https://github.com/meetecho/janus-gateway/pull/536","id":156285830,"number":536,"title":"Event handler plugins","user":{"login":"lminiero","id":3684796,"avatar_url":"https://avatars.githubusercontent.com/u/3684796?v=3","gravatar_id":"","url":"https://api.github.com/users/lminiero","html_url":"https://github.com/lminiero","followers_url":"https://api.github.com/users/lminiero/followers","following_url":"https://api.github.com/users/lminiero/following{/other_user}","gists_url":"https://api.github.com/users/lminiero/gists{/gist_id}","starred_url":"https://api.github.com/users/lminiero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lminiero/subscriptions","organizations_url":"https://api.github.com/users/lminiero/orgs","repos_url":"https://api.github.com/users/lminiero/repos","events_url":"https://api.github.com/users/lminiero/events{/privacy}","received_events_url":"https://api.github.com/users/lminiero/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":64,"created_at":"2016-05-23T13:52:25Z","updated_at":"2016-10-26T23:59:30Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/meetecho/janus-gateway/pulls/536","html_url":"https://github.com/meetecho/janus-gateway/pull/536","diff_url":"https://github.com/meetecho/janus-gateway/pull/536.diff","patch_url":"https://github.com/meetecho/janus-gateway/pull/536.patch"},"body":"Hi all,\r\n\r\nthis is a pretty major change in Janus, as it introduces a new category of plugins: event handlers. More specifically, now Janus (and plugins) can originate events of specific types (e.g., session created, handle detached, peerconnection going up, media available, packet loss, plugin-specific data, etc.). These events are then passed to all the event handler plugins that subscribe to that type, so that they can be, indeed, \"handled\" somehow.\r\n\r\nSince the Admin API is pull-based, this should be useful/helpful for several reasons. You may want, for instance, to use some events as a \"trigger\" to do an Admin API pull, rather than polling continuously; or you may want to save these events to a database, so that you have a view of what happened in each media session; or send them to an external component for processing; etc. I expect a typical usage will actually be implementing plugins that interface to external services (e.g., Homer, callstats.io, etc.) in order to have them analyzed/processed there.\r\n\r\nAt the moment I only implemented a very simple sample plugin, that basically forwards any event it receives to a web backend via HTTP POST. If you want to give it a quick test, enable it in `janus.eventhandler.sampleevh.cfg` (it's disabled by default), set a valid backend (e.g., `http://localhost:8080`), and wait for events to be received by your web server. A very simple way to set one up just for testing purposes is this node.js code, that will just print whatever it receives on standard output:\r\n\r\n\tvar http = require('http');\r\n\r\n\thttp.createServer(function (req, res) {\r\n\t\tvar body = \"\";\r\n\t\treq.on('data', function (chunk) {\r\n\t\t\tbody += chunk;\r\n\t\t});\r\n\t\treq.on('end', function () {\r\n\t\t\tconsole.log(body);\r\n\t\t\tres.writeHead(200);\r\n\t\t\tres.end();\r\n\t\t});\r\n\t}).listen(8080);\r\n\r\nThat said, this is a very rough first attempt at defining how this new interface could be implemented. For instance, the available event types are limited to what came to my mind when I started writing the code, and so can definitely be extended. At the moment the kind of events (macrocategory, if you will) you can receive in an event handler are:\r\n\r\n* session related events (e.g., session created/destroyed, etc.)\r\n* handle related events (e.g., handle attached/detached, etc.)\r\n* JSEP related events (e.g., got/sent offer/answer)\r\n* WebRTC related events (e.g., PeerConnection up/down, ICE updates, DTLS updates, etc.)\r\n* media related events (e.g., media started/stopped flowing, stats on packets/bytes, etc.)\r\n* events originated by plugins (at the moment, all of them, no way to pick)\r\n* events originated by transports (at the moment, all of them, no way to pick)\r\n\r\nNot sure if we can have more, or if they can fit in there. The idea is that the core generates events of a certain macrocategory, each with different details. Event handler plugins can then subscribe to those macrocategories (none, some, all) in order to be notified. I realize this is pretty crude (you can't filter which plugins you're interested in, for instance: it's all or none), but for the sake of simplicity (handler plugins edit a mask, the core checks it before relaying) I chose to do it the way it is now for the moment.\r\n\r\nEvents are always provided as `json_t` objects: I chose to do that instead of creating a new struct for the purpose as (i) a JSON object is generic enough to cover pretty much everything, and (ii) its `json_incref`/`json_decref` mechanism allowed for a much easier management of referencing these events (the same event may be shared across multiple handlers, which may handle them at any time). Besides, it makes it easier for plugins that just want to serialize the data and shoot it somewhere (as my sample plugin does, incidentally). I tried to define a common syntax to refer to, though, and specifically:\r\n\r\n\t{\r\n\t\t\"type\" : <numeric event type identifier>,\r\n\t\t\"timestamp\" : <monotonic time of when the event was generated>,\r\n\t\t\"session_id\" : <unique session identifier>,\r\n\t\t\"handle_id\" : <unique handle identifier, if provided/available>,\r\n\t\t\"event\" : {\r\n\t\t\t<event body, custom depending on event type>\r\n\t\t}\r\n\t}\r\n\r\nThe monotonic timer is there in case event handler plugins need to know how much time passed since the event was generated and when they actually managed to handle it. You can see the exact format of the events I implemented so far by looking at the messages my sample plugin handler shoots, as explained above.\r\n\r\nAs to which events are available, I only implemented only a few of them, in this first version, and specifically:\r\n\r\n* session created/destroyed/timeout\r\n* handle attached/detached\r\n* JSEP local/remote offer/answer\r\n* ICE states, DTLS states, selected ICE pair, PeerConnection up/down\r\n* media started/stopped flowing\r\n\r\nRelevant messages that are missing are, for instance, statistics of some sort, and other will definitely come to mind, but I guess it's a good start for the moment. As a side note, for what concerns originating events in other plugins I only integrated the \"notify event\" feature in the EchoTest plugin, for the moment: specifically, any time you send the EchoTest plugin a new message, it sends the core the current internal configuration (audio/video/bitrate) as an event. Of course the plan is to use the feature for many more events that may be relevant to specific plugins, e.g., SIP calls in the SIP plugin, people joining/leaving/publishing/etc. in any of the communication plugins, and so on. This really will depend on what people will think is necessary, and what isn't.\r\n\r\nI believe that's all, for now. I don't expect it to be very optimized right now, and I haven't even started figuring out if it has any impact on performance, and how much. At the moment each event is generated as soon as the \"thing\" happens, and as such is originated by whatever thread is in control of the activity. No message bus or anything of that sort is implemented: this means that, when receiving an event, handler plugins MUST just increase the reference to the object and just enqueue it, to then process it ASAP in a thread of their own (pretty much as most of the media plugins do anyway). This should help avoid slowing Janus down, as doing any I/O or event processing in the callback itself would instead.\r\n\r\nAny comment? Feedback? Suggestion?"},"comment":{"url":"https://api.github.com/repos/meetecho/janus-gateway/issues/comments/256510861","html_url":"https://github.com/meetecho/janus-gateway/pull/536#issuecomment-256510861","issue_url":"https://api.github.com/repos/meetecho/janus-gateway/issues/536","id":256510861,"user":{"login":"joshdickson40","id":2819631,"avatar_url":"https://avatars.githubusercontent.com/u/2819631?v=3","gravatar_id":"","url":"https://api.github.com/users/joshdickson40","html_url":"https://github.com/joshdickson40","followers_url":"https://api.github.com/users/joshdickson40/followers","following_url":"https://api.github.com/users/joshdickson40/following{/other_user}","gists_url":"https://api.github.com/users/joshdickson40/gists{/gist_id}","starred_url":"https://api.github.com/users/joshdickson40/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joshdickson40/subscriptions","organizations_url":"https://api.github.com/users/joshdickson40/orgs","repos_url":"https://api.github.com/users/joshdickson40/repos","events_url":"https://api.github.com/users/joshdickson40/events{/privacy}","received_events_url":"https://api.github.com/users/joshdickson40/received_events","type":"User","site_admin":false},"created_at":"2016-10-26T23:59:30Z","updated_at":"2016-10-26T23:59:30Z","body":"It sounds like we agree on timestamp. The main issue there is that you may retroactively realize you are out of sync, but you might have already acted on what you thought was the next state change in real time. I'm sure you could build some difficult, convoluted logic to sort things out API-side and get back in state, but that's obviously not a good way to do this and would be quite error-prone.\r\n\r\nA retry happens for anything that's not a 2XX response inside the allowed request period (I'm forgetting specifics but I think you set it to 10 seconds). If you get a 404, 500, 503, 504, etc, those all trigger a retry. If the timeout period halts the request (i.e. the request is hanging for a substantial amount of time), the retry occurs.\r\n\r\nI am not 100% sure what you mean regarding broken backends / etc. My assumption here is that:\r\n- At any one point in time, Janus is sending at most one set of events to the API backend and enqueueing any other events that are ready to be sent while waiting for the result of the outbound message (I'm assuming batching here)\r\n- If the outbound message succeeds, Janus checks for any enqueued events, and if some exist, sends them immediately - if not, it stops for a short amount of time before re-checking the queue.\r\n- If the outbound message fails to see a valid 2XX response within the timeout period (10 seconds), Janus retries it as is (i.e. it does NOT rebuild the message), and continues to retry it so long as it continues to fail up to X times (~10 is probably fair... ) *Note: after each failing try, Janus should wait a short amount of time, this way if Janus sees 10 failing gateway errors in ~3 seconds, we don't necessarily conclude that the backend is down.... there should be a time component, not just retry number to this logic*\r\n- If the outbound message eventually succeeds, Janus checks for any enqueued events, and if some exist, sends them immediately - if not, it stops for a short amount of time before re-checking the queue (it might be smart here to have an upper bound on how many events we can send in each request, so that a 1 min outage is not followed by a message that is *very* large - perhaps 50-100 events/message is the batching upper bound. Obviously the number must be high enough that we know that even with all events enabled, we're able to work through a large queue)\r\n- If the outbound message never succeeds over ~10x retries over the course of ~1-2 mins, assume that the backend is down. Delete the events that we were trying to send, and empty the queue. This ensures that for very long outages or situations where files are misconfigured, we don't have a permanently increasing internal queue of pending events. At this point, Janus can wait a short amount of time and begin checking the queue, attempting to send the events, etc....\r\n\r\nBasically Janus has no idea of how \"down\" / \"broken\" the backend is. The message send logic is just centered around trying to keep the outgoing event queue at 0.\r\n"}}
{"actor":{"display_login":"joshdickson40"}}