{"action":"created","issue":{"url":"https://api.github.com/repos/mcollina/pino/issues/99","repository_url":"https://api.github.com/repos/mcollina/pino","labels_url":"https://api.github.com/repos/mcollina/pino/issues/99/labels{/name}","comments_url":"https://api.github.com/repos/mcollina/pino/issues/99/comments","events_url":"https://api.github.com/repos/mcollina/pino/issues/99/events","html_url":"https://github.com/mcollina/pino/pull/99","id":180119826,"number":99,"title":"Support log message filtering","user":{"login":"bripkens","id":596443,"avatar_url":"https://avatars.githubusercontent.com/u/596443?v=3","gravatar_id":"","url":"https://api.github.com/users/bripkens","html_url":"https://github.com/bripkens","followers_url":"https://api.github.com/users/bripkens/followers","following_url":"https://api.github.com/users/bripkens/following{/other_user}","gists_url":"https://api.github.com/users/bripkens/gists{/gist_id}","starred_url":"https://api.github.com/users/bripkens/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bripkens/subscriptions","organizations_url":"https://api.github.com/users/bripkens/orgs","repos_url":"https://api.github.com/users/bripkens/repos","events_url":"https://api.github.com/users/bripkens/events{/privacy}","received_events_url":"https://api.github.com/users/bripkens/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-09-29T18:01:34Z","updated_at":"2016-09-29T18:25:36Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/mcollina/pino/pulls/99","html_url":"https://github.com/mcollina/pino/pull/99","diff_url":"https://github.com/mcollina/pino/pull/99.diff","patch_url":"https://github.com/mcollina/pino/pull/99.patch"},"body":"Sometimes, especially when processing a large number of events /\r\nrequests, applications can go wild and start logging the same stuff\r\nover and over again. It is desirable to filter messages in these\r\ncases, so as to avoid filling up hard drives or to avoid exceeding\r\nlog aggregation limits. In other cases, sensitive information\r\ncould be filtered out.\r\n\r\nThis commit adds basic facilities for log message filtering. These\r\nallow the registration of a log message filter which is applied\r\nbefore (potentially) expensive formatting and serialization\r\ntasks are executed. This is similar to other logging libraries\r\nsuch as logback in the Java world.\r\n\r\nIn the end, I'd like to use this as the foundation for a duplicate log\r\nmessage filter similar to one available in logback:\r\n\r\nhttp://logback.qos.ch/manual/filters.html#DuplicateMessageFilter"},"comment":{"url":"https://api.github.com/repos/mcollina/pino/issues/comments/250550515","html_url":"https://github.com/mcollina/pino/pull/99#issuecomment-250550515","issue_url":"https://api.github.com/repos/mcollina/pino/issues/99","id":250550515,"user":{"login":"bripkens","id":596443,"avatar_url":"https://avatars.githubusercontent.com/u/596443?v=3","gravatar_id":"","url":"https://api.github.com/users/bripkens","html_url":"https://github.com/bripkens","followers_url":"https://api.github.com/users/bripkens/followers","following_url":"https://api.github.com/users/bripkens/following{/other_user}","gists_url":"https://api.github.com/users/bripkens/gists{/gist_id}","starred_url":"https://api.github.com/users/bripkens/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bripkens/subscriptions","organizations_url":"https://api.github.com/users/bripkens/orgs","repos_url":"https://api.github.com/users/bripkens/repos","events_url":"https://api.github.com/users/bripkens/events{/privacy}","received_events_url":"https://api.github.com/users/bripkens/received_events","type":"User","site_admin":false},"created_at":"2016-09-29T18:25:36Z","updated_at":"2016-09-29T18:25:36Z","body":"> However, have you benchmarked your changes relative to the master branch?\r\n\r\nWithout any filtering active:\r\n\r\n```\r\n$ git checkout master\r\n$ node benchmarks/basic.js\r\nbenchBunyan*10000: 1493.528ms\r\nbenchWinston*10000: 2211.598ms\r\nbenchBole*10000: 301.056ms\r\nbenchDebug*10000: 442.542ms\r\nbenchLogLevel*10000: 322.385ms\r\nbenchPino*10000: 277.271ms\r\nbenchPinoExreme*10000: 105.704ms\r\nbenchBunyan*10000: 1499.082ms\r\nbenchWinston*10000: 2215.568ms\r\nbenchBole*10000: 281.044ms\r\nbenchDebug*10000: 432.720ms\r\nbenchLogLevel*10000: 308.749ms\r\nbenchPino*10000: 262.547ms\r\nbenchPinoExreme*10000: 106.387ms\r\n\r\n$ git checkout filtering\r\n$ node benchmarks/basic.js\r\nbenchBunyan*10000: 1488.006ms\r\nbenchWinston*10000: 2245.172ms\r\nbenchBole*10000: 298.442ms\r\nbenchDebug*10000: 442.922ms\r\nbenchLogLevel*10000: 326.200ms\r\nbenchPino*10000: 288.153ms\r\nbenchPinoExreme*10000: 110.538ms\r\nbenchBunyan*10000: 1445.158ms\r\nbenchWinston*10000: 2176.479ms\r\nbenchBole*10000: 273.919ms\r\nbenchDebug*10000: 420.733ms\r\nbenchLogLevel*10000: 304.727ms\r\nbenchPino*10000: 266.270ms\r\nbenchPinoExreme*10000: 107.102ms\r\n```\r\n\r\n> My first inclination on this is that it should be handled separately\r\n\r\nIn theory, and as far as SOC is concerned, I absolutely agree with you. Unfortunately doing it further downstream has two major drawbacks:\r\n\r\n 1. Expensive serialization and formatting logic gets executed unnecessarily.\r\n 2. Log message analysis becomes harder as the raw log message is no longer accessible. This makes it expensive to develop log message similarity based approach for filtering."}}
{"actor":{"display_login":"bripkens"}}