{"action":"created","issue":{"url":"https://api.github.com/repos/Normation/rudder/issues/1969","repository_url":"https://api.github.com/repos/Normation/rudder","labels_url":"https://api.github.com/repos/Normation/rudder/issues/1969/labels{/name}","comments_url":"https://api.github.com/repos/Normation/rudder/issues/1969/comments","events_url":"https://api.github.com/repos/Normation/rudder/issues/1969/events","html_url":"https://github.com/Normation/rudder/pull/1969","id":332406781,"node_id":"MDExOlB1bGxSZXF1ZXN0MTk0ODk2Mzc4","number":1969,"title":"Fixes #12719: Some reports are duplicated between agent and postgres leading to \"unexpected\" compliance ","user":{"login":"fanf","id":44649,"node_id":"MDQ6VXNlcjQ0NjQ5","avatar_url":"https://avatars1.githubusercontent.com/u/44649?v=4","gravatar_id":"","url":"https://api.github.com/users/fanf","html_url":"https://github.com/fanf","followers_url":"https://api.github.com/users/fanf/followers","following_url":"https://api.github.com/users/fanf/following{/other_user}","gists_url":"https://api.github.com/users/fanf/gists{/gist_id}","starred_url":"https://api.github.com/users/fanf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fanf/subscriptions","organizations_url":"https://api.github.com/users/fanf/orgs","repos_url":"https://api.github.com/users/fanf/repos","events_url":"https://api.github.com/users/fanf/events{/privacy}","received_events_url":"https://api.github.com/users/fanf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-06-14T13:37:23Z","updated_at":"2018-06-19T07:15:58Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/Normation/rudder/pulls/1969","html_url":"https://github.com/Normation/rudder/pull/1969","diff_url":"https://github.com/Normation/rudder/pull/1969.diff","patch_url":"https://github.com/Normation/rudder/pull/1969.patch"},"body":"https://www.rudder-project.org/redmine/issues/12719\r\n\r\nNot yet finished (remains the we selection part in Setting). But it's a fairly big change. \r\nI'm quite confident on it, because we have a very good test coverage on that part. \r\n\r\nSo, basically, I completly changed the way we pair report <-> expected component value. This was needed to allows to be more lenient in case of ducplicate keys and unbounded variable size (from iterators). \r\n\r\nThe main modification are: \r\n- we have a parameter set to keep what is the unexpected interpretation: UnexpectedReportInterpretation\r\n- we don't sort anymore the kind of component values in 3 cases (None, simple value, var pattern). In place of that, we create for each component value a Value utility class with the following info: \r\n  -  value / unexpanded value: the two component value\r\n  -  cardinality: number of expected reports, most of the time '1' but can be increased for out of bound vars (with iterators)\r\n  -  numberDuplicates: the number of dropped duplicated message for that pairing, so that we can know how bad syslog is. Should be 0. This is used to warn looder and looder the user when we have more duplicates\r\n  -  isVar: a boolean telling us if the expected reports contains a variable pattern. \r\n  -  pattern: the value as a pattern. This is defined in *all* cases, even for None and simple value - they just have a very specific pattern that only match the corresponding exact value. This is the main difference compared to before, and it allows to process ALL value with the same algo. \r\n  - specificity: how specific the pattern is. That allows to start with more specific patterns and avoid bug https://www.rudder-project.org/redmine/issues/7758. For ex, `.*` is 0 (not specific at all), `foobarbaz` is 9.\r\n  - matchingReports : the list of reports whose keyValue match pattern. Most of the time, we only have one. More lead to unexpected (modulo unexpected special interpretations). \r\n\r\nWith that, and as introduced in \"pattern\" description, we are able to process all reports of a component in only one pass with the `recPairReports` algorithm. \r\nThe mains ideas of the algo are: \r\n\r\n- we sort expected values and reports by specificy to avoid bug 7758, \r\n- we have two stack of values: the one not yet paired (all values at begining), and the one already paired (empty at begining)\r\n- then, for each report we look for the first not-yet-paired value whose pattern accept report keyValue. \r\n- if non not-yet-paired value matches, we try the same thing with already paired (but that will lead to an unexpected modulo interpretation settings)\r\n- if nothing matches, it's a real unexpected reports. \r\n\r\nOnce all reports are processed, we merge the two stack of values, and we transform them in components value status (the still free value lead to missing, the one with one report get the report status, etc). And THAT'S ALL. \r\n\r\nThere's a subpart of the algo which is in charge of finding the matching value for the current report: `recPairReports`. It's almost just a loop on all values to consider, with some special cases about what to do in case of a duplicate key or an out of bound var and Rudder settings. \r\n"},"comment":{"url":"https://api.github.com/repos/Normation/rudder/issues/comments/398299057","html_url":"https://github.com/Normation/rudder/pull/1969#issuecomment-398299057","issue_url":"https://api.github.com/repos/Normation/rudder/issues/1969","id":398299057,"node_id":"MDEyOklzc3VlQ29tbWVudDM5ODI5OTA1Nw==","user":{"login":"fanf","id":44649,"node_id":"MDQ6VXNlcjQ0NjQ5","avatar_url":"https://avatars1.githubusercontent.com/u/44649?v=4","gravatar_id":"","url":"https://api.github.com/users/fanf","html_url":"https://github.com/fanf","followers_url":"https://api.github.com/users/fanf/followers","following_url":"https://api.github.com/users/fanf/following{/other_user}","gists_url":"https://api.github.com/users/fanf/gists{/gist_id}","starred_url":"https://api.github.com/users/fanf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fanf/subscriptions","organizations_url":"https://api.github.com/users/fanf/orgs","repos_url":"https://api.github.com/users/fanf/repos","events_url":"https://api.github.com/users/fanf/events{/privacy}","received_events_url":"https://api.github.com/users/fanf/received_events","type":"User","site_admin":false},"created_at":"2018-06-19T07:15:58Z","updated_at":"2018-06-19T07:15:58Z","author_association":"MEMBER","body":"PR rebased"}}
{"actor":{"display_login":"fanf"}}