{"action":"created","issue":{"url":"https://api.github.com/repos/matomo-org/matomo/issues/12622","repository_url":"https://api.github.com/repos/matomo-org/matomo","labels_url":"https://api.github.com/repos/matomo-org/matomo/issues/12622/labels{/name}","comments_url":"https://api.github.com/repos/matomo-org/matomo/issues/12622/comments","events_url":"https://api.github.com/repos/matomo-org/matomo/issues/12622/events","html_url":"https://github.com/matomo-org/matomo/issues/12622","id":305914022,"number":12622,"title":"log import: avoid importing duplicates ","user":{"login":"sebalis","id":8028279,"avatar_url":"https://avatars1.githubusercontent.com/u/8028279?v=4","gravatar_id":"","url":"https://api.github.com/users/sebalis","html_url":"https://github.com/sebalis","followers_url":"https://api.github.com/users/sebalis/followers","following_url":"https://api.github.com/users/sebalis/following{/other_user}","gists_url":"https://api.github.com/users/sebalis/gists{/gist_id}","starred_url":"https://api.github.com/users/sebalis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sebalis/subscriptions","organizations_url":"https://api.github.com/users/sebalis/orgs","repos_url":"https://api.github.com/users/sebalis/repos","events_url":"https://api.github.com/users/sebalis/events{/privacy}","received_events_url":"https://api.github.com/users/sebalis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-03-16T12:43:57Z","updated_at":"2018-03-18T22:35:01Z","closed_at":null,"author_association":"NONE","body":"If the import script is called on the same logfile more than once, entries imported in the first run and still present in the file during the second run are imported twice (at least when I tested it on Apache logs). This creates a few problems: running the script has to be tied to the web server rotating the file, and new entries can not be seen in Matomo until the log has been rotated and the importer run again.\r\n\r\nIt would be much better if it were possible to run the importer frequently, maybe even every minute. In order to do this, the importer needs to be able to know which entries have already been imported. I don’t know how to implement this in detail. I suppose it should be reasonable to expect that the log entries have time data with one-second granularity. So by remembering the time of the latest entry already imported and perhaps adding some logic for re-identifying entries from that last second, it would become feasible to run the importer as often as one would like without messing up the data. What do you think? For me this might make the difference between choosing Matomo or some other log analysis software."},"comment":{"url":"https://api.github.com/repos/matomo-org/matomo/issues/comments/374059356","html_url":"https://github.com/matomo-org/matomo/issues/12622#issuecomment-374059356","issue_url":"https://api.github.com/repos/matomo-org/matomo/issues/12622","id":374059356,"user":{"login":"sebalis","id":8028279,"avatar_url":"https://avatars1.githubusercontent.com/u/8028279?v=4","gravatar_id":"","url":"https://api.github.com/users/sebalis","html_url":"https://github.com/sebalis","followers_url":"https://api.github.com/users/sebalis/followers","following_url":"https://api.github.com/users/sebalis/following{/other_user}","gists_url":"https://api.github.com/users/sebalis/gists{/gist_id}","starred_url":"https://api.github.com/users/sebalis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sebalis/subscriptions","organizations_url":"https://api.github.com/users/sebalis/orgs","repos_url":"https://api.github.com/users/sebalis/repos","events_url":"https://api.github.com/users/sebalis/events{/privacy}","received_events_url":"https://api.github.com/users/sebalis/received_events","type":"User","site_admin":false},"created_at":"2018-03-18T22:35:01Z","updated_at":"2018-03-18T22:35:01Z","author_association":"NONE","body":"Matomo looks very good, I would like to use it – although for privacy reasons I will restrict myself to the log importer, which reduces the difference to other products. Also this makes it all the more important to get as much accuracy and timeliness out of the importer as possible."}}
{"actor":{"display_login":"sebalis"}}