{"action":"created","issue":{"url":"https://api.github.com/repos/snowplow/snowplow-rdb-loader/issues/93","repository_url":"https://api.github.com/repos/snowplow/snowplow-rdb-loader","labels_url":"https://api.github.com/repos/snowplow/snowplow-rdb-loader/issues/93/labels{/name}","comments_url":"https://api.github.com/repos/snowplow/snowplow-rdb-loader/issues/93/comments","events_url":"https://api.github.com/repos/snowplow/snowplow-rdb-loader/issues/93/events","html_url":"https://github.com/snowplow/snowplow-rdb-loader/issues/93","id":305197069,"number":93,"title":"RDB Shredder: consider disabling validation against JSON Schema","user":{"login":"chuwy","id":681171,"avatar_url":"https://avatars2.githubusercontent.com/u/681171?v=4","gravatar_id":"","url":"https://api.github.com/users/chuwy","html_url":"https://github.com/chuwy","followers_url":"https://api.github.com/users/chuwy/followers","following_url":"https://api.github.com/users/chuwy/following{/other_user}","gists_url":"https://api.github.com/users/chuwy/gists{/gist_id}","starred_url":"https://api.github.com/users/chuwy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chuwy/subscriptions","organizations_url":"https://api.github.com/users/chuwy/orgs","repos_url":"https://api.github.com/users/chuwy/repos","events_url":"https://api.github.com/users/chuwy/events{/privacy}","received_events_url":"https://api.github.com/users/chuwy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-03-14T15:02:17Z","updated_at":"2018-03-15T10:38:25Z","closed_at":null,"author_association":"MEMBER","body":"From my experience, enriched data assumes that raw data was not just enriched, but also validated - we never add invalid contexts/unstruct events to final enriched event.\r\n\r\nYet validation is quite compute-heavy process (not compared to distributed IO, but still). So in the end we're wasting our resources doing double-validating on already valid data.\r\n\r\nQuestion here to someone who aware of bad rows. What is the most-common type of errors in shredded/bad bucket.\r\n\r\nE.g. Snowflake Transformer does not do any validation and seems quite happy with that."},"comment":{"url":"https://api.github.com/repos/snowplow/snowplow-rdb-loader/issues/comments/373332630","html_url":"https://github.com/snowplow/snowplow-rdb-loader/issues/93#issuecomment-373332630","issue_url":"https://api.github.com/repos/snowplow/snowplow-rdb-loader/issues/93","id":373332630,"user":{"login":"BenFradet","id":1737211,"avatar_url":"https://avatars2.githubusercontent.com/u/1737211?v=4","gravatar_id":"","url":"https://api.github.com/users/BenFradet","html_url":"https://github.com/BenFradet","followers_url":"https://api.github.com/users/BenFradet/followers","following_url":"https://api.github.com/users/BenFradet/following{/other_user}","gists_url":"https://api.github.com/users/BenFradet/gists{/gist_id}","starred_url":"https://api.github.com/users/BenFradet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BenFradet/subscriptions","organizations_url":"https://api.github.com/users/BenFradet/orgs","repos_url":"https://api.github.com/users/BenFradet/repos","events_url":"https://api.github.com/users/BenFradet/events{/privacy}","received_events_url":"https://api.github.com/users/BenFradet/received_events","type":"User","site_admin":false},"created_at":"2018-03-15T10:38:25Z","updated_at":"2018-03-15T10:38:25Z","author_association":"MEMBER","body":"Isn't removing validation introducing coupling between the two?\r\n\r\nAlso, somewhat unfortunately, I don't think bypassing validation will save us a lot of time / resources."}}
{"actor":{"display_login":"BenFradet"}}