{"action":"created","issue":{"url":"https://api.github.com/repos/eel-sdk/eel/issues/138","repository_url":"https://api.github.com/repos/eel-sdk/eel","labels_url":"https://api.github.com/repos/eel-sdk/eel/issues/138/labels{/name}","comments_url":"https://api.github.com/repos/eel-sdk/eel/issues/138/comments","events_url":"https://api.github.com/repos/eel-sdk/eel/issues/138/events","html_url":"https://github.com/eel-sdk/eel/issues/138","id":140315407,"number":138,"title":"Collect Parquet statistics (including Hive DBs)","user":{"login":"Antwnis","id":3376244,"avatar_url":"https://avatars.githubusercontent.com/u/3376244?v=3","gravatar_id":"","url":"https://api.github.com/users/Antwnis","html_url":"https://github.com/Antwnis","followers_url":"https://api.github.com/users/Antwnis/followers","following_url":"https://api.github.com/users/Antwnis/following{/other_user}","gists_url":"https://api.github.com/users/Antwnis/gists{/gist_id}","starred_url":"https://api.github.com/users/Antwnis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Antwnis/subscriptions","organizations_url":"https://api.github.com/users/Antwnis/orgs","repos_url":"https://api.github.com/users/Antwnis/repos","events_url":"https://api.github.com/users/Antwnis/events{/privacy}","received_events_url":"https://api.github.com/users/Antwnis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-03-11T23:12:37Z","updated_at":"2016-03-15T23:51:50Z","closed_at":null,"body":"If not using `Impala` running `select count (*) from xx_table` could require a coffee break to get back results. Even Spark is super slow !\r\n\r\nWith `eel` we could easily check either a file / or check recursive an entire folder, or an entire hive DB and tables - and if files are Parquet - read super-efficiently and with multiple threads the `stats` from the Parquet footers (!)\r\n\r\nIf doing that we could easily create a flow\r\n\r\ncollect-hive-stats ~> etl-ingestion ~> collect-hive-stats ~> display diff of stats !    "},"comment":{"url":"https://api.github.com/repos/eel-sdk/eel/issues/comments/197071839","html_url":"https://github.com/eel-sdk/eel/issues/138#issuecomment-197071839","issue_url":"https://api.github.com/repos/eel-sdk/eel/issues/138","id":197071839,"user":{"login":"Antwnis","id":3376244,"avatar_url":"https://avatars.githubusercontent.com/u/3376244?v=3","gravatar_id":"","url":"https://api.github.com/users/Antwnis","html_url":"https://github.com/Antwnis","followers_url":"https://api.github.com/users/Antwnis/followers","following_url":"https://api.github.com/users/Antwnis/following{/other_user}","gists_url":"https://api.github.com/users/Antwnis/gists{/gist_id}","starred_url":"https://api.github.com/users/Antwnis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Antwnis/subscriptions","organizations_url":"https://api.github.com/users/Antwnis/orgs","repos_url":"https://api.github.com/users/Antwnis/repos","events_url":"https://api.github.com/users/Antwnis/events{/privacy}","received_events_url":"https://api.github.com/users/Antwnis/received_events","type":"User","site_admin":false},"created_at":"2016-03-15T23:51:50Z","updated_at":"2016-03-15T23:51:50Z","body":"High level and stats - if run against a table - num of partitions - rows per partitions \r\nif run against a db - stats per table\r\n\r\nin any case summarize - aggregate per table / db to keep track of totals"}}