{"action":"created","issue":{"url":"https://api.github.com/repos/linkedin/gobblin/issues/833","repository_url":"https://api.github.com/repos/linkedin/gobblin","labels_url":"https://api.github.com/repos/linkedin/gobblin/issues/833/labels{/name}","comments_url":"https://api.github.com/repos/linkedin/gobblin/issues/833/comments","events_url":"https://api.github.com/repos/linkedin/gobblin/issues/833/events","html_url":"https://github.com/linkedin/gobblin/issues/833","id":140071568,"number":833,"title":"TimebasedAvroPartitioner casting issue","user":{"login":"sbjroll","id":604768,"avatar_url":"https://avatars.githubusercontent.com/u/604768?v=3","gravatar_id":"","url":"https://api.github.com/users/sbjroll","html_url":"https://github.com/sbjroll","followers_url":"https://api.github.com/users/sbjroll/followers","following_url":"https://api.github.com/users/sbjroll/following{/other_user}","gists_url":"https://api.github.com/users/sbjroll/gists{/gist_id}","starred_url":"https://api.github.com/users/sbjroll/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sbjroll/subscriptions","organizations_url":"https://api.github.com/users/sbjroll/orgs","repos_url":"https://api.github.com/users/sbjroll/repos","events_url":"https://api.github.com/users/sbjroll/events{/privacy}","received_events_url":"https://api.github.com/users/sbjroll/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/linkedin/gobblin/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-03-11T02:57:40Z","updated_at":"2016-03-15T00:11:23Z","closed_at":null,"body":"In [TimebasedAvroPartitioner.java](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/writer/partitioner/TimeBasedAvroWriterPartitioner.java)\r\n\r\n`  private long getRecordTimestamp(Optional<Object> writerPartitionColumnValue) {\r\n    return writerPartitionColumnValue.orNull() instanceof Long ? (Long) writerPartitionColumnValue.get()\r\n        : System.currentTimeMillis();\r\n  }`\r\n\r\nif the partitioning column is not instance of Long, it will return the currently timestamp.\r\n\r\nIt works generally, however, if there system failure for more then one \"partitioning time\", let's say partition by hour, then all the data from previous partitions will all goto the current partition.\r\n\r\n\r\nIn our use case, our partitioning column is type of integer. So it always picks current timestamp.\r\n\r\nI think it's better to throw the exceptions or let the developer to do work around."},"comment":{"url":"https://api.github.com/repos/linkedin/gobblin/issues/comments/196578334","html_url":"https://github.com/linkedin/gobblin/issues/833#issuecomment-196578334","issue_url":"https://api.github.com/repos/linkedin/gobblin/issues/833","id":196578334,"user":{"login":"zliu41","id":6342538,"avatar_url":"https://avatars.githubusercontent.com/u/6342538?v=3","gravatar_id":"","url":"https://api.github.com/users/zliu41","html_url":"https://github.com/zliu41","followers_url":"https://api.github.com/users/zliu41/followers","following_url":"https://api.github.com/users/zliu41/following{/other_user}","gists_url":"https://api.github.com/users/zliu41/gists{/gist_id}","starred_url":"https://api.github.com/users/zliu41/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zliu41/subscriptions","organizations_url":"https://api.github.com/users/zliu41/orgs","repos_url":"https://api.github.com/users/zliu41/repos","events_url":"https://api.github.com/users/zliu41/events{/privacy}","received_events_url":"https://api.github.com/users/zliu41/received_events","type":"User","site_admin":false},"created_at":"2016-03-15T00:11:23Z","updated_at":"2016-03-15T00:11:23Z","body":"@sahilTakiar how about moving `long getRecordTimestamp(Optional<Object> writerPartitionColumnValue)` from `TimeBasedAvroWriterPartitioner` to the parent class `TimeBasedWriterPartitioner` and make it protected, so that one can override it and do whatever he wants, as long as it returns a `long`."}}