{"action":"created","issue":{"url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/1134","repository_url":"https://api.github.com/repos/OpenTSDB/opentsdb","labels_url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/1134/labels{/name}","comments_url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/1134/comments","events_url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/1134/events","html_url":"https://github.com/OpenTSDB/opentsdb/issues/1134","id":290894489,"node_id":"MDU6SXNzdWUyOTA4OTQ0ODk=","number":1134,"title":"Fsck finding negative timestamps, crashes without fixing the issue","user":{"login":"heaje","id":3622298,"node_id":"MDQ6VXNlcjM2MjIyOTg=","avatar_url":"https://avatars2.githubusercontent.com/u/3622298?v=4","gravatar_id":"","url":"https://api.github.com/users/heaje","html_url":"https://github.com/heaje","followers_url":"https://api.github.com/users/heaje/followers","following_url":"https://api.github.com/users/heaje/following{/other_user}","gists_url":"https://api.github.com/users/heaje/gists{/gist_id}","starred_url":"https://api.github.com/users/heaje/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/heaje/subscriptions","organizations_url":"https://api.github.com/users/heaje/orgs","repos_url":"https://api.github.com/users/heaje/repos","events_url":"https://api.github.com/users/heaje/events{/privacy}","received_events_url":"https://api.github.com/users/heaje/received_events","type":"User","site_admin":false},"labels":[{"id":70105,"node_id":"MDU6TGFiZWw3MDEwNQ==","url":"https://api.github.com/repos/OpenTSDB/opentsdb/labels/bug","name":"bug","color":"ff0000","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/OpenTSDB/opentsdb/milestones/4","html_url":"https://github.com/OpenTSDB/opentsdb/milestone/4","labels_url":"https://api.github.com/repos/OpenTSDB/opentsdb/milestones/4/labels","id":1654994,"node_id":"MDk6TWlsZXN0b25lMTY1NDk5NA==","number":4,"title":"v2.4.0","description":"","creator":{"login":"johann8384","id":154673,"node_id":"MDQ6VXNlcjE1NDY3Mw==","avatar_url":"https://avatars2.githubusercontent.com/u/154673?v=4","gravatar_id":"","url":"https://api.github.com/users/johann8384","html_url":"https://github.com/johann8384","followers_url":"https://api.github.com/users/johann8384/followers","following_url":"https://api.github.com/users/johann8384/following{/other_user}","gists_url":"https://api.github.com/users/johann8384/gists{/gist_id}","starred_url":"https://api.github.com/users/johann8384/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johann8384/subscriptions","organizations_url":"https://api.github.com/users/johann8384/orgs","repos_url":"https://api.github.com/users/johann8384/repos","events_url":"https://api.github.com/users/johann8384/events{/privacy}","received_events_url":"https://api.github.com/users/johann8384/received_events","type":"User","site_admin":false},"open_issues":32,"closed_issues":11,"state":"open","created_at":"2016-03-19T15:29:45Z","updated_at":"2018-12-11T23:37:57Z","due_on":"2017-11-30T08:00:00Z","closed_at":null},"comments":1,"created_at":"2018-01-23T16:06:52Z","updated_at":"2018-12-11T23:38:01Z","closed_at":null,"author_association":"NONE","body":"I don't know how this occurred, but when I attempt to run a fsck with any of the flags enabled, I get errors like this:\r\n\r\n<pre>2018-01-23 07:48:54,438 ERROR [Fsck #14] Fsck: Shouldn't be here\r\norg.hbase.async.RemoteException: java.io.IOException: Timestamp cannot be negative. ts=-9223372036854775808\r\n        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2236)\r\n        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)\r\n        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)\r\n        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)\r\nCaused by: java.lang.IllegalArgumentException: Timestamp cannot be negative. ts=-9223372036854775808\r\n        at org.apache.hadoop.hbase.client.Put.addImmutable(Put.java:306)\r\n        at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toPut(ProtobufUtil.java:619)\r\n        at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2213)\r\n        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33646)\r\n        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)\r\n        ... 3 more\r\n        at org.hbase.async.RegionClient.makeException(RegionClient.java:1758) ~[asynchbase-1.8.0.jar:na]\r\n        at org.hbase.async.RegionClient.decodeException(RegionClient.java:1776) ~[asynchbase-1.8.0.jar:na]\r\n        at org.hbase.async.RegionClient.decode(RegionClient.java:1488) ~[asynchbase-1.8.0.jar:na]\r\n        at org.hbase.async.RegionClient.decode(RegionClient.java:88) ~[asynchbase-1.8.0.jar:na]\r\n        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na]\r\n        at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1226) ~[asynchbase-1.8.0.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2236)\r\n        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)\r\n        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)\r\n        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)\r\n        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2236)\r\n        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)\r\n        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)\r\n        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)\r\nCaused by: java.lang.IllegalArgumentException: Timestamp cannot be negative. ts=-9223372036854775808\r\n        at org.apache.hadoop.hbase.client.Put.addImmutable(Put.java:306)\r\n        at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toPut(ProtobufUtil.java:619)\r\n        at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2213)\r\n        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33646)\r\n        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)\r\n        ... 3 more\r\n        at org.hbase.async.RegionClient.makeException(RegionClient.java:1758) ~[asynchbase-1.8.0.jar:na]\r\n        at org.hbase.async.RegionClient.decodeException(RegionClient.java:1776) ~[asynchbase-1.8.0.jar:na]\r\n        at org.hbase.async.RegionClient.decode(RegionClient.java:1488) ~[asynchbase-1.8.0.jar:na]\r\n        at org.hbase.async.RegionClient.decode(RegionClient.java:88) ~[asynchbase-1.8.0.jar:na]\r\n        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na]\r\n        at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1226) ~[asynchbase-1.8.0.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[netty-3.9.4.Final.jar:na]\r\n        at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3678) ~[asynchbase-1.8.0.jar:na]\r\n        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[netty-3.9.4.Final.jar:na]\r\n        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[netty-3.9.4.Final.jar:na]</pre>\r\n\r\nI am currently running OpenTSDB 2.4.0 RC2 using Oracle JDK 1.8u152.  I have run fsck like this:\r\n`tsdb fsck --config=</path/to/config.conf> --fix-all --full-scan`\r\n\r\nI have found that the problem only occurs if the \"--fix\" or \"--fix-all\" flags are provided."},"comment":{"url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/comments/446404639","html_url":"https://github.com/OpenTSDB/opentsdb/issues/1134#issuecomment-446404639","issue_url":"https://api.github.com/repos/OpenTSDB/opentsdb/issues/1134","id":446404639,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NjQwNDYzOQ==","user":{"login":"manolama","id":1377700,"node_id":"MDQ6VXNlcjEzNzc3MDA=","avatar_url":"https://avatars2.githubusercontent.com/u/1377700?v=4","gravatar_id":"","url":"https://api.github.com/users/manolama","html_url":"https://github.com/manolama","followers_url":"https://api.github.com/users/manolama/followers","following_url":"https://api.github.com/users/manolama/following{/other_user}","gists_url":"https://api.github.com/users/manolama/gists{/gist_id}","starred_url":"https://api.github.com/users/manolama/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manolama/subscriptions","organizations_url":"https://api.github.com/users/manolama/orgs","repos_url":"https://api.github.com/users/manolama/repos","events_url":"https://api.github.com/users/manolama/events{/privacy}","received_events_url":"https://api.github.com/users/manolama/received_events","type":"User","site_admin":false},"created_at":"2018-12-11T23:38:01Z","updated_at":"2018-12-11T23:38:01Z","author_association":"MEMBER","body":"Yeah that's a bad rowkey or value so we'll need to catch and drop it."}}
{"actor":{"display_login":"manolama"}}