{"action":"created","issue":{"url":"https://api.github.com/repos/tuplejump/FiloDB/issues/92","repository_url":"https://api.github.com/repos/tuplejump/FiloDB","labels_url":"https://api.github.com/repos/tuplejump/FiloDB/issues/92/labels{/name}","comments_url":"https://api.github.com/repos/tuplejump/FiloDB/issues/92/comments","events_url":"https://api.github.com/repos/tuplejump/FiloDB/issues/92/events","html_url":"https://github.com/tuplejump/FiloDB/issues/92","id":149897918,"number":92,"title":"Lost rows during the same data ingesting into FiloDB","user":{"login":"java8964","id":1430098,"avatar_url":"https://avatars.githubusercontent.com/u/1430098?v=3","gravatar_id":"","url":"https://api.github.com/users/java8964","html_url":"https://github.com/java8964","followers_url":"https://api.github.com/users/java8964/followers","following_url":"https://api.github.com/users/java8964/following{/other_user}","gists_url":"https://api.github.com/users/java8964/gists{/gist_id}","starred_url":"https://api.github.com/users/java8964/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/java8964/subscriptions","organizations_url":"https://api.github.com/users/java8964/orgs","repos_url":"https://api.github.com/users/java8964/repos","events_url":"https://api.github.com/users/java8964/events{/privacy}","received_events_url":"https://api.github.com/users/java8964/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-04-20T21:57:06Z","updated_at":"2016-04-27T20:53:14Z","closed_at":null,"body":"I am testing FiloDB with latest trunk, include PR-91, and  can produce this problem that I saw the filodb lost raw after the data ingested. Below is the test case I run, and just want to make sure that it is a bug, not something I misunderstand how filodb is supposed to work.\r\n\r\nI have this 1m records of unique emails, with domain. I will use the first 2 characters as the partition key, and use the domain as the segment key, and unique email address itself as the row key. I tested with in memory.\r\n1) After I save the data from spark into filoDB, if I load the data back in one DF, the count is 1M.\r\n2) Then I save the exactly same data from spark into filoDB again in the same spark shell, and load the data back in one DF, the count is less than 1M.\r\n\r\nI enable the log at debug level for filoDB package, and the full log is attached.\r\nFrom the log, you can see what is my steps, and the log output from filoDB, and the 2nd time, the count is not correct. \r\n\r\nI will also dig the log more, as want to understand more internal of filoDB. Hope the log is helpful to identify the root cause of this case.\r\n\r\nThanks\r\nYong\r\n\r\n[filodb-lost-update-bug.log.gz](https://github.com/tuplejump/FiloDB/files/228873/filodb-lost-update-bug.log.gz)\r\n"},"comment":{"url":"https://api.github.com/repos/tuplejump/FiloDB/issues/comments/215225723","html_url":"https://github.com/tuplejump/FiloDB/issues/92#issuecomment-215225723","issue_url":"https://api.github.com/repos/tuplejump/FiloDB/issues/92","id":215225723,"user":{"login":"velvia","id":1062875,"avatar_url":"https://avatars.githubusercontent.com/u/1062875?v=3","gravatar_id":"","url":"https://api.github.com/users/velvia","html_url":"https://github.com/velvia","followers_url":"https://api.github.com/users/velvia/followers","following_url":"https://api.github.com/users/velvia/following{/other_user}","gists_url":"https://api.github.com/users/velvia/gists{/gist_id}","starred_url":"https://api.github.com/users/velvia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/velvia/subscriptions","organizations_url":"https://api.github.com/users/velvia/orgs","repos_url":"https://api.github.com/users/velvia/repos","events_url":"https://api.github.com/users/velvia/events{/privacy}","received_events_url":"https://api.github.com/users/velvia/received_events","type":"User","site_admin":false},"created_at":"2016-04-27T20:53:14Z","updated_at":"2016-04-27T20:53:14Z","body":"@java8964 I have some changes in a new branch, which adds \"clustering\" support.\r\nvelvia/clustering-support-90\r\n\r\nIf you run in local / in-memory store, like your test in the other thread, you should find that it consistently returns 1 million rows now.   Basically one change is that it makes sure to flush any remaining rows it finds in the memtable at the end of ingestion.\r\n\r\nIf you test code in this branch on a cluster, be ware that it will require an extra port to be opened.  Right now the port assignment is random, and I'll need to add config to make sure it's not random.\r\n\r\nIt will be possible to connect in the future remotely to this port and get monitoring/stats out."}}