{"action":"reopened","issue":{"url":"https://api.github.com/repos/orientechnologies/orientdb/issues/6434","repository_url":"https://api.github.com/repos/orientechnologies/orientdb","labels_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/6434/labels{/name}","comments_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/6434/comments","events_url":"https://api.github.com/repos/orientechnologies/orientdb/issues/6434/events","html_url":"https://github.com/orientechnologies/orientdb/issues/6434","id":166308832,"number":6434,"title":"cannot rebuild unique index, but cannot find offending duplicate records","user":{"login":"rdelangh","id":15123823,"avatar_url":"https://avatars.githubusercontent.com/u/15123823?v=3","gravatar_id":"","url":"https://api.github.com/users/rdelangh","html_url":"https://github.com/rdelangh","followers_url":"https://api.github.com/users/rdelangh/followers","following_url":"https://api.github.com/users/rdelangh/following{/other_user}","gists_url":"https://api.github.com/users/rdelangh/gists{/gist_id}","starred_url":"https://api.github.com/users/rdelangh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rdelangh/subscriptions","organizations_url":"https://api.github.com/users/rdelangh/orgs","repos_url":"https://api.github.com/users/rdelangh/repos","events_url":"https://api.github.com/users/rdelangh/events{/privacy}","received_events_url":"https://api.github.com/users/rdelangh/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/orientechnologies/orientdb/labels/storage%20team","name":"storage team","color":"c5def5"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/orientechnologies/orientdb/milestones/88","html_url":"https://github.com/orientechnologies/orientdb/milestones/2.2.x%20(next%20hotfix)","labels_url":"https://api.github.com/repos/orientechnologies/orientdb/milestones/88/labels","id":1768027,"number":88,"title":"2.2.x (next hotfix)","description":"","creator":{"login":"robfrank","id":413587,"avatar_url":"https://avatars.githubusercontent.com/u/413587?v=3","gravatar_id":"","url":"https://api.github.com/users/robfrank","html_url":"https://github.com/robfrank","followers_url":"https://api.github.com/users/robfrank/followers","following_url":"https://api.github.com/users/robfrank/following{/other_user}","gists_url":"https://api.github.com/users/robfrank/gists{/gist_id}","starred_url":"https://api.github.com/users/robfrank/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robfrank/subscriptions","organizations_url":"https://api.github.com/users/robfrank/orgs","repos_url":"https://api.github.com/users/robfrank/repos","events_url":"https://api.github.com/users/robfrank/events{/privacy}","received_events_url":"https://api.github.com/users/robfrank/received_events","type":"User","site_admin":false},"open_issues":162,"closed_issues":0,"state":"open","created_at":"2016-05-16T08:28:14Z","updated_at":"2016-07-19T13:55:56Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2016-07-19T11:30:00Z","updated_at":"2016-07-19T13:55:56Z","closed_at":null,"body":"Hi,\r\n\r\nrunning the standard version of OrientDB v2.2.3 currently on a single Ubuntu 14.04 LTS server with 128GB installed RAM, I have a database \"cdrarch\", with class \"@mds_csv_cf14\" with all STRING properties except one DATETIME property which is named \"START_DATETIME\". \r\n\r\nInitially (no records in this class yet) I defined a unique index \"idx_mds_csv_cf14_0\" on the combination of [ START_DATETIME, FILENAME, RECORD_NUMBER ] :\r\nCREATE INDEX idx_mds_csv_cf14_0 ON mds_csv_cf14 (START_DATETIME, FILENAME, RECORD_NUMBER) UNIQUE\r\n\r\nSo far so good, then I started loading records into the class during 2 weeks. Currently there is about 70M records in this table.\r\n\r\nFor some reasons, the Java process of the OrientDB was leaking (?) or at least growing terribly in consumed memory (see below log extract), so that it finally became inaccessible from a data-loading client program (via REST-api), console, or web interface. \r\n\r\n2016-07-18 09:50:02:610 INFO  Database 'OSystem' uses 101,525MB/119,482MB of DISKCACHE memory, while Heap is not completely used (usedHeap=272\r\n6MB maxHeap=7282MB). To improve performance set maxHeap to 3271MB and DISKCACHE to 123492MB [OAbstractProfiler$MemoryChecker]\r\n2016-07-18 09:50:02:611 INFO  -> Open server.sh (or server.bat on Windows) and change the following variables: 1) MAXHEAP=-Xmx3271M 2) MAXDISK\r\nCACHE=123492 [OAbstractProfiler$MemoryChecker]\r\n2016-07-18 09:50:02:612 INFO  Database 'cdrarch' uses 101,525MB/119,482MB of DISKCACHE memory, while Heap is not completely used (usedHeap=272\r\n6MB maxHeap=7282MB). To improve performance set maxHeap to 3271MB and DISKCACHE to 123492MB [OAbstractProfiler$MemoryChecker]\r\n2016-07-18 09:50:02:612 INFO  -> Open server.sh (or server.bat on Windows) and change the following variables: 1) MAXHEAP=-Xmx3271M 2) MAXDISK\r\nCACHE=123492 [OAbstractProfiler$MemoryChecker]\r\n\r\nHad to stop the server process to unblock it.\r\n\r\nI found that the \"server.sh\" script did not have any explicit setting for \"MAXDISKCACHE\", which -according to certain articles on the web- might have led to this growing Java process.\r\n\r\nSo I gave that parameter the following value:\r\nMAXDISKCACHE=\"-Dstorage.diskCache.bufferSize=8192\"\r\nthen restarted the server program.\r\n\r\nThen I restarted the data-loading program which is attempting to insert new records into \"@mds_csv_cf14\".\r\n\r\nBut somehow the unique index got corrupted, because loading new records into this class fails with the message\r\n\r\n2016-07-19 12:07:07:160 SEVER Internal server error:\r\ncom.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #218:1248232: found duplicated key 'OCompositeKey{keys=[Thu Jun 30 23:52:57 CEST 2016, C22N010000000014.001.00665346, 1]}' in index 'idx_mds_csv_cf14_0' previously assigned to the record #214:1248231\r\n\r\nWhen I search for duplicate records in the class for the combination of the unique index, i.e. [ START_DATETIME, FILENAME, RECORD_NUMBER ], around the timestamp indicated in the log message (Jun 30 23:52:57 CEST 2016), there aren't any : \r\n\r\nSELECT * FROM (\r\nSELECT START_DATETIME, FILENAME, RECORD_NUMBER, count(*) AS cnt FROM mds_csv_cf14 \r\nWHERE \r\nSTART_DATETIME BETWEEN DATE('20160630','yyyyMMdd') AND DATE('20160701','yyyyMMdd') \r\nGROUP BY START_DATETIME, FILENAME, RECORD_NUMBER ) \r\nWHERE cnt > 1\r\n\r\n=> no records returned from this query\r\n\r\nSo, again based on articles on the web describing that the index sometimes contains data which is not referring to anything really in the class, I dropped the index, and started it re-creation.\r\n\r\nThis failed again on exactly the same spot : \r\n\r\n2016-07-19 12:07:07:160 SEVER Internal server error:\r\ncom.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #218:1248232: found duplicated key 'OCompositeKey{keys=[Thu Jun 30 23:52:57 CEST 2016, C22N010000000014.001.00665346, 1]}' in index 'idx_mds_csv_cf14_0' previously assigned to the record #214:1248231\r\n        DB name=\"cdrarch\"INDEX=idx_mds_csv_cf14_0 RID=#214:1248231 \r\n\r\n**=> there are no duplicate records found from the \"GROUP BY\" query earlier, but according to the index-creating there would be duplicates anyhow ??!!**\r\n\r\n## Steps to reproduce the problem\r\nRemove the setting for MAXDISKCACHE in the \"server.sh\" file, and start stuffing tons of records to the database. Hopefully this reproduces the problem. I won't try it, because it risks my database to become even more corrupted.\r\n\r\n### OrientDB Version\r\nv2.2.3\r\n\r\n### Operating System\r\nUbuntu 14.04\r\n\r\n### Java Version\r\n1.8.0_92\r\n"}}
{"actor":{"display_login":"lvca"}}