{"action":"created","issue":{"url":"https://api.github.com/repos/influxdata/influxdb/issues/5865","repository_url":"https://api.github.com/repos/influxdata/influxdb","labels_url":"https://api.github.com/repos/influxdata/influxdb/issues/5865/labels{/name}","comments_url":"https://api.github.com/repos/influxdata/influxdb/issues/5865/comments","events_url":"https://api.github.com/repos/influxdata/influxdb/issues/5865/events","html_url":"https://github.com/influxdata/influxdb/issues/5865","id":137350394,"number":5865,"title":"Conversion to tsm fails with exceeds max index value","user":{"login":"pbooth","id":34048,"avatar_url":"https://avatars.githubusercontent.com/u/34048?v=3","gravatar_id":"","url":"https://api.github.com/users/pbooth","html_url":"https://github.com/pbooth","followers_url":"https://api.github.com/users/pbooth/followers","following_url":"https://api.github.com/users/pbooth/following{/other_user}","gists_url":"https://api.github.com/users/pbooth/gists{/gist_id}","starred_url":"https://api.github.com/users/pbooth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pbooth/subscriptions","organizations_url":"https://api.github.com/users/pbooth/orgs","repos_url":"https://api.github.com/users/pbooth/repos","events_url":"https://api.github.com/users/pbooth/events{/privacy}","received_events_url":"https://api.github.com/users/pbooth/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/influxdata/influxdb/labels/tsm","name":"tsm","color":"e11d21"}],"state":"open","locked":false,"assignee":{"login":"joelegasse","id":1391698,"avatar_url":"https://avatars.githubusercontent.com/u/1391698?v=3","gravatar_id":"","url":"https://api.github.com/users/joelegasse","html_url":"https://github.com/joelegasse","followers_url":"https://api.github.com/users/joelegasse/followers","following_url":"https://api.github.com/users/joelegasse/following{/other_user}","gists_url":"https://api.github.com/users/joelegasse/gists{/gist_id}","starred_url":"https://api.github.com/users/joelegasse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joelegasse/subscriptions","organizations_url":"https://api.github.com/users/joelegasse/orgs","repos_url":"https://api.github.com/users/joelegasse/repos","events_url":"https://api.github.com/users/joelegasse/events{/privacy}","received_events_url":"https://api.github.com/users/joelegasse/received_events","type":"User","site_admin":false},"milestone":null,"comments":3,"created_at":"2016-02-29T19:39:55Z","updated_at":"2016-03-02T21:54:50Z","closed_at":null,"body":"Hello, I have a 0.9.6 influxdb that I have upgraded to    0.10.1-1 \r\nWhen I try to upgrade the shard in a 30GB DB I get the following:\r\n\r\n    2016/02/29 11:50:16.988594 Still Working: Completed Shards: 8/29 Points read/written: 306704235/306704235\r\n    2016/02/29 11:50:17.510485 Failed to convert    /data/fusionio/fio01/apps/tpeng/influxdb_lab15/data/prism/default/2210: Conversion of /data/fusionio/fio01/apps/tpeng/influxdb_lab15/data/prism/default/2210 failed: key 'latency_micros,cgroups=both,env=perf,instance=ampstestlab8,stor=nonpersist#!~#value' exceeds max index entries: 218400 > 65535\r\n\r\n\r\nIs there something that I can with the bz shard to shrink it?"},"comment":{"url":"https://api.github.com/repos/influxdata/influxdb/issues/comments/191450204","html_url":"https://github.com/influxdata/influxdb/issues/5865#issuecomment-191450204","issue_url":"https://api.github.com/repos/influxdata/influxdb/issues/5865","id":191450204,"user":{"login":"joelegasse","id":1391698,"avatar_url":"https://avatars.githubusercontent.com/u/1391698?v=3","gravatar_id":"","url":"https://api.github.com/users/joelegasse","html_url":"https://github.com/joelegasse","followers_url":"https://api.github.com/users/joelegasse/followers","following_url":"https://api.github.com/users/joelegasse/following{/other_user}","gists_url":"https://api.github.com/users/joelegasse/gists{/gist_id}","starred_url":"https://api.github.com/users/joelegasse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joelegasse/subscriptions","organizations_url":"https://api.github.com/users/joelegasse/orgs","repos_url":"https://api.github.com/users/joelegasse/repos","events_url":"https://api.github.com/users/joelegasse/events{/privacy}","received_events_url":"https://api.github.com/users/joelegasse/received_events","type":"User","site_admin":false},"created_at":"2016-03-02T21:54:50Z","updated_at":"2016-03-02T21:54:50Z","body":"@pbooth Thank you for finding this. @jonseymour is right, we need to handle this in the conversion tool, and there's nothing you should need to do with your files.\r\n\r\nThis is the side effect of a design decision in TSM, specifically the `tsm1` format's block index for a given key. Using the default values, it's limited to 65535 (a 16-bit index key) blocks of up to 1000 entries (values) per block, per TSM file. This means that over 65 million points would need to be stored for a single field in a single series in a single shard without going over the file limit for this to be an issue. As @jonseymour mentioned, usually the file-size limit is hit first, and even at the highest compaction, that index limit isn't hit.\r\n\r\nI should have a fix merged shortly, and we're planning a `0.10.2` release in the near future that will also include some other fixes/optimizations for the conversion tool as well."}}