{"action":"created","issue":{"url":"https://api.github.com/repos/OpenGrok/OpenGrok/issues/1170","repository_url":"https://api.github.com/repos/OpenGrok/OpenGrok","labels_url":"https://api.github.com/repos/OpenGrok/OpenGrok/issues/1170/labels{/name}","comments_url":"https://api.github.com/repos/OpenGrok/OpenGrok/issues/1170/comments","events_url":"https://api.github.com/repos/OpenGrok/OpenGrok/issues/1170/events","html_url":"https://github.com/OpenGrok/OpenGrok/issues/1170","id":172126929,"number":1170,"title":"make parsers stop producing tokens > 32766 chars","user":{"login":"tarzanek","id":504773,"avatar_url":"https://avatars.githubusercontent.com/u/504773?v=3","gravatar_id":"","url":"https://api.github.com/users/tarzanek","html_url":"https://github.com/tarzanek","followers_url":"https://api.github.com/users/tarzanek/followers","following_url":"https://api.github.com/users/tarzanek/following{/other_user}","gists_url":"https://api.github.com/users/tarzanek/gists{/gist_id}","starred_url":"https://api.github.com/users/tarzanek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tarzanek/subscriptions","organizations_url":"https://api.github.com/users/tarzanek/orgs","repos_url":"https://api.github.com/users/tarzanek/repos","events_url":"https://api.github.com/users/tarzanek/events{/privacy}","received_events_url":"https://api.github.com/users/tarzanek/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/OpenGrok/OpenGrok/labels/stopper","name":"stopper","color":"eb6420"}],"state":"open","locked":false,"assignee":{"login":"tarzanek","id":504773,"avatar_url":"https://avatars.githubusercontent.com/u/504773?v=3","gravatar_id":"","url":"https://api.github.com/users/tarzanek","html_url":"https://github.com/tarzanek","followers_url":"https://api.github.com/users/tarzanek/followers","following_url":"https://api.github.com/users/tarzanek/following{/other_user}","gists_url":"https://api.github.com/users/tarzanek/gists{/gist_id}","starred_url":"https://api.github.com/users/tarzanek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tarzanek/subscriptions","organizations_url":"https://api.github.com/users/tarzanek/orgs","repos_url":"https://api.github.com/users/tarzanek/repos","events_url":"https://api.github.com/users/tarzanek/events{/privacy}","received_events_url":"https://api.github.com/users/tarzanek/received_events","type":"User","site_admin":false},"assignees":[{"login":"tarzanek","id":504773,"avatar_url":"https://avatars.githubusercontent.com/u/504773?v=3","gravatar_id":"","url":"https://api.github.com/users/tarzanek","html_url":"https://github.com/tarzanek","followers_url":"https://api.github.com/users/tarzanek/followers","following_url":"https://api.github.com/users/tarzanek/following{/other_user}","gists_url":"https://api.github.com/users/tarzanek/gists{/gist_id}","starred_url":"https://api.github.com/users/tarzanek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tarzanek/subscriptions","organizations_url":"https://api.github.com/users/tarzanek/orgs","repos_url":"https://api.github.com/users/tarzanek/repos","events_url":"https://api.github.com/users/tarzanek/events{/privacy}","received_events_url":"https://api.github.com/users/tarzanek/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/OpenGrok/OpenGrok/milestones/2","html_url":"https://github.com/OpenGrok/OpenGrok/milestones/0.13","labels_url":"https://api.github.com/repos/OpenGrok/OpenGrok/milestones/2/labels","id":360045,"number":2,"title":"0.13","description":"0.13 release","creator":{"login":"vladak","id":2934284,"avatar_url":"https://avatars.githubusercontent.com/u/2934284?v=3","gravatar_id":"","url":"https://api.github.com/users/vladak","html_url":"https://github.com/vladak","followers_url":"https://api.github.com/users/vladak/followers","following_url":"https://api.github.com/users/vladak/following{/other_user}","gists_url":"https://api.github.com/users/vladak/gists{/gist_id}","starred_url":"https://api.github.com/users/vladak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vladak/subscriptions","organizations_url":"https://api.github.com/users/vladak/orgs","repos_url":"https://api.github.com/users/vladak/repos","events_url":"https://api.github.com/users/vladak/events{/privacy}","received_events_url":"https://api.github.com/users/vladak/received_events","type":"User","site_admin":false},"open_issues":36,"closed_issues":144,"state":"open","created_at":"2013-06-20T10:24:36Z","updated_at":"2016-08-19T13:05:07Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2016-08-19T13:04:23Z","updated_at":"2016-08-19T13:06:02Z","closed_at":null,"body":"last lucene versions imposed a limit on how big terms can be\r\nso you can easily end up with:\r\n\r\n> Aug 12, 2016 3:07:27 PM org.opensolaris.opengrok.index.IndexDatabase indexDown\r\n> WARNING: Failed to add file /source/foobar.xml\r\n> java.lang.IllegalArgumentException: Document contains at least one immense term in field=\"full\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[101, 57, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 102, 48, 48, 48, 48, 48, 48, 48, 102, 48, 49, 55, 48]...', original message: bytes can be at most 32766 in length; got 89226\r\n>         at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:772)\r\n>         at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:417)\r\n>         at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:373)\r\n>         at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)\r\n>         at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:449)\r\n>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1492)\r\n>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1271)\r\n>         at org.opensolaris.opengrok.index.IndexDatabase.addFile(IndexDatabase.java:635)\r\n>         at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:883)\r\n>         at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:848)\r\n>         at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:848)\r\n>         at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:848)\r\n>         at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:848)\r\n>         at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:848)\r\n>         at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:848)\r\n>         at org.opensolaris.opengrok.index.IndexDatabase.update(IndexDatabase.java:397)\r\n>         at org.opensolaris.opengrok.index.IndexDatabase$2.run(IndexDatabase.java:184)\r\n>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\r\n>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)\r\n>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n>         at java.lang.Thread.run(Thread.java:745)\r\n> Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 89226\r\n>         at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:263)\r\n>         at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:150)\r\n>         at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:762)\r\n>         ... 21 more\r\n\r\nso parsers can try hard and still this will be dropped from index\r\n- we can have files where this will take e.g. 4h for a simple file to be indexed (imagine xml file with <blob>0032023040a00f000h0004949d32323b0000 .... </blob>  - the chars between blob are hex numbers without any whitespace and it's more than 32766 of them ) - which will eventually be dropped - so once this is fixed indexing should be faster and skip/encounter such dummy binary files much earlier"},"comment":{"url":"https://api.github.com/repos/OpenGrok/OpenGrok/issues/comments/241013059","html_url":"https://github.com/OpenGrok/OpenGrok/issues/1170#issuecomment-241013059","issue_url":"https://api.github.com/repos/OpenGrok/OpenGrok/issues/1170","id":241013059,"user":{"login":"tarzanek","id":504773,"avatar_url":"https://avatars.githubusercontent.com/u/504773?v=3","gravatar_id":"","url":"https://api.github.com/users/tarzanek","html_url":"https://github.com/tarzanek","followers_url":"https://api.github.com/users/tarzanek/followers","following_url":"https://api.github.com/users/tarzanek/following{/other_user}","gists_url":"https://api.github.com/users/tarzanek/gists{/gist_id}","starred_url":"https://api.github.com/users/tarzanek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tarzanek/subscriptions","organizations_url":"https://api.github.com/users/tarzanek/orgs","repos_url":"https://api.github.com/users/tarzanek/repos","events_url":"https://api.github.com/users/tarzanek/events{/privacy}","received_events_url":"https://api.github.com/users/tarzanek/received_events","type":"User","site_admin":false},"created_at":"2016-08-19T13:06:02Z","updated_at":"2016-08-19T13:06:02Z","body":"(so I hope once fixed my internal setup will get the indexing done in faster time, if I'd get 20% faster out of this fix, I'd be happy :) )"}}
{"actor":{"display_login":"tarzanek"}}