{"action":"created","issue":{"url":"https://api.github.com/repos/hankcs/hanlp-solr-plugin/issues/10","repository_url":"https://api.github.com/repos/hankcs/hanlp-solr-plugin","labels_url":"https://api.github.com/repos/hankcs/hanlp-solr-plugin/issues/10/labels{/name}","comments_url":"https://api.github.com/repos/hankcs/hanlp-solr-plugin/issues/10/comments","events_url":"https://api.github.com/repos/hankcs/hanlp-solr-plugin/issues/10/events","html_url":"https://github.com/hankcs/hanlp-solr-plugin/issues/10","id":170811742,"number":10,"title":"偏移量错误","user":{"login":"drmam","id":19721891,"avatar_url":"https://avatars.githubusercontent.com/u/19721891?v=3","gravatar_id":"","url":"https://api.github.com/users/drmam","html_url":"https://github.com/drmam","followers_url":"https://api.github.com/users/drmam/followers","following_url":"https://api.github.com/users/drmam/following{/other_user}","gists_url":"https://api.github.com/users/drmam/gists{/gist_id}","starred_url":"https://api.github.com/users/drmam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drmam/subscriptions","organizations_url":"https://api.github.com/users/drmam/orgs","repos_url":"https://api.github.com/users/drmam/repos","events_url":"https://api.github.com/users/drmam/events{/privacy}","received_events_url":"https://api.github.com/users/drmam/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":27,"created_at":"2016-08-12T07:09:59Z","updated_at":"2016-08-12T08:27:24Z","closed_at":null,"body":"作者你好,我用tika读取文件之后就会出现这个错误  tika是1.6的，hanlp-solr是最新版，下面代码就是您文件中单元测试的代码。测试文件test\r\n测试结果如下：\r\n       软件工程专业是2002年国家教育部新增专业，随着计算机应用领域的不\r\n断扩大及中国经济建设的不断发展，软件工程专业将成为一个新的热门专业。软\r\n件工程专业以计算机科学与技术学科为基础，\r\n\r\nQuery = text:专业\r\n命中：1\r\n测试回测换行符 , 0.2795085\r\n文中有3个词汇”专业“只命中一个，如果使用ik分词器是就不会报错的，希望解决一下。 \r\n\r\n[test.docx](https://github.com/hankcs/hanlp-solr-plugin/files/414755/test.docx)\r\n\r\norg.apache.lucene.search.highlight.InvalidTokenOffsetsException: Token ， exceeds length of provided text sized 98\r\n\tat org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:225)\r\n\tat org.apache.lucene.search.highlight.Highlighter.getBestFragments(Highlighter.java:156)\r\n\tat org.apache.lucene.search.highlight.Highlighter.getBestFragment(Highlighter.java:102)\r\n\tat org.apache.lucene.search.highlight.Highlighter.getBestFragment(Highlighter.java:80)\r\n\tat com.lib.service.user.HighLighterTest.displayHtmlHighlight(HighLighterTest.java:181)\r\n\tat com.lib.service.user.HighLighterTest.testHightlight(HighLighterTest.java:116)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:483)\r\n\tat junit.framework.TestCase.runTest(TestCase.java:176)\r\n\tat junit.framework.TestCase.runBare(TestCase.java:141)\r\n\tat junit.framework.TestResult$1.protect(TestResult.java:122)\r\n\tat junit.framework.TestResult.runProtected(TestResult.java:142)\r\n\tat junit.framework.TestResult.run(TestResult.java:125)\r\n\tat junit.framework.TestCase.run(TestCase.java:129)\r\n\tat junit.framework.TestSuite.runTest(TestSuite.java:255)\r\n\tat junit.framework.TestSuite.run(TestSuite.java:250)\r\n\tat org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)\r\n\tat org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)\r\n\tat org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)\r\n\tat org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)\r\n\tat org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)\r\n\tat org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)\r\n\tat org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)\r\n\r\n\r\n\r\n\r\n\r\n/*\r\n * <summary></summary>\r\n * <author>He Han</author>\r\n * <email>me@hankcs.com</email>\r\n * <create-date>2015/10/22 11:37</create-date>\r\n *\r\n * <copyright file=\"HighLighterDemo.java\" company=\"码农场\">\r\n * Copyright (c) 2008-2015, 码农场. All Right Reserved, http://www.hankcs.com/\r\n * This source is subject to Hankcs. Please contact Hankcs to get more information.\r\n * </copyright>\r\n */\r\npackage com.lib.service.user;\r\n\r\nimport junit.framework.TestCase;\r\nimport org.apache.lucene.analysis.Analyzer;\r\nimport org.apache.lucene.document.Document;\r\nimport org.apache.lucene.document.Field;\r\nimport org.apache.lucene.document.TextField;\r\nimport org.apache.lucene.index.*;\r\nimport org.apache.lucene.queryparser.classic.ParseException;\r\nimport org.apache.lucene.queryparser.classic.QueryParser;\r\nimport org.apache.lucene.search.*;\r\nimport org.apache.lucene.search.highlight.*;\r\nimport org.apache.lucene.store.Directory;\r\nimport org.apache.lucene.store.LockObtainFailedException;\r\nimport org.apache.lucene.store.RAMDirectory;\r\nimport org.apache.tika.metadata.Metadata;\r\nimport org.apache.tika.parser.AutoDetectParser;\r\nimport org.apache.tika.parser.ParseContext;\r\nimport org.apache.tika.parser.Parser;\r\nimport org.apache.tika.parser.txt.TXTParser;\r\nimport org.apache.tika.sax.BodyContentHandler;\r\nimport org.junit.Test;\r\nimport org.xml.sax.ContentHandler;\r\n\r\nimport com.hankcs.lucene.HanLPAnalyzer;\r\n\r\nimport java.io.File;\r\nimport java.io.FileInputStream;\r\nimport java.io.IOException;\r\nimport java.io.InputStream;\r\nimport java.util.Scanner;\r\n\r\n/**\r\n * 演示高亮搜索结果\r\n *\r\n * @author hankcs\r\n */\r\npublic class HighLighterTest extends TestCase\r\n{\r\n\t@Test\r\n    public  void testHightlight() throws Exception\r\n    {\r\n        // Lucene Document的主要域名\r\n        String fieldName = \"text\";\r\n\r\n        // 实例化Analyzer分词器\r\n        Analyzer analyzer = new HanLPAnalyzer();\r\n\r\n        Directory directory = null;\r\n        IndexWriter iwriter;\r\n        IndexReader ireader = null;\r\n        IndexSearcher isearcher;\r\n        try\r\n        {\r\n            //索引过程**********************************\r\n            //建立内存索引对象\r\n            directory = new RAMDirectory();\r\n\r\n            //配置IndexWriterConfig\r\n            IndexWriterConfig iwConfig = new IndexWriterConfig(analyzer);\r\n            iwConfig.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);\r\n            iwriter = new IndexWriter(directory, iwConfig);\r\n            {\r\n                // 加入一个文档\r\n                Document doc = new Document();\r\n                String fileName=Parser(\"C:\\\\Users\\\\drmam\\\\Desktop\\\\test.docx\");\r\n                doc.add(new TextField(fieldName, fileName, Field.Store.YES));\r\n                doc.add(new TextField(\"title\", \"测试回测换行符\", Field.Store.YES));\r\n                iwriter.addDocument(doc);\r\n            }\r\n            {\r\n                // 再加入一个\r\n                Document doc = new Document();\r\n                doc.add(new TextField(fieldName, \"\\n\\n   \\n程序员\\n\\n喜欢黑夜\", Field.Store.YES));\r\n                doc.add(new TextField(\"title\", \"关于程序员\", Field.Store.YES));\r\n                iwriter.addDocument(doc);\r\n            }\r\n            iwriter.close();\r\n\r\n            //搜索过程**********************************\r\n            //实例化搜索器\r\n            ireader = DirectoryReader.open(directory);\r\n            isearcher = new IndexSearcher(ireader);\r\n\r\n            String keyword = \"专业\";\r\n            //使用QueryParser查询分析器构造Query对象\r\n            QueryParser qp = new QueryParser(fieldName, analyzer);\r\n            Query query = qp.parse(keyword);\r\n            System.out.println(\"Query = \" + query);\r\n\r\n            //搜索相似度最高的5条记录\r\n            TopDocs topDocs = isearcher.search(query, 5);\r\n            System.out.println(\"命中：\" + topDocs.totalHits);\r\n            //输出结果\r\n            ScoreDoc[] scoreDocs = topDocs.scoreDocs;\r\n\r\n            for (int i = 0; i < Math.min(5, scoreDocs.length); ++i)\r\n            {\r\n                Document targetDoc = isearcher.doc(scoreDocs[i].doc);\r\n                System.out.print(targetDoc.getField(\"title\").stringValue());\r\n                System.out.println(\" , \" + scoreDocs[i].score);\r\n\r\n                String text = targetDoc.get(fieldName);\r\n                \r\n                System.out.println(displayHtmlHighlight(query, analyzer, fieldName, text, 200));\r\n                \r\n            }\r\n        }\r\n        catch (CorruptIndexException e)\r\n        {\r\n            e.printStackTrace();\r\n        }\r\n        catch (LockObtainFailedException e)\r\n        {\r\n            e.printStackTrace();\r\n        }\r\n        catch (IOException e)\r\n        {\r\n            e.printStackTrace();\r\n        }\r\n        catch (ParseException e)\r\n        {\r\n            e.printStackTrace();\r\n        }\r\n        finally\r\n        {\r\n            if (ireader != null)\r\n            {\r\n                try\r\n                {\r\n                    ireader.close();\r\n                }\r\n                catch (IOException e)\r\n                {\r\n                    e.printStackTrace();\r\n                }\r\n            }\r\n            if (directory != null)\r\n            {\r\n                try\r\n                {\r\n                    directory.close();\r\n                }\r\n                catch (IOException e)\r\n                {\r\n                    e.printStackTrace();\r\n                }\r\n            }\r\n        }\r\n    }\r\n\r\n    /**\r\n     * 获取高亮显示结果的html代码\r\n     *\r\n     * @param query        查询\r\n     * @param analyzer     分词器\r\n     * @param fieldName    域名\r\n     * @param fieldContent 域内容\r\n     * @param fragmentSize 结果的长度（不含html标签长度）\r\n     * @return 结果（一段html代码）\r\n     * @throws IOException\r\n     * @throws InvalidTokenOffsetsException\r\n     */\r\n    static String displayHtmlHighlight(Query query, Analyzer analyzer, String fieldName, String fieldContent, int fragmentSize) throws IOException, InvalidTokenOffsetsException\r\n    {\r\n        //创建一个高亮器\r\n        Highlighter highlighter = new Highlighter(new SimpleHTMLFormatter(\"【\", \"】\"), new QueryScorer(query));\r\n        Fragmenter fragmenter = new SimpleFragmenter(fragmentSize);\r\n        highlighter.setTextFragmenter(fragmenter);\r\n        return highlighter.getBestFragment(analyzer, fieldName, fieldContent);\r\n    }\r\n    public static String Parser(String filePath){\r\n\t\t\r\n\t\t\r\n\t\tParser parser = new AutoDetectParser();\r\n        InputStream input = null;  \r\n        try{  \r\n            Metadata metadata = new Metadata();  \r\n            metadata.set(Metadata.CONTENT_ENCODING, \"utf-8\");  \r\n             //metadata.set(Metadata.RESOURCE_NAME_KEY, file.getName());  \r\n            input = new FileInputStream(new File(filePath));  \r\n            ContentHandler handler = new BodyContentHandler(1024*1024*10);//当文件大于100000时，new BodyContentHandler(1024*1024*10);   \r\n            ParseContext context = new ParseContext();  \r\n            //context.set(Parser.class,parser);  \r\n            parser.parse(input,handler, metadata,context);  \r\n            /*for(String name:metadata.names()) {  \r\n                System.out.println(name+\":\"+metadata.get(name));  \r\n            } */\r\n            System.out.println(handler.toString());\r\n            return handler.toString();  \r\n        }catch (Exception e){  \r\n            e.printStackTrace();  \r\n        }finally {  \r\n            try {  \r\n                if(input!=null)input.close();  \r\n            } catch (IOException e) {  \r\n                e.printStackTrace();  \r\n            }  \r\n        }  \r\n        return null;  \r\n\t\t\r\n\t}\r\n}\r\n"},"comment":{"url":"https://api.github.com/repos/hankcs/hanlp-solr-plugin/issues/comments/239390013","html_url":"https://github.com/hankcs/hanlp-solr-plugin/issues/10#issuecomment-239390013","issue_url":"https://api.github.com/repos/hankcs/hanlp-solr-plugin/issues/10","id":239390013,"user":{"login":"drmam","id":19721891,"avatar_url":"https://avatars.githubusercontent.com/u/19721891?v=3","gravatar_id":"","url":"https://api.github.com/users/drmam","html_url":"https://github.com/drmam","followers_url":"https://api.github.com/users/drmam/followers","following_url":"https://api.github.com/users/drmam/following{/other_user}","gists_url":"https://api.github.com/users/drmam/gists{/gist_id}","starred_url":"https://api.github.com/users/drmam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drmam/subscriptions","organizations_url":"https://api.github.com/users/drmam/orgs","repos_url":"https://api.github.com/users/drmam/repos","events_url":"https://api.github.com/users/drmam/events{/privacy}","received_events_url":"https://api.github.com/users/drmam/received_events","type":"User","site_admin":false},"created_at":"2016-08-12T08:27:23Z","updated_at":"2016-08-12T08:27:23Z","body":" 好的我先试下，因为我用的eclipse，所以不知道还可以那样。我先下载个ide。麻烦了"}}
{"actor":{"display_login":"drmam"}}