{"action":"opened","issue":{"url":"https://api.github.com/repos/NationalSecurityAgency/datawave/issues/234","repository_url":"https://api.github.com/repos/NationalSecurityAgency/datawave","labels_url":"https://api.github.com/repos/NationalSecurityAgency/datawave/issues/234/labels{/name}","comments_url":"https://api.github.com/repos/NationalSecurityAgency/datawave/issues/234/comments","events_url":"https://api.github.com/repos/NationalSecurityAgency/datawave/issues/234/events","html_url":"https://github.com/NationalSecurityAgency/datawave/issues/234","id":371795348,"node_id":"MDU6SXNzdWUzNzE3OTUzNDg=","number":234,"title":"Update severely out-of-date Lucene dependency","user":{"login":"ctubbsii","id":1280725,"node_id":"MDQ6VXNlcjEyODA3MjU=","avatar_url":"https://avatars3.githubusercontent.com/u/1280725?v=4","gravatar_id":"","url":"https://api.github.com/users/ctubbsii","html_url":"https://github.com/ctubbsii","followers_url":"https://api.github.com/users/ctubbsii/followers","following_url":"https://api.github.com/users/ctubbsii/following{/other_user}","gists_url":"https://api.github.com/users/ctubbsii/gists{/gist_id}","starred_url":"https://api.github.com/users/ctubbsii/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ctubbsii/subscriptions","organizations_url":"https://api.github.com/users/ctubbsii/orgs","repos_url":"https://api.github.com/users/ctubbsii/repos","events_url":"https://api.github.com/users/ctubbsii/events{/privacy}","received_events_url":"https://api.github.com/users/ctubbsii/received_events","type":"User","site_admin":false},"labels":[{"id":800865048,"node_id":"MDU6TGFiZWw4MDA4NjUwNDg=","url":"https://api.github.com/repos/NationalSecurityAgency/datawave/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"open","locked":false,"assignee":{"login":"keith-ratcliffe","id":1606834,"node_id":"MDQ6VXNlcjE2MDY4MzQ=","avatar_url":"https://avatars2.githubusercontent.com/u/1606834?v=4","gravatar_id":"","url":"https://api.github.com/users/keith-ratcliffe","html_url":"https://github.com/keith-ratcliffe","followers_url":"https://api.github.com/users/keith-ratcliffe/followers","following_url":"https://api.github.com/users/keith-ratcliffe/following{/other_user}","gists_url":"https://api.github.com/users/keith-ratcliffe/gists{/gist_id}","starred_url":"https://api.github.com/users/keith-ratcliffe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keith-ratcliffe/subscriptions","organizations_url":"https://api.github.com/users/keith-ratcliffe/orgs","repos_url":"https://api.github.com/users/keith-ratcliffe/repos","events_url":"https://api.github.com/users/keith-ratcliffe/events{/privacy}","received_events_url":"https://api.github.com/users/keith-ratcliffe/received_events","type":"User","site_admin":false},"assignees":[{"login":"keith-ratcliffe","id":1606834,"node_id":"MDQ6VXNlcjE2MDY4MzQ=","avatar_url":"https://avatars2.githubusercontent.com/u/1606834?v=4","gravatar_id":"","url":"https://api.github.com/users/keith-ratcliffe","html_url":"https://github.com/keith-ratcliffe","followers_url":"https://api.github.com/users/keith-ratcliffe/followers","following_url":"https://api.github.com/users/keith-ratcliffe/following{/other_user}","gists_url":"https://api.github.com/users/keith-ratcliffe/gists{/gist_id}","starred_url":"https://api.github.com/users/keith-ratcliffe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keith-ratcliffe/subscriptions","organizations_url":"https://api.github.com/users/keith-ratcliffe/orgs","repos_url":"https://api.github.com/users/keith-ratcliffe/repos","events_url":"https://api.github.com/users/keith-ratcliffe/events{/privacy}","received_events_url":"https://api.github.com/users/keith-ratcliffe/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2018-10-19T03:41:39Z","updated_at":"2018-10-19T03:41:39Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"The Lucene dependency is severely out-of-date, and has published CVEs. I made a pass at updating Lucene in the parent POM, but it is overridden in the warehouse/pom.xml back to 4.7.1. I can't update it there, because stuff in warehouse/ingest-core still depends on tokenizer behavior from versions that are no longer supported in newer versions of Lucene. I'm not familiar enough with the code to understand the consequences of aggressively dropping that behavior, and using the newer, supported, tokenizer behavior.\r\n\r\nThis is the diff I have so far:\r\n\r\n```diff\r\ndiff --git a/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/DefaultTokenSearch.java b/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/DefaultTokenSearch.java\r\nindex 8a8269e5..2d25c29b 100644\r\n--- a/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/DefaultTokenSearch.java\r\n+++ b/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/DefaultTokenSearch.java\r\n@@ -11,7 +11,7 @@ import java.util.List;\r\n import java.util.Set;\r\n import java.util.regex.Pattern;\r\n \r\n-import org.apache.lucene.analysis.util.CharArraySet;\r\n+import org.apache.lucene.analysis.CharArraySet;\r\n import org.slf4j.Logger;\r\n import org.slf4j.LoggerFactory;\r\n \r\ndiff --git a/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/StandardAnalyzer.java b/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/StandardAnalyzer.java\r\nindex 02eed1fa..b68ea470 100644\r\n--- a/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/StandardAnalyzer.java\r\n+++ b/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/StandardAnalyzer.java\r\n@@ -8,8 +8,8 @@ import org.apache.lucene.analysis.core.LowerCaseFilter;\r\n import org.apache.lucene.analysis.core.StopAnalyzer;\r\n import org.apache.lucene.analysis.core.StopFilter;\r\n import org.apache.lucene.analysis.standard.StandardFilter;\r\n-import org.apache.lucene.analysis.util.CharArraySet;\r\n-import org.apache.lucene.analysis.util.StopwordAnalyzerBase;\r\n+import org.apache.lucene.analysis.CharArraySet;\r\n+import org.apache.lucene.analysis.StopwordAnalyzerBase;\r\n import org.apache.lucene.util.Version;\r\n \r\n /**\r\ndiff --git a/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/StandardTokenizer.java b/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/StandardTokenizer.java\r\nindex 004a4bdb..5899775f 100644\r\n--- a/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/StandardTokenizer.java\r\n+++ b/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/StandardTokenizer.java\r\n@@ -24,6 +24,7 @@ import java.util.Map;\r\n \r\n import datawave.util.ObjectFactory;\r\n \r\n+import org.apache.lucene.util.AttributeFactory;\r\n import org.apache.lucene.analysis.Tokenizer;\r\n import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;\r\n import org.apache.lucene.analysis.tokenattributes.OffsetAttribute;\r\ndiff --git a/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/TokenSearch.java b/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/TokenSearch.java\r\nindex c42e8b9e..3468a774 100644\r\n--- a/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/TokenSearch.java\r\n+++ b/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/TokenSearch.java\r\n@@ -9,8 +9,8 @@ import java.util.List;\r\n \r\n import datawave.util.ObjectFactory;\r\n \r\n-import org.apache.lucene.analysis.util.CharArraySet;\r\n-import org.apache.lucene.analysis.util.WordlistLoader;\r\n+import org.apache.lucene.analysis.CharArraySet;\r\n+import org.apache.lucene.analysis.WordlistLoader;\r\n import org.apache.lucene.util.IOUtils;\r\n import org.apache.lucene.util.Version;\r\n import org.slf4j.Logger;\r\ndiff --git a/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/TokenizationHelper.java b/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/TokenizationHelper.java\r\nindex a79d242c..16b882ea 100644\r\n--- a/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/TokenizationHelper.java\r\n+++ b/warehouse/ingest-core/src/main/java/datawave/ingest/data/tokenize/TokenizationHelper.java\r\n@@ -8,7 +8,7 @@ import datawave.ingest.data.config.DataTypeHelper;\r\n import org.apache.hadoop.conf.Configuration;\r\n import org.apache.log4j.Logger;\r\n import org.apache.lucene.analysis.Analyzer;\r\n-import org.apache.lucene.analysis.util.CharArraySet;\r\n+import org.apache.lucene.analysis.CharArraySet;\r\n \r\n public class TokenizationHelper {\r\n     \r\ndiff --git a/warehouse/ingest-core/src/main/java/datawave/ingest/mapreduce/handler/tokenize/ExtendedContentIndexingColumnBasedHandler.java b/warehouse/ingest-core/src/main/java/datawave/ingest/mapreduce/handler/tokenize/ExtendedContentIndexingColumnBasedHandler.java\r\nindex 6ee3863c..9ba5480c 100644\r\n--- a/warehouse/ingest-core/src/main/java/datawave/ingest/mapreduce/handler/tokenize/ExtendedContentIndexingColumnBasedHandler.java\r\n+++ b/warehouse/ingest-core/src/main/java/datawave/ingest/mapreduce/handler/tokenize/ExtendedContentIndexingColumnBasedHandler.java\r\n@@ -54,7 +54,7 @@ import org.apache.hadoop.mapreduce.TaskAttemptContext;\r\n import org.apache.hadoop.mapreduce.TaskInputOutputContext;\r\n import org.apache.hadoop.util.bloom.BloomFilter;\r\n import org.apache.log4j.Logger;\r\n-import org.apache.lucene.analysis.util.CharArraySet;\r\n+import org.apache.lucene.analysis.CharArraySet;\r\n import org.infinispan.commons.util.Base64;\r\n \r\n import com.google.common.collect.Multimap;\r\ndiff --git a/warehouse/pom.xml b/warehouse/pom.xml\r\nindex 08228c48..56b20acc 100644\r\n--- a/warehouse/pom.xml\r\n+++ b/warehouse/pom.xml\r\n@@ -45,7 +45,6 @@\r\n         <version.basis-rlp>7.5.0</version.basis-rlp>\r\n         <version.hadoop.processors>2.2.3</version.hadoop.processors>\r\n         <version.hamcrest>1.3</version.hamcrest>\r\n-        <version.lucene>4.7.1</version.lucene>\r\n     </properties>\r\n     <dependencyManagement>\r\n         <dependencies>\r\n```"}}
{"actor":{"display_login":"ctubbsii"}}