{"action":"created","issue":{"url":"https://api.github.com/repos/codice/ddf/issues/3553","repository_url":"https://api.github.com/repos/codice/ddf","labels_url":"https://api.github.com/repos/codice/ddf/issues/3553/labels{/name}","comments_url":"https://api.github.com/repos/codice/ddf/issues/3553/comments","events_url":"https://api.github.com/repos/codice/ddf/issues/3553/events","html_url":"https://github.com/codice/ddf/pull/3553","id":347569033,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA2MTY0MzQ2","number":3553,"title":"DDF-3938 Solr query performance improvements","user":{"login":"pklinef","id":700518,"node_id":"MDQ6VXNlcjcwMDUxOA==","avatar_url":"https://avatars1.githubusercontent.com/u/700518?v=4","gravatar_id":"","url":"https://api.github.com/users/pklinef","html_url":"https://github.com/pklinef","followers_url":"https://api.github.com/users/pklinef/followers","following_url":"https://api.github.com/users/pklinef/following{/other_user}","gists_url":"https://api.github.com/users/pklinef/gists{/gist_id}","starred_url":"https://api.github.com/users/pklinef/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pklinef/subscriptions","organizations_url":"https://api.github.com/users/pklinef/orgs","repos_url":"https://api.github.com/users/pklinef/repos","events_url":"https://api.github.com/users/pklinef/events{/privacy}","received_events_url":"https://api.github.com/users/pklinef/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"bdeining","id":6551038,"node_id":"MDQ6VXNlcjY1NTEwMzg=","avatar_url":"https://avatars2.githubusercontent.com/u/6551038?v=4","gravatar_id":"","url":"https://api.github.com/users/bdeining","html_url":"https://github.com/bdeining","followers_url":"https://api.github.com/users/bdeining/followers","following_url":"https://api.github.com/users/bdeining/following{/other_user}","gists_url":"https://api.github.com/users/bdeining/gists{/gist_id}","starred_url":"https://api.github.com/users/bdeining/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdeining/subscriptions","organizations_url":"https://api.github.com/users/bdeining/orgs","repos_url":"https://api.github.com/users/bdeining/repos","events_url":"https://api.github.com/users/bdeining/events{/privacy}","received_events_url":"https://api.github.com/users/bdeining/received_events","type":"User","site_admin":false},"assignees":[{"login":"bdeining","id":6551038,"node_id":"MDQ6VXNlcjY1NTEwMzg=","avatar_url":"https://avatars2.githubusercontent.com/u/6551038?v=4","gravatar_id":"","url":"https://api.github.com/users/bdeining","html_url":"https://github.com/bdeining","followers_url":"https://api.github.com/users/bdeining/followers","following_url":"https://api.github.com/users/bdeining/following{/other_user}","gists_url":"https://api.github.com/users/bdeining/gists{/gist_id}","starred_url":"https://api.github.com/users/bdeining/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdeining/subscriptions","organizations_url":"https://api.github.com/users/bdeining/orgs","repos_url":"https://api.github.com/users/bdeining/repos","events_url":"https://api.github.com/users/bdeining/events{/privacy}","received_events_url":"https://api.github.com/users/bdeining/received_events","type":"User","site_admin":false},{"login":"rzwiefel","id":9013416,"node_id":"MDQ6VXNlcjkwMTM0MTY=","avatar_url":"https://avatars0.githubusercontent.com/u/9013416?v=4","gravatar_id":"","url":"https://api.github.com/users/rzwiefel","html_url":"https://github.com/rzwiefel","followers_url":"https://api.github.com/users/rzwiefel/followers","following_url":"https://api.github.com/users/rzwiefel/following{/other_user}","gists_url":"https://api.github.com/users/rzwiefel/gists{/gist_id}","starred_url":"https://api.github.com/users/rzwiefel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rzwiefel/subscriptions","organizations_url":"https://api.github.com/users/rzwiefel/orgs","repos_url":"https://api.github.com/users/rzwiefel/repos","events_url":"https://api.github.com/users/rzwiefel/events{/privacy}","received_events_url":"https://api.github.com/users/rzwiefel/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2018-08-04T00:25:34Z","updated_at":"2018-08-06T20:36:48Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/codice/ddf/pulls/3553","html_url":"https://github.com/codice/ddf/pull/3553","diff_url":"https://github.com/codice/ddf/pull/3553.diff","patch_url":"https://github.com/codice/ddf/pull/3553.patch"},"body":"#### What does this PR do?\r\nMakes changes to Solr schema.xml and solrconfig.xml to improve query performance.  Enabled docValues (improved sorting and faceting), switched to RptWithGeometrySpatialField (improved Intersects), replaced Trie fields with Point fields (improved range queries), removed sort keys with new multivalued doc value sorting (smaller ingest/index), replaced guava cache with Solr real time get (enable longer auto soft commit), and increased auto commit times (improved Solr cache hit ratio).\r\n\r\n#### Who is reviewing it? \r\n<!--(please choose AT LEAST two reviewers that need to approve the PR before it can get merged)-->\r\n@brendan-hofmann \r\n@paouelle \r\n\r\n#### Select relevant component teams: \r\nhttps://github.com/orgs/codice/teams/solr\r\n\r\n#### Ask 2 committers to review/merge the PR and tag them here.\r\n<!--\r\nIf you don't know who to ask, you can request reviews in https://groups.google.com/forum/#!forum/ddf-developers .\r\n(please choose ONLY two committers from below, delete the rest)\r\n-->\r\n@bdeining\r\n@rzwiefel\r\n\r\n#### How should this be tested?\r\n<!--(List steps with links to updated documentation)-->\r\nCompare query performance with large dataset before and after.  Constantly ingest data (at least one per second or faster) during query performance testing to verify improved caching.  I used Apache Bench to get the mean response time and response percentiles but JMeter and others should work too.  Here is an example of doing a query for `test`, sorting by a date, and default page size of 250 as generated by Intrigue.  Using 10 concurrent workers over 5000 requests.\r\n\r\n`ab -n 5000 -c 10 \"http://localhost:8994/solr/catalog/select?q=+(++(*:*+-validation-errors_txt:[*+TO+*])++AND++(++(+metacard-tags_txt_tokenized:%22resource%22+OR++(*:*+-metacard-tags_txt:[*+TO+*])++)++AND+(point-of-contact_txt_ws:%22test%22+source-id_txt_ws:%22test%22+ext.feature-code_txt_ws:%22test%22+contact.publisher-name_txt_ws:%22test%22+validation-errors_txt_ws:%22test%22+media.type_txt_ws:%22test%22+ext.feature-class_txt_ws:%22test%22+id_txt_ws:%22test%22+contact.creator-email_txt_ws:%22test%22+contact.creator-address_txt_ws:%22test%22+checksum_txt_ws:%22test%22+metacard.version.edited-by_txt_ws:%22test%22+checksum-algorithm_txt_ws:%22test%22+metacard.version.resource-uri_txt_ws:%22test%22+metadata-content-type-version_txt_ws:%22test%22+security.access-groups_txt_ws:%22test%22+title_txt_ws:%22test%22+validation-warnings_txt_ws:%22test%22+media.format_txt_ws:%22test%22+metacard.owner_txt_ws:%22test%22+contact.publisher-address_txt_ws:%22test%22+ext.import-location_txt_ws:%22test%22+contact.creator-phone_txt_ws:%22test%22+contact.publisher-email_txt_ws:%22test%22+topic.keyword_txt_ws:%22test%22+security.access-individuals_txt_ws:%22test%22+failed-validators-warnings_txt_ws:%22test%22+topic.category_txt_ws:%22test%22+media.encoding_txt_ws:%22test%22+resource.derived-download-url_txt_ws:%22test%22+metacard.associations.derived_txt_ws:%22test%22+contact.contributor-email_txt_ws:%22test%22+contact.point-of-contact-name_txt_ws:%22test%22+failed-validators-errors_txt_ws:%22test%22+metacard.version.type_txt_ws:%22test%22+resource-size_txt_ws:%22test%22+media.compression_txt_ws:%22test%22+metacard.associations.related_txt_ws:%22test%22+metadata-content-type_txt_ws:%22test%22+location.country-code_txt_ws:%22test%22+contact.creator-name_txt_ws:%22test%22+contact.contributor-phone_txt_ws:%22test%22+contact.publisher-phone_txt_ws:%22test%22+metadata_txt_ws:%22test%22+topic.vocabulary_txt_ws:%22test%22+contact.point-of-contact-email_txt_ws:%22test%22+metacard.version.tags_txt_ws:%22test%22+contact.point-of-contact-address_txt_ws:%22test%22+metacard.version.action_txt_ws:%22test%22+language_txt_ws:%22test%22+metacard.version.id_txt_ws:%22test%22+datatype_txt_ws:%22test%22+media.scanning-mode_txt_ws:%22test%22+description_txt_ws:%22test%22+associations.external_txt_ws:%22test%22+datetime.name_txt_ws:%22test%22+metacard-tags_txt_ws:%22test%22+resource-uri_txt_ws:%22test%22+contact.contributor-name_txt_ws:%22test%22+location.crs-name_txt_ws:%22test%22+contact.point-of-contact-phone_txt_ws:%22test%22+resource.derived-uri_txt_ws:%22test%22+resource-download-url_txt_ws:%22test%22+contact.contributor-address_txt_ws:%22test%22+metadata-target-namespace_txt_ws:%22test%22+media.format-version_txt_ws:%22test%22+location.crs-code_txt_ws:%22test%22)+)++)+&fl=point-of-contact_txt,ext.feature-code_txt,source-id_txt,contact.publisher-name_txt,validation-errors_txt,media.type_txt,ext.feature-class_txt,id_txt,contact.creator-email_txt,metacard_type_name_txt,contact.creator-address_txt,checksum_txt,metacard.version.edited-by_txt,metacard.sharing_xml,checksum-algorithm_txt,metacard.version.resource-uri_txt,queries_xml,security.access-groups_txt,title_txt,validation-warnings_txt,media.format_txt,metacard.owner_txt,contact.publisher-address_txt,ext.import-location_txt,contact.creator-phone_txt,security.access-individuals_txt,topic.keyword_txt,contact.publisher-email_txt,topic.category_txt,media.encoding_txt,failed-validators-warnings_txt,resource.derived-download-url_txt,metacard.associations.derived_txt,contact.contributor-email_txt,contact.point-of-contact-name_txt,metacard.version.type_txt,failed-validators-errors_txt,media.compression_txt,resource-size_txt,metacard.associations.related_txt,location.country-code_txt,contact.creator-name_txt,contact.contributor-phone_txt,contact.publisher-phone_txt,topic.vocabulary_txt,contact.point-of-contact-email_txt,metacard.version.tags_txt,metacard.version.action_txt,contact.point-of-contact-address_txt,language_txt,metacard.version.id_txt,media.scanning-mode_txt,datatype_txt,description_txt,associations.external_txt,datetime.name_txt,metacard-tags_txt,resource-uri_txt,contact.contributor-name_txt,location.crs-name_txt,contact.point-of-contact-phone_txt,resource-download-url_txt,resource.derived-uri_txt,contact.contributor-address_txt,lists_xml,media.format-version_txt,location.crs-code_txt,*_geo,*_bln,*_tdt,*_shr,*_flt,*_dbl,*_int,*_lng,*_bin,*_obj,score&start=0&sort=modified_tdt+desc&rows=250&wt=javabin&version=2\"`\r\n\r\n#### Any background context you want to provide?\r\n\r\n#### What are the relevant tickets?\r\n[DDF-3938](https://codice.atlassian.net/browse/DDF-3938)\r\n\r\n#### Screenshots\r\n<!--(if appropriate)-->\r\n\r\n#### Checklist:\r\n- [ n/a ] Documentation Updated\r\n- [ n/a ] Update / Add Unit Tests\r\n- [ n/a ] Update / Add Integration Tests\r\n\r\n#### Notes on Review Process\r\nPlease see [Notes on Review Process](https://codice.atlassian.net/wiki/spaces/DDF/pages/71946981/Pull+Request+Guidelines) for further guidance on requirements for merging and abbreviated reviews. \r\n\r\n#### Review Comment Legend:\r\n- ✏️ (Pencil) This comment is a nitpick or style suggestion, no action required for approval. This comment should provide a suggestion either as an in line code snippet or a gist. \r\n- ❓ (Question Mark) This comment is to gain a clearer understanding of design or code choices, clarification is required but action may not be necessary for approval.\r\n- ❗ (Exclamation Mark) This comment is critical and requires clarification or action before approval.\r\n"},"comment":{"url":"https://api.github.com/repos/codice/ddf/issues/comments/410844987","html_url":"https://github.com/codice/ddf/pull/3553#issuecomment-410844987","issue_url":"https://api.github.com/repos/codice/ddf/issues/3553","id":410844987,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMDg0NDk4Nw==","user":{"login":"codacy-bot","id":19940114,"node_id":"MDQ6VXNlcjE5OTQwMTE0","avatar_url":"https://avatars1.githubusercontent.com/u/19940114?v=4","gravatar_id":"","url":"https://api.github.com/users/codacy-bot","html_url":"https://github.com/codacy-bot","followers_url":"https://api.github.com/users/codacy-bot/followers","following_url":"https://api.github.com/users/codacy-bot/following{/other_user}","gists_url":"https://api.github.com/users/codacy-bot/gists{/gist_id}","starred_url":"https://api.github.com/users/codacy-bot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/codacy-bot/subscriptions","organizations_url":"https://api.github.com/users/codacy-bot/orgs","repos_url":"https://api.github.com/users/codacy-bot/repos","events_url":"https://api.github.com/users/codacy-bot/events{/privacy}","received_events_url":"https://api.github.com/users/codacy-bot/received_events","type":"User","site_admin":false},"created_at":"2018-08-06T20:36:48Z","updated_at":"2018-08-06T20:36:48Z","author_association":"NONE","body":"\n![Codacy](https://app.codacy.com/assets/images/favicon.png) Here is an overview of what got changed by this pull request:\n\n\n```diff\n\nComplexity decreasing per file\n==============================\n+ catalog/core/catalog-core-solr/src/main/java/ddf/catalog/source/solr/DynamicSchemaResolver.java  -2\n         \n```\n\n\nSee the complete overview on [Codacy](https://app.codacy.com/app/Codice/ddf/pullRequest?prid=1997077&bid=8274979)\n        "}}
{"actor":{"display_login":"codacy-bot"}}