{"action":"created","issue":{"url":"https://api.github.com/repos/berkmancenter/mediacloud/issues/24","repository_url":"https://api.github.com/repos/berkmancenter/mediacloud","labels_url":"https://api.github.com/repos/berkmancenter/mediacloud/issues/24/labels{/name}","comments_url":"https://api.github.com/repos/berkmancenter/mediacloud/issues/24/comments","events_url":"https://api.github.com/repos/berkmancenter/mediacloud/issues/24/events","html_url":"https://github.com/berkmancenter/mediacloud/issues/24","id":160316646,"number":24,"title":"Upgrade Solr to 6.x, address OOM problems","user":{"login":"pypt","id":181949,"avatar_url":"https://avatars.githubusercontent.com/u/181949?v=3","gravatar_id":"","url":"https://api.github.com/users/pypt","html_url":"https://github.com/pypt","followers_url":"https://api.github.com/users/pypt/followers","following_url":"https://api.github.com/users/pypt/following{/other_user}","gists_url":"https://api.github.com/users/pypt/gists{/gist_id}","starred_url":"https://api.github.com/users/pypt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pypt/subscriptions","organizations_url":"https://api.github.com/users/pypt/orgs","repos_url":"https://api.github.com/users/pypt/repos","events_url":"https://api.github.com/users/pypt/events{/privacy}","received_events_url":"https://api.github.com/users/pypt/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/berkmancenter/mediacloud/labels/solr","name":"solr","color":"fbca04"}],"state":"open","locked":false,"assignee":{"login":"pypt","id":181949,"avatar_url":"https://avatars.githubusercontent.com/u/181949?v=3","gravatar_id":"","url":"https://api.github.com/users/pypt","html_url":"https://github.com/pypt","followers_url":"https://api.github.com/users/pypt/followers","following_url":"https://api.github.com/users/pypt/following{/other_user}","gists_url":"https://api.github.com/users/pypt/gists{/gist_id}","starred_url":"https://api.github.com/users/pypt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pypt/subscriptions","organizations_url":"https://api.github.com/users/pypt/orgs","repos_url":"https://api.github.com/users/pypt/repos","events_url":"https://api.github.com/users/pypt/events{/privacy}","received_events_url":"https://api.github.com/users/pypt/received_events","type":"User","site_admin":false},"assignees":[{"login":"pypt","id":181949,"avatar_url":"https://avatars.githubusercontent.com/u/181949?v=3","gravatar_id":"","url":"https://api.github.com/users/pypt","html_url":"https://github.com/pypt","followers_url":"https://api.github.com/users/pypt/followers","following_url":"https://api.github.com/users/pypt/following{/other_user}","gists_url":"https://api.github.com/users/pypt/gists{/gist_id}","starred_url":"https://api.github.com/users/pypt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pypt/subscriptions","organizations_url":"https://api.github.com/users/pypt/orgs","repos_url":"https://api.github.com/users/pypt/repos","events_url":"https://api.github.com/users/pypt/events{/privacy}","received_events_url":"https://api.github.com/users/pypt/received_events","type":"User","site_admin":false}],"milestone":null,"comments":10,"created_at":"2016-06-15T00:50:55Z","updated_at":"2016-08-04T21:25:36Z","closed_at":null,"body":"We're currently running Solr 4.6.0, so before doing any performance improvements we should upgrade to Solr 6 (6.0.1 currently).\r\n\r\nIt is recommended to upgrade between major Solr versions by doing the upgrade incrementally, i.e. version 4 -> version 5 -> version 6, not version 4 -> version 6.\r\n\r\nLoose checklist:\r\n\r\n1. Install Solr using Gradle (#23) for easier upgrades.\r\n\r\n2. Upgrade 4.6.0 to 4.10.4 in `solr_gradle/4.10.4` branch\r\n\r\n    Should be pretty safe and easy, basically a version number change in Gradle's config file.\r\n\r\n3. Upgrade 4.10.4 to 5.5.1 in `solr_gradle/5.5.1` branch\r\n\r\n    Will need more extensive testing and some time with the changelog.\r\n\r\n4. Upgrade 5.5.1 to 6.0.1 in `solr_gradle/6.0.1` branch\r\n\r\n    Ditto with more extensive testing.\r\n\r\n5. After testing the whole 4.6.0 -> 4.10.4 -> 5.5.1 -> 6.0.1 upgrade path, merge branches one by one to `release` and deploy to `mcquery*`.\r\n"},"comment":{"url":"https://api.github.com/repos/berkmancenter/mediacloud/issues/comments/237688377","html_url":"https://github.com/berkmancenter/mediacloud/issues/24#issuecomment-237688377","issue_url":"https://api.github.com/repos/berkmancenter/mediacloud/issues/24","id":237688377,"user":{"login":"hroberts","id":1830735,"avatar_url":"https://avatars.githubusercontent.com/u/1830735?v=3","gravatar_id":"","url":"https://api.github.com/users/hroberts","html_url":"https://github.com/hroberts","followers_url":"https://api.github.com/users/hroberts/followers","following_url":"https://api.github.com/users/hroberts/following{/other_user}","gists_url":"https://api.github.com/users/hroberts/gists{/gist_id}","starred_url":"https://api.github.com/users/hroberts/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hroberts/subscriptions","organizations_url":"https://api.github.com/users/hroberts/orgs","repos_url":"https://api.github.com/users/hroberts/repos","events_url":"https://api.github.com/users/hroberts/events{/privacy}","received_events_url":"https://api.github.com/users/hroberts/received_events","type":"User","site_admin":false},"created_at":"2016-08-04T21:25:36Z","updated_at":"2016-08-04T21:25:36Z","body":"Here are the fields we're keeping in the index, with comments for each:\r\n\r\n* solr_id (stored)- this is required, and we use a concatenation of the ss_id and the s_id in a special format that instructs solr to store sentences from the same story on the same shard, which allows us to quickly count story matches; we don't need this to be stored for any reason, but solr may require it to be stored as the primary id;\r\n* story_sentences_id (stored) - we need this as a stored value so that we can lookup returned sentences; we don't every actually query this in practice, so we could stop indexing it, but I can imagine cases in which it would be useful to query;\r\n* media_id (not stored) - query by media_id is a core function;\r\n* stories_id (stored) - query by stories_id is a core function; we need to store this so that we can quickly process queries grouped by stories_id, otherwise we would have to get the ss_id back and then query the ss table for the stories_id values, which would slow things down a lot;\r\n* sentence_number (not stored) - I can imagine wanting to query this, but I don't think we actually ever have, so I think we could drop it;\r\n* processed_stories_id (stored) - we need this indexed and stored to be able to implement the query streaming features in the api (last_processed_stories_id=);\r\n* sentence (stored) - query by sentence is a core function, but we could rewrite the perl Solr.pm module to query sentences from the ss table instead using sentences returned by solr; we will need to do this to have room to fit the story_text: field into the index;\r\n* title (stored) - (see sentence)\r\n* publish_date (not stored) - query by date is a core function\r\n* publish_day (not stored) - faster version of publish_date for querying by day\r\n* language (not stored) - query by language is a core function\r\n* bitly_click_count (stored) - we need this to be able to sort by bitly clicks, which we need if we ever want to display influential stories list on the dashboard; I don't think we need to store this, though.\r\n* media_sets_id (stored) - obsolete;  we should get rid of it, but we don't actually write this to the db any more so it won't save any space.\r\n* tags_id_stories|media (not stored) - core query\r\n* tags_id_story_senences (stored) - core query; I think we need these stored values to implement sentences/field_counts\r\n"}}
{"actor":{"display_login":"hroberts"}}