{"action":"closed","number":156,"pull_request":{"url":"https://api.github.com/repos/EducationalTestingService/rsmtool/pulls/156","id":168315587,"html_url":"https://github.com/EducationalTestingService/rsmtool/pull/156","diff_url":"https://github.com/EducationalTestingService/rsmtool/pull/156.diff","patch_url":"https://github.com/EducationalTestingService/rsmtool/pull/156.patch","issue_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/issues/156","number":156,"state":"closed","locked":false,"title":"Improve SKLL integration with RSMTool","user":{"login":"desilinguist","id":161680,"avatar_url":"https://avatars2.githubusercontent.com/u/161680?v=4","gravatar_id":"","url":"https://api.github.com/users/desilinguist","html_url":"https://github.com/desilinguist","followers_url":"https://api.github.com/users/desilinguist/followers","following_url":"https://api.github.com/users/desilinguist/following{/other_user}","gists_url":"https://api.github.com/users/desilinguist/gists{/gist_id}","starred_url":"https://api.github.com/users/desilinguist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/desilinguist/subscriptions","organizations_url":"https://api.github.com/users/desilinguist/orgs","repos_url":"https://api.github.com/users/desilinguist/repos","events_url":"https://api.github.com/users/desilinguist/events{/privacy}","received_events_url":"https://api.github.com/users/desilinguist/received_events","type":"User","site_admin":false},"body":"This PR improves integration of RSMTool with SKLL in addition to addressing issue #129.\r\n\r\n#### Code\r\n\r\n1. Remove static list of SKLL learners and use `importlib` to check any given model name against the `skll.learner` module directly in a new function called `is_skll_model()`. This means that any and all SKLL learners (classifiers and regressors) are now available to RSMTool. \r\n2. For consistency, also create a new function called `is_builtin_model()`. \r\n3. Allow computation of expected scores (probability-weighted averages over score points) from probabilistic SKLL classifiers for both `rsmtool` and `rsmpredict`.\r\n   - Add a new `predict_expected_scores` option to the configuration files for `rsmtool` and `rsmpredict`, which is `False` by default. \r\n    - Add checks to configuration parser to quickly raise an error if the specified model name for RSMTool experiments are not probabilistic classifiers. The same check is used to raise an error for built-in models since they are not classifiers.\r\n    - Modify `Modeler.train_skll_model()` to train with probabilities enabled if we want expected scores. \r\n    - Modify `Modeler.predict()` to take in the min and max scores and the `predict_expected_scores` flag.\r\n   - Add a new general purpose function called `compute_expected_scores_from_model()` which takes in a SKLL model, a `FeatureSet` object, the min and max scores, and computes expected scores using probability distributions obtained from the model. It also explicitly checks for errors and raises exceptions. \r\n    - Call this function from both `rsmtool` and `rsmpredict` in the appropriate places. \r\n4. Have `train_skll_model()` choose `f1_score_micro` as the default objective if the SKLL learner is a classifier. Have it also return the objective which we can then pass into the report and then not have to worry about checking the objective in the notebooks. \r\n5. Improve calls to `train_skll_model()` and `train_builtin_model()` to use keyword arguments correctly. \r\n6. Modify log messages to show \"expected scores\" when generating predictions.\r\n7. Add a note to the top of the report if the experiment used expected scores.\r\n8. Some other minor changes for readability and style.\r\n\r\n#### Tests\r\n\r\nAdded several new tests.\r\n- Tests for the general function `compute_expected_scores_from_model()`.\r\n- Tests for the configuration parsing piece.\r\n- Several new experiment tests for classification, classification with custom objectives, and classification with expected scores for both rsmtool as well as rsmpredict experiments.\r\n- Tests for rsmpredict to catch various errors with expected scores. \r\n\r\n#### Documentation\r\n\r\n- Add documentation about expected scores for both `rsmtool` and `rsmpredict` and add explicit notes about the expected scores potentially being different on different machines due to small probability values.\r\n- Move `skll_objective` back down to optional fields since we want all 4 required `rsmtool` argument to be together. Added links from the `model` field to `skll_objective` and `predict_expected_scores` fields. ","created_at":"2018-02-09T18:53:06Z","updated_at":"2018-02-12T21:30:49Z","closed_at":"2018-02-12T21:30:48Z","merged_at":"2018-02-12T21:30:48Z","merge_commit_sha":"a83b36f215f7afcc1f63933c5b0f942a1f7dd9bd","assignee":{"login":"desilinguist","id":161680,"avatar_url":"https://avatars2.githubusercontent.com/u/161680?v=4","gravatar_id":"","url":"https://api.github.com/users/desilinguist","html_url":"https://github.com/desilinguist","followers_url":"https://api.github.com/users/desilinguist/followers","following_url":"https://api.github.com/users/desilinguist/following{/other_user}","gists_url":"https://api.github.com/users/desilinguist/gists{/gist_id}","starred_url":"https://api.github.com/users/desilinguist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/desilinguist/subscriptions","organizations_url":"https://api.github.com/users/desilinguist/orgs","repos_url":"https://api.github.com/users/desilinguist/repos","events_url":"https://api.github.com/users/desilinguist/events{/privacy}","received_events_url":"https://api.github.com/users/desilinguist/received_events","type":"User","site_admin":false},"assignees":[{"login":"desilinguist","id":161680,"avatar_url":"https://avatars2.githubusercontent.com/u/161680?v=4","gravatar_id":"","url":"https://api.github.com/users/desilinguist","html_url":"https://github.com/desilinguist","followers_url":"https://api.github.com/users/desilinguist/followers","following_url":"https://api.github.com/users/desilinguist/following{/other_user}","gists_url":"https://api.github.com/users/desilinguist/gists{/gist_id}","starred_url":"https://api.github.com/users/desilinguist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/desilinguist/subscriptions","organizations_url":"https://api.github.com/users/desilinguist/orgs","repos_url":"https://api.github.com/users/desilinguist/repos","events_url":"https://api.github.com/users/desilinguist/events{/privacy}","received_events_url":"https://api.github.com/users/desilinguist/received_events","type":"User","site_admin":false}],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/pulls/156/commits","review_comments_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/pulls/156/comments","review_comment_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/pulls/comments{/number}","comments_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/issues/156/comments","statuses_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/statuses/d8e17285a9859da3e308379e8ffe44206b199d8e","head":{"label":"EducationalTestingService:allow-use-of-expected-values","ref":"allow-use-of-expected-values","sha":"d8e17285a9859da3e308379e8ffe44206b199d8e","user":{"login":"EducationalTestingService","id":4822867,"avatar_url":"https://avatars0.githubusercontent.com/u/4822867?v=4","gravatar_id":"","url":"https://api.github.com/users/EducationalTestingService","html_url":"https://github.com/EducationalTestingService","followers_url":"https://api.github.com/users/EducationalTestingService/followers","following_url":"https://api.github.com/users/EducationalTestingService/following{/other_user}","gists_url":"https://api.github.com/users/EducationalTestingService/gists{/gist_id}","starred_url":"https://api.github.com/users/EducationalTestingService/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/EducationalTestingService/subscriptions","organizations_url":"https://api.github.com/users/EducationalTestingService/orgs","repos_url":"https://api.github.com/users/EducationalTestingService/repos","events_url":"https://api.github.com/users/EducationalTestingService/events{/privacy}","received_events_url":"https://api.github.com/users/EducationalTestingService/received_events","type":"Organization","site_admin":false},"repo":{"id":55423129,"name":"rsmtool","full_name":"EducationalTestingService/rsmtool","owner":{"login":"EducationalTestingService","id":4822867,"avatar_url":"https://avatars0.githubusercontent.com/u/4822867?v=4","gravatar_id":"","url":"https://api.github.com/users/EducationalTestingService","html_url":"https://github.com/EducationalTestingService","followers_url":"https://api.github.com/users/EducationalTestingService/followers","following_url":"https://api.github.com/users/EducationalTestingService/following{/other_user}","gists_url":"https://api.github.com/users/EducationalTestingService/gists{/gist_id}","starred_url":"https://api.github.com/users/EducationalTestingService/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/EducationalTestingService/subscriptions","organizations_url":"https://api.github.com/users/EducationalTestingService/orgs","repos_url":"https://api.github.com/users/EducationalTestingService/repos","events_url":"https://api.github.com/users/EducationalTestingService/events{/privacy}","received_events_url":"https://api.github.com/users/EducationalTestingService/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/EducationalTestingService/rsmtool","description":"RSMTool is a python package for facilitating research on building and evaluating automated scoring models.","fork":false,"url":"https://api.github.com/repos/EducationalTestingService/rsmtool","forks_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/forks","keys_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/keys{/key_id}","collaborators_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/teams","hooks_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/hooks","issue_events_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/issues/events{/number}","events_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/events","assignees_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/assignees{/user}","branches_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/branches{/branch}","tags_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/tags","blobs_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/git/refs{/sha}","trees_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/git/trees{/sha}","statuses_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/statuses/{sha}","languages_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/languages","stargazers_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/stargazers","contributors_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/contributors","subscribers_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/subscribers","subscription_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/subscription","commits_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/commits{/sha}","git_commits_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/git/commits{/sha}","comments_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/comments{/number}","issue_comment_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/issues/comments{/number}","contents_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/contents/{+path}","compare_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/compare/{base}...{head}","merges_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/merges","archive_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/downloads","issues_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/issues{/number}","pulls_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/pulls{/number}","milestones_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/milestones{/number}","notifications_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/labels{/name}","releases_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/releases{/id}","deployments_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/deployments","created_at":"2016-04-04T15:35:11Z","updated_at":"2018-02-06T19:23:12Z","pushed_at":"2018-02-12T21:30:48Z","git_url":"git://github.com/EducationalTestingService/rsmtool.git","ssh_url":"git@github.com:EducationalTestingService/rsmtool.git","clone_url":"https://github.com/EducationalTestingService/rsmtool.git","svn_url":"https://github.com/EducationalTestingService/rsmtool","homepage":"rsmtool.readthedocs.io","size":8412,"stargazers_count":23,"watchers_count":23,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":5,"mirror_url":null,"archived":false,"open_issues_count":21,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":5,"open_issues":21,"watchers":23,"default_branch":"master"}},"base":{"label":"EducationalTestingService:rsmtool-refactor","ref":"rsmtool-refactor","sha":"e4dcd049340a024a017974425ac7a5cd422a43f0","user":{"login":"EducationalTestingService","id":4822867,"avatar_url":"https://avatars0.githubusercontent.com/u/4822867?v=4","gravatar_id":"","url":"https://api.github.com/users/EducationalTestingService","html_url":"https://github.com/EducationalTestingService","followers_url":"https://api.github.com/users/EducationalTestingService/followers","following_url":"https://api.github.com/users/EducationalTestingService/following{/other_user}","gists_url":"https://api.github.com/users/EducationalTestingService/gists{/gist_id}","starred_url":"https://api.github.com/users/EducationalTestingService/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/EducationalTestingService/subscriptions","organizations_url":"https://api.github.com/users/EducationalTestingService/orgs","repos_url":"https://api.github.com/users/EducationalTestingService/repos","events_url":"https://api.github.com/users/EducationalTestingService/events{/privacy}","received_events_url":"https://api.github.com/users/EducationalTestingService/received_events","type":"Organization","site_admin":false},"repo":{"id":55423129,"name":"rsmtool","full_name":"EducationalTestingService/rsmtool","owner":{"login":"EducationalTestingService","id":4822867,"avatar_url":"https://avatars0.githubusercontent.com/u/4822867?v=4","gravatar_id":"","url":"https://api.github.com/users/EducationalTestingService","html_url":"https://github.com/EducationalTestingService","followers_url":"https://api.github.com/users/EducationalTestingService/followers","following_url":"https://api.github.com/users/EducationalTestingService/following{/other_user}","gists_url":"https://api.github.com/users/EducationalTestingService/gists{/gist_id}","starred_url":"https://api.github.com/users/EducationalTestingService/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/EducationalTestingService/subscriptions","organizations_url":"https://api.github.com/users/EducationalTestingService/orgs","repos_url":"https://api.github.com/users/EducationalTestingService/repos","events_url":"https://api.github.com/users/EducationalTestingService/events{/privacy}","received_events_url":"https://api.github.com/users/EducationalTestingService/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/EducationalTestingService/rsmtool","description":"RSMTool is a python package for facilitating research on building and evaluating automated scoring models.","fork":false,"url":"https://api.github.com/repos/EducationalTestingService/rsmtool","forks_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/forks","keys_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/keys{/key_id}","collaborators_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/teams","hooks_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/hooks","issue_events_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/issues/events{/number}","events_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/events","assignees_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/assignees{/user}","branches_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/branches{/branch}","tags_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/tags","blobs_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/git/refs{/sha}","trees_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/git/trees{/sha}","statuses_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/statuses/{sha}","languages_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/languages","stargazers_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/stargazers","contributors_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/contributors","subscribers_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/subscribers","subscription_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/subscription","commits_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/commits{/sha}","git_commits_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/git/commits{/sha}","comments_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/comments{/number}","issue_comment_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/issues/comments{/number}","contents_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/contents/{+path}","compare_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/compare/{base}...{head}","merges_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/merges","archive_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/downloads","issues_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/issues{/number}","pulls_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/pulls{/number}","milestones_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/milestones{/number}","notifications_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/labels{/name}","releases_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/releases{/id}","deployments_url":"https://api.github.com/repos/EducationalTestingService/rsmtool/deployments","created_at":"2016-04-04T15:35:11Z","updated_at":"2018-02-06T19:23:12Z","pushed_at":"2018-02-12T21:30:48Z","git_url":"git://github.com/EducationalTestingService/rsmtool.git","ssh_url":"git@github.com:EducationalTestingService/rsmtool.git","clone_url":"https://github.com/EducationalTestingService/rsmtool.git","svn_url":"https://github.com/EducationalTestingService/rsmtool","homepage":"rsmtool.readthedocs.io","size":8412,"stargazers_count":23,"watchers_count":23,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":5,"mirror_url":null,"archived":false,"open_issues_count":21,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":5,"open_issues":21,"watchers":23,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/EducationalTestingService/rsmtool/pulls/156"},"html":{"href":"https://github.com/EducationalTestingService/rsmtool/pull/156"},"issue":{"href":"https://api.github.com/repos/EducationalTestingService/rsmtool/issues/156"},"comments":{"href":"https://api.github.com/repos/EducationalTestingService/rsmtool/issues/156/comments"},"review_comments":{"href":"https://api.github.com/repos/EducationalTestingService/rsmtool/pulls/156/comments"},"review_comment":{"href":"https://api.github.com/repos/EducationalTestingService/rsmtool/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/EducationalTestingService/rsmtool/pulls/156/commits"},"statuses":{"href":"https://api.github.com/repos/EducationalTestingService/rsmtool/statuses/d8e17285a9859da3e308379e8ffe44206b199d8e"}},"author_association":"OWNER","merged":true,"mergeable":null,"rebaseable":null,"mergeable_state":"unknown","merged_by":{"login":"desilinguist","id":161680,"avatar_url":"https://avatars2.githubusercontent.com/u/161680?v=4","gravatar_id":"","url":"https://api.github.com/users/desilinguist","html_url":"https://github.com/desilinguist","followers_url":"https://api.github.com/users/desilinguist/followers","following_url":"https://api.github.com/users/desilinguist/following{/other_user}","gists_url":"https://api.github.com/users/desilinguist/gists{/gist_id}","starred_url":"https://api.github.com/users/desilinguist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/desilinguist/subscriptions","organizations_url":"https://api.github.com/users/desilinguist/orgs","repos_url":"https://api.github.com/users/desilinguist/repos","events_url":"https://api.github.com/users/desilinguist/events{/privacy}","received_events_url":"https://api.github.com/users/desilinguist/received_events","type":"User","site_admin":false},"comments":2,"review_comments":5,"maintainer_can_modify":false,"commits":46,"additions":19449,"deletions":81,"changed_files":185}}
{"actor":{"display_login":"desilinguist"}}