{"action":"created","comment":{"url":"https://api.github.com/repos/mldbai/mldb/pulls/comments/49207674","id":49207674,"diff_hunk":"@@ -325,13 +390,23 @@ apply(const FunctionApplier & applier,\n         break;\n     }\n \n+    RowValue values;\n+    Date ts = inputVal.getEffectiveTimestamp();\n+\n     // Compute the score for every word in the input\n+    //cerr << \"corpus size \" << corpusSize << \" document size \" << documentSize << endl;\n+\n     for (auto& col : inputVal.getRow() ) {\n-        Utf8String colName = std::get<0>(col).toUtf8String();\n+        Utf8String term = std::get<0>(col).toUtf8String(); // the term is the columnName\n+        double frequency = (double) std::get<1>(col).getAtom().toUInt() / documentSize;\n \n-        double frequency = std::get<1>(col).toDouble() / documentCount;","path":"plugins/tfidf.cc","position":389,"original_position":389,"commit_id":"e2bc326cc6cebfb9f78e5c3f64bde92470648801","original_commit_id":"e2bc326cc6cebfb9f78e5c3f64bde92470648801","user":{"login":"guyd","id":683788,"avatar_url":"https://avatars.githubusercontent.com/u/683788?v=3","gravatar_id":"","url":"https://api.github.com/users/guyd","html_url":"https://github.com/guyd","followers_url":"https://api.github.com/users/guyd/followers","following_url":"https://api.github.com/users/guyd/following{/other_user}","gists_url":"https://api.github.com/users/guyd/gists{/gist_id}","starred_url":"https://api.github.com/users/guyd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guyd/subscriptions","organizations_url":"https://api.github.com/users/guyd/orgs","repos_url":"https://api.github.com/users/guyd/repos","events_url":"https://api.github.com/users/guyd/events{/privacy}","received_events_url":"https://api.github.com/users/guyd/received_events","type":"User","site_admin":false},"body":"@mailletf @steadtler Here we divide the term frequency (# of times a term appears in the document) by the document size (the number of terms it contains).  This is a good approach in some cases especially when documents are varying in size greatly but it may not be what the user wants (BM25 is likely a better option).  Since the term frequencies are passed as an input to the tf-idf function, I suggest we do not perform that normalization here and let the user decide if he/she wants to use raw term frequencies or normalized ones.  Do you agree?","created_at":"2016-01-08T16:34:42Z","updated_at":"2016-01-08T16:34:42Z","html_url":"https://github.com/mldbai/mldb/pull/56#discussion_r49207674","pull_request_url":"https://api.github.com/repos/mldbai/mldb/pulls/56","_links":{"self":{"href":"https://api.github.com/repos/mldbai/mldb/pulls/comments/49207674"},"html":{"href":"https://github.com/mldbai/mldb/pull/56#discussion_r49207674"},"pull_request":{"href":"https://api.github.com/repos/mldbai/mldb/pulls/56"}}},"pull_request":{"url":"https://api.github.com/repos/mldbai/mldb/pulls/56","id":55480476,"html_url":"https://github.com/mldbai/mldb/pull/56","diff_url":"https://github.com/mldbai/mldb/pull/56.diff","patch_url":"https://github.com/mldbai/mldb/pull/56.patch","issue_url":"https://api.github.com/repos/mldbai/mldb/issues/56","number":56,"state":"open","locked":false,"title":"Mldb 1226 persist model","user":{"login":"guyd","id":683788,"avatar_url":"https://avatars.githubusercontent.com/u/683788?v=3","gravatar_id":"","url":"https://api.github.com/users/guyd","html_url":"https://github.com/guyd","followers_url":"https://api.github.com/users/guyd/followers","following_url":"https://api.github.com/users/guyd/following{/other_user}","gists_url":"https://api.github.com/users/guyd/gists{/gist_id}","starred_url":"https://api.github.com/users/guyd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guyd/subscriptions","organizations_url":"https://api.github.com/users/guyd/orgs","repos_url":"https://api.github.com/users/guyd/repos","events_url":"https://api.github.com/users/guyd/events{/privacy}","received_events_url":"https://api.github.com/users/guyd/received_events","type":"User","site_admin":false},"body":"- modified the kmeans procedure to persist the centroids as a model file - that model is necessary to create a kmeans function\r\n- similarly modified tf-idf procedure to persist the document frequencies of the terms in the corpus along with the corpus size - that model is also necessary when creating a kmeans function\r\n- adapted and augmented the test coverage of the kmeans and tf-idf functions","created_at":"2016-01-08T16:25:48Z","updated_at":"2016-01-08T16:34:42Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b551c634c6245e2642dc926828d88153d758efa8","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/mldbai/mldb/pulls/56/commits","review_comments_url":"https://api.github.com/repos/mldbai/mldb/pulls/56/comments","review_comment_url":"https://api.github.com/repos/mldbai/mldb/pulls/comments{/number}","comments_url":"https://api.github.com/repos/mldbai/mldb/issues/56/comments","statuses_url":"https://api.github.com/repos/mldbai/mldb/statuses/e2bc326cc6cebfb9f78e5c3f64bde92470648801","head":{"label":"mldbai:MLDB-1226-persist-model","ref":"MLDB-1226-persist-model","sha":"e2bc326cc6cebfb9f78e5c3f64bde92470648801","user":{"login":"mldbai","id":11510172,"avatar_url":"https://avatars.githubusercontent.com/u/11510172?v=3","gravatar_id":"","url":"https://api.github.com/users/mldbai","html_url":"https://github.com/mldbai","followers_url":"https://api.github.com/users/mldbai/followers","following_url":"https://api.github.com/users/mldbai/following{/other_user}","gists_url":"https://api.github.com/users/mldbai/gists{/gist_id}","starred_url":"https://api.github.com/users/mldbai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mldbai/subscriptions","organizations_url":"https://api.github.com/users/mldbai/orgs","repos_url":"https://api.github.com/users/mldbai/repos","events_url":"https://api.github.com/users/mldbai/events{/privacy}","received_events_url":"https://api.github.com/users/mldbai/received_events","type":"Organization","site_admin":false},"repo":{"id":47634692,"name":"mldb","full_name":"mldbai/mldb","owner":{"login":"mldbai","id":11510172,"avatar_url":"https://avatars.githubusercontent.com/u/11510172?v=3","gravatar_id":"","url":"https://api.github.com/users/mldbai","html_url":"https://github.com/mldbai","followers_url":"https://api.github.com/users/mldbai/followers","following_url":"https://api.github.com/users/mldbai/following{/other_user}","gists_url":"https://api.github.com/users/mldbai/gists{/gist_id}","starred_url":"https://api.github.com/users/mldbai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mldbai/subscriptions","organizations_url":"https://api.github.com/users/mldbai/orgs","repos_url":"https://api.github.com/users/mldbai/repos","events_url":"https://api.github.com/users/mldbai/events{/privacy}","received_events_url":"https://api.github.com/users/mldbai/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mldbai/mldb","description":"MLDB is the Machine Learning Database","fork":false,"url":"https://api.github.com/repos/mldbai/mldb","forks_url":"https://api.github.com/repos/mldbai/mldb/forks","keys_url":"https://api.github.com/repos/mldbai/mldb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mldbai/mldb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mldbai/mldb/teams","hooks_url":"https://api.github.com/repos/mldbai/mldb/hooks","issue_events_url":"https://api.github.com/repos/mldbai/mldb/issues/events{/number}","events_url":"https://api.github.com/repos/mldbai/mldb/events","assignees_url":"https://api.github.com/repos/mldbai/mldb/assignees{/user}","branches_url":"https://api.github.com/repos/mldbai/mldb/branches{/branch}","tags_url":"https://api.github.com/repos/mldbai/mldb/tags","blobs_url":"https://api.github.com/repos/mldbai/mldb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mldbai/mldb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mldbai/mldb/git/refs{/sha}","trees_url":"https://api.github.com/repos/mldbai/mldb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mldbai/mldb/statuses/{sha}","languages_url":"https://api.github.com/repos/mldbai/mldb/languages","stargazers_url":"https://api.github.com/repos/mldbai/mldb/stargazers","contributors_url":"https://api.github.com/repos/mldbai/mldb/contributors","subscribers_url":"https://api.github.com/repos/mldbai/mldb/subscribers","subscription_url":"https://api.github.com/repos/mldbai/mldb/subscription","commits_url":"https://api.github.com/repos/mldbai/mldb/commits{/sha}","git_commits_url":"https://api.github.com/repos/mldbai/mldb/git/commits{/sha}","comments_url":"https://api.github.com/repos/mldbai/mldb/comments{/number}","issue_comment_url":"https://api.github.com/repos/mldbai/mldb/issues/comments{/number}","contents_url":"https://api.github.com/repos/mldbai/mldb/contents/{+path}","compare_url":"https://api.github.com/repos/mldbai/mldb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mldbai/mldb/merges","archive_url":"https://api.github.com/repos/mldbai/mldb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mldbai/mldb/downloads","issues_url":"https://api.github.com/repos/mldbai/mldb/issues{/number}","pulls_url":"https://api.github.com/repos/mldbai/mldb/pulls{/number}","milestones_url":"https://api.github.com/repos/mldbai/mldb/milestones{/number}","notifications_url":"https://api.github.com/repos/mldbai/mldb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mldbai/mldb/labels{/name}","releases_url":"https://api.github.com/repos/mldbai/mldb/releases{/id}","created_at":"2015-12-08T16:34:16Z","updated_at":"2015-12-31T04:07:08Z","pushed_at":"2016-01-08T16:25:48Z","git_url":"git://github.com/mldbai/mldb.git","ssh_url":"git@github.com:mldbai/mldb.git","clone_url":"https://github.com/mldbai/mldb.git","svn_url":"https://github.com/mldbai/mldb","homepage":"http://mldb.ai","size":8261,"stargazers_count":11,"watchers_count":11,"language":"C++","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":9,"forks":1,"open_issues":9,"watchers":11,"default_branch":"master"}},"base":{"label":"mldbai:master","ref":"master","sha":"883c4f0bb409e630aa0d713e030cd284758f10d6","user":{"login":"mldbai","id":11510172,"avatar_url":"https://avatars.githubusercontent.com/u/11510172?v=3","gravatar_id":"","url":"https://api.github.com/users/mldbai","html_url":"https://github.com/mldbai","followers_url":"https://api.github.com/users/mldbai/followers","following_url":"https://api.github.com/users/mldbai/following{/other_user}","gists_url":"https://api.github.com/users/mldbai/gists{/gist_id}","starred_url":"https://api.github.com/users/mldbai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mldbai/subscriptions","organizations_url":"https://api.github.com/users/mldbai/orgs","repos_url":"https://api.github.com/users/mldbai/repos","events_url":"https://api.github.com/users/mldbai/events{/privacy}","received_events_url":"https://api.github.com/users/mldbai/received_events","type":"Organization","site_admin":false},"repo":{"id":47634692,"name":"mldb","full_name":"mldbai/mldb","owner":{"login":"mldbai","id":11510172,"avatar_url":"https://avatars.githubusercontent.com/u/11510172?v=3","gravatar_id":"","url":"https://api.github.com/users/mldbai","html_url":"https://github.com/mldbai","followers_url":"https://api.github.com/users/mldbai/followers","following_url":"https://api.github.com/users/mldbai/following{/other_user}","gists_url":"https://api.github.com/users/mldbai/gists{/gist_id}","starred_url":"https://api.github.com/users/mldbai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mldbai/subscriptions","organizations_url":"https://api.github.com/users/mldbai/orgs","repos_url":"https://api.github.com/users/mldbai/repos","events_url":"https://api.github.com/users/mldbai/events{/privacy}","received_events_url":"https://api.github.com/users/mldbai/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mldbai/mldb","description":"MLDB is the Machine Learning Database","fork":false,"url":"https://api.github.com/repos/mldbai/mldb","forks_url":"https://api.github.com/repos/mldbai/mldb/forks","keys_url":"https://api.github.com/repos/mldbai/mldb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mldbai/mldb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mldbai/mldb/teams","hooks_url":"https://api.github.com/repos/mldbai/mldb/hooks","issue_events_url":"https://api.github.com/repos/mldbai/mldb/issues/events{/number}","events_url":"https://api.github.com/repos/mldbai/mldb/events","assignees_url":"https://api.github.com/repos/mldbai/mldb/assignees{/user}","branches_url":"https://api.github.com/repos/mldbai/mldb/branches{/branch}","tags_url":"https://api.github.com/repos/mldbai/mldb/tags","blobs_url":"https://api.github.com/repos/mldbai/mldb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mldbai/mldb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mldbai/mldb/git/refs{/sha}","trees_url":"https://api.github.com/repos/mldbai/mldb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mldbai/mldb/statuses/{sha}","languages_url":"https://api.github.com/repos/mldbai/mldb/languages","stargazers_url":"https://api.github.com/repos/mldbai/mldb/stargazers","contributors_url":"https://api.github.com/repos/mldbai/mldb/contributors","subscribers_url":"https://api.github.com/repos/mldbai/mldb/subscribers","subscription_url":"https://api.github.com/repos/mldbai/mldb/subscription","commits_url":"https://api.github.com/repos/mldbai/mldb/commits{/sha}","git_commits_url":"https://api.github.com/repos/mldbai/mldb/git/commits{/sha}","comments_url":"https://api.github.com/repos/mldbai/mldb/comments{/number}","issue_comment_url":"https://api.github.com/repos/mldbai/mldb/issues/comments{/number}","contents_url":"https://api.github.com/repos/mldbai/mldb/contents/{+path}","compare_url":"https://api.github.com/repos/mldbai/mldb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mldbai/mldb/merges","archive_url":"https://api.github.com/repos/mldbai/mldb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mldbai/mldb/downloads","issues_url":"https://api.github.com/repos/mldbai/mldb/issues{/number}","pulls_url":"https://api.github.com/repos/mldbai/mldb/pulls{/number}","milestones_url":"https://api.github.com/repos/mldbai/mldb/milestones{/number}","notifications_url":"https://api.github.com/repos/mldbai/mldb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mldbai/mldb/labels{/name}","releases_url":"https://api.github.com/repos/mldbai/mldb/releases{/id}","created_at":"2015-12-08T16:34:16Z","updated_at":"2015-12-31T04:07:08Z","pushed_at":"2016-01-08T16:25:48Z","git_url":"git://github.com/mldbai/mldb.git","ssh_url":"git@github.com:mldbai/mldb.git","clone_url":"https://github.com/mldbai/mldb.git","svn_url":"https://github.com/mldbai/mldb","homepage":"http://mldb.ai","size":8261,"stargazers_count":11,"watchers_count":11,"language":"C++","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":9,"forks":1,"open_issues":9,"watchers":11,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/mldbai/mldb/pulls/56"},"html":{"href":"https://github.com/mldbai/mldb/pull/56"},"issue":{"href":"https://api.github.com/repos/mldbai/mldb/issues/56"},"comments":{"href":"https://api.github.com/repos/mldbai/mldb/issues/56/comments"},"review_comments":{"href":"https://api.github.com/repos/mldbai/mldb/pulls/56/comments"},"review_comment":{"href":"https://api.github.com/repos/mldbai/mldb/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/mldbai/mldb/pulls/56/commits"},"statuses":{"href":"https://api.github.com/repos/mldbai/mldb/statuses/e2bc326cc6cebfb9f78e5c3f64bde92470648801"}}}}