{"action":"created","comment":{"url":"https://api.github.com/repos/airbnb/aerosolve/pulls/comments/71268543","id":71268543,"diff_hunk":"@@ -119,74 +123,198 @@ object LinearRankerUtils {\n     sum\n   }\n \n-  // Makes an example pointwise while preserving the float features.\n+  /**\n+    * Makes an example pointwise by combining context and flatmap with feature transforms.\n+    * Each observation is further flattened to [[SparseLabeledPoint]] format where features are indexed\n+    * by a specified [[AdditiveModel]].\n+    */\n+  def makePointwiseFloatVector(examples: RDD[Example],\n+                               transformer: Transformer,\n+                               params: AdditiveModelTrainer.AdditiveTrainerParams,\n+                               model: AdditiveModel,\n+                               isTraining: Example => Boolean = _ => true,\n+                               groupSize: Int = 100): RDD[SparseLabeledPoint] = {\n+    val assemblerTimer = examples.sparkContext.accumulator(0L, \"pointAssembler\")","path":"training/src/main/scala/com/airbnb/aerosolve/training/LinearRankerUtils.scala","position":173,"original_position":173,"commit_id":"f4032b826bd8ec5950111316e5b1d2562c092182","original_commit_id":"f4032b826bd8ec5950111316e5b1d2562c092182","user":{"login":"jq","id":78691,"avatar_url":"https://avatars.githubusercontent.com/u/78691?v=3","gravatar_id":"","url":"https://api.github.com/users/jq","html_url":"https://github.com/jq","followers_url":"https://api.github.com/users/jq/followers","following_url":"https://api.github.com/users/jq/following{/other_user}","gists_url":"https://api.github.com/users/jq/gists{/gist_id}","starred_url":"https://api.github.com/users/jq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jq/subscriptions","organizations_url":"https://api.github.com/users/jq/orgs","repos_url":"https://api.github.com/users/jq/repos","events_url":"https://api.github.com/users/jq/events{/privacy}","received_events_url":"https://api.github.com/users/jq/received_events","type":"User","site_admin":false},"body":"this is for debug speed?","created_at":"2016-07-19T03:40:27Z","updated_at":"2016-07-19T03:40:27Z","html_url":"https://github.com/airbnb/aerosolve/pull/229#discussion_r71268543","pull_request_url":"https://api.github.com/repos/airbnb/aerosolve/pulls/229","_links":{"self":{"href":"https://api.github.com/repos/airbnb/aerosolve/pulls/comments/71268543"},"html":{"href":"https://github.com/airbnb/aerosolve/pull/229#discussion_r71268543"},"pull_request":{"href":"https://api.github.com/repos/airbnb/aerosolve/pulls/229"}}},"pull_request":{"url":"https://api.github.com/repos/airbnb/aerosolve/pulls/229","id":77868907,"html_url":"https://github.com/airbnb/aerosolve/pull/229","diff_url":"https://github.com/airbnb/aerosolve/pull/229.diff","patch_url":"https://github.com/airbnb/aerosolve/pull/229.patch","issue_url":"https://api.github.com/repos/airbnb/aerosolve/issues/229","number":229,"state":"open","locked":false,"title":"Optimize AdditiveModelTrainer using sparse representation","user":{"login":"saurfang","id":4317392,"avatar_url":"https://avatars.githubusercontent.com/u/4317392?v=3","gravatar_id":"","url":"https://api.github.com/users/saurfang","html_url":"https://github.com/saurfang","followers_url":"https://api.github.com/users/saurfang/followers","following_url":"https://api.github.com/users/saurfang/following{/other_user}","gists_url":"https://api.github.com/users/saurfang/gists{/gist_id}","starred_url":"https://api.github.com/users/saurfang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saurfang/subscriptions","organizations_url":"https://api.github.com/users/saurfang/orgs","repos_url":"https://api.github.com/users/saurfang/repos","events_url":"https://api.github.com/users/saurfang/events{/privacy}","received_events_url":"https://api.github.com/users/saurfang/received_events","type":"User","site_admin":false},"body":"This is a huge one. Thanks for reviewing it in advance. You might want to examine commit by commit and also add `&w=1` or `?w=1` to the URL to ignore white spaces in the diff due to reformatting.\r\n\r\nAside a few minor optimizations mentioned in the commits, the major changes are:\r\n1. Add `SparseLabeledPoint` for space efficient representation of feature vector\r\n2. Add validation split into model trainer for better understanding of model convergence\r\n\r\n## Sparse Representation\r\nIt shall be noted this implementation is not ideal in any sense but as a compromise to best fit in Aerosolve existing architecture without rewriting all the transforms.\r\n\r\n`SparseLabeledPoint` store feature vector in the form of\r\n```java\r\n    public final int[] indices;\r\n    public final float[] values;\r\n    public final int[] denseIndices;\r\n    public final float[][] denseValues;\r\n```\r\nwhere float and string features are stored in `indices,values` and dense features are stored in `denseIndices,denseValues`. The index of each feature is backed by `featureIndexer` in AdditiveModel of type `Map<String, Map<String, Integer>>`.\r\n\r\nSo instead of\r\n```\r\nMap(\r\n  'family1' -> Map(\r\n    `feature1' -> value1,\r\n    `feature4' -> value2,\r\n   ...),\r\n  ...\r\n)\r\n```\r\nwe store features as\r\n```\r\nindices: [1, 4, ...]\r\nvalues: [1, 2, ...]\r\n```\r\nFurthermore, to support efficient model updates with sparsely represented feature vector, we collect all features into `Function[] weightVector` where features are directly indexed by their corresponding index in the `featureIndexer`.\r\n\r\n## Validation Split\r\nBecause SGD loss is rather jittery in short window frame and we perform averaging of model weights among other regularizations during each iterations, training loss appears to be a poor proxy to model convergence.\r\nThis PR adds an optional `isTraining` function to split training data into training and validation. After each iteration, validation loss is calculated without updating the model itself.\r\nThis gives a much clearer guidance to both learning rate specification, its appropriate decay rate, early stopping, and if model of any iterations might have been overfitted.\r\nThe final model is picked from the iteration with lowest validation loss instead.\r\n\r\n## Speed Optimization\r\nWith validation loss, we were able to experiment with parameters and sampling technique with much better confidence. \r\nThis PR also adds option to `sample_by_partition` which take a sample of partitions. This played nicely when there are lots of partitions and data is randomly distributed when they are pulled. It appears model can converge much faster when a smaller sample is chosen and run with a few more iterations.\r\n\r\n\r\n@deerzq @jq @timyitong\r\n\r\np.s. I will update hackpad once this is merged to outline a few techniques and findings.\r\n","created_at":"2016-07-18T19:57:43Z","updated_at":"2016-07-19T03:40:27Z","closed_at":null,"merged_at":null,"merge_commit_sha":"206fd3fd9f1d99163a09d3e75cce6f8e46d47ddb","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/airbnb/aerosolve/pulls/229/commits","review_comments_url":"https://api.github.com/repos/airbnb/aerosolve/pulls/229/comments","review_comment_url":"https://api.github.com/repos/airbnb/aerosolve/pulls/comments{/number}","comments_url":"https://api.github.com/repos/airbnb/aerosolve/issues/229/comments","statuses_url":"https://api.github.com/repos/airbnb/aerosolve/statuses/f4032b826bd8ec5950111316e5b1d2562c092182","head":{"label":"airbnb:additive_optim","ref":"additive_optim","sha":"f4032b826bd8ec5950111316e5b1d2562c092182","user":{"login":"airbnb","id":698437,"avatar_url":"https://avatars.githubusercontent.com/u/698437?v=3","gravatar_id":"","url":"https://api.github.com/users/airbnb","html_url":"https://github.com/airbnb","followers_url":"https://api.github.com/users/airbnb/followers","following_url":"https://api.github.com/users/airbnb/following{/other_user}","gists_url":"https://api.github.com/users/airbnb/gists{/gist_id}","starred_url":"https://api.github.com/users/airbnb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airbnb/subscriptions","organizations_url":"https://api.github.com/users/airbnb/orgs","repos_url":"https://api.github.com/users/airbnb/repos","events_url":"https://api.github.com/users/airbnb/events{/privacy}","received_events_url":"https://api.github.com/users/airbnb/received_events","type":"Organization","site_admin":false},"repo":{"id":35507603,"name":"aerosolve","full_name":"airbnb/aerosolve","owner":{"login":"airbnb","id":698437,"avatar_url":"https://avatars.githubusercontent.com/u/698437?v=3","gravatar_id":"","url":"https://api.github.com/users/airbnb","html_url":"https://github.com/airbnb","followers_url":"https://api.github.com/users/airbnb/followers","following_url":"https://api.github.com/users/airbnb/following{/other_user}","gists_url":"https://api.github.com/users/airbnb/gists{/gist_id}","starred_url":"https://api.github.com/users/airbnb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airbnb/subscriptions","organizations_url":"https://api.github.com/users/airbnb/orgs","repos_url":"https://api.github.com/users/airbnb/repos","events_url":"https://api.github.com/users/airbnb/events{/privacy}","received_events_url":"https://api.github.com/users/airbnb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/airbnb/aerosolve","description":"A machine learning package built for humans.","fork":false,"url":"https://api.github.com/repos/airbnb/aerosolve","forks_url":"https://api.github.com/repos/airbnb/aerosolve/forks","keys_url":"https://api.github.com/repos/airbnb/aerosolve/keys{/key_id}","collaborators_url":"https://api.github.com/repos/airbnb/aerosolve/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/airbnb/aerosolve/teams","hooks_url":"https://api.github.com/repos/airbnb/aerosolve/hooks","issue_events_url":"https://api.github.com/repos/airbnb/aerosolve/issues/events{/number}","events_url":"https://api.github.com/repos/airbnb/aerosolve/events","assignees_url":"https://api.github.com/repos/airbnb/aerosolve/assignees{/user}","branches_url":"https://api.github.com/repos/airbnb/aerosolve/branches{/branch}","tags_url":"https://api.github.com/repos/airbnb/aerosolve/tags","blobs_url":"https://api.github.com/repos/airbnb/aerosolve/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/airbnb/aerosolve/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/airbnb/aerosolve/git/refs{/sha}","trees_url":"https://api.github.com/repos/airbnb/aerosolve/git/trees{/sha}","statuses_url":"https://api.github.com/repos/airbnb/aerosolve/statuses/{sha}","languages_url":"https://api.github.com/repos/airbnb/aerosolve/languages","stargazers_url":"https://api.github.com/repos/airbnb/aerosolve/stargazers","contributors_url":"https://api.github.com/repos/airbnb/aerosolve/contributors","subscribers_url":"https://api.github.com/repos/airbnb/aerosolve/subscribers","subscription_url":"https://api.github.com/repos/airbnb/aerosolve/subscription","commits_url":"https://api.github.com/repos/airbnb/aerosolve/commits{/sha}","git_commits_url":"https://api.github.com/repos/airbnb/aerosolve/git/commits{/sha}","comments_url":"https://api.github.com/repos/airbnb/aerosolve/comments{/number}","issue_comment_url":"https://api.github.com/repos/airbnb/aerosolve/issues/comments{/number}","contents_url":"https://api.github.com/repos/airbnb/aerosolve/contents/{+path}","compare_url":"https://api.github.com/repos/airbnb/aerosolve/compare/{base}...{head}","merges_url":"https://api.github.com/repos/airbnb/aerosolve/merges","archive_url":"https://api.github.com/repos/airbnb/aerosolve/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/airbnb/aerosolve/downloads","issues_url":"https://api.github.com/repos/airbnb/aerosolve/issues{/number}","pulls_url":"https://api.github.com/repos/airbnb/aerosolve/pulls{/number}","milestones_url":"https://api.github.com/repos/airbnb/aerosolve/milestones{/number}","notifications_url":"https://api.github.com/repos/airbnb/aerosolve/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/airbnb/aerosolve/labels{/name}","releases_url":"https://api.github.com/repos/airbnb/aerosolve/releases{/id}","deployments_url":"https://api.github.com/repos/airbnb/aerosolve/deployments","created_at":"2015-05-12T19:11:57Z","updated_at":"2016-07-18T19:44:55Z","pushed_at":"2016-07-18T19:57:43Z","git_url":"git://github.com/airbnb/aerosolve.git","ssh_url":"git@github.com:airbnb/aerosolve.git","clone_url":"https://github.com/airbnb/aerosolve.git","svn_url":"https://github.com/airbnb/aerosolve","homepage":"http://airbnb.github.io/aerosolve/","size":5661,"stargazers_count":3240,"watchers_count":3240,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":380,"mirror_url":null,"open_issues_count":2,"forks":380,"open_issues":2,"watchers":3240,"default_branch":"master"}},"base":{"label":"airbnb:master","ref":"master","sha":"5092b3c2415055d6fbbf1d7df04e1567d9093fd0","user":{"login":"airbnb","id":698437,"avatar_url":"https://avatars.githubusercontent.com/u/698437?v=3","gravatar_id":"","url":"https://api.github.com/users/airbnb","html_url":"https://github.com/airbnb","followers_url":"https://api.github.com/users/airbnb/followers","following_url":"https://api.github.com/users/airbnb/following{/other_user}","gists_url":"https://api.github.com/users/airbnb/gists{/gist_id}","starred_url":"https://api.github.com/users/airbnb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airbnb/subscriptions","organizations_url":"https://api.github.com/users/airbnb/orgs","repos_url":"https://api.github.com/users/airbnb/repos","events_url":"https://api.github.com/users/airbnb/events{/privacy}","received_events_url":"https://api.github.com/users/airbnb/received_events","type":"Organization","site_admin":false},"repo":{"id":35507603,"name":"aerosolve","full_name":"airbnb/aerosolve","owner":{"login":"airbnb","id":698437,"avatar_url":"https://avatars.githubusercontent.com/u/698437?v=3","gravatar_id":"","url":"https://api.github.com/users/airbnb","html_url":"https://github.com/airbnb","followers_url":"https://api.github.com/users/airbnb/followers","following_url":"https://api.github.com/users/airbnb/following{/other_user}","gists_url":"https://api.github.com/users/airbnb/gists{/gist_id}","starred_url":"https://api.github.com/users/airbnb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airbnb/subscriptions","organizations_url":"https://api.github.com/users/airbnb/orgs","repos_url":"https://api.github.com/users/airbnb/repos","events_url":"https://api.github.com/users/airbnb/events{/privacy}","received_events_url":"https://api.github.com/users/airbnb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/airbnb/aerosolve","description":"A machine learning package built for humans.","fork":false,"url":"https://api.github.com/repos/airbnb/aerosolve","forks_url":"https://api.github.com/repos/airbnb/aerosolve/forks","keys_url":"https://api.github.com/repos/airbnb/aerosolve/keys{/key_id}","collaborators_url":"https://api.github.com/repos/airbnb/aerosolve/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/airbnb/aerosolve/teams","hooks_url":"https://api.github.com/repos/airbnb/aerosolve/hooks","issue_events_url":"https://api.github.com/repos/airbnb/aerosolve/issues/events{/number}","events_url":"https://api.github.com/repos/airbnb/aerosolve/events","assignees_url":"https://api.github.com/repos/airbnb/aerosolve/assignees{/user}","branches_url":"https://api.github.com/repos/airbnb/aerosolve/branches{/branch}","tags_url":"https://api.github.com/repos/airbnb/aerosolve/tags","blobs_url":"https://api.github.com/repos/airbnb/aerosolve/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/airbnb/aerosolve/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/airbnb/aerosolve/git/refs{/sha}","trees_url":"https://api.github.com/repos/airbnb/aerosolve/git/trees{/sha}","statuses_url":"https://api.github.com/repos/airbnb/aerosolve/statuses/{sha}","languages_url":"https://api.github.com/repos/airbnb/aerosolve/languages","stargazers_url":"https://api.github.com/repos/airbnb/aerosolve/stargazers","contributors_url":"https://api.github.com/repos/airbnb/aerosolve/contributors","subscribers_url":"https://api.github.com/repos/airbnb/aerosolve/subscribers","subscription_url":"https://api.github.com/repos/airbnb/aerosolve/subscription","commits_url":"https://api.github.com/repos/airbnb/aerosolve/commits{/sha}","git_commits_url":"https://api.github.com/repos/airbnb/aerosolve/git/commits{/sha}","comments_url":"https://api.github.com/repos/airbnb/aerosolve/comments{/number}","issue_comment_url":"https://api.github.com/repos/airbnb/aerosolve/issues/comments{/number}","contents_url":"https://api.github.com/repos/airbnb/aerosolve/contents/{+path}","compare_url":"https://api.github.com/repos/airbnb/aerosolve/compare/{base}...{head}","merges_url":"https://api.github.com/repos/airbnb/aerosolve/merges","archive_url":"https://api.github.com/repos/airbnb/aerosolve/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/airbnb/aerosolve/downloads","issues_url":"https://api.github.com/repos/airbnb/aerosolve/issues{/number}","pulls_url":"https://api.github.com/repos/airbnb/aerosolve/pulls{/number}","milestones_url":"https://api.github.com/repos/airbnb/aerosolve/milestones{/number}","notifications_url":"https://api.github.com/repos/airbnb/aerosolve/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/airbnb/aerosolve/labels{/name}","releases_url":"https://api.github.com/repos/airbnb/aerosolve/releases{/id}","deployments_url":"https://api.github.com/repos/airbnb/aerosolve/deployments","created_at":"2015-05-12T19:11:57Z","updated_at":"2016-07-18T19:44:55Z","pushed_at":"2016-07-18T19:57:43Z","git_url":"git://github.com/airbnb/aerosolve.git","ssh_url":"git@github.com:airbnb/aerosolve.git","clone_url":"https://github.com/airbnb/aerosolve.git","svn_url":"https://github.com/airbnb/aerosolve","homepage":"http://airbnb.github.io/aerosolve/","size":5661,"stargazers_count":3240,"watchers_count":3240,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":380,"mirror_url":null,"open_issues_count":2,"forks":380,"open_issues":2,"watchers":3240,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/airbnb/aerosolve/pulls/229"},"html":{"href":"https://github.com/airbnb/aerosolve/pull/229"},"issue":{"href":"https://api.github.com/repos/airbnb/aerosolve/issues/229"},"comments":{"href":"https://api.github.com/repos/airbnb/aerosolve/issues/229/comments"},"review_comments":{"href":"https://api.github.com/repos/airbnb/aerosolve/pulls/229/comments"},"review_comment":{"href":"https://api.github.com/repos/airbnb/aerosolve/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/airbnb/aerosolve/pulls/229/commits"},"statuses":{"href":"https://api.github.com/repos/airbnb/aerosolve/statuses/f4032b826bd8ec5950111316e5b1d2562c092182"}}}}
{"actor":{"display_login":"jq"}}