{"action":"created","comment":{"url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/pulls/comments/84765152","pull_request_review_id":5522174,"id":84765152,"diff_hunk":"@@ -0,0 +1,768 @@\n+#!/usr/bin/env python3.5\n+import os\n+import pandas as pd\n+import pdb","path":"fpsd/features.py","position":4,"original_position":4,"commit_id":"b768850162266ece3607656e68c39a76e6a1b430","original_commit_id":"b768850162266ece3607656e68c39a76e6a1b430","user":{"login":"fowlslegs","id":3538418,"avatar_url":"https://avatars.githubusercontent.com/u/3538418?v=3","gravatar_id":"","url":"https://api.github.com/users/fowlslegs","html_url":"https://github.com/fowlslegs","followers_url":"https://api.github.com/users/fowlslegs/followers","following_url":"https://api.github.com/users/fowlslegs/following{/other_user}","gists_url":"https://api.github.com/users/fowlslegs/gists{/gist_id}","starred_url":"https://api.github.com/users/fowlslegs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fowlslegs/subscriptions","organizations_url":"https://api.github.com/users/fowlslegs/orgs","repos_url":"https://api.github.com/users/fowlslegs/repos","events_url":"https://api.github.com/users/fowlslegs/events{/privacy}","received_events_url":"https://api.github.com/users/fowlslegs/received_events","type":"User","site_admin":false},"body":"Sorry, this import too.","created_at":"2016-10-24T19:31:35Z","updated_at":"2016-10-24T19:31:35Z","html_url":"https://github.com/freedomofpress/FingerprintSecureDrop/pull/30#discussion_r84765152","pull_request_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/pulls/30","_links":{"self":{"href":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/pulls/comments/84765152"},"html":{"href":"https://github.com/freedomofpress/FingerprintSecureDrop/pull/30#discussion_r84765152"},"pull_request":{"href":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/pulls/30"}}},"pull_request":{"url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/pulls/30","id":84711419,"html_url":"https://github.com/freedomofpress/FingerprintSecureDrop/pull/30","diff_url":"https://github.com/freedomofpress/FingerprintSecureDrop/pull/30.diff","patch_url":"https://github.com/freedomofpress/FingerprintSecureDrop/pull/30.patch","issue_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/issues/30","number":30,"state":"open","locked":false,"title":"Feature generation","user":{"login":"redshiftzero","id":7832803,"avatar_url":"https://avatars.githubusercontent.com/u/7832803?v=3","gravatar_id":"","url":"https://api.github.com/users/redshiftzero","html_url":"https://github.com/redshiftzero","followers_url":"https://api.github.com/users/redshiftzero/followers","following_url":"https://api.github.com/users/redshiftzero/following{/other_user}","gists_url":"https://api.github.com/users/redshiftzero/gists{/gist_id}","starred_url":"https://api.github.com/users/redshiftzero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/redshiftzero/subscriptions","organizations_url":"https://api.github.com/users/redshiftzero/orgs","repos_url":"https://api.github.com/users/redshiftzero/repos","events_url":"https://api.github.com/users/redshiftzero/events{/privacy}","received_events_url":"https://api.github.com/users/redshiftzero/received_events","type":"User","site_admin":false},"body":"The PR adds feature generation code and tests. It takes data from the `raw` schema, generates all features relevant to Tor traffic from Wang et al. 2014, and stores the results in the `features` schema.\r\n\r\nOne can generate (or regenerate) all features using:\r\n\r\n```\r\n$ python3 features.py\r\n```\r\n\r\nMost of the feature generation code - with the exception of some preprocessing for the “burst” features - has been rewritten in SQL for speed. The burst preprocessing (in `compute_bursts()`) is done in Python and is thus slow (Issue #29 now exists to address this). Otherwise, each feature generation method is a Python wrapper for a SQL query. Where possible, the feature generation code is parametrized such that variants of the features used in Wang et al. 2014 can be easily generated. For example, instead of binning our traces into windows of size 30 and computing counts of outgoing cells in the first 100 bins, we may wish to vary both the window size and number of bins. As such, both of these parameters are passed to the feature generation method for this feature `generate_table_binned_counts(num_features=100, size_window=30)`. \r\n\r\nThe calling of these various functions is done in the `main()` function but I would not futz too much with that as all these tasks will be integrated with [a data science pipeline tool](https://github.com/spotify/luigi) to automate their execution upon some condition e.g. new rows entering the `raw` schema. \r\n\r\nThis PR also adds tests for each feature generation function (both Python or SQL using a test database) in `test/test_features.py`. This is implemented by using a testing database that has the same structure as the production database but is empty. Each test inserts some rows into the test database during `setUp()`, runs the test, and then removes those rows and any tables that have been created by the feature generation function during `tearDown()`. Run all tests with:\r\n\r\n```\r\n$ python3 -m unittest -v test/test_features.py\r\n```\r\n\r\nA couple of packages are added to the requirements files (in `ml-requirements`):\r\n\r\n* `pandas`, a general data analysis library\r\n* `tqdm`, a nice progress bar used to display progress on the burst generation computation\r\n\r\n\r\nFinally, details of how the test database is set up are written out in `test_db_setup.md` which we can add to be configured in the vagrant setup after branch `sorter-db-integration` is merged. \r\n","created_at":"2016-09-09T20:11:29Z","updated_at":"2016-10-24T19:31:35Z","closed_at":null,"merged_at":null,"merge_commit_sha":"77c3f6094f0f059135d7a6a092e263d45910515c","assignee":{"login":"conorsch","id":657862,"avatar_url":"https://avatars.githubusercontent.com/u/657862?v=3","gravatar_id":"","url":"https://api.github.com/users/conorsch","html_url":"https://github.com/conorsch","followers_url":"https://api.github.com/users/conorsch/followers","following_url":"https://api.github.com/users/conorsch/following{/other_user}","gists_url":"https://api.github.com/users/conorsch/gists{/gist_id}","starred_url":"https://api.github.com/users/conorsch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/conorsch/subscriptions","organizations_url":"https://api.github.com/users/conorsch/orgs","repos_url":"https://api.github.com/users/conorsch/repos","events_url":"https://api.github.com/users/conorsch/events{/privacy}","received_events_url":"https://api.github.com/users/conorsch/received_events","type":"User","site_admin":false},"assignees":[{"login":"conorsch","id":657862,"avatar_url":"https://avatars.githubusercontent.com/u/657862?v=3","gravatar_id":"","url":"https://api.github.com/users/conorsch","html_url":"https://github.com/conorsch","followers_url":"https://api.github.com/users/conorsch/followers","following_url":"https://api.github.com/users/conorsch/following{/other_user}","gists_url":"https://api.github.com/users/conorsch/gists{/gist_id}","starred_url":"https://api.github.com/users/conorsch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/conorsch/subscriptions","organizations_url":"https://api.github.com/users/conorsch/orgs","repos_url":"https://api.github.com/users/conorsch/repos","events_url":"https://api.github.com/users/conorsch/events{/privacy}","received_events_url":"https://api.github.com/users/conorsch/received_events","type":"User","site_admin":false},{"login":"fowlslegs","id":3538418,"avatar_url":"https://avatars.githubusercontent.com/u/3538418?v=3","gravatar_id":"","url":"https://api.github.com/users/fowlslegs","html_url":"https://github.com/fowlslegs","followers_url":"https://api.github.com/users/fowlslegs/followers","following_url":"https://api.github.com/users/fowlslegs/following{/other_user}","gists_url":"https://api.github.com/users/fowlslegs/gists{/gist_id}","starred_url":"https://api.github.com/users/fowlslegs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fowlslegs/subscriptions","organizations_url":"https://api.github.com/users/fowlslegs/orgs","repos_url":"https://api.github.com/users/fowlslegs/repos","events_url":"https://api.github.com/users/fowlslegs/events{/privacy}","received_events_url":"https://api.github.com/users/fowlslegs/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/pulls/30/commits","review_comments_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/pulls/30/comments","review_comment_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/pulls/comments{/number}","comments_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/issues/30/comments","statuses_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/statuses/b768850162266ece3607656e68c39a76e6a1b430","head":{"label":"freedomofpress:feature-generation","ref":"feature-generation","sha":"b768850162266ece3607656e68c39a76e6a1b430","user":{"login":"freedomofpress","id":5388147,"avatar_url":"https://avatars.githubusercontent.com/u/5388147?v=3","gravatar_id":"","url":"https://api.github.com/users/freedomofpress","html_url":"https://github.com/freedomofpress","followers_url":"https://api.github.com/users/freedomofpress/followers","following_url":"https://api.github.com/users/freedomofpress/following{/other_user}","gists_url":"https://api.github.com/users/freedomofpress/gists{/gist_id}","starred_url":"https://api.github.com/users/freedomofpress/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freedomofpress/subscriptions","organizations_url":"https://api.github.com/users/freedomofpress/orgs","repos_url":"https://api.github.com/users/freedomofpress/repos","events_url":"https://api.github.com/users/freedomofpress/events{/privacy}","received_events_url":"https://api.github.com/users/freedomofpress/received_events","type":"Organization","site_admin":false},"repo":{"id":55802855,"name":"FingerprintSecureDrop","full_name":"freedomofpress/FingerprintSecureDrop","owner":{"login":"freedomofpress","id":5388147,"avatar_url":"https://avatars.githubusercontent.com/u/5388147?v=3","gravatar_id":"","url":"https://api.github.com/users/freedomofpress","html_url":"https://github.com/freedomofpress","followers_url":"https://api.github.com/users/freedomofpress/followers","following_url":"https://api.github.com/users/freedomofpress/following{/other_user}","gists_url":"https://api.github.com/users/freedomofpress/gists{/gist_id}","starred_url":"https://api.github.com/users/freedomofpress/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freedomofpress/subscriptions","organizations_url":"https://api.github.com/users/freedomofpress/orgs","repos_url":"https://api.github.com/users/freedomofpress/repos","events_url":"https://api.github.com/users/freedomofpress/events{/privacy}","received_events_url":"https://api.github.com/users/freedomofpress/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/freedomofpress/FingerprintSecureDrop","description":"Hardening the SecureDrop whistleblower platform against website fingerprinting attacks","fork":false,"url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop","forks_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/forks","keys_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/keys{/key_id}","collaborators_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/teams","hooks_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/hooks","issue_events_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/issues/events{/number}","events_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/events","assignees_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/assignees{/user}","branches_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/branches{/branch}","tags_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/tags","blobs_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/git/refs{/sha}","trees_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/git/trees{/sha}","statuses_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/statuses/{sha}","languages_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/languages","stargazers_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/stargazers","contributors_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/contributors","subscribers_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/subscribers","subscription_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/subscription","commits_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/commits{/sha}","git_commits_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/git/commits{/sha}","comments_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/comments{/number}","issue_comment_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/issues/comments{/number}","contents_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/contents/{+path}","compare_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/compare/{base}...{head}","merges_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/merges","archive_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/downloads","issues_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/issues{/number}","pulls_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/pulls{/number}","milestones_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/milestones{/number}","notifications_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/labels{/name}","releases_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/releases{/id}","deployments_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/deployments","created_at":"2016-04-08T19:01:22Z","updated_at":"2016-10-24T18:33:07Z","pushed_at":"2016-10-24T19:30:46Z","git_url":"git://github.com/freedomofpress/FingerprintSecureDrop.git","ssh_url":"git@github.com:freedomofpress/FingerprintSecureDrop.git","clone_url":"https://github.com/freedomofpress/FingerprintSecureDrop.git","svn_url":"https://github.com/freedomofpress/FingerprintSecureDrop","homepage":"","size":1217,"stargazers_count":11,"watchers_count":11,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":33,"forks":1,"open_issues":33,"watchers":11,"default_branch":"master"}},"base":{"label":"freedomofpress:master","ref":"master","sha":"4083aa61a64eb85e589be0ffa0cecb6143db9ae4","user":{"login":"freedomofpress","id":5388147,"avatar_url":"https://avatars.githubusercontent.com/u/5388147?v=3","gravatar_id":"","url":"https://api.github.com/users/freedomofpress","html_url":"https://github.com/freedomofpress","followers_url":"https://api.github.com/users/freedomofpress/followers","following_url":"https://api.github.com/users/freedomofpress/following{/other_user}","gists_url":"https://api.github.com/users/freedomofpress/gists{/gist_id}","starred_url":"https://api.github.com/users/freedomofpress/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freedomofpress/subscriptions","organizations_url":"https://api.github.com/users/freedomofpress/orgs","repos_url":"https://api.github.com/users/freedomofpress/repos","events_url":"https://api.github.com/users/freedomofpress/events{/privacy}","received_events_url":"https://api.github.com/users/freedomofpress/received_events","type":"Organization","site_admin":false},"repo":{"id":55802855,"name":"FingerprintSecureDrop","full_name":"freedomofpress/FingerprintSecureDrop","owner":{"login":"freedomofpress","id":5388147,"avatar_url":"https://avatars.githubusercontent.com/u/5388147?v=3","gravatar_id":"","url":"https://api.github.com/users/freedomofpress","html_url":"https://github.com/freedomofpress","followers_url":"https://api.github.com/users/freedomofpress/followers","following_url":"https://api.github.com/users/freedomofpress/following{/other_user}","gists_url":"https://api.github.com/users/freedomofpress/gists{/gist_id}","starred_url":"https://api.github.com/users/freedomofpress/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freedomofpress/subscriptions","organizations_url":"https://api.github.com/users/freedomofpress/orgs","repos_url":"https://api.github.com/users/freedomofpress/repos","events_url":"https://api.github.com/users/freedomofpress/events{/privacy}","received_events_url":"https://api.github.com/users/freedomofpress/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/freedomofpress/FingerprintSecureDrop","description":"Hardening the SecureDrop whistleblower platform against website fingerprinting attacks","fork":false,"url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop","forks_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/forks","keys_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/keys{/key_id}","collaborators_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/teams","hooks_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/hooks","issue_events_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/issues/events{/number}","events_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/events","assignees_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/assignees{/user}","branches_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/branches{/branch}","tags_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/tags","blobs_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/git/refs{/sha}","trees_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/git/trees{/sha}","statuses_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/statuses/{sha}","languages_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/languages","stargazers_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/stargazers","contributors_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/contributors","subscribers_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/subscribers","subscription_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/subscription","commits_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/commits{/sha}","git_commits_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/git/commits{/sha}","comments_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/comments{/number}","issue_comment_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/issues/comments{/number}","contents_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/contents/{+path}","compare_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/compare/{base}...{head}","merges_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/merges","archive_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/downloads","issues_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/issues{/number}","pulls_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/pulls{/number}","milestones_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/milestones{/number}","notifications_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/labels{/name}","releases_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/releases{/id}","deployments_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/deployments","created_at":"2016-04-08T19:01:22Z","updated_at":"2016-10-24T18:33:07Z","pushed_at":"2016-10-24T19:30:46Z","git_url":"git://github.com/freedomofpress/FingerprintSecureDrop.git","ssh_url":"git@github.com:freedomofpress/FingerprintSecureDrop.git","clone_url":"https://github.com/freedomofpress/FingerprintSecureDrop.git","svn_url":"https://github.com/freedomofpress/FingerprintSecureDrop","homepage":"","size":1217,"stargazers_count":11,"watchers_count":11,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":33,"forks":1,"open_issues":33,"watchers":11,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/pulls/30"},"html":{"href":"https://github.com/freedomofpress/FingerprintSecureDrop/pull/30"},"issue":{"href":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/issues/30"},"comments":{"href":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/issues/30/comments"},"review_comments":{"href":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/pulls/30/comments"},"review_comment":{"href":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/pulls/30/commits"},"statuses":{"href":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/statuses/b768850162266ece3607656e68c39a76e6a1b430"}}}}
{"actor":{"display_login":"fowlslegs"}}