{"action":"created","issue":{"url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/issues/30","repository_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop","labels_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/issues/30/labels{/name}","comments_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/issues/30/comments","events_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/issues/30/events","html_url":"https://github.com/freedomofpress/FingerprintSecureDrop/pull/30","id":176098381,"number":30,"title":"Feature generation","user":{"login":"redshiftzero","id":7832803,"avatar_url":"https://avatars.githubusercontent.com/u/7832803?v=3","gravatar_id":"","url":"https://api.github.com/users/redshiftzero","html_url":"https://github.com/redshiftzero","followers_url":"https://api.github.com/users/redshiftzero/followers","following_url":"https://api.github.com/users/redshiftzero/following{/other_user}","gists_url":"https://api.github.com/users/redshiftzero/gists{/gist_id}","starred_url":"https://api.github.com/users/redshiftzero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/redshiftzero/subscriptions","organizations_url":"https://api.github.com/users/redshiftzero/orgs","repos_url":"https://api.github.com/users/redshiftzero/repos","events_url":"https://api.github.com/users/redshiftzero/events{/privacy}","received_events_url":"https://api.github.com/users/redshiftzero/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-09-09T20:11:29Z","updated_at":"2016-09-21T20:27:07Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/pulls/30","html_url":"https://github.com/freedomofpress/FingerprintSecureDrop/pull/30","diff_url":"https://github.com/freedomofpress/FingerprintSecureDrop/pull/30.diff","patch_url":"https://github.com/freedomofpress/FingerprintSecureDrop/pull/30.patch"},"body":"The PR adds feature generation code and tests. It takes data from the `raw` schema, generates all features relevant to Tor traffic from Wang et al. 2014, and stores the results in the `features` schema.\r\n\r\nOne can generate (or regenerate) all features using:\r\n\r\n```\r\n$ python3 features.py\r\n```\r\n\r\nMost of the feature generation code - with the exception of some preprocessing for the “burst” features - has been rewritten in SQL for speed. The burst preprocessing (in `compute_bursts()`) is done in Python and is thus slow (Issue #29 now exists to address this). Otherwise, each feature generation method is a Python wrapper for a SQL query. Where possible, the feature generation code is parametrized such that variants of the features used in Wang et al. 2014 can be easily generated. For example, instead of binning our traces into windows of size 30 and computing counts of outgoing cells in the first 100 bins, we may wish to vary both the window size and number of bins. As such, both of these parameters are passed to the feature generation method for this feature `generate_table_binned_counts(num_features=100, size_window=30)`. \r\n\r\nThe calling of these various functions is done in the `main()` function but I would not futz too much with that as all these tasks will be integrated with [a data science pipeline tool](https://github.com/spotify/luigi) to automate their execution upon some condition e.g. new rows entering the `raw` schema. \r\n\r\nThis PR also adds tests for each feature generation function (both Python or SQL using a test database) in `test/test_features.py`. This is implemented by using a testing database that has the same structure as the production database but is empty. Each test inserts some rows into the test database during `setUp()`, runs the test, and then removes those rows and any tables that have been created by the feature generation function during `tearDown()`. Run all tests with:\r\n\r\n```\r\n$ python3 -m unittest -v test/test_features.py\r\n```\r\n\r\nA couple of packages are added to the requirements files (in `ml-requirements`):\r\n\r\n* `pandas`, a general data analysis library\r\n* `tqdm`, a nice progress bar used to display progress on the burst generation computation\r\n\r\n\r\nFinally, details of how the test database is set up are written out in `test_db_setup.md` which we can add to be configured in the vagrant setup after branch `sorter-db-integration` is merged. \r\n"},"comment":{"url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/issues/comments/248732668","html_url":"https://github.com/freedomofpress/FingerprintSecureDrop/pull/30#issuecomment-248732668","issue_url":"https://api.github.com/repos/freedomofpress/FingerprintSecureDrop/issues/30","id":248732668,"user":{"login":"redshiftzero","id":7832803,"avatar_url":"https://avatars.githubusercontent.com/u/7832803?v=3","gravatar_id":"","url":"https://api.github.com/users/redshiftzero","html_url":"https://github.com/redshiftzero","followers_url":"https://api.github.com/users/redshiftzero/followers","following_url":"https://api.github.com/users/redshiftzero/following{/other_user}","gists_url":"https://api.github.com/users/redshiftzero/gists{/gist_id}","starred_url":"https://api.github.com/users/redshiftzero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/redshiftzero/subscriptions","organizations_url":"https://api.github.com/users/redshiftzero/orgs","repos_url":"https://api.github.com/users/redshiftzero/repos","events_url":"https://api.github.com/users/redshiftzero/events{/privacy}","received_events_url":"https://api.github.com/users/redshiftzero/received_events","type":"User","site_admin":false},"created_at":"2016-09-21T20:27:07Z","updated_at":"2016-09-21T20:27:07Z","body":"Ok so after discussion, the TODOs in `fpsd/test_db_setup.md` need to be converted to Ansible tasks (and it looks like since this PR was submitted some of them now do exist in master so that's good), but let me do that before review of this PR. "}}
{"actor":{"display_login":"redshiftzero"}}