{"action":"created","issue":{"url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/272","repository_url":"https://api.github.com/repos/edx/edx-analytics-pipeline","labels_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/272/labels{/name}","comments_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/272/comments","events_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/272/events","html_url":"https://github.com/edx/edx-analytics-pipeline/pull/272","id":168255149,"number":272,"title":"Appends 10 auth_userprofile fields to ModuleEngagementRosterRecord","user":{"login":"pomegranited","id":7556571,"avatar_url":"https://avatars.githubusercontent.com/u/7556571?v=3","gravatar_id":"","url":"https://api.github.com/users/pomegranited","html_url":"https://github.com/pomegranited","followers_url":"https://api.github.com/users/pomegranited/followers","following_url":"https://api.github.com/users/pomegranited/following{/other_user}","gists_url":"https://api.github.com/users/pomegranited/gists{/gist_id}","starred_url":"https://api.github.com/users/pomegranited/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pomegranited/subscriptions","organizations_url":"https://api.github.com/users/pomegranited/orgs","repos_url":"https://api.github.com/users/pomegranited/repos","events_url":"https://api.github.com/users/pomegranited/events{/privacy}","received_events_url":"https://api.github.com/users/pomegranited/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/edx/edx-analytics-pipeline/labels/needs%20triage","name":"needs triage","color":"e11d21"},{"url":"https://api.github.com/repos/edx/edx-analytics-pipeline/labels/open-source-contribution","name":"open-source-contribution","color":"bfe5bf"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-07-29T05:45:58Z","updated_at":"2016-07-29T06:20:20Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/edx/edx-analytics-pipeline/pulls/272","html_url":"https://github.com/edx/edx-analytics-pipeline/pull/272","diff_url":"https://github.com/edx/edx-analytics-pipeline/pull/272.diff","patch_url":"https://github.com/edx/edx-analytics-pipeline/pull/272.patch"},"body":"Adds the following fields from the `auth_userprofile` table to the `ModuleEngagementRosterRecord`:\r\n\r\n* user_id\r\n* language\r\n* location\r\n* year_of_birth\r\n* level_of_education\r\n* gender\r\n* mailing_address\r\n* city\r\n* country\r\n* goals\r\n\r\nAppends the new fields to the end of the hive table to facilitate migrations for persistent clusters.\r\n\r\n**Discussions**: This change is related to [edx-analytics-data-api PR #128](https://github.com/edx/edx-analytics-data-api/pull/128), which has been discussed extensively with the Analytics team.\r\n\r\n**Dependencies**: None\r\n\r\n**Sandbox URL**: TBD - sandbox is being provisioned.\r\n\r\n**Partner information**: 3rd party-hosted open edX instance\r\n\r\n**Testing instructions**:\r\n\r\n1. See [edx-analytics-data-api PR #128](https://github.com/edx/edx-analytics-data-api/pull/128) for testing setup instructions.  \r\n1. If you want to see the data flow through to the Analytics API, install the branch from [edx-analytics-data-api PR #129 ](https://github.com/edx/edx-analytics-data-api/pull/129) and restart the analytics API.\r\n1. Reinstall the analytics pipeline from this PR branch on vagrant instance under the previously-installed location:\r\n\r\n    ```\r\nssh vagrant\r\ncd /var/lib/analytics-tasks/analyticstack/repo\r\nsource /var/lib/analytics-tasks/analyticstack/venv/bin/activate\r\ngit remote add pr https://github.com/open-craft/edx-analytics-data-api.git\r\ngit fetch pr\r\ngit checkout jill/module-engagement-add-fields\r\nmake bootstrap\r\n```\r\n1. To preserve your existing hdfs data, run these hive SQL queries.\r\n\r\n    *NB* This step is only required for persistent clusters, like the analyticstack.  Most production analytics clusters are run using transient clusters.  To simulate the transient cluster effect, DROP the following tables instead of ALTERing them.\r\n\r\n    ```\r\nsudo su hadoop\r\nhive    \r\nhive> ALTER TABLE module_engagement_roster\r\nADD COLUMNS (\r\n    user_id int,\r\n    language string,\r\n    location string,\r\n    year_of_birth int,\r\n    level_of_education string,\r\n    gender string,\r\n    mailing_address string,\r\n    city string,\r\n    country string,\r\n    goals string\r\n);\r\n\r\nhive> ALTER TABLE auth_userprofile\r\nADD COLUMNS (\r\n    language string,\r\n    location string,\r\n    mailing_address string,\r\n    city string,\r\n    country string,\r\n    goals string\r\n);\r\n```\r\n\r\n1. Re-run the pipeline tasks, using the `--overwrite` flag to overwrite existing data.\r\n\r\n    *NB* @haikuginger I made a mistake with the initial PR description, and used `ModuleEngagementWorkflowTask --date $TOMORROW`.  This doesn't work, as the query that constructs this table relies on today's data from the `auth_userprofile` table.\r\n\r\n    ```\r\nsudo su hadoop\r\nsource /var/lib/analytics-tasks/analyticstack/venv/bin/activate\r\nTODAY=`date +%Y-%m-%d`\r\nTOMORROW=`date --date=\"tomorrow\" +%Y-%m-%d`\r\n\r\nlaunch-task ImportEnrollmentsIntoMysql \\\r\n    --interval 2016-07-01-$TOMORROW \\\r\n    --local-scheduler --n-reduce-tasks 2 \\\r\n    --remote-log-level DEBUG\r\n\r\nlaunch-task ModuleEngagementWorkflowTask \\\r\n     --local-scheduler --n-reduce-tasks 2 \\\r\n     --remote-log-level DEBUG \\\r\n     --date $TODAY --overwrite  # NOTE\r\n```\r\n1. Check the ElasticSearch index to ensure the new fields are populated:\r\n\r\n    ```\r\nhttp://localhost:9223/roster/_search/?course_id=<your URL-encoded course ID>\r\n```\r\n1. Save changes to your test user profiles from the `Accounts` page.\r\n1. Wait 1 min for tracking logs to be copied to hdfs (cron'd).\r\n1. Re-run steps 3-5 to ensure the updated data is visible in the hive table.\r\n\r\n**Author Notes and Concerns**:\r\n\r\n1. The `auth_userprofile.language` field doesn't seem to be populated by the Accounts page anymore; instead it feeds a one-to-many relationship with `student_languageproficiency`.  Will need to discuss with the client whether this field is required, and if so, import the `student_languageproficiency` into hive and add a `JOIN` to the roster query.\r\n1. The `auth_userprofile.location` field doesn't seem to be populated by the Accounts page anymore either.  It may be deprecated; will discuss with client.\r\n\r\n**Reviewers**\r\n- [ ] @haikuginger\r\n- [ ] edX reviewer[s] TBD\r\n"},"comment":{"url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/comments/236103634","html_url":"https://github.com/edx/edx-analytics-pipeline/pull/272#issuecomment-236103634","issue_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/272","id":236103634,"user":{"login":"edx-webhook","id":8964148,"avatar_url":"https://avatars.githubusercontent.com/u/8964148?v=3","gravatar_id":"","url":"https://api.github.com/users/edx-webhook","html_url":"https://github.com/edx-webhook","followers_url":"https://api.github.com/users/edx-webhook/followers","following_url":"https://api.github.com/users/edx-webhook/following{/other_user}","gists_url":"https://api.github.com/users/edx-webhook/gists{/gist_id}","starred_url":"https://api.github.com/users/edx-webhook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edx-webhook/subscriptions","organizations_url":"https://api.github.com/users/edx-webhook/orgs","repos_url":"https://api.github.com/users/edx-webhook/repos","events_url":"https://api.github.com/users/edx-webhook/events{/privacy}","received_events_url":"https://api.github.com/users/edx-webhook/received_events","type":"User","site_admin":false},"created_at":"2016-07-29T06:20:20Z","updated_at":"2016-07-29T06:20:20Z","body":"Thanks for the pull request, @pomegranited! I've created [OSPR-1366](https://openedx.atlassian.net/browse/OSPR-1366) to keep track of it in JIRA. JIRA is a place for product owners to prioritize feature reviews by the engineering development teams.\n\nFeel free to add as much of the following information to the ticket:\n- supporting documentation\n- edx-code email threads\n- timeline information (\"this must be merged by XX date\", and why that is)\n- partner information (\"this is a course on edx.org\")\n- any other information that can help Product understand the context for the PR\n\nAll technical communication about the code itself will still be done via the GitHub pull request interface. As a reminder, [our process documentation is here](http://edx-developer-guide.readthedocs.org/en/latest/process/overview.html).\n\n\n\nIf you like, you can add yourself to the [AUTHORS](https://github.com/edx/edx-analytics-pipeline/blob/master/AUTHORS)      file for this repo, though that isn't required.     Please see the [CONTRIBUTING](https://github.com/edx/edx-platform/blob/master/CONTRIBUTING.rst)     file for more information."}}
{"actor":{"display_login":"edx-webhook"}}