{"action":"closed","issue":{"url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/issues/1371","repository_url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data","labels_url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/issues/1371/labels{/name}","comments_url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/issues/1371/comments","events_url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/issues/1371/events","html_url":"https://github.com/california-civic-data-coalition/django-calaccess-raw-data/issues/1371","id":148689608,"number":1371,"title":"Add choices and documentcloud_pages to FilerInterestsCd.interest_cd (in other.py)","user":{"login":"gordonje","id":4304548,"avatar_url":"https://avatars.githubusercontent.com/u/4304548?v=3","gravatar_id":"","url":"https://api.github.com/users/gordonje","html_url":"https://github.com/gordonje","followers_url":"https://api.github.com/users/gordonje/followers","following_url":"https://api.github.com/users/gordonje/following{/other_user}","gists_url":"https://api.github.com/users/gordonje/gists{/gist_id}","starred_url":"https://api.github.com/users/gordonje/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gordonje/subscriptions","organizations_url":"https://api.github.com/users/gordonje/orgs","repos_url":"https://api.github.com/users/gordonje/repos","events_url":"https://api.github.com/users/gordonje/events{/privacy}","received_events_url":"https://api.github.com/users/gordonje/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/labels/documentation","name":"documentation","color":"eb6420"},{"url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/labels/small","name":"small","color":"207de5"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-04-15T15:23:54Z","updated_at":"2016-04-15T16:34:52Z","closed_at":"2016-04-15T16:34:52Z","body":"## Overview\n\nBasically, we think **interest_cd** should be defined as a \"choice field\" (in Django parlance), which just means it's a field with a defined list of valid values and an intelligible definition for each value. \n\nOur goal is for every potential choice field to include references to locations in the [official documentation](http://django-calaccess-raw-data.californiacivicdata.org/en/latest/officialdocumentation.html) where the field's valid values and their meanings are described AND to have those choices included in the field's definition.\n\nAnd we need your help!\n\n## What to do\n\n**Step 1**: Claim this ticket by adding yourself as an **Assignee** (to the right)\n\n**Step 2**: Find where in the [official documentation](http://django-calaccess-raw-data.californiacivicdata.org/en/latest/officialdocumentation.html) interest_cd's list of valid values is defined. \n\nThe best place to look first is in the .CAL layout documents: [.CAL Format version 2.01](http://www.documentcloud.org/documents/2712034-Cal-Format-201.html#document/p1) and [.CAL Format version 1.05.02](http://www.documentcloud.org/documents/2712033-Cal-Format-1-05-02.html#document/p1). These are documents that describe the layout of the .CAL file format, which is the required electronic format of any filings ingested by the CAL-ACCESS system (more on that [here](http://django-calaccess-raw-data.californiacivicdata.org/en/latest/officialdocumentation.html#cal-file-format) if you are interested).\n\nYou can start by opening the document and just searching for \"interest_cd\". For example, if you were assigned to tackle the stmt_type field on the CvrCampaignDisclosureCd model, you would be looking for something like this::\n\n![Docs](https://dl.dropboxusercontent.com/s/jue8oa2btod3i0v/stmt_type.png?dl=0)\n\nBut note that the same column might appear on multiple tables, and the list of valid values may vary depending on the table. We want to be sure to find the list of valid values for the interest_cd on the FilerInterestsCd model.\n\nAnother good place to look is in the [MapCalFormat2Fields](http://www.documentcloud.org/documents/2711616-MapCalFormat2Fields.html#document/p1) document, which describes how .CAL format fields are mapped to CAL-ACCESS database table columns. Note the _Valid_V_ column on the far right.\n\n**Step 3**: Copy the DocumentCloud ID and the page number(s) where interest_cd's list of valid values is defined.\n\nThe DocumentCloud ID can be found the in the URL that points to the document. For example, the url to the .CAL Format version 2.01 doc is `http://www.documentcloud.org/documents/2712034-Cal-Format-201.html#document/p1`, and the id is `2712034-Cal-Format-201`. \n\nAnd the start page we want to grab is the DocumentCloud page number, not the page number printed on the .PDF. If the list of valid values spans multiple pages, then grab the page where the list ends as well.\n\n**Step 4**: Find where interest_cd is defined in other.py in the [/calaccess_raw/models/other.py](https://github.com/california-civic-data-coalition/django-calaccess-raw-data/blob/master/calaccess_raw/models/other.py) file underneath the FilerInterestsCd class.\n\nSince the model files are rather long, take care to be sure you find the where interest_cd is defined for the FilerInterestsCd model. It should look something like this:\n```python\n    class FilerInterestsCd(CalAccessBaseModel):\n        # ...\n        # several lines of code later...\n        # ...\n        interest_cd = fields.IntegerField(\n            max_length=None,\n            db_column='INTEREST_CD',\n            blank=True,\n            verbose_name='interest code',\n            help_text='',\n        )\n```\n**Step 5**: Add the document references\n\nSay you found the list of valid values listed in three places. You would list them in the `documentcloud_pages` attribute like this:\n```python\n    class FilerInterestsCd(CalAccessBaseModel):\n        # ...\n        # several lines of code later...\n        # ...\n        interest_cd = fields.IntegerField(\n            max_length=None,\n            db_column='INTEREST_CD',\n            blank=True,\n            verbose_name='interest code',\n            help_text='',\n            documentcloud_pages=[\n                DocumentCloud(id='2711616-MapCalFormat2Fields', start_page=10),\n                DocumentCloud(id='2712033-Cal-Format-1-05-02', start_page=19),\n                DocumentCloud(id='2712034-Cal-Format-201', start_page=24, end_page=25),\n            ]\n        )\n```\nNote: You only need to specify the `end_page` if the list spans multiple pages.\n**Step 6**: Update the interest_cd's `help_text` (if necessary)\n\nIt might be missing or it might be an empty string or it might not be consistent with what you see in the documentation. Note: The entire help_text string needs to be enclosed in single- or double-quotes like this:\n```python\n    help_text='This is the help text',\n```\n\nIf the help text is really long (80+ characters), then it needs to span multiple lines like this:\n```python\n    help_text='This is the help text...blah...blah...blah...blah...blah...blah...\\ \n...blah...blah...blah',\n```\n\n**If you can,** check the interest_cd's defined choices against the actual values in interest_cd's database column.\n\nThis will require you to install the app and run the `updatecalaccessrawdata` command. Then you can get the distinct column values with a `GROUP BY` query like this:\n```sql\n    SELECT INTEREST_CD, COUNT(*)\n    FROM FILER_INTERESTS_CD\n    GROUP BY 1\n    ORDER BY 1;\n```\nIt's likely that you'll find undocumented values in database column, some of which are variants of the valid values. You can note these by repeating the same definition for each variant like this:\n```python\n    OFFICE_CD_CHOICES = (\n        ('GOV', 'Governor'),\n        ('gov', 'Governor'),\n        ('GUV', 'Governor'),\n        # ...\n    ),\n```\nIf you're dealing with a set of choices that's been consolidated into [calaccess_raw/annotations/choices.py](https://github.com/california-civic-data-coalition/django-calaccess-raw-data/blob/master/calaccess_raw/annotations/choices.py), you can map the different variants to the same definition using the key of the choices dict like this:\n\n```python\n    OFFICE_CD_CHOICES = (\n        ('gov', look_ups.OFFICE_CODES['GOV']),\n        ('GUV', look_ups.OFFICE_CODES['GOV']),\n        # ...\n    ),\n```\nIf you're not able to check the database column values for whatever reason, don't sweat it. We'll deal with it in a later issue.\n\n**Wrap up**: Review your changes and create a pull request \n\n![Pull request](https://dl.dropboxusercontent.com/u/3640647/ScreenCloud/1440368058.52.png)\n\nThat's it!\n\n**PS**: If any of this feels confusing, or if it doesn't seem to match up with what you are seeing, or if you have any suggestions about another approach, don't hesitate to speak up. You can post a comment below, or hit up the #california-civic-data channel on the News Nerdery Slack.\n"}}