{"action":"created","issue":{"url":"https://api.github.com/repos/open-source-economics/taxdata/issues/265","repository_url":"https://api.github.com/repos/open-source-economics/taxdata","labels_url":"https://api.github.com/repos/open-source-economics/taxdata/issues/265/labels{/name}","comments_url":"https://api.github.com/repos/open-source-economics/taxdata/issues/265/comments","events_url":"https://api.github.com/repos/open-source-economics/taxdata/issues/265/events","html_url":"https://github.com/open-source-economics/taxdata/pull/265","id":344144329,"node_id":"MDExOlB1bGxSZXF1ZXN0MjAzNjA5Mjc1","number":265,"title":"Different test results on pr-261-MH branch","user":{"login":"martinholmer","id":12170745,"node_id":"MDQ6VXNlcjEyMTcwNzQ1","avatar_url":"https://avatars1.githubusercontent.com/u/12170745?v=4","gravatar_id":"","url":"https://api.github.com/users/martinholmer","html_url":"https://github.com/martinholmer","followers_url":"https://api.github.com/users/martinholmer/followers","following_url":"https://api.github.com/users/martinholmer/following{/other_user}","gists_url":"https://api.github.com/users/martinholmer/gists{/gist_id}","starred_url":"https://api.github.com/users/martinholmer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martinholmer/subscriptions","organizations_url":"https://api.github.com/users/martinholmer/orgs","repos_url":"https://api.github.com/users/martinholmer/repos","events_url":"https://api.github.com/users/martinholmer/events{/privacy}","received_events_url":"https://api.github.com/users/martinholmer/received_events","type":"User","site_admin":false},"labels":[{"id":552692119,"node_id":"MDU6TGFiZWw1NTI2OTIxMTk=","url":"https://api.github.com/repos/open-source-economics/taxdata/labels/in%20progress","name":"in progress","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":24,"created_at":"2018-07-24T18:04:23Z","updated_at":"2018-07-27T14:39:00Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/open-source-economics/taxdata/pulls/265","html_url":"https://github.com/open-source-economics/taxdata/pull/265","diff_url":"https://github.com/open-source-economics/taxdata/pull/265.diff","patch_url":"https://github.com/open-source-economics/taxdata/pull/265.patch"},"body":"I'm getting different PUF test_data results on a local branch that includes the code changes in PR #261.\r\nHere is what I've done:\r\n1. Include all recent code changes on master branch in my local master branch\r\n2. Execute `make git-pr N=261`\r\n3. Execute `git merge master` to include recent master changes in the local copy of PR #261\r\n4. Execute `make puf_data/cps-matched-puf.csv`\r\n5. Execute `make puf_data/puf.csv`\r\n6. Execute `test_pufcsv_data`\r\n7. Copy  `puf_agg_actual.txt` to `puf_agg_expect.txt`\r\n\r\n@anderson, note the maximum `age_spouse` difference and others.\r\nHave I done something wrong?  Why are the test results different?\r\nCan it be that the `float_format='%.2f'` used to write the `cps-matched-puf.csv` file makes a difference?\r\nWhat version of pandas are you using?  I'm using 0.23.0 pandas.\r\nDo the `test_pufcsv_data` results I'm getting look \"fishy\" in any way?\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/open-source-economics/taxdata/issues/comments/408438546","html_url":"https://github.com/open-source-economics/taxdata/pull/265#issuecomment-408438546","issue_url":"https://api.github.com/repos/open-source-economics/taxdata/issues/265","id":408438546,"node_id":"MDEyOklzc3VlQ29tbWVudDQwODQzODU0Ng==","user":{"login":"martinholmer","id":12170745,"node_id":"MDQ6VXNlcjEyMTcwNzQ1","avatar_url":"https://avatars1.githubusercontent.com/u/12170745?v=4","gravatar_id":"","url":"https://api.github.com/users/martinholmer","html_url":"https://github.com/martinholmer","followers_url":"https://api.github.com/users/martinholmer/followers","following_url":"https://api.github.com/users/martinholmer/following{/other_user}","gists_url":"https://api.github.com/users/martinholmer/gists{/gist_id}","starred_url":"https://api.github.com/users/martinholmer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martinholmer/subscriptions","organizations_url":"https://api.github.com/users/martinholmer/orgs","repos_url":"https://api.github.com/users/martinholmer/repos","events_url":"https://api.github.com/users/martinholmer/events{/privacy}","received_events_url":"https://api.github.com/users/martinholmer/received_events","type":"User","site_admin":false},"created_at":"2018-07-27T14:39:00Z","updated_at":"2018-07-27T14:39:00Z","author_association":"CONTRIBUTOR","body":"@andersonfrailey said:\r\n> If I do, I think we can say with confidence that the `taxdata-dev` environment you and I are running are at the root cause of our troubles.\r\n\r\nI would not be surprised if this turns out to be true. \r\n\r\nBut I think you're not giving enough consideration to another possibility: that the Matching code (and there is a lot of it) has been written in a way that does not respect the differences between Python 2.7 and Python 3.6. \r\nI actually have proof, as of this morning, that this is a fact.  On my local branch of PR#261,  I generate a `puf.csv` file with this checksum:\r\n```\r\niMac2:taxdata mrh$ md5 puf_data/puf*csv\r\nMD5 (puf_data/puf-MH-0724.csv) = 33797d7ae7fc098fc8df468de7c17ce5\r\nMD5 (puf_data/puf-MH.csv) = 33797d7ae7fc098fc8df468de7c17ce5\r\n```\r\nBut when I add this line at the top of each Python file in the Matching subdirectory, I generate a different `puf.csv` file:\r\n```\r\nfrom __future__ import division\r\n```\r\nHere is the info on my newly generated (under Python 2.7) `puf.csv` file now that integer division produces a floating-point values as on Python 3.6:\r\n```\r\niMac2:taxdata mrh$ md5 puf_data/puf*csv\r\nMD5 (puf_data/puf-MH-0724.csv) = 33797d7ae7fc098fc8df468de7c17ce5\r\nMD5 (puf_data/puf-MH.csv) = 33797d7ae7fc098fc8df468de7c17ce5\r\nMD5 (puf_data/puf.csv) = 6cac486cbec54b5c0d5c378910672e69\r\n```\r\nWhy is the `puf.csv` file different?   Probably because some of the Matching code divides two integers and expects to get a floating-point result (which is what happens in Python 3.6).  But under Python 2.7 the result is integer division.   Here's some examples of lines of code that might fail in this regard:\r\n```\r\ngrep -nH -e \" / \" *py\r\nadd_cps_vars.py:25:    puffile['wt'] = puffile['s006'] / 100\r\ncps_rets.py:355:                record['130'] = wasp / float(totalwas)\r\ncps_rets.py:692:                        if income / float(totincx + income) < 0.5:\r\ncps_rets.py:775:            if indjs == 1 and float(totincx) / income > 0.99:\r\nphase1.py:70:        wageshr = was / tpi\r\nphase1.py:71:        capshr = (intst + texint + dbe) / tpi\r\nphase1.py:159:                                countx['SOI_wgt'] / countx['CPS_wgt'], 0)\r\nphase2.py:21:        factor = SOI['wt'].sum() / CPS['wt'].sum()\r\nsoi_rets.py:59:    wt = SOI['s006'] / 100\r\n```\r\nIt looks to me that the `S006` values in the `puf2011.csv` file are all integer.  So, the line above from the `add_cps_vars.py` file gives different results on Python 2.7 (without the future import) and Python 3.6.\r\nFor example, if puffile['s006'] is 2090, the Python 3.6 division result is 20.90 while under Python 2.7 it is the integer 20.\r\n\r\nThe fact that adding `from __future__ import division` did not produce the same MD5 checksum as from the docker environment suggests, to me, that there are 2.7-vs-3.6 incompatibilities in the Matching code other than integer division.\r\n\r\n@hdoupe \r\n"}}
{"actor":{"display_login":"martinholmer"}}