{"action":"created","comment":{"url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/pulls/comments/89704610","pull_request_review_id":10242545,"id":89704610,"diff_hunk":"@@ -152,36 +156,38 @@\n     \"    return dtypedict\\n\",\n     \"\\n\",\n     \"\\n\",\n-    \"def hpd_csv2sql(description, input_csv_file, sep_char, output_pickle,\\\\\\n\",\n+    \"def hpd_csv2sql(description, input_csv_url, sep_char,\\\\\\n\",\n     \"            table_name, dtype_dict, load_pickle, \\\\\\n\",\n-    \"                input_pickle, db_action, truncate_columns, date_time_columns,\\\\\\n\",\n+    \"                pickle_file, db_action, truncate_columns, date_time_columns,\\\\\\n\",\n     \"               chunk_size, keep_cols, max_col_len=255):\\n\",\n-    \"\\n\",\n+    \"    \\n\",\n     \"    log.info(\\\"Beginning {} Import {}\\\".format(description,datetime.datetime.now()))\\n\",\n     \"    \\n\",\n-    \"    if os.path.isfile(input_pickle):\\n\",\n-    \"        log.info(\\\"Flagged load of PICKLE: {} = True\\\".format(input_pickle))\\n\",\n+    \"    if load_pickle: #IF FLAGGED TO LOAD PICKLE AS TRUE\\n\",\n+    \"        log.info(\\\"Flagged load of PICKLE: {} = True\\\".format(pickle_file))\\n\",\n     \"        \\n\",\n-    \"        with open(input_pickle, 'r') as picklefile:\\n\",\n-    \"            log.info(\\\"Begin OPEN {} Pickle: {}\\\".format(input_pickle, datetime.datetime.now()))\\n\",\n-    \"            log.info(\\\"Great we have a pickle file...Loading from {}\\\".format(input_pickle))\\n\",\n+    \"        with open(pickle_file, 'r') as picklefile:\\n\",\n+    \"            log.info(\\\"Begin OPEN {} Pickle: {}\\\".format(pickle_file, datetime.datetime.now()))\\n\",\n+    \"            log.info(\\\"Great we have a pickle file...Loading from {}\\\".format(pickle_file))\\n\",\n     \"            df = pickle.load(picklefile)\\n\",\n     \"\\n\",\n     \"    else: \\n\",\n-    \"        log.info(\\\"Reading CSV from {} .. This may take a while...\\\".format(input_csv_file))\\n\",\n+    \"        log.info(\\\"Reading CSV from {} .. This may take a while...\\\".format(input_csv_url))\\n\",\n     \"        \\n\",\n-    \"        with open(input_csv_file, 'r') as input_csv: ## should just change to IF EXISTS rather than open()???\\n\",\n-    \"            df = pd.read_csv(input_csv_file , sep=sep_char, dtype=dtype_dict)\\n\",\n-    \"                \\n\",\n-    \"        log.info(\\\"Why don't we save {} for next time\\\".format(output_pickle))\\n\",\n+    \"        #SWITCHING TO URL DIRECTLY, CAN'T CHECK FOR LOCAL FILE ANYMORE\\n\",","path":"JPF/mysql.skeleton.ipynb","position":99,"original_position":99,"commit_id":"f9b237a285d693d6ebfce8d01c53620a2f71d8df","original_commit_id":"dcbd19606a792303f2044327295b587c5dc72c38","user":{"login":"chrishenry","id":35760,"avatar_url":"https://avatars.githubusercontent.com/u/35760?v=3","gravatar_id":"","url":"https://api.github.com/users/chrishenry","html_url":"https://github.com/chrishenry","followers_url":"https://api.github.com/users/chrishenry/followers","following_url":"https://api.github.com/users/chrishenry/following{/other_user}","gists_url":"https://api.github.com/users/chrishenry/gists{/gist_id}","starred_url":"https://api.github.com/users/chrishenry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrishenry/subscriptions","organizations_url":"https://api.github.com/users/chrishenry/orgs","repos_url":"https://api.github.com/users/chrishenry/repos","events_url":"https://api.github.com/users/chrishenry/events{/privacy}","received_events_url":"https://api.github.com/users/chrishenry/received_events","type":"User","site_admin":false},"body":"I'd argue that we should try to maintain a workflow of download -> pickle -> load, with checks that will skip the download and pickle steps if on disk.\r\n\r\nThe biggest reason being that opening the file via network every time is pretty inefficient. Any issue in pickling or loading and you have re-download and re-pickle again. \r\n\r\nAlso, while I'm not super familiar with pandas internals, opening via http probably doesn't give us the option to read the csv in chunks, resulting in a more memory efficient load process. For example, I've been watching this process for ~4 hours now.","created_at":"2016-11-28T00:18:39Z","updated_at":"2016-11-28T00:18:39Z","html_url":"https://github.com/chrishenry/deltanyc-heatseek/pull/8#discussion_r89704610","pull_request_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/pulls/8","_links":{"self":{"href":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/pulls/comments/89704610"},"html":{"href":"https://github.com/chrishenry/deltanyc-heatseek/pull/8#discussion_r89704610"},"pull_request":{"href":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/pulls/8"}}},"pull_request":{"url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/pulls/8","id":95125027,"html_url":"https://github.com/chrishenry/deltanyc-heatseek/pull/8","diff_url":"https://github.com/chrishenry/deltanyc-heatseek/pull/8.diff","patch_url":"https://github.com/chrishenry/deltanyc-heatseek/pull/8.patch","issue_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/issues/8","number":8,"state":"open","locked":false,"title":"Big changes: load from URLS, pickle cleaned up, tested to HPD Violations","user":{"login":"jpfreeley","id":21322912,"avatar_url":"https://avatars.githubusercontent.com/u/21322912?v=3","gravatar_id":"","url":"https://api.github.com/users/jpfreeley","html_url":"https://github.com/jpfreeley","followers_url":"https://api.github.com/users/jpfreeley/followers","following_url":"https://api.github.com/users/jpfreeley/following{/other_user}","gists_url":"https://api.github.com/users/jpfreeley/gists{/gist_id}","starred_url":"https://api.github.com/users/jpfreeley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jpfreeley/subscriptions","organizations_url":"https://api.github.com/users/jpfreeley/orgs","repos_url":"https://api.github.com/users/jpfreeley/repos","events_url":"https://api.github.com/users/jpfreeley/events{/privacy}","received_events_url":"https://api.github.com/users/jpfreeley/received_events","type":"User","site_admin":false},"body":"I made a slew of changes ... I've switched everything to Download URL. I'm paving the way to diff-only updates. \r\n\r\nI cleaned up pickle process by eliminating a pickle_file\r\n\r\n11/26: I've added an SQL script which begins to create a \"bad buildings\" table from counts of various violations, permits, complaints, litigations. \r\n\r\n11/26: Added a few notes to NB and SQL\r\n\r\n11/26: Added PubAdvWebScrape which scrapes Public Advocate's worst landlords buildings search page and captures, BuildingAddress, Landlord, #HPDViolations, #DOBViolations\r\n\r\n11/26: Added PubAdv Worst Landlords to bad_buildings SQL file. \r\n\r\n11/27: Updated PubAdvWebScrape .. more descriptive VIO COUNT columns","created_at":"2016-11-24T06:47:31Z","updated_at":"2016-11-28T00:18:39Z","closed_at":null,"merged_at":null,"merge_commit_sha":"3d35beadf0fc6997ab3834170ec9ff2bf95b404b","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/pulls/8/commits","review_comments_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/pulls/8/comments","review_comment_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/pulls/comments{/number}","comments_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/issues/8/comments","statuses_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/statuses/f9b237a285d693d6ebfce8d01c53620a2f71d8df","head":{"label":"jpfreeley:master","ref":"master","sha":"f9b237a285d693d6ebfce8d01c53620a2f71d8df","user":{"login":"jpfreeley","id":21322912,"avatar_url":"https://avatars.githubusercontent.com/u/21322912?v=3","gravatar_id":"","url":"https://api.github.com/users/jpfreeley","html_url":"https://github.com/jpfreeley","followers_url":"https://api.github.com/users/jpfreeley/followers","following_url":"https://api.github.com/users/jpfreeley/following{/other_user}","gists_url":"https://api.github.com/users/jpfreeley/gists{/gist_id}","starred_url":"https://api.github.com/users/jpfreeley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jpfreeley/subscriptions","organizations_url":"https://api.github.com/users/jpfreeley/orgs","repos_url":"https://api.github.com/users/jpfreeley/repos","events_url":"https://api.github.com/users/jpfreeley/events{/privacy}","received_events_url":"https://api.github.com/users/jpfreeley/received_events","type":"User","site_admin":false},"repo":{"id":74075127,"name":"deltanyc-heatseek","full_name":"jpfreeley/deltanyc-heatseek","owner":{"login":"jpfreeley","id":21322912,"avatar_url":"https://avatars.githubusercontent.com/u/21322912?v=3","gravatar_id":"","url":"https://api.github.com/users/jpfreeley","html_url":"https://github.com/jpfreeley","followers_url":"https://api.github.com/users/jpfreeley/followers","following_url":"https://api.github.com/users/jpfreeley/following{/other_user}","gists_url":"https://api.github.com/users/jpfreeley/gists{/gist_id}","starred_url":"https://api.github.com/users/jpfreeley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jpfreeley/subscriptions","organizations_url":"https://api.github.com/users/jpfreeley/orgs","repos_url":"https://api.github.com/users/jpfreeley/repos","events_url":"https://api.github.com/users/jpfreeley/events{/privacy}","received_events_url":"https://api.github.com/users/jpfreeley/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/jpfreeley/deltanyc-heatseek","description":"Delta.NYC Cohort 2 Deliverables","fork":true,"url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek","forks_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/forks","keys_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/teams","hooks_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/hooks","issue_events_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/issues/events{/number}","events_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/events","assignees_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/assignees{/user}","branches_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/branches{/branch}","tags_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/tags","blobs_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/git/refs{/sha}","trees_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/statuses/{sha}","languages_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/languages","stargazers_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/stargazers","contributors_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/contributors","subscribers_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/subscribers","subscription_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/subscription","commits_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/commits{/sha}","git_commits_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/git/commits{/sha}","comments_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/comments{/number}","issue_comment_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/issues/comments{/number}","contents_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/contents/{+path}","compare_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/merges","archive_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/downloads","issues_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/issues{/number}","pulls_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/pulls{/number}","milestones_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/milestones{/number}","notifications_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/labels{/name}","releases_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/releases{/id}","deployments_url":"https://api.github.com/repos/jpfreeley/deltanyc-heatseek/deployments","created_at":"2016-11-17T23:02:32Z","updated_at":"2016-11-18T18:36:01Z","pushed_at":"2016-11-27T23:46:51Z","git_url":"git://github.com/jpfreeley/deltanyc-heatseek.git","ssh_url":"git@github.com:jpfreeley/deltanyc-heatseek.git","clone_url":"https://github.com/jpfreeley/deltanyc-heatseek.git","svn_url":"https://github.com/jpfreeley/deltanyc-heatseek","homepage":null,"size":99,"stargazers_count":0,"watchers_count":0,"language":"Jupyter Notebook","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"chrishenry:master","ref":"master","sha":"e57d14f39fa4517927fb2f64d8b0b2a6c4766e5e","user":{"login":"chrishenry","id":35760,"avatar_url":"https://avatars.githubusercontent.com/u/35760?v=3","gravatar_id":"","url":"https://api.github.com/users/chrishenry","html_url":"https://github.com/chrishenry","followers_url":"https://api.github.com/users/chrishenry/followers","following_url":"https://api.github.com/users/chrishenry/following{/other_user}","gists_url":"https://api.github.com/users/chrishenry/gists{/gist_id}","starred_url":"https://api.github.com/users/chrishenry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrishenry/subscriptions","organizations_url":"https://api.github.com/users/chrishenry/orgs","repos_url":"https://api.github.com/users/chrishenry/repos","events_url":"https://api.github.com/users/chrishenry/events{/privacy}","received_events_url":"https://api.github.com/users/chrishenry/received_events","type":"User","site_admin":false},"repo":{"id":74074732,"name":"deltanyc-heatseek","full_name":"chrishenry/deltanyc-heatseek","owner":{"login":"chrishenry","id":35760,"avatar_url":"https://avatars.githubusercontent.com/u/35760?v=3","gravatar_id":"","url":"https://api.github.com/users/chrishenry","html_url":"https://github.com/chrishenry","followers_url":"https://api.github.com/users/chrishenry/followers","following_url":"https://api.github.com/users/chrishenry/following{/other_user}","gists_url":"https://api.github.com/users/chrishenry/gists{/gist_id}","starred_url":"https://api.github.com/users/chrishenry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrishenry/subscriptions","organizations_url":"https://api.github.com/users/chrishenry/orgs","repos_url":"https://api.github.com/users/chrishenry/repos","events_url":"https://api.github.com/users/chrishenry/events{/privacy}","received_events_url":"https://api.github.com/users/chrishenry/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/chrishenry/deltanyc-heatseek","description":"Delta.NYC Cohort 2 Deliverables","fork":false,"url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek","forks_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/forks","keys_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/keys{/key_id}","collaborators_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/teams","hooks_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/hooks","issue_events_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/issues/events{/number}","events_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/events","assignees_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/assignees{/user}","branches_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/branches{/branch}","tags_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/tags","blobs_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/git/refs{/sha}","trees_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/git/trees{/sha}","statuses_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/statuses/{sha}","languages_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/languages","stargazers_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/stargazers","contributors_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/contributors","subscribers_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/subscribers","subscription_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/subscription","commits_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/commits{/sha}","git_commits_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/git/commits{/sha}","comments_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/comments{/number}","issue_comment_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/issues/comments{/number}","contents_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/contents/{+path}","compare_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/compare/{base}...{head}","merges_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/merges","archive_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/downloads","issues_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/issues{/number}","pulls_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/pulls{/number}","milestones_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/milestones{/number}","notifications_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/labels{/name}","releases_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/releases{/id}","deployments_url":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/deployments","created_at":"2016-11-17T22:55:36Z","updated_at":"2016-11-22T23:51:20Z","pushed_at":"2016-11-27T23:46:51Z","git_url":"git://github.com/chrishenry/deltanyc-heatseek.git","ssh_url":"git@github.com:chrishenry/deltanyc-heatseek.git","clone_url":"https://github.com/chrishenry/deltanyc-heatseek.git","svn_url":"https://github.com/chrishenry/deltanyc-heatseek","homepage":null,"size":86,"stargazers_count":2,"watchers_count":2,"language":"Jupyter Notebook","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":5,"mirror_url":null,"open_issues_count":1,"forks":5,"open_issues":1,"watchers":2,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/pulls/8"},"html":{"href":"https://github.com/chrishenry/deltanyc-heatseek/pull/8"},"issue":{"href":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/issues/8"},"comments":{"href":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/issues/8/comments"},"review_comments":{"href":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/pulls/8/comments"},"review_comment":{"href":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/pulls/8/commits"},"statuses":{"href":"https://api.github.com/repos/chrishenry/deltanyc-heatseek/statuses/f9b237a285d693d6ebfce8d01c53620a2f71d8df"}}}}
{"actor":{"display_login":"chrishenry"}}