{"action":"created","comment":{"url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/pulls/comments/187973243","pull_request_review_id":119842269,"id":187973243,"diff_hunk":"@@ -1,37 +1,29 @@\n using ..DBUtils\n using SQLite\n using MySQL\n+using EzXML\n+using DataFrames\n \n \"\"\"\n pubmed_save_efetch(efetch_dict, conn)\n \n Save the results (dictionary) of an entrez-pubmed fetch to the input database.\n \"\"\"\n-function save_efetch!(conn::Union{MySQL.Connection, SQLite.DB}, efetch_dict, verbose=false)\n+function save_efetch!(conn::Union{MySQL.Connection, SQLite.DB}, articles::EzXML.Node, verbose=false, cleanup=false)\n \n     #Decide type of article based on structrure of efetch\n-    articles = nothing\n-    if haskey(efetch_dict, \"PubmedArticle\")\n-        TypeArticle = PubMedArticle\n-        articles = efetch_dict[\"PubmedArticle\"]\n-    else\n-        println(efetch_dict)\n+\n+    if nodename(articles) != \"PubmedArticleSet\"\n+        println(articles)\n         error(\"Save efetch is only supported for PubMed searches\")\n     end\n \n-    println(\"Saving \" , length(articles) ,  \" articles to database\")\n-    for xml_article in articles\n-        article = TypeArticle(xml_article)\n-        # println(\"=============Article=====================\")\n-        # println(article)\n-        db_insert!(conn, article, verbose)\n+    println(\"Saving \" , countelements(articles) ,  \" articles to database\")\n \n-        #-------MeshHeadingList\n-        mesh_heading_list = MeshHeadingList(xml_article)\n-        db_insert!(conn, article.pmid, mesh_heading_list, verbose)\n-    end\n+    parsed = pubmed_to_dfs(articles)\n+\n+    db_insert!(conn, parsed, cleanup=true)\n \n-    conn\n end","path":"src/PubMed/eutils_sql_save.jl","position":45,"original_position":45,"commit_id":"b53b40544af3dcd3b204dcf943a81ac7ffdb208a","original_commit_id":"b53b40544af3dcd3b204dcf943a81ac7ffdb208a","user":{"login":"mirestrepo","id":1238840,"avatar_url":"https://avatars3.githubusercontent.com/u/1238840?v=4","gravatar_id":"","url":"https://api.github.com/users/mirestrepo","html_url":"https://github.com/mirestrepo","followers_url":"https://api.github.com/users/mirestrepo/followers","following_url":"https://api.github.com/users/mirestrepo/following{/other_user}","gists_url":"https://api.github.com/users/mirestrepo/gists{/gist_id}","starred_url":"https://api.github.com/users/mirestrepo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mirestrepo/subscriptions","organizations_url":"https://api.github.com/users/mirestrepo/orgs","repos_url":"https://api.github.com/users/mirestrepo/repos","events_url":"https://api.github.com/users/mirestrepo/events{/privacy}","received_events_url":"https://api.github.com/users/mirestrepo/received_events","type":"User","site_admin":false},"body":"I feel like save_efetch! is no longer needed. \r\n\r\nInstead there is a `PubMed.parsexml` or `Pubmed.parse` that corresponds to pubmed_to_dfs. Then in the processes  - we have only one process search_and_parse. The user can manually call `db_insert!(conn, parsed, cleanup=true)` we just give an example. \r\nFor large queries do you see and advantage on saving every 10_000/30_000 instead of at the end?","created_at":"2018-05-14T14:27:00Z","updated_at":"2018-05-14T14:36:28Z","html_url":"https://github.com/bcbi/BioMedQuery.jl/pull/18#discussion_r187973243","pull_request_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/pulls/18","author_association":"OWNER","_links":{"self":{"href":"https://api.github.com/repos/bcbi/BioMedQuery.jl/pulls/comments/187973243"},"html":{"href":"https://github.com/bcbi/BioMedQuery.jl/pull/18#discussion_r187973243"},"pull_request":{"href":"https://api.github.com/repos/bcbi/BioMedQuery.jl/pulls/18"}}},"pull_request":{"url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/pulls/18","id":187620043,"html_url":"https://github.com/bcbi/BioMedQuery.jl/pull/18","diff_url":"https://github.com/bcbi/BioMedQuery.jl/pull/18.diff","patch_url":"https://github.com/bcbi/BioMedQuery.jl/pull/18.patch","issue_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/issues/18","number":18,"state":"open","locked":false,"title":"Dfs backend","user":{"login":"mcmcgrath13","id":24885580,"avatar_url":"https://avatars3.githubusercontent.com/u/24885580?v=4","gravatar_id":"","url":"https://api.github.com/users/mcmcgrath13","html_url":"https://github.com/mcmcgrath13","followers_url":"https://api.github.com/users/mcmcgrath13/followers","following_url":"https://api.github.com/users/mcmcgrath13/following{/other_user}","gists_url":"https://api.github.com/users/mcmcgrath13/gists{/gist_id}","starred_url":"https://api.github.com/users/mcmcgrath13/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mcmcgrath13/subscriptions","organizations_url":"https://api.github.com/users/mcmcgrath13/orgs","repos_url":"https://api.github.com/users/mcmcgrath13/repos","events_url":"https://api.github.com/users/mcmcgrath13/events{/privacy}","received_events_url":"https://api.github.com/users/mcmcgrath13/received_events","type":"User","site_admin":false},"body":"Changes parsing of pubmed xml from a structure and dictionary based approach to parsing xml elements directly into dataframes.  Added load_medline and pubmed_search_and_parse processes and updated all other processes and functions to use the dataframe backend instead of structures.","created_at":"2018-05-12T12:53:03Z","updated_at":"2018-05-14T14:36:28Z","closed_at":null,"merged_at":null,"merge_commit_sha":"41f42455b558794d988b9f88fa3b662ec4843dcb","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/pulls/18/commits","review_comments_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/pulls/18/comments","review_comment_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/pulls/comments{/number}","comments_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/issues/18/comments","statuses_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/statuses/b53b40544af3dcd3b204dcf943a81ac7ffdb208a","head":{"label":"bcbi:dfs_backend","ref":"dfs_backend","sha":"b53b40544af3dcd3b204dcf943a81ac7ffdb208a","user":{"login":"bcbi","id":16926523,"avatar_url":"https://avatars3.githubusercontent.com/u/16926523?v=4","gravatar_id":"","url":"https://api.github.com/users/bcbi","html_url":"https://github.com/bcbi","followers_url":"https://api.github.com/users/bcbi/followers","following_url":"https://api.github.com/users/bcbi/following{/other_user}","gists_url":"https://api.github.com/users/bcbi/gists{/gist_id}","starred_url":"https://api.github.com/users/bcbi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bcbi/subscriptions","organizations_url":"https://api.github.com/users/bcbi/orgs","repos_url":"https://api.github.com/users/bcbi/repos","events_url":"https://api.github.com/users/bcbi/events{/privacy}","received_events_url":"https://api.github.com/users/bcbi/received_events","type":"Organization","site_admin":false},"repo":{"id":59500020,"name":"BioMedQuery.jl","full_name":"bcbi/BioMedQuery.jl","owner":{"login":"bcbi","id":16926523,"avatar_url":"https://avatars3.githubusercontent.com/u/16926523?v=4","gravatar_id":"","url":"https://api.github.com/users/bcbi","html_url":"https://github.com/bcbi","followers_url":"https://api.github.com/users/bcbi/followers","following_url":"https://api.github.com/users/bcbi/following{/other_user}","gists_url":"https://api.github.com/users/bcbi/gists{/gist_id}","starred_url":"https://api.github.com/users/bcbi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bcbi/subscriptions","organizations_url":"https://api.github.com/users/bcbi/orgs","repos_url":"https://api.github.com/users/bcbi/repos","events_url":"https://api.github.com/users/bcbi/events{/privacy}","received_events_url":"https://api.github.com/users/bcbi/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/bcbi/BioMedQuery.jl","description":"Julia utilities to interact with BioMedical Databases and APIs","fork":false,"url":"https://api.github.com/repos/bcbi/BioMedQuery.jl","forks_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/forks","keys_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/keys{/key_id}","collaborators_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/teams","hooks_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/hooks","issue_events_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/issues/events{/number}","events_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/events","assignees_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/assignees{/user}","branches_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/branches{/branch}","tags_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/tags","blobs_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/git/refs{/sha}","trees_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/git/trees{/sha}","statuses_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/statuses/{sha}","languages_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/languages","stargazers_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/stargazers","contributors_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/contributors","subscribers_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/subscribers","subscription_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/subscription","commits_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/commits{/sha}","git_commits_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/git/commits{/sha}","comments_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/comments{/number}","issue_comment_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/issues/comments{/number}","contents_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/contents/{+path}","compare_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/compare/{base}...{head}","merges_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/merges","archive_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/downloads","issues_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/issues{/number}","pulls_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/pulls{/number}","milestones_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/milestones{/number}","notifications_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/labels{/name}","releases_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/releases{/id}","deployments_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/deployments","created_at":"2016-05-23T16:37:54Z","updated_at":"2018-04-20T16:34:32Z","pushed_at":"2018-05-12T12:53:04Z","git_url":"git://github.com/bcbi/BioMedQuery.jl.git","ssh_url":"git@github.com:bcbi/BioMedQuery.jl.git","clone_url":"https://github.com/bcbi/BioMedQuery.jl.git","svn_url":"https://github.com/bcbi/BioMedQuery.jl","homepage":"","size":2547,"stargazers_count":9,"watchers_count":9,"language":"Julia","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":5,"mirror_url":null,"archived":false,"open_issues_count":5,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit"},"forks":5,"open_issues":5,"watchers":9,"default_branch":"master"}},"base":{"label":"bcbi:master","ref":"master","sha":"f0bd0e459eb594dae161cf1d0f34802c6a820ef8","user":{"login":"bcbi","id":16926523,"avatar_url":"https://avatars3.githubusercontent.com/u/16926523?v=4","gravatar_id":"","url":"https://api.github.com/users/bcbi","html_url":"https://github.com/bcbi","followers_url":"https://api.github.com/users/bcbi/followers","following_url":"https://api.github.com/users/bcbi/following{/other_user}","gists_url":"https://api.github.com/users/bcbi/gists{/gist_id}","starred_url":"https://api.github.com/users/bcbi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bcbi/subscriptions","organizations_url":"https://api.github.com/users/bcbi/orgs","repos_url":"https://api.github.com/users/bcbi/repos","events_url":"https://api.github.com/users/bcbi/events{/privacy}","received_events_url":"https://api.github.com/users/bcbi/received_events","type":"Organization","site_admin":false},"repo":{"id":59500020,"name":"BioMedQuery.jl","full_name":"bcbi/BioMedQuery.jl","owner":{"login":"bcbi","id":16926523,"avatar_url":"https://avatars3.githubusercontent.com/u/16926523?v=4","gravatar_id":"","url":"https://api.github.com/users/bcbi","html_url":"https://github.com/bcbi","followers_url":"https://api.github.com/users/bcbi/followers","following_url":"https://api.github.com/users/bcbi/following{/other_user}","gists_url":"https://api.github.com/users/bcbi/gists{/gist_id}","starred_url":"https://api.github.com/users/bcbi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bcbi/subscriptions","organizations_url":"https://api.github.com/users/bcbi/orgs","repos_url":"https://api.github.com/users/bcbi/repos","events_url":"https://api.github.com/users/bcbi/events{/privacy}","received_events_url":"https://api.github.com/users/bcbi/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/bcbi/BioMedQuery.jl","description":"Julia utilities to interact with BioMedical Databases and APIs","fork":false,"url":"https://api.github.com/repos/bcbi/BioMedQuery.jl","forks_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/forks","keys_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/keys{/key_id}","collaborators_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/teams","hooks_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/hooks","issue_events_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/issues/events{/number}","events_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/events","assignees_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/assignees{/user}","branches_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/branches{/branch}","tags_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/tags","blobs_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/git/refs{/sha}","trees_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/git/trees{/sha}","statuses_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/statuses/{sha}","languages_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/languages","stargazers_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/stargazers","contributors_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/contributors","subscribers_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/subscribers","subscription_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/subscription","commits_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/commits{/sha}","git_commits_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/git/commits{/sha}","comments_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/comments{/number}","issue_comment_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/issues/comments{/number}","contents_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/contents/{+path}","compare_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/compare/{base}...{head}","merges_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/merges","archive_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/downloads","issues_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/issues{/number}","pulls_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/pulls{/number}","milestones_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/milestones{/number}","notifications_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/labels{/name}","releases_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/releases{/id}","deployments_url":"https://api.github.com/repos/bcbi/BioMedQuery.jl/deployments","created_at":"2016-05-23T16:37:54Z","updated_at":"2018-04-20T16:34:32Z","pushed_at":"2018-05-12T12:53:04Z","git_url":"git://github.com/bcbi/BioMedQuery.jl.git","ssh_url":"git@github.com:bcbi/BioMedQuery.jl.git","clone_url":"https://github.com/bcbi/BioMedQuery.jl.git","svn_url":"https://github.com/bcbi/BioMedQuery.jl","homepage":"","size":2547,"stargazers_count":9,"watchers_count":9,"language":"Julia","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":5,"mirror_url":null,"archived":false,"open_issues_count":5,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit"},"forks":5,"open_issues":5,"watchers":9,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/bcbi/BioMedQuery.jl/pulls/18"},"html":{"href":"https://github.com/bcbi/BioMedQuery.jl/pull/18"},"issue":{"href":"https://api.github.com/repos/bcbi/BioMedQuery.jl/issues/18"},"comments":{"href":"https://api.github.com/repos/bcbi/BioMedQuery.jl/issues/18/comments"},"review_comments":{"href":"https://api.github.com/repos/bcbi/BioMedQuery.jl/pulls/18/comments"},"review_comment":{"href":"https://api.github.com/repos/bcbi/BioMedQuery.jl/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/bcbi/BioMedQuery.jl/pulls/18/commits"},"statuses":{"href":"https://api.github.com/repos/bcbi/BioMedQuery.jl/statuses/b53b40544af3dcd3b204dcf943a81ac7ffdb208a"}},"author_association":"COLLABORATOR"}}
{"actor":{"display_login":"mirestrepo"}}