title: How to use entrezpy and Biopython Entrez libraries to access ClinVar data from genomic position of variant
body: <p>[Disclaimer: I have published this question 3 weeks ago in biostars, with no answers yet. I really would like to get some ideas/discussion to find a solution, so I post also here.
  biostars post link: <a href="https://www.biostars.org/p/447413/" rel="nofollow noreferrer">https://www.biostars.org/p/447413/</a>]</p>
  <p>For one of my projects of my PhD, I would like to access all variants, found in ClinVar db, that are in the same genomic position as the variant in each row of the input GSVar file. The language constraint is Python.</p>
  <p>Up to now I have used entrezpy module: entrezpy.esearch.esearcher. Please see more for entrezpy at: <a href="https://entrezpy.readthedocs.io/en/master/" rel="nofollow noreferrer">https://entrezpy.readthedocs.io/en/master/</a></p>
  <p>From the entrezpy docs I have followed this guide to access UIDs using the genomic position of a variant: <a href="https://entrezpy.readthedocs.io/en/master/tutorials/esearch/esearch_uids.html" rel="nofollow noreferrer">https://entrezpy.readthedocs.io/en/master/tutorials/esearch/esearch_uids.html</a> in code:</p>
  <pre><code># first get UIDs for clinvar records of the same position
  # credits: credits: https://entrezpy.readthedocs.io/en/master/tutorials/esearch/esearch_uids.html
  chr = variants[&quot;chr&quot;].split(&quot;chr&quot;)[1]
  start, end = str(variants[&quot;start&quot;]), str(variants[&quot;end&quot;])
  
  es = entrezpy.esearch.esearcher.Esearcher('esearcher', self.entrez_email)
  genomic_pos = chr + &quot;[chr]&quot; + &quot; AND &quot; + start + &quot;:&quot; + end  # + &quot;[chrpos37]&quot;
  entrez_query = es.inquire(
      {'db': 'clinvar',
       'term': genomic_pos,
       'retmax': 100000,
       'retstart': 0,
       'rettype': 'uilist'})  # 'usehistory': False
  entrez_uids = entrez_query.get_result().uids
  </code></pre>
  <p>Then I have used Entrez from BioPython to get the available ClinVar records:</p>
  <pre><code># process each VariationArchive of each UID
      handle = Entrez.efetch(db='clinvar', id=current_entrez_uids, rettype='vcv')
      clinvar_records = {}
      tree = ET.parse(handle)
      root = tree.getroot()
  </code></pre>
  <p>This approach is working. However, I have two main drawbacks:</p>
  <ul>
  <li><p>entrezpy fulls up my log file recording all interaction with Entrez making the log file too big to be read by the hospital collaborator, who is variant curator.</p>
  </li>
  <li><p>entrezpy function, entrez_query.get_result().uids, will return all UIDs retrieved so far from all the requests (say a request for each variant in GSvar), thus this space inefficient retrieval. That is the entrez_uids list will quickly grow a lot as I process all variants from a GSVar file. The simple solution that I have implenented is to check which UIDs are new from the current request and then keep only those for Entrez.fetch(). However, I still need to keep all seen UIDs, from previous variants in order to be able to know which is the new UIDs. I do this in code by:</p>
  <pre><code># first snippet's first lines go here
  entrez_uids = entrez_query.get_result().uids
  current_entrez_uids = [uid for uid in entrez_uids if uid not in self.all_entrez_uids_gsvar_file]
  self.all_entrez_uids_gsvar_file += current_entrez_uids
  </code></pre>
  </li>
  </ul>
  <p>Does anyone have suggestion(s) on how to address these two presented drawbacks?</p>