text: user4441: I currently use the 'UniProt website REST API' (example) to collect the UniParc IDs of all the members of the given cluster. Fetching each entry individually is very slow though, so I wanted to ask if there is a way that I can do this request in batches.
  What I have already tried:
  There are pages mentioning a batch retrieval of entries. As I understood it, all the examples only adress the ID mapping service. Here I have tried adjusting the given example to the following
  import urllib.parse
  import urllib.request
  
  url = 'https://www.uniprot.org/uploadlists/'
  
  params = {
  'from': 'NF100',
  'to': 'UPARC',
  'format': 'tab',
  'query': 'UniRef100_Q8RKD8'
  }
  
  data = urllib.parse.urlencode(params)
  data = data.encode('utf-8')
  req = urllib.request.Request(url, data)
  with urllib.request.urlopen(req) as f:
     response = f.read()
  print(response.decode('utf-8'))
  
  which resulted in the output
  From    To
  UniRef100_Q8RKD8        null
  
  I also checked the interface of this service on the website and there I can't seem to map NF100 to UPARC, so I'm assuming that is why I'm not getting a result via the API.
  I also know that the UniRef Id contains either a UniParc ID or the UniProtKB accession, so I could use the UniParc ID directly or fetch it for the UniProtKB via the proteins api with the uniparc endpoint, but from what I can see I only get the UniParc ID/entry of one of the members.
  
  user4441: Update 17.10.22
  It seems that the api changed and the previous solution doesn't work anymore. I updated my solution by using the Uniprot id mapping service. All the following instructions are copied from the documentation page but added here because link answers are discouraged. The documentation page also includes an implementation in python. The basic example is as follows (using commandline tools)
  $ curl 
      --request POST 'https://rest.uniprot.org/idmapping/run' 
      --form 'ids="P99999,P31946"' 
      --form 'from="UniProtKB_AC-ID"' 
      --form 'to="UniRef100"'
  
  This returns a job id as a response
  {"jobId":"27a020f6334184c4eb382111fbcad0e848f40300"}
  
  You then have to poll the api for the status of the job
  $ curl -i 'https://rest.uniprot.org/idmapping/status/{jobId}'
  
  Which returns you a status of FINISHED and the url where you can fetch the result in the HTTP Header. Note that since the job might take some time to finish you need to poll this url until you get the status FINISHED.
  HTTP/1.1 303
  Server: nginx/1.17.7
  Location: https://rest.uniprot.org/idmapping/uniref/results/27a020f6334184c4eb382111fbcad0e848f40300
  Content-Type: application/json
  Access-Control-Allow-Origin: *
  ...
  {"jobStatus":"FINISHED"}  
  
  Access the result url with curl
  $ curl -s "https://rest.uniprot.org/idmapping/uniref/results/27a020f6334184c4eb382111fbcad0e848f40300"
  
  Which gives you a json dictionary
  { "results": [...], "failedIds": []}
  
  Each entry in the result list is a dictionary similar to the following example
  {
      'from': 'P99999'\n\t
      , 'to': {
          'id': 'UniRef100_P99999'
          , 'memberCount': 17
          , 'representativeMember': {
              'uniparcId': 'UPI000013EAA0'
              , ...
          }
          , 'memberIdTypes': [
              'UniProtKB Unreviewed (TrEMBL)'
              , 'UniProtKB Reviewed (Swiss-Prot)'
          ]
          , 'members': [
              'P99999,0'
              , 'Q6WUX8,0'
              , 'P99998,0'
              , 'Q5RFH4,0'
              , 'G4XXL9,1'
              , 'A0A2J8VDX6,1'
              , 'K7CAI8,1'
              , 'A0A663D840,1'
              , 'G4XXM0,1'
              , 'G4XXM2,1'
              , 'G4XXM4,1'
              , 'G1RY04,1'
              , 'G4XXM3,1'
              , 'G4XXM6,1'
              , 'C9JFR7,1'
              , 'A0A2J8VE06,1'
              , 'A0A2J8KZC0,1'
          ]
          , ...
      }
  }
  
  The uniparc id of the cluster is given in the representative Member.
  If you want the group members directly they are given as a list of strings that contain the accession and the memberIdType index separated with a comma. You would use the index to look up the type of the accession in the memberIdTypes array.
  The result also contains information such as the sequence of the representative Member, or the number of organisms in the cluster.
  Note: From what I can tell in the UniRef100 Clusters there will only be one Uniparc Id which you will find with the representative Member. If you want to map to UniRef90 or UniRef50 there will be other Uniparc ids in the member list, however the representative Uniparc Id will still only be in the representative Member, so you it's probably safest to parse both.
  
  Old solution
  I found a way to batch the mapping from uniref cluster ids to the ids of all members. The url
  https://www.uniprot.org/uniref/?query=UniRef100_A0A003&format=tab&columns=id,members
  can be used to get the link from one uniref Cluster to it's member ids. Multiple uniref cluster can be fetched at once by connecting the ids with a +OR+ like so https://www.uniprot.org/uniref/?query=UniRef100_P99999+OR+UniRef100_P31946&format=tab&columns=id,members
  Because the uniref cluster ids are not stable I ended up using the following syntax, where instead of querying the UniRef cluster by it's id, I query by one of it's members, namely the one that was used to create the current uniref cluster id: https://www.uniprot.org/uniref/?query=*&format=tab&limit=10&columns=id,members&fil=identity:1.0+AND+member:(P99999+OR+P31946)
  In the notes section of the 'Retrieve / ID mapping' documentation of the uniprot they talk about fetching chunks up to 20.000 identifiers at a time. I am not sure if I use the service they are talking about though, so I'm keeping my requests at 100 identifiers per request and that seems to work.
  One thing I noticed was that fetching results via the uniref endpoint is faster for small sets of identifiers (<20) since the fetch time of the batch request is similar for smaller and larger (<100) sets. So batching only makes sense for more than 20 identifiers at a time. (Disclaimer: I have only done very basic tests on this)
  Once I have all the members of a cluster I can use the proteins/uniparc endpoint of the proteins api to fetch the uniparc data. For uniparc accessions (starting with the string 'UPI') I use the GET /uniparc/upi/{upi} endpoint. For the other accession, which should all be Uniprot accessions, I use the GET /uniparc/accession/{accession} endpoint.
  This doesn't answer the question of a direct mapping between the uniref ids and uniparc member ids, but it is a workable solution.
  
original_text: 4441: <p>I currently use the 'UniProt website REST API' (<a href="https://www.uniprot.org/uniref/UniRef100_P31946.xml" rel="nofollow noreferrer">example</a>) to collect the UniParc IDs of all the members of the given cluster. Fetching each entry individually is very slow though, so I wanted to ask if there is a way that I can do this request in batches.</p>
  <p><strong>What I have already tried:</strong>
  There are pages mentioning a <a href="https://www.uniprot.org/help/api_batch_retrieval" rel="nofollow noreferrer">batch retrieval of entries</a>. As I understood it, all the examples only adress the ID mapping service. Here I have tried adjusting the given example to the following</p>
  <pre><code>import urllib.parse
  import urllib.request
  
  url = 'https://www.uniprot.org/uploadlists/'
  
  params = {
  'from': 'NF100',
  'to': 'UPARC',
  'format': 'tab',
  'query': 'UniRef100_Q8RKD8'
  }
  
  data = urllib.parse.urlencode(params)
  data = data.encode('utf-8')
  req = urllib.request.Request(url, data)
  with urllib.request.urlopen(req) as f:
     response = f.read()
  print(response.decode('utf-8'))
  </code></pre>
  <p>which resulted in the output</p>
  <pre><code>From    To
  UniRef100_Q8RKD8        null
  </code></pre>
  <p>I also checked the interface of this service on the website and there I can't seem to map NF100 to UPARC, so I'm assuming that is why I'm not getting a result via the API.</p>
  <p>I also know that the UniRef Id contains either a UniParc ID or the UniProtKB accession, so I could use the UniParc ID directly or fetch it for the UniProtKB via the <a href="https://www.ebi.ac.uk/proteins/api/uniparc/accession/P31946.json" rel="nofollow noreferrer">proteins api with the uniparc endpoint</a>, but from what I can see I only get the UniParc ID/entry of one of the members.</p>
   4441: <p><strong>Update 17.10.22</strong></p>
  <p>It seems that the api changed and the previous solution doesn't work anymore. I updated my solution by using the <a href="https://www.uniprot.org/help/id_mapping" rel="nofollow noreferrer">Uniprot id mapping service</a>. All the following instructions are copied from the documentation page but added here because link answers are discouraged. The documentation page also includes an implementation in python. The basic example is as follows (using commandline tools)</p>
  <pre><code>$ curl 
      --request POST 'https://rest.uniprot.org/idmapping/run' 
      --form 'ids=&quot;P99999,P31946&quot;' 
      --form 'from=&quot;UniProtKB_AC-ID&quot;' 
      --form 'to=&quot;UniRef100&quot;'
  </code></pre>
  <p>This returns a job id as a response</p>
  <pre><code>{&quot;jobId&quot;:&quot;27a020f6334184c4eb382111fbcad0e848f40300&quot;}
  </code></pre>
  <p>You then have to poll the api for the status of the job</p>
  <pre><code>$ curl -i 'https://rest.uniprot.org/idmapping/status/{jobId}'
  </code></pre>
  <p>Which returns you a status of FINISHED and the url where you can fetch the result in the HTTP Header. Note that since the job might take some time to finish you need to poll this url until you get the status FINISHED.</p>
  <pre><code>HTTP/1.1 303
  Server: nginx/1.17.7
  Location: https://rest.uniprot.org/idmapping/uniref/results/27a020f6334184c4eb382111fbcad0e848f40300
  Content-Type: application/json
  Access-Control-Allow-Origin: *
  ...
  {&quot;jobStatus&quot;:&quot;FINISHED&quot;}  
  </code></pre>
  <p>Access the result url with curl</p>
  <pre><code>$ curl -s &quot;https://rest.uniprot.org/idmapping/uniref/results/27a020f6334184c4eb382111fbcad0e848f40300&quot;
  </code></pre>
  <p>Which gives you a json dictionary</p>
  <pre><code>{ &quot;results&quot;: [...], &quot;failedIds&quot;: []}
  </code></pre>
  <p>Each entry in the result list is a dictionary similar to the following example</p>
  <pre><code>{
      'from': 'P99999'\n\t
      , 'to': {
          'id': 'UniRef100_P99999'
          , 'memberCount': 17
          , 'representativeMember': {
              'uniparcId': 'UPI000013EAA0'
              , ...
          }
          , 'memberIdTypes': [
              'UniProtKB Unreviewed (TrEMBL)'
              , 'UniProtKB Reviewed (Swiss-Prot)'
          ]
          , 'members': [
              'P99999,0'
              , 'Q6WUX8,0'
              , 'P99998,0'
              , 'Q5RFH4,0'
              , 'G4XXL9,1'
              , 'A0A2J8VDX6,1'
              , 'K7CAI8,1'
              , 'A0A663D840,1'
              , 'G4XXM0,1'
              , 'G4XXM2,1'
              , 'G4XXM4,1'
              , 'G1RY04,1'
              , 'G4XXM3,1'
              , 'G4XXM6,1'
              , 'C9JFR7,1'
              , 'A0A2J8VE06,1'
              , 'A0A2J8KZC0,1'
          ]
          , ...
      }
  }
  </code></pre>
  <p>The uniparc id of the cluster is given in the representative Member.</p>
  <p>If you want the group members directly they are given as a list of strings that contain the accession and the memberIdType index separated with a comma. You would use the index to look up the type of the accession in the memberIdTypes array.</p>
  <p>The result also contains information such as the sequence of the representative Member, or the number of organisms in the cluster.</p>
  <p><strong>Note:</strong> From what I can tell in the UniRef100 Clusters there will only be one Uniparc Id which you will find with the representative Member. If you want to map to UniRef90 or UniRef50 there will be other Uniparc ids in the member list, however the representative Uniparc Id will still only be in the representative Member, so you it's probably safest to parse both.</p>
  <hr />
  <p><strong>Old solution</strong></p>
  <p>I found a way to batch the mapping from uniref cluster ids to the ids of all members. The url
  <code>https://www.uniprot.org/uniref/?query=UniRef100_A0A003&amp;format=tab&amp;columns=id,members</code>
  can be used to get the link from one uniref Cluster to it's member ids. Multiple uniref cluster can be fetched at once by connecting the ids with a <code>+OR+</code> like so <code>https://www.uniprot.org/uniref/?query=UniRef100_P99999+OR+UniRef100_P31946&amp;format=tab&amp;columns=id,members</code></p>
  <p>Because the uniref cluster ids are not stable I ended up using the following syntax, where instead of querying the UniRef cluster by it's id, I query by one of it's members, namely the one that was used to create the current uniref cluster id: <code>https://www.uniprot.org/uniref/?query=*&amp;format=tab&amp;limit=10&amp;columns=id,members&amp;fil=identity:1.0+AND+member:(P99999+OR+P31946)</code></p>
  <p>In the notes section of the <a href="https://www.uniprot.org/help/uploadlists" rel="nofollow noreferrer">'Retrieve / ID mapping'</a> documentation of the uniprot they talk about fetching chunks up to 20.000 identifiers at a time. I am not sure if I use the service they are talking about though, so I'm keeping my requests at 100 identifiers per request and that seems to work.</p>
  <p>One thing I noticed was that fetching results via the <a href="https://www.uniprot.org/uniref/UniRef100_P31946.xml" rel="nofollow noreferrer">uniref endpoint</a> is faster for small sets of identifiers (&lt;20) since the fetch time of the batch request is similar for smaller and larger (&lt;100) sets. So batching only makes sense for more than 20 identifiers at a time. (Disclaimer: I have only done very basic tests on this)</p>
  <p>Once I have all the members of a cluster I can use the proteins/uniparc endpoint of the <a href="https://www.ebi.ac.uk/proteins/api/doc/" rel="nofollow noreferrer">proteins api</a> to fetch the uniparc data. For uniparc accessions (starting with the string 'UPI') I use the <code>GET /uniparc/upi/{upi}</code> endpoint. For the other accession, which should all be Uniprot accessions, I use the <code>GET /uniparc/accession/{accession}</code> endpoint.</p>
  <p>This doesn't answer the question of a direct mapping between the uniref ids and uniparc member ids, but it is a workable solution.</p>
  