title: Export/stream massive results from splunk REST API
body: <p>I need to export a massive number of events from splunk. Hence for performance reasons i resorted to directly using the REST API in my python code rather than using the Splunk SDK itself. </p>
  
  <p>I <a href="https://answers.splunk.com/answers/232330/how-to-use-the-rest-api-to-just-run-a-search-and-s.html" rel="nofollow noreferrer">found</a> the following curl command to export results. This is also available <a href="https://stackoverflow.com/questions/13368407/search-splunk-with-rest-api-without-a-saved-search">here</a>:-</p>
  
  <pre><code>  curl -ku username:password
  https://splunk_host:port/servicesNS/admin/search/search/jobs/export -d
  search=“search index%3D_internal | head 3” -d output_mode=json
  </code></pre>
  
  <p>My attempt at simulating this using python's http functions is as follows:-</p>
  
  <pre><code>//assume i have authenticated to splunk and have a session key
  base_url = "http://splunkhost:port"
  
  search_job_urn = '/services/search/jobs/export'
  
  myhttp = httplib2.Http(disable_ssl_certificate_validation=True)
  
  searchjob = myhttp.request(base_url + search_job_urn, 'POST', headers=
  {'Authorization': 'Splunk %s' % sessionKey},
  body=urllib.urlencode({'search':'search index=indexname sourcetype=sourcename'}))[1]
  
  print searchjob
  </code></pre>
  
  <p>The last print keeps printing all results until done. For large queries i get "Memory Errors". I need to be able to read results in chunks (say 50,000) and write them to a file and reset the buffer for searchjob. How can i accomplish that?</p>