title: tqdm slows down my program at a factor of at least 8
body: <p>To give a bit of context, I have a converter that request JSON data in two steps, one step 1 where it request the complete amount of Data_points and the second where per data point the details are requested. To keep track of the progress, because I am expecting to request a lot of data from this point. That is why I use tqdm, the problem it really is slowing my program done by at least a factor of 8.</p>
  <pre><code>import requests
  import json
  import os
  import time
  from datetime import timedelta
  from datetime import datetime
  from datetime import date
  import pandas as pd
  import shutil
  import zipfile
  import smtplib, ssl
  from progress.bar import Bar
  from tqdm import tqdm
  from time import sleep
  </code></pre>
  <p>Here is the code:</p>
  <pre><code>def fetch_data_points(url: str):
      limit_request = 100
      # Placeholder for limit: please do not remove = 1000000000 -JJ
      folder_path_reset(&quot;api_request_jsons&quot;,&quot;csv&quot;,&quot;Geographic_information&quot;)
      total_start_time = start_time_measure()
      start_time = start_time_measure(
          'Starting Phase 1: First request from API: Data Points')
  
      for i in tqdm(range(limit_request)):
          response = requests.get(url,params={&quot;limit&quot;: limit_request})
      API_status_report(response)
      end_time_measure(total_start_time, &quot;Request completed: &quot;)
      end_time_measure(total_start_time, &quot;End of Phase 1, completed in: &quot;)
      return response.json()
  </code></pre>
  <p>Keep note of the time here:</p>
  <p>Here is the console working with tqdm.</p>
  <pre><code>Starting Phase 1: First request from API: Data Points
  100%|██████████| 100/100 [00:21&lt;00:00,  4.69it/s]Successfull connection!
  
  Request completed: 0:00:21.359000
  End of Phase 1, completed in: 0:00:21.359000
  Saving points
  Exported_data\api_request_jsons\Fetch_points\Points.json saved
  Point saved: 0:00:00.016000
  Data saved. Total time of program run: 0:00:00.016000
  Starting Phase 2: Second request from API: 100 requested
    9%|▉         | 9/100 [02:12&lt;22:17, 14.69s/it]
  </code></pre>
  <p>And here is the console without tqdm:</p>
  <pre><code>Starting Phase 1: First request from API: Data Points
  Successfull connection!
  Request completed: 0:00:00.297000
  End of Phase 1, completed in: 0:00:00.297000
  Saving points
  Exported_data\api_request_jsons\Fetch_points\Points.json saved
  Point saved: 0:00:00.015000
  Data saved. Total time of program run: 0:00:00.015000
  Starting Phase 2: Second request from API: 100 requested
   10%|█         | 10/100 [01:54&lt;16:52, 11.25s/it]
  </code></pre>
  <p><a href="https://i.stack.imgur.com/qTv2i.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/qTv2i.png" alt="enter image description here" /></a></p>
  <p>As you see here the program has slowed down by almost factor of ten. A request of 100 points takes normally 00.297000 seconds. But with TQDM that is 0:00:21.359000. More than five times slower than it should be. I would expect two times slower but five times its bit too much.. Can anyone give me some pointers to minimize this slow down as much as possible.</p>
  <p>Edit: Okay I have decided to abandon this the tqdm measure on the first function, I just cannot get it done right. It takes too much tweaking and I have noticed that when I adjust the amount of data being requested the data is clearly inconsistent.</p>
  <p>So I tried on the second function the code that is relevant is this:</p>
  <p>To explain it request the details of each data-point and puts them into an array for later use:</p>
  <pre><code>def fetch_details_of_data_points(url: str):
      input_json = fetch_data_points(url)
      fetch_points_save(input_json)
      all_location_points_details = []
      amount_of_objects = len(input_json)
      total_start_time = start_time_measure()
      start_time = start_time_measure(f'Starting Phase 2: Second request from API: {str(amount_of_objects)} requested')
      for i in tqdm(range(amount_of_objects),miniters=1):
          for obj in input_json:
  </code></pre>
  <p>all_location_points_details.append(fetch_details(obj.get(&quot;detail&quot;)))</p>
  <pre><code>def fetch_details(url: str):
      response = requests.get(url)
      # Makes request call to get the data of detail
      # save_file(folder_path,GipodId,text2)
      # any other processe
      return response.json()
  </code></pre>
  <p>but I am getting an error here:</p>
  <pre><code> Message=('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
    Source=C:\Users\xxxxxx\GIPOD_REQUEST_CONVERSION.py
    StackTrace:
    File &quot;C:\Users\QF6207\xxxxxx\GIPOD_REQUEST_CONVERSION.py&quot;, line 195, in fetch_details
      response = requests.get(url)
    File &quot;C:\Users\xxxxxx\GIPOD_REQUEST_CONVERSION.py&quot;, line 361, in fetch_details_of_data_points
      all_location_points_details.append(fetch_details(obj.get(&quot;detail&quot;)))
    File &quot;C:\Users\xxxxxx\GIPOD_REQUEST_CONVERSION.py&quot;, line 446, in &lt;module&gt; (Current frame)
      fetch_details_of_data_points(api_response_url)
  </code></pre>
  <p>From what I can understand apparently the Request for one point datapoint just takes way too long causing the disconnect to happen.
  Of note I know from experience that the request of one Datapoint it takes roughly 0.25 seconds for the data to be requested. So in theory the Progress bar should update and count up by one increment and update for every 0.25 seconds.</p>
  <p>Now if this can be kind of be resolved by having the response time as the of the get command being the update time that would help a lot of making the progress bar a bit more accurate.</p>
  <p>So how do I do this?</p>
  <p>Edit: I have found a solution for my problem without actually doing much delay, after reading through I have found a creative way to the manual update after the function is done.</p>
  <pre><code>with tqdm (total=limit) as firstrequest:
      all_location_points_details = fetch_details_of_data_points(url,limit)
      firstrequest.update(limit)
  
  
  with tqdm(total=amount_of_objects) as second_request:
      for obj in input_json: 
          all_location_points_details.append(fetch_details(obj.get(&quot;detail&quot;)))
          second_request.update(1)
  </code></pre>