title: Python 3.x Segmentation Fault with Threading module
body: <p>Machine: Raspberry Pi 3B+ <br>
  System: Raspbian GNU/Linux 10 (buster)<br>
  Python version: Python 3.7.3</p>
  <p>I have created code, that downloads chapters for novel from internet site protected by Cloudflare. Since downloading 400 sites with 5 seconds delay each time took over 1.5 hour i rewrote the code to use it with threading module.
  It was a success. With 4 threads i managed to take it down to 16 minutes.</p>
  <p>But from time to time Segmentation Fault occured. Also, when i increased thread count, it occured more often/sooner in the progress(of download) and with 6 it was almost certain, that it will occur.</p>
  <p>First thing i tried to do(which i found somewhere on Raspberry Pi forums) was to increase stack size with <code>ulimit -s (stack size)</code>. It didn't help. I had a feeling, that it increased how often it occured.</p>
  <p>After that i found <code>faulthandler</code> module <a href="https://blog.richard.do/2018/03/18/how-to-debug-segmentation-fault-in-python/" rel="nofollow noreferrer">here</a>. I enabled it with</p>
  <pre><code>import faulthandler
  faulthandler.enable()
  PYTHONFAULTHANDLER = 1
  </code></pre>
  <p>and it gave me this once fault occured (core is function defined in the script):</p>
  <pre><code>    Fatal Python error: Segmentation fault
  
  Thread 0x6f6ff460 (most recent call first):
    File &quot;new Tensura download.py&quot;, line 89 in core
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 865 in run
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 917 in _bootstrap_inner
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 885 in _bootstrap
  
  Current thread 0x700ff460 (most recent call first):
    File &quot;/usr/lib/python3/dist-packages/OpenSSL/SSL.py&quot;, line 1914 in do_handshake
    File &quot;/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py&quot;, line 453 in wrap_socket
    File &quot;/usr/lib/python3/dist-packages/urllib3/util/ssl_.py&quot;, line 344 in ssl_wrap_socket
    File &quot;/usr/lib/python3/dist-packages/urllib3/connection.py&quot;, line 344 in connect
    File &quot;/usr/lib/python3/dist-packages/urllib3/connectionpool.py&quot;, line 841 in _validate_conn
    File &quot;/usr/lib/python3/dist-packages/urllib3/connectionpool.py&quot;, line 343 in _make_request
    File &quot;/usr/lib/python3/dist-packages/urllib3/connectionpool.py&quot;, line 600 in urlopen
    File &quot;/home/pi/.local/lib/python3.7/site-packages/requests/adapters.py&quot;, line 449 in send
    File &quot;/home/pi/.local/lib/python3.7/site-packages/requests/sessions.py&quot;, line 646 in send
    File &quot;/home/pi/.local/lib/python3.7/site-packages/requests/sessions.py&quot;, line 533 in request
    File &quot;/home/pi/.local/lib/python3.7/site-packages/cfscrape/__init__.py&quot;, line 121 in request
    File &quot;/home/pi/.local/lib/python3.7/site-packages/requests/sessions.py&quot;, line 546 in get
    File &quot;new Tensura download.py&quot;, line 49 in core
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 865 in run
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 917 in _bootstrap_inner
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 885 in _bootstrap
  
  Thread 0x70aff460 (most recent call first):
    File &quot;/usr/lib/python3/dist-packages/bs4/builder/_lxml.py&quot;, line 287 in feed
    File &quot;/usr/lib/python3/dist-packages/bs4/__init__.py&quot;, line 342 in _feed
    File &quot;/usr/lib/python3/dist-packages/bs4/__init__.py&quot;, line 281 in __init__
    File &quot;new Tensura download.py&quot;, line 50 in core
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 865 in run
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 917 in _bootstrap_inner
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 885 in _bootstrap
  
  Thread 0x714ff460 (most recent call first):
    File &quot;/usr/lib/python3/dist-packages/OpenSSL/SSL.py&quot;, line 773 in load_verify_locations
    File &quot;/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py&quot;, line 428 in load_verify_locations
    File &quot;/usr/lib/python3/dist-packages/urllib3/util/ssl_.py&quot;, line 321 in ssl_wrap_socket
    File &quot;/usr/lib/python3/dist-packages/urllib3/connection.py&quot;, line 344 in connect
    File &quot;/usr/lib/python3/dist-packages/urllib3/connectionpool.py&quot;, line 841 in _validate_conn
    File &quot;/usr/lib/python3/dist-packages/urllib3/connectionpool.py&quot;, line 343 in _make_request
    File &quot;/usr/lib/python3/dist-packages/urllib3/connectionpool.py&quot;, line 600 in urlopen
    File &quot;/home/pi/.local/lib/python3.7/site-packages/requests/adapters.py&quot;, line 449 in send
    File &quot;/home/pi/.local/lib/python3.7/site-packages/requests/sessions.py&quot;, line 646 in send
    File &quot;/home/pi/.local/lib/python3.7/site-packages/requests/sessions.py&quot;, line 533 in request
    File &quot;/home/pi/.local/lib/python3.7/site-packages/cfscrape/__init__.py&quot;, line 121 in request
    File &quot;/home/pi/.local/lib/python3.7/site-packages/cfscrape/__init__.py&quot;, line 219 in solve_cf_challenge
    File &quot;/home/pi/.local/lib/python3.7/site-packages/cfscrape/__init__.py&quot;, line 129 in request
    File &quot;/home/pi/.local/lib/python3.7/site-packages/requests/sessions.py&quot;, line 546 in get
    File &quot;new Tensura download.py&quot;, line 49 in core
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 865 in run
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 917 in _bootstrap_inner
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 885 in _bootstrap
  
  Thread 0x71eff460 (most recent call first):
    File &quot;new Tensura download.py&quot;, line 253 in core
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 865 in run
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 917 in _bootstrap_inner
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 885 in _bootstrap
  
  Thread 0x728ff460 (most recent call first):
    File &quot;/usr/lib/python3.7/selectors.py&quot;, line 415 in select
    File &quot;/usr/lib/python3.7/subprocess.py&quot;, line 1681 in _communicate
    File &quot;/usr/lib/python3.7/subprocess.py&quot;, line 939 in communicate
    File &quot;/home/pi/.local/lib/python3.7/site-packages/cfscrape/__init__.py&quot;, line 322 in solve_challenge
    File &quot;/home/pi/.local/lib/python3.7/site-packages/cfscrape/__init__.py&quot;, line 204 in solve_cf_challenge
    File &quot;/home/pi/.local/lib/python3.7/site-packages/cfscrape/__init__.py&quot;, line 129 in request
    File &quot;/home/pi/.local/lib/python3.7/site-packages/cfscrape/__init__.py&quot;, line 237 in solve_cf_challenge
    File &quot;/home/pi/.local/lib/python3.7/site-packages/cfscrape/__init__.py&quot;, line 129 in request
    File &quot;/home/pi/.local/lib/python3.7/site-packages/requests/sessions.py&quot;, line 546 in get
    File &quot;new Tensura download.py&quot;, line 49 in core
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 865 in run
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 917 in _bootstrap_inner
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 885 in _bootstrap
  
  Thread 0x7329c460 (most recent call first):
    File &quot;/usr/lib/python3/dist-packages/bs4/builder/_lxml.py&quot;, line 287 in feed
    File &quot;/usr/lib/python3/dist-packages/bs4/__init__.py&quot;, line 342 in _feed
    File &quot;/usr/lib/python3/dist-packages/bs4/__init__.py&quot;, line 281 in __init__
    File &quot;new Tensura download.py&quot;, line 50 in core
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 865 in run
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 917 in _bootstrap_inner
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 885 in _bootstrap
  
  Thread 0x76fe5ad0 (most recent call first):
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 1048 in _wait_for_tstate_lock
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 1032 in join
    File &quot;/usr/lib/python3.7/threading.py&quot;, line 1281 in _shutdown
  bash: linia 1: 18565 Naruszenie ochrony pamięci   /usr/bin/python3 &quot;new Tensura download.py&quot;
  </code></pre>
  <p>I see, that the fault is caused by threading module, but i have no idea how to fix it.<br>
  If you have any clue, on how to eliminate this, i will be thankful.</p>
  <p>Here is the whole code if somebody wants it the threading initiator is at the very end (note, that <strong>enscript</strong> and <strong>ps2pdf</strong> terminal programs are required to run core and <strong>cfscrape</strong> module is request module for Cloudflare protected websites):</p>
  <pre><code>#improvment from 1h 40min to 1h 10 min with new version
  #with 2 threads this goes down to 36 min
  #with 4 down to 16 min (if i remember correctly)
  #with 5 down to 13 min 40 s
  #with 6 down to 13 min (segmentation faults occur)
  
  #handling segmentation fault
  import faulthandler
  faulthandler.enable()
  PYTHONFAULTHANDLER = 1
  
  import cfscrape
  from bs4 import BeautifulSoup
  import threading
  import subprocess
  from time import time
  
  #------time tracking class------
  class time_check():
      def __init__(self):      #starts time tracking
          self.start_t = time()
      def check(self):         #prints current time from start
          end_t = time()
          t = end_t - self.start_t
          
          t_sec = int(t % 60)
          t_min = int(((t - t_sec) / 60) % 60)
          t_hr = int(((((t - t % 60) / 60) - t_min) / 60) % 24)
          
          print(f'Operation took {t_hr} hours {t_min} minuts {t_sec} seconds.')
  
  #----------------------------creating scraper------------------------
  print('scrape s')
  scrape = cfscrape.create_scraper()
  source = scrape.get('https://www.readlightnovel.org/tensei-shitara-slime-datta-ken-wn', headers={&quot;User-Agent&quot;: &quot;XZ&quot;}).content
  print('scrape e')
  
  soup = BeautifulSoup(source, 'lxml')
  
  #---------------------char table-----------------------------
                    #   V    Vnbsp-V would you believe? there are more than 1 type of spaces. one is 160(noble space), other 32 in hex, another 12288
  chars = ['\t', '\n', '\r', ' ', ' ', '　', '', '「', '“', '」', '”', '『', '』'] #         VV VV these are not the same markers
  for x in '''QWERTYUIOPASDFGHJKLZXCVBNMqwertyuiopasdfghjklzxcvbnm-_=+,＜&lt;《〈≪.≫〉》&gt;＞[{]};:'&quot;/?`‘’~1234567890!！@#$%^&amp;*＊※()\|…–—−ー・→°✦✧''':
      chars.append(x)
  
  #-----------------------getting more links from main site------------------------------------
  print('link s')
  links, link = [], []
  table = soup.find_all('ul', class_='chapter-chs')[:4]
  for x in table:
      x = x.find_all('a')
      links += x
      
  for x in links:
      link += [x['href']]
  print('link e')
  
  #--------------------core------------------------------
  #purely esthetic purpose, brerak lines, changes characters, nicely breaks lines and saves it to files
  def core(links, thr_id='?'):
      ttime = time_check()
      for num, link in enumerate(links):
          
          #--------------------------request--------------------------------
          source = scrape.get(link, headers={&quot;User-Agent&quot;: &quot;XZ&quot;}).content
          soup = BeautifulSoup(source, 'lxml')
          
          
          #---------------------some deleting of useless data---------------------------
          for trin in soup.find_all('div', class_='trinity-player-iframe-wrapper'):
              trin.replace_with('')
          for scr in soup.find_all('script'):
              scr.replace_with('')
          for scr in soup.find_all('text-center add11'):
              scr.replace_with('')
          for scr in soup.find_all('text-center'):
              scr.replace_with('')
          for div in soup.find_all('div', 'hidden'):
              div.replace_with('')
          
          #----------getting title for chapter---------------------------
          try:
              title = soup.find('div', class_='block-title').h1.text
          except:
              try:
                  title = f'v6 Tensei Shitara Slime Datta Ken (WN) - Chapter {num.name}'
              except:
                  title = f'v6 Tensei Shitara Slime Datta Ken (WN) - Chapter {num}'
          
          #-------------------getting wanted part of the site--------------------
          desc = soup.find('div', class_='desc').text
          
          #----------------new line/dialogue breaker------------------------------
          beg_list, end_list, tetr, out = ['「', '(', '&lt;', '“', '＜', '《', '≪'], ['」', ')', '&gt;', '”', '》', '≫'], '   ', ''
          dialog = 0 
          
          #it is based on cart/sight/focus basically there is this tetr variable.
          #At the beginning it changes its focus from lets say characters 21,22,23 to 22,23,24, and at the end addes first one to the output(in this case it would add 22).
  
          #thats only if nothing would be changed.
          #But each time newest letter (24 for this example) is added, it goes through a scanner.
          
          #Some(all) of them manually alter the focus, so that new letters can be added(mostly \n).
          for x in desc:
              tetr = tetr[1:] + x
                          
              if tetr in [' . ', ' . ', '　.　'] and dialog == 0:     #line end after dot v1
                  tetr = ' \n '
                  out += '.'
              
              elif tetr[1:] in ['. ', '. ', '.　'] and dialog == 0:   #line end after dot v2
                  out += tetr[0]
                  tetr = tetr[0] + '.\n'
              
              elif x in beg_list and dialog == 0:     #beginning of dialog/thought or something similar
                  dialog = 1
                  out += tetr[0:2]
                  tetr = tetr[2] + '\n' + x
                  
              elif x in end_list:                           #end of above
                  dialog = 0
                  out += tetr[0:2]
                  tetr = tetr[2] + x + '\n'
              
              elif x in ['!', '！', '?'] and dialog == 0:        #breaker for ! and ?
                  out += tetr[0:2]
                  tetr = tetr[1] + x + '\n'
              
              else:                                       #for all other letters
                  out += tetr[0]
                  
          desc = out
          del out, beg_list, end_list, tetr, dialog
          
          #------------------character changer------------------------
          out, tetr = '', '   '
          for x in desc:
              tetr = tetr[1:] + x
              
              
              if tetr[0] not in chars:
                  out += '\\'
                  continue
              
              if tetr[0] in ['△'] :
                  out += '$'
                  
              elif tetr[0] in ['’', '‘'] :
                  out += &quot;'&quot;
                  
              elif tetr[0] in ['「'] :
                  out += '['
                  
              elif tetr[0] in ['」'] :
                  out += ']'
                  
              elif tetr[0] in ['＞', '〉'] :
                  out += '&gt;'
                  
              elif tetr[0] in ['＜', '〈'] :
                  out += '&lt;'
                  
              elif tetr[0] in ['“', '”'] :
                  out += '&quot;'
                  
              elif tetr[0] in ['＊'] :
                  out += '*'
                  
              elif tetr[0] in ['※']:
                  out += '--'
                  
              elif tetr[0] in ['…'] :
                  out += '...'
                  
              elif tetr[0] in ['『', '“', '《', '≪', '〈'] :
                  out += '&lt;&lt;'
                  
              elif tetr[0] in ['』', '”', '》', '≫', '〉'] :
                  out += '&gt;&gt;'
                  
              elif tetr[0] in ['–', '—', 'ー', '・', '−'] :
                  out += '-'
                  
              elif tetr[0] in ['→'] :
                  out += '-&gt;'
                  
              elif tetr[0] in [' ', '　']:
                  out += ' '
                  
              elif tetr[0] in ['\t']:
                  out += '    '
                  
              elif tetr[0] in ['！']:
                  out += '!'
                  
              elif tetr[0] in ['é']:
                  out += 'e'
                  
              elif tetr[0] in ['Ō']:
                  out += 'Oo'
                  
              elif tetr[0] in ['✦', '✧']:
                  out += '*'
                  
              elif tetr[0] in ['°']:
                  out += ' degrees '
              
              else:
                  out += tetr[0]
               
          desc = out
          del out, tetr
          
          #--------------------nice line break---------------------------------
          line, f_list = '', []
          for x in desc:            #converting long text into list of shorter strings
              if x == '\n':
                  f_list.append(line)
                  line = ''
              line += x
          
          for y, line in enumerate(f_list):
              if len(line) &gt; 91:
                  for z in range(91, -1, -1):
                      v = line[z]
                      if v in [' ', ' ']:              #looking for closest space from the (length) character to first
                          sep = z    #+1          #separator
                          break
                  else:                       #or taking the (width) bit
                      sep = 90
                      
                  if sep == 0:
                      sep = 1
                      
                  part1 = line[:sep]   #separating line
                  part2 = line[sep:]
                  f_list[y] = part1 + '\n'      #and adding it to menu  (part1 to place on old line)
                  f_list.insert(y+1, part2)                   #(part 2 to next line)
          
          out = ''
          for line in f_list:
              out += line
              
          desc = out
          del out, sep, part1, part2, f_list, line
          
          #-------------title adder------------------
          desc = title + '\n' + desc
          
          #----------------file creator------------------------
          file = f'v7 {title}'
          with open(f'v7/{file}.txt', 'w') as file:
              file.write(desc)
          file = f'v7 {title}'
          
          #----------------------------ps creeation--------------------------
          n_file_ps = f'{file}.ps'
          proc = subprocess.Popen(['enscript', '-p', f'psv7/{n_file_ps}', f'v7/{file}.txt', f'--header={file}'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
          proc.wait()
          
          #---------------------------pdf creation----------------------------------
          n_file_pdf = f'{file}.pdf'
          proc = subprocess.Popen(['ps2pdf', f'psv7/{n_file_ps}', f'pdfv7/{n_file_pdf}'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
          proc.wait()
          
          print(thr_id, n_file_pdf, 'done')
          
      ttime.check()
  
  #-----------------------------folders--------------------------------
  try:       #deleing existing files
      proc = subprocess.Popen('cd pdfv7 &amp;&amp; rm v7*', stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, shell=True)
      proc = subprocess.Popen('cd psv7 &amp;&amp; rm v7*', stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, shell=True)
      proc = subprocess.Popen('cd v7 &amp;&amp; rm v7*', stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, shell=True)
  except:
      pass
  
  for fold in ['psv7', 'pdfv7', 'v7']:
      try:    #creating new folders
          proc = subprocess.Popen(['mkdir', fold], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
      except:
          pass
      proc.wait()
  
  #---------------------threads creation------------------------------------------
  #for fixing segmentation error i used terminal command(8192 default) &quot;ulimit -s 16284&quot; &lt;---no, i changed it back
  thr_times = 7          #6 sometimes throws up segmentation fault (like 5 but more often)(4 also does that, but more chaotically)
  
  threads = {}
  for x in range(thr_times):    #creating threads                                     #V V V splitting links equally for all of the threads
      threads[str(x)] = threading.Thread(target=core, name=f'Tensura thr.{x}', args=([link[x::thr_times], x])) #&lt;-- x == thread id
  
  for thread in threads:
      threads[thread].start()
  
  
  # core([link[80]])
  </code></pre>
  <p><br><br>
  Edit:<br>
  Okay, since no one was answering i created my workaround.<br>
  If you can't capture segmentation fault within script just create another script that launches and monitors if script is running. Pretty neat huh?<br>
  Problem is if script is killed near the end, as it will have to start all over, but you can go away from keyboard and if Segmentation Fault is you only problem, this script will have the work done(unless something else happens).<br></p>
  <pre><code>import subprocess
  import time
  import psutil
  from fnctime import time_check
  
  def kill(proc_pid):
      process = psutil.Process(proc_pid)
      for proc in process.children(recursive=True):
          proc.kill()
      process.kill()
  
  ttime = time_check()
  
  #begin process
  p0 = subprocess.Popen('cd /home/pi/scripts/&quot;Tensura Beautiful soup&quot; &amp;&amp; python3.7 &quot;new Tensura download.py&quot;', stderr=subprocess.PIPE, text=True, shell=True)
  #define variable for checking if p0 is alive
  status = p0.poll()
  
  try:
      while True:
          if status != None:          #if process is dead  start it again
              print('\nStart it again')
              p0 = subprocess.Popen('cd /home/pi/scripts/&quot;Tensura Beautiful soup&quot; &amp;&amp; python3.7 &quot;new Tensura download.py&quot;', stderr=subprocess.PIPE, text=True, shell=True)
              status = p0.poll()      #and renew status
              
          while status == None:                        #while it is alive
              print(p0.stdout if p0.stdout else '', end='')     #print what it has printed
              time.sleep(1)                    #wait
              status = p0.poll()              #check again
              
          a, b = p0.communicate()
          
          if b == 'Segmentation fault\n':        #if Segmentation fault occured: 
              print('--Segmentation fault--')
              continue                           #go again
          if 'UnicodeDecodeError' in b:
              print('Unicode Decode Error')
              continue
          print(f'--b: {b} --')
          print('--all done--')
          break                        #break whole loop
  except KeyboardInterrupt:          #if ctrl^C was clicked
      try:
          kill(p0.pid)           #kill the process
      except:
          print('already dead')      #unless it's dead already
  
  ttime.check()
  </code></pre>
  <p>I hope i helped in some way to live with this problem without solving it.</p>