title: Design pattern for non-blocking nested function calls in main thread of GUI?
body: <p>I need some help with identifying a design pattern or a technique to make a PyQt GUI application non-blocking, when it has multiple nested calls which invoke one external subprocess after another. The pseudocode is:</p>
  <pre><code>class MainWindow:
      def executeInSubprocess(self, theCommand):
          subprocess.run(theCommand, check = True)
          #program control reaches here only after the command executed by subprocess.run completes execution
    
      def function1(self):
          commandToExecute = ['python', '-m', 'someProgram.py', '--CLA_option']
          self.executeInSubprocess(commandToExecute)
          self.function2()
          #some commands to update the GUI widgets
      
      def function2(self):
          #Some lines here gather data generated by running function1
          self.function3(data)
    
      def function3(self, data): 
          commandToExecute = ['python', '-m', 'anotherProgram.py', '--CLA_option']
          for oneData in data:
              newCommand = commandToExecute[:] #copy contents of list
              newCommand.append('--someOption')
              newCommand.append(oneData)
              self.executeInSubprocess(newCommand)
  </code></pre>
  <p>When the user clicks a button on the GUI, the <code>function1</code> gets invoked. The button remains disabled until the <code>#some commands to update the GUI widgets</code> line, where the button is re-enabled.</p>
  <p><strong>The problem:</strong></p>
  <ol>
  <li>How to design it in such a way that <code>executeInSubprocess</code> is non-blocking? I'd like to try replacing <code>subprocess.run</code> with <code>QProcess</code>, which would not wait until the external command completes, thereby not freezing the GUI.</li>
  <li><code>executeInSubprocess</code> is called inside a <code>for</code> loop multiple times in <code>function3</code>, and is dependent on the data generated by <code>function1</code>.</li>
  </ol>
  <p>These functions are not too large, so it shouldn't be an issue refactoring it. What I need, is knowledge on a good way to design such a scenario. I've considered putting references to <code>function1</code>, <code>function2</code> and <code>function3</code> in a list and having a function in a class call them one by one, but then the <code>executeInSubprocess</code> in <code>function3</code> complicates things.</p>
  <p>Update: I've provided an example of how complex and intertwined the code can get.</p>
  <pre><code>#!/usr/bin/python
  # -*- coding: ascii -*-
  from PyQt5.QtWidgets import (QApplication, QMainWindow, QPushButton, QPlainTextEdit, QVBoxLayout, QWidget)
  from PyQt5.QtCore import QProcess
  from collections import deque
  import sys
  
  # Note: The objective was to design a way to execute a function and return control back to the main thread so that the GUI does not freeze.
  # But it had to be done in a way that a function could invoke another function, which could invoke another function. Each of these functions
  # can have one or more external commands that they need to run. So after invoking each external process, control has to return to the main thread
  # and then after running the external process completes, the next command or function needs to run and return control to the main thread and 
  # so on until all functions and external commands are exhausted. What's given below is a very crude, tangled design that needs to be improved.
  
  class ExternalProcessInvoker:
      def __init__(self):
          self.externalProcess = None
          self.currentFunction = None
          self.functionQueue = deque() #a sequence of functions to be executed
          self.commandQueue = dict() #{the key is a function reference: the value is a list of commands [['python', 'prog1.py'], ['python', 'prog2.py'], ...]}
          self.FIRST_ELEMENT_OF_LIST = 0
          self.UTF8 = 'utf8'        
          
      def registerTheMessageFunction(self, functionReference):
          self.message = functionReference
      
      def startTheProcess(self):
          while self.functionQueue:
              #print('Elements in queue are: ', self.functionQueue)
              theFunctionToExecute = self.functionQueue.popleft()
              #---run the function
              theFunctionToExecute(theFunctionToExecute) #The function may populate commandQueue when run
              self.currentFunction = theFunctionToExecute
              self.checkForMoreFunctionPopulatedCommandsToExecute()
  
      def checkForMoreFunctionPopulatedCommandsToExecute(self):
          #---check if commandQueue is populated with any external process that needs to be run    
          if self.currentFunction in self.commandQueue:
              if self.commandQueue[self.currentFunction]:#contains some command
                  command = self.commandQueue[self.currentFunction].popleft()
                  #command = self.convertToUTF8(command)
                  if self.externalProcess is None:  # No process running.
                      #---run the command
                      self.message('Executing ' + str(command)); print('Executing ', command)
                      self.externalProcess = QProcess() 
                      self.externalProcess.readyReadStandardOutput.connect(self.functionForStdout)
                      self.externalProcess.readyReadStandardError.connect(self.functionForStderr)
                      self.externalProcess.stateChanged.connect(self.functionForStateChange)
                      self.externalProcess.finished.connect(self.processCompleted)  # Clean up once complete.
                      self.externalProcess.start(command[self.FIRST_ELEMENT_OF_LIST], command[self.FIRST_ELEMENT_OF_LIST:]) #the first argument is the first command element (usually 'python'). The second argument is a list of all the remaining elements of the command
                      print('Process ID: ', self.externalProcess.processId())    
          else:
              self.currentFunction = None              
      
      def convertToUTF8(self, command):
          return [oneString.encode('utf-8') for oneString in command]
  
      def stopTheProcess(self):
          self.message('Attempting to stop the processes')
          if self.externalProcess:
              self.externalProcess.kill()
          self.message('Also cleared ' + len(self.functionQueue) + ' items from execution queue')
          self.functionQueue.clear()
  
      def functionForStderr(self):
          data = self.externalProcess.readAllStandardError()
          stderr = bytes(data).decode(self.UTF8)
          self.message(stderr)
  
      def functionForStdout(self):
          data = self.externalProcess.readAllStandardOutput()
          stdout = bytes(data).decode(self.UTF8)
          self.message(stdout)
  
      def functionForStateChange(self, state):
          states = {QProcess.NotRunning: 'Not running', QProcess.Starting: 'Starting', QProcess.Running: 'Running'}
          state_name = states[state]
          self.message(f'State changed: {state_name}')
  
      def processCompleted(self):
          self.message('Process finished.')
          self.externalProcess = None
          self.checkForMoreFunctionPopulatedCommandsToExecute()
  
      def addThisFunctionToProcessingQueue(self, functionReference):
          #print('Received request to add this function to processing queue: ', functionReference)
          #print('What's already in processing queue: ', self.functionQueue)
          self.functionQueue.append(functionReference)
  
      def addThisCommandToProcessingList(self, command, functionReference):
          if functionReference not in self.commandQueue:
              self.commandQueue[functionReference] = deque()
          self.commandQueue[functionReference].append(command)
          #print('\nfunc ref: ', functionReference, ' command: ', command)
          self.message('Added ' + str(command) + ' for processing')
          #print(self.commandQueue)
  
  
  class MainWindow(QMainWindow):
      def __init__(self):
          super().__init__()
          self.proc = ExternalProcessInvoker()
          self.proc.registerTheMessageFunction(self.message)
  
          self.executeButton = QPushButton('Execute') #TODO: Disable button until processing completes or processes are killed
          self.executeButton.pressed.connect(self.initiateProcessing)
          self.killButton = QPushButton('Kill process') #TODO: Disable button until processing is started
          self.killButton.pressed.connect(self.killProcessing)        
          self.textDisplay = QPlainTextEdit()
          self.textDisplay.setReadOnly(True)
  
          boxLayout = QVBoxLayout()
          boxLayout.addWidget(self.executeButton)
          boxLayout.addWidget(self.killButton)
          boxLayout.addWidget(self.textDisplay)
  
          theWidget = QWidget()
          theWidget.setLayout(boxLayout)
  
          self.setCentralWidget(theWidget)
          
      def message(self, mssg):
          self.textDisplay.appendPlainText(mssg)
          
      def initiateProcessing(self):
          print('Processing initiated')
          #---first populate the functions and commands to execute
          self.proc.addThisFunctionToProcessingQueue(self.function1)
          self.proc.addThisFunctionToProcessingQueue(self.function2)
          self.proc.addThisFunctionToProcessingQueue(self.function3)
          self.proc.startTheProcess()
          
      def function1(self, functionReference):
          self.message('Executing function1')
          command = ['python', 'dummyScript1.py']
          self.proc.addThisCommandToProcessingList(command, functionReference)
          
      def function2(self, functionReference):
          self.message('Executing function2')
          
      def function3(self, functionReference):
          self.message('Executing function3')
          maxInvocations = 5
          secondsToSleep = 1
          for _ in range(maxInvocations):
              command = ['python', 'dummyScript2.py', str(secondsToSleep)]
              self.proc.addThisCommandToProcessingList(command, functionReference) 
              secondsToSleep = secondsToSleep + 1
  
          self.message('Processing complete')
          print('Completed')
      
      def killProcessing(self):
          self.proc.stopTheProcess()
  
  app = QApplication(sys.argv)
  w = MainWindow()
  w.show()
  app.exec_()
  </code></pre>
  <p>Dummy script 1:</p>
  <pre><code>#!/usr/bin/python
  # -*- coding: ascii -*-
  import time
  import datetime
  
  sec = 3
  print(datetime.datetime.now(), &quot; Going to sleep for &quot;, sec, &quot; seconds&quot;)
  time.sleep(sec)
  print(datetime.datetime.now(), &quot; Done sleeping dummy script1&quot;)
  </code></pre>
  <p>Dummy script 2:</p>
  <pre><code>#!/usr/bin/python
  # -*- coding: ascii -*-
  import time
  
  def gotosleep(sleepTime):
      print(&quot;Going to sleep for &quot;, sleepTime, &quot; seconds&quot;)
      time.sleep(sleepTime)
      print(&quot;Done sleeping for &quot;, sleepTime, &quot; seconds in dummy script2&quot;)
  
  if __name__ == &quot;__main__&quot;:
      FIRST_EXTRA_ARG = 1
      sleepTime = sys.argv[FIRST_EXTRA_ARG]
      gotosleep(sleepTime)
  </code></pre>