title: Servo/audio synchronization in animatronics on RPi. Considering FFT solution
body: <p>I've been working on an animatronic shooting gallery using a Raspberry Pi B+ and an Arduino Mega. Overall, things are going well except for one detail. I'm having trouble keeping the motor movements in sync with sound. </p>
  
  <p>For example, say I got this talking pig. You hit the target and the pig says "Hey, Kid! Watch were you're pointing that thing! You'll put your eye out!" or something like that. Problem is, the motors making the pig move can't keep up with the audio. They lag behind, getting out of sync with the audio. The longer the routine, the further they fall behind. What's more, it doesn't lag behind at a consistent rate. It seems to depend on what else the computer is doing at the time. For example, I had Audacity running at the same time as my program was running, and there was noticeably more lag then when my program was the only program running. I also notice a slight difference in lag between running my program in IDLE vs running from terminal.</p>
  
  <p>Some of my targets are more sensitive to synchronization than others, for example, if I had a rocket that went up to a whooshing sound, it's not really an issue if the motor moves a little faster or slower, but mouth movements look terrible if they're off by more than maybe a tenth of a second.</p>
  
  <p>To control my motors, I use a pickled list of tuples each containing 0: time in ms after beginning of routine for the position signal to be sent, 1: the number of the motor for the signal to be sent to, and 2: the position of the motor. The main program loops through all the targets and gives each a chance to see if it is time for it to send the next motor position command.</p>
  
  <p>Here is my current code:</p>
  
  <pre><code>import time
  import pygame.mixer
  import os
  import cPickle as pickle
  import RPi.GPIO as GPIO
  from Adafruit_PWM_Servo_Driver import PWM
  
  GPIO.setmode(GPIO.BOARD)
  chan_list = [17,18]
  GPIO.setup(chan_list, GPIO.IN, pull_up_down=GPIO.PUD_UP)
  GPIO.add_event_detect(17, GPIO.FALLING)
  GPIO.add_event_detect(18, GPIO.FALLING)
  
  
  pygame.mixer.init(channels=1, frequency=22050)
  pygame.mixer.set_num_channels(10)
  pwm = PWM(0x40)
  pwm.setPWMFreq(60)
  
  game_on = 1
  
  class Bot:
      def __init__(self, folder, servopins):
  
          self.startExecute = 0
          self.finishExecute = 0
  
          self.folder = folder # name of folder containing routine sound and motor position data
          self.servoPins = servopins #list of servo motor pins for this bot
  ##        self.IRpin = IRpin
          self.startTime = 0 # start time of curent routine in ms. begins at 0ms
          self.counter = 0 #next frame of routine to be executed
          self.isRunning = 0 #1 if bot is running a a routine 0 if not target inactive if 1
          self.routine = 0 # position of current routine in self.routines
  ##        self.numRoutines = 0 
          self.sounds = [] #list containing sound objects for routines. One per routine
          self.routines = [] #list of lists of tuples containing time, motor number, and motor position. One per routine
          self.list_dir = os.listdir(self.folder) # creates a list containing the names of all routine sound and motor position data files
          self.list_dir.sort()
          self.currentFrame = () # tuple containing current routine frame containing execution time, motor number and motor position waiting to be executed
  
          # appends all routine sound files into a list
          for filo in self.list_dir:
              if filo.endswith('.wav'):
                  self.sounds.append(pygame.mixer.Sound(self.folder + filo))
                  print filo
          #appends all routine motor position files into a list
          for filo in self.list_dir:
              if filo.endswith('.pkl'):
                  self.incoming = open(self.folder + filo)
                  self.pklroutine = pickle.load(self.incoming)
                  self.routines.append(self.pklroutine)
                  self.incoming.close()
                  print filo
  #        self.sound = pygame.mixer.Sound(str(self.routine) + '.wav')
  #        self.motorfile = open(str(self.routine) + '.pkl')
  
      # starts a routine running. resets counter to first frame of routine.
      #Starts routine timer. Starts routine sound object playing. Loads first frame of routine
      def run(self):
          self.isRunning = 1
          self.startTime = round(time.clock() * 1000)
          self.sounds[self.routine].play()
          self.currentFrame = self.routines[self.routine][self.counter]
  
  
      def execute(self):
  
          if self.counter == 0:
  
              self.startExecute = round(time.clock() * 1000)
  
          if self.currentFrame[0] &lt;= (round(time.clock() * 1000) - self.startTime):
              pwm.setPWM(self.servoPins[self.currentFrame[1]-1], 0, int(round(150 + ((self.currentFrame[2] * 2.6)))))
              if self.counter &lt; (len(self.routines[self.routine]) -2):
                  self.counter = self.counter + 1
                  self.currentFrame = self.routines[self.routine][self.counter]
              else:
                  print (round(time.clock() * 1000) - self.startTime)
  
                  self.finishExecute = round(time.clock() * 1000)
                  self.counter = 0
                  self.isRunning = 0
                  if self.routine &lt; (len(self.routines) - 1):
                      self.routine = self.routine + 1
                  else:
                      self.routine = 0
  
  
  
  bot1 = Bot('Pig/', [0,1])
  bot2 = Bot('Goat/', [2,3])
  
  while 1:
  
      if game_on:
          if GPIO.event_detected(17):
              bot1.run()
          if GPIO.event_detected(18):
              bot2.run()
  
  
  
      if bot1.isRunning:
          bot1.execute()
      if bot2.isRunning == 1:
          bot2.execute()
  </code></pre>
  
  <p>The list of tuples is created using two other programs, one on the Arduino Mega, and another on my PC using Python. The arduino is connected to a couple of potentiometers connected to +5V and ground, with the signal voltage coming from the middle terminal into an analog input. The arduino converts this analog value to a motor position in degrees, and sends that over the serial port to the PC that saves it in a tuple along with the motor number, and time that the byte was received, appends the tuple to a list, then at the end of the program, saves it in a pkl file. I will provide that code if somebody really wants to see it, but I don't believe the problem lies within the data produced by this process.</p>
  
  <p>If you have any suggestions about how I could fix this code to make it work, that would be the preferable solution, because the solution I'm contemplating now seems like it might just be one massive PITA.</p>
  
  <p>I'm thinking that I could use one audio channel (say, the left channel) for my audio, then in the other channel, I could have sine waves of different frequencies and amplitudes with each frequency being assigned to a specific motor. Say for example I have this one motor that moves a mouth. This motor will have an assigned frequency of 1000 Hz. Whenever the mouth needs to be open, there will be a 1000 Hz sine wave in the motor control channel. I would use an Arduino Mega to detect that sine wave, and send the PWM signal to the motor to move to a certain position. The position <em>may</em> even be determined by the amplitude of the sine wave. To make things a little simpler, I could just have 3 positions for the mouth, closed (no signal), open a little (small amplitude signal), and open a little more (larger amplitude signal).</p>
  
  <p>All of this would be done using this FFT library: <a href="http://wiki.openmusiclabs.com/wiki/ArduinoFFT" rel="nofollow noreferrer">http://wiki.openmusiclabs.com/wiki/ArduinoFFT</a></p>
  
  <p>I would need to build a circuit to convert the AC signal from the RPi to a 0-5V signal. I have found this circuit <a href="https://i.stack.imgur.com/cQ68q.png" rel="nofollow noreferrer">here</a> that looks like it plausibly might work, assuming I can find a way to cleanly amplify the signal from the RPi to +/- 2.5V. <a href="https://electronics.stackexchange.com/questions/148085/fft-beat-detection-circuit">Source</a></p>
  
  <p>Other than that, I don't know how difficult or effective it would be to implement this solution. For one, I'm not sure if I can send PWM signals to the motors and run the FFT library at the same time, and I've never worked with this library, or had any experience with FFT at all.</p>
  
  <p>My fallback solution is to design the shooting gallery so that it is less dependent on synchronization. This would involve such compromises such as just making the mouth open and close at a steady rate rather than trying to make it move in sync with audio, and keeping triggered routines short (&lt;= 5 seconds)</p>
  
  <p>Okay, problem defined, now on to specific questions:</p>
  
  <p>What can I do to make the current version of the program work without having to resort to analyzing a bunch of sine waves using FFT?</p>
  
  <p>Failing that, any suggestions on the input circuit? What could I use to cleanly amplify the RPi signal to +/- 2.5V, and will the circuit I linked to actually work to convert that signal to a 0V-5V signal readable by an Arduino analog input?</p>
  
  <p>Will using the FFT library interfere with sending the PWM signals to the motors?</p>
  
  <p>How much of a PITA will it be to extract usable data from a bunch of sine waves using FFT?</p>
  
  <p>The documentation for the FFT library is a little lacking, How do I set the range of frequencies to analyze? I see that it starts at 0 Hz(DC), but what's the high end? Can I set that? I only need maybe 30 usable bins. What frequencies should I use to get the clearest signals? I'm guessing I don't want to use the highest and lowest, because the lowest would take longer to detect, and the highest will be more distorted due to the poor quality of the audio coming out of the RPi. Should I set it to 64 bins and use the middle 30? And again, how do I determine the center frequency of each bin?</p>