title: Go routine for processing files in a work flow. What am I missing?
body: <p>I am building something to monitor a directory for file uploads.  Right now I am using a for {} loop to continuously read the directory for testing purposes with the plan to use cron or something in the future to launch my application.</p>
  <p>The goal is to monitor an upload directory, ensure files have finished copying, then move the files to another directory for processing.  The files themselves range from 15GB to about 50GB and we will be receiving hundreds daily.</p>
  <p>This is my first foray into go routines.  I am not sure if I am completely misunderstanding go routines, channels and wait groups or something but I had thought that as I loop through a list of files, each file gets processed by a go routine function independently.  However when I run the below code it grabs a file but only acknowledges the first file it finds in the directory.  I noticed though once the first file finishes other files are acknowledged as completed.</p>
  <pre><code>package main
  
  import (
          &quot;flag&quot;
          &quot;fmt&quot;
          &quot;io/ioutil&quot;
          &quot;log&quot;
          &quot;os&quot;
          &quot;sync&quot;
          &quot;time&quot;
  
          &quot;gopkg.in/yaml.v2&quot;
  )
  
  type Config struct {
          LogFileName  string `yaml:&quot;logfilename&quot;`
          LogFilePath  string `yaml:&quot;logfilepath&quot;`
          UploadRoot   string `yaml:&quot;upload_root&quot;`
          TPUploadTool string `yaml:&quot;tp_upload_tool&quot;`
  }
  
  var wg sync.WaitGroup
  
  const WORKERS = 5
  
  func getConfig(fileName string) (*Config, error) {
          conf := &amp;Config{}
          yamlFile, err := os.Open(fileName)
          if err != nil {
                  fmt.Printf(&quot;Error reading YAML file: %s\n&quot;, err)
                  os.Exit(1)
          }
  
          defer yamlFile.Close()
          yaml_decoder := yaml.NewDecoder(yamlFile)
  
          if err := yaml_decoder.Decode(conf); err != nil {
                  return nil, err
          }
          return conf, err
  }
  
  func getFileData(fileToUpload string, fileStatus chan string) {
          var newSize int64
          var currentSize int64
  
          currentSize = 0
          newSize = 0
          fmt.Printf(&quot;Uploading: %s\n&quot;, fileToUpload)
          fileDone := false
          for !fileDone {
                  fileToUploadStat, _ := os.Stat(fileToUpload)
                  currentSize = fileToUploadStat.Size()
                  //fmt.Printf(&quot;%s current size is: %d\n&quot;, fileToUpload, currentSize)
                  //fmt.Println(&quot;New size &quot;, newSize)
                  if currentSize != 0 {
                          if currentSize &gt; newSize {
                                  newSize = currentSize
                          } else if newSize == currentSize {
                                  fileStatus &lt;- &quot;Done&quot;
                                  fileDone = true
                                  wg.Done()
                          }
                  }
                  time.Sleep(1 * time.Second)
          }
  
  }
  
  func sendToCDS() {
          fmt.Println(&quot;Sending To CDS&quot;)
  }
  
  func main() {
  
          fileStatus := make(chan string)
          configFileName := flag.String(&quot;config&quot;, &quot;&quot;, &quot;YAML configuration file.\n&quot;)
          flag.Parse()
  
          if *configFileName == &quot;&quot; {
                  flag.PrintDefaults()
                  os.Exit(1)
          }
  
          UploaderConfig, err := getConfig(*configFileName)
          if err != nil {
                  log.Fatal(&quot;Error reading configuration file.&quot;)
          }
  
          for {
                  fmt.Print(&quot;Checking for new files..&quot;)
                  uploadFiles, err := ioutil.ReadDir(UploaderConfig.UploadRoot)
                  if err != nil {
                          log.Fatal(err)
                  }
                  if len(uploadFiles) == 0 {
                          fmt.Println(&quot;..no files to transfer.\n&quot;)
                  }
                  for _, uploadFile := range uploadFiles {
                          wg.Add(1)
                          fmt.Println(&quot;...adding&quot;, uploadFile.Name())
                          if err != nil {
                                  log.Fatalln(&quot;Unable to read file information.&quot;)
                          }
                          ff := UploaderConfig.UploadRoot + &quot;/&quot; + uploadFile.Name()
  
                          go getFileData(ff, fileStatus)
                          status := &lt;-fileStatus
  
                          if status == &quot;Done&quot; {
                                  fmt.Printf(&quot;%s is done.\n&quot;, uploadFile.Name())
                                  os.Remove(ff)
                          }
                  }
                  wg.Wait()
          }
  }
  </code></pre>
  <p>I had thought about using channels for a thread safe queueing mechanism that loads up with the files in the directory and then the files get picked up by workers.  I have done similar things in Python.</p>