{"action":"opened","issue":{"url":"https://api.github.com/repos/QutBioacoustics/baw-workers/issues/38","repository_url":"https://api.github.com/repos/QutBioacoustics/baw-workers","labels_url":"https://api.github.com/repos/QutBioacoustics/baw-workers/issues/38/labels{/name}","comments_url":"https://api.github.com/repos/QutBioacoustics/baw-workers/issues/38/comments","events_url":"https://api.github.com/repos/QutBioacoustics/baw-workers/issues/38/events","html_url":"https://github.com/QutBioacoustics/baw-workers/issues/38","id":152673043,"number":38,"title":"Non-zero exits codes from AP.exe not honored","user":{"login":"atruskie","id":1452968,"avatar_url":"https://avatars.githubusercontent.com/u/1452968?v=3","gravatar_id":"","url":"https://api.github.com/users/atruskie","html_url":"https://github.com/atruskie","followers_url":"https://api.github.com/users/atruskie/followers","following_url":"https://api.github.com/users/atruskie/following{/other_user}","gists_url":"https://api.github.com/users/atruskie/gists{/gist_id}","starred_url":"https://api.github.com/users/atruskie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/atruskie/subscriptions","organizations_url":"https://api.github.com/users/atruskie/orgs","repos_url":"https://api.github.com/users/atruskie/repos","events_url":"https://api.github.com/users/atruskie/events{/privacy}","received_events_url":"https://api.github.com/users/atruskie/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/QutBioacoustics/baw-workers/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-05-03T00:04:52Z","updated_at":"2016-05-03T00:04:52Z","closed_at":null,"body":"In this scenario the `ffprobe` executable was missing. AP.exe correctly failed to execute and return an `ERRORLEVEL` of `52`. This non-zero exit code was not detected and the job appears to have registered as successful. There is a `job.success` file, the action does not get moved to the failed queue, and the `worker.log` file thinks the run is successful as well.\r\n\r\nEnsure the job fails is the executable script that is run fails.\r\n\r\nAP.exe log:\r\n```\r\n2016-05-02T09:35:05.1706370+00:00 [1] INFO  LoggedConsole - QUT Bioacoustic Analysis Program - version 15.10.3309.0 (RELEASE build, 10/30/2015 06:54) \r\nGit branch-version: master-7cd0fa5e863158369c8d7043266a6f36342c1cf5\r\nCopyright QUT 2016\r\n2016-05-02T09:35:05.1836340+00:00 [1] INFO  LogFileOnly - Executable called with these arguments: \r\n/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/programs/AnalysisPrograms/AnalysisPrograms.exe audio2csv /source:/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav /config:/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config /tempdir:/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp /output:/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2\r\n\r\n2016-05-02T09:35:05.7220910+00:00 [1] INFO  LoggedConsole - # PROCESS LONG RECORDING\r\n2016-05-02T09:35:05.7224820+00:00 [1] INFO  LoggedConsole - # DATE AND TIME: 05/02/2016 09:35:05\r\n2016-05-02T09:35:05.7225780+00:00 [1] INFO  LoggedConsole - # Recording file:      /home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav\r\n2016-05-02T09:35:05.7228770+00:00 [1] INFO  LoggedConsole - # Configuration file:  /mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config\r\n2016-05-02T09:35:05.7229810+00:00 [1] INFO  LoggedConsole - # Output folder:       /home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2\r\n2016-05-02T09:35:05.7231240+00:00 [1] INFO  LoggedConsole - # Temp File Directory: /mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp\r\n2016-05-02T09:35:05.8615460+00:00 [1] INFO  AnalysisPrograms.AnalyseLongRecordings.AnalyseLongRecording - Minimum event threshold has been set to 0.2\r\n2016-05-02T09:35:05.8811140+00:00 [1] INFO  AnalysisBase.FileSegment - Parsed file start date as 2015-09-06T21:00:00.0000000+00:00\r\n2016-05-02T09:35:05.8815530+00:00 [1] WARN  AnalysisPrograms.AnalyseLongRecordings.AnalyseLongRecording - Neither start nor end segment offsets provided. Therefore ignored\r\n2016-05-02T09:35:06.0472510+00:00 [1] INFO  AnalysisPrograms.Acoustic - Image tiling is enabled\r\n2016-05-02T09:35:06.0507030+00:00 [1] INFO  LoggedConsole - STARTING ANALYSIS ...\r\n2016-05-02T09:35:06.0783340+00:00 [1] INFO  LoggedConsole - FATAL ERROR:\r\n\tCould not find exe: /usr/local/bin/ffprobe\r\n2016-05-02T09:35:06.1169140+00:00 [1] INFO  LoggedConsole - Usage: AnalysisPrograms <action> options\r\n\r\nGlobal options:\r\n    -d   -debug     [switch]        *    \r\n    -n   -nodebug   [switch]        *    \r\n    -l   -loglevel  [logverbosity]  *    \r\n    -v   -verbose   [switch]        *    \r\n    -vv  -vverbose  [switch]        *    \r\nAction: \r\n  audio2csv - Calls AnalyseLongRecording.Execute(): Outputs acoustic indices and LD false-colour spectrograms.\r\n    -so  -source              file         1  The source audio file to operate on                                                                                                                                                                           \r\n    -c   -config              file         2  The path to the config file                                                                                                                                                                                   \r\n    -o   -output              directory    3  A directory to write output to                                                                                                                                                                                \r\n    -t   -tempdir             [directory]  *  A TEMP directory where cut files will be stored. Use this option for efficiency (e.g. write to a RAM Disk).                                                                                                   \r\n    -s   -startoffset         [double]     *  The start offset to start analyzing from (in seconds)                                                                                                                                                         \r\n    -e   -endoffset           [double]     *  The end offset to stop analyzing (in seconds)                                                                                                                                                                 \r\n    -w   -whenexitcopylog     [switch]     *  If true, attempts to copy the executable's log file to output directory. If it can't determine an output directory, it copies to the working directory.                                                       \r\n    -wh  -whenexitcopyconfig  [switch]     *  If true, attempts to copy the executable's config file to output directory. If it can't determine an output directory, it copies to the working directory. If it can't find a config file, nothing is copied  \r\n\r\n2016-05-02T09:35:06.1175700+00:00 [1] INFO  LogFileOnly - ERRORLEVEL: 52\r\n```\r\n\r\nworker.log:\r\n```\r\n# Logfile created on 2016-05-02 09:35:04 +0000 by logger.rb/47272\r\n2016-05-02T09:35:04.936+0000 [ INFO-BawWorkers::Analysis::Runner-005860] Executing mono \"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/programs/AnalysisPrograms/AnalysisPrograms.exe\" audio2csv /source:\"/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav\" /config:\"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config\" /tempdir:\"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp\" /output:\"/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2\".\r\n2016-05-02T09:35:06.453+0000 [DEBUG-BawAudioTools::RunExternalProgram-005860] External Program: status=52;killed=false;pid=5864;time_out_sec=7200;time_taken_sec=1.516216453164816;timed_out=false;exceptions=[];command=mono \"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/programs/AnalysisPrograms/AnalysisPrograms.exe\" audio2csv /source:\"/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav\" /config:\"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config\" /tempdir:\"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp\" /output:\"/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2\"\r\n\tStandard output: QUT Bioacoustic Analysis Program - version 15.10.3309.0 (RELEASE build, 10/30/2015 06:54) \r\nGit branch-version: master-7cd0fa5e863158369c8d7043266a6f36342c1cf5\r\nCopyright QUT 2016\r\n# PROCESS LONG RECORDING\r\n# DATE AND TIME: 05/02/2016 09:35:05\r\n# Recording file:      /home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav\r\n# Configuration file:  /mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config\r\n# Output folder:       /home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2\r\n# Temp File Directory: /mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp\r\n[2016-05-02T09:35:05.8615460+00:00] INFO  - Minimum event threshold has been set to 0.2\r\n[2016-05-02T09:35:05.8811140+00:00] INFO  - Parsed file start date as 2015-09-06T21:00:00.0000000+00:00\r\n[2016-05-02T09:35:05.8815530+00:00] WARN  - Neither start nor end segment offsets provided. Therefore ignored\r\n[2016-05-02T09:35:06.0472510+00:00] INFO  - Image tiling is enabled\r\nSTARTING ANALYSIS ...\r\nFATAL ERROR:\r\n\tCould not find exe: /usr/local/bin/ffprobe\r\nUsage: AnalysisPrograms <action> options\r\n\r\nGlobal options:\r\n    -d   -debug     [switch]        *    \r\n    -n   -nodebug   [switch]        *    \r\n    -l   -loglevel  [logverbosity]  *    \r\n    -v   -verbose   [switch]        *    \r\n    -vv  -vverbose  [switch]        *    \r\nAction: \r\n  audio2csv - Calls AnalyseLongRecording.Execute(): Outputs acoustic indices and LD false-colour spectrograms.\r\n    -so  -source              file         1  The source audio file to operate on                                                                                                                                                                           \r\n    -c   -config              file         2  The path to the config file                                                                                                                                                                                   \r\n    -o   -output              directory    3  A directory to write output to                                                                                                                                                                                \r\n    -t   -tempdir             [directory]  *  A TEMP directory where cut files will be stored. Use this option for efficiency (e.g. write to a RAM Disk).                                                                                                   \r\n    -s   -startoffset         [double]     *  The start offset to start analyzing from (in seconds)                                                                                                                                                         \r\n    -e   -endoffset           [double]     *  The end offset to stop analyzing (in seconds)                                                                                                                                                                 \r\n    -w   -whenexitcopylog     [switch]     *  If true, attempts to copy the executable's log file to output directory. If it can't determine an output directory, it copies to the working directory.                                                       \r\n    -wh  -whenexitcopyconfig  [switch]     *  If true, attempts to copy the executable's config file to output directory. If it can't determine an output directory, it copies to the working directory. If it can't find a config file, nothing is copied  \r\n\r\n\r\n\tStandard Error: \r\n2016-05-02T09:35:06.453+0000 [DEBUG-BawWorkers::Analysis::Runner-005860] Successfully executed {:command=>\"mono \\\"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/programs/AnalysisPrograms/AnalysisPrograms.exe\\\" audio2csv /source:\\\"/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav\\\" /config:\\\"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config\\\" /tempdir:\\\"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp\\\" /output:\\\"/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2\\\"\", :stdout=>\"QUT Bioacoustic Analysis Program - version 15.10.3309.0 (RELEASE build, 10/30/2015 06:54) \\nGit branch-version: master-7cd0fa5e863158369c8d7043266a6f36342c1cf5\\nCopyright QUT 2016\\n# PROCESS LONG RECORDING\\n# DATE AND TIME: 05/02/2016 09:35:05\\n# Recording file:      /home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav\\n# Configuration file:  /mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config\\n# Output folder:       /home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2\\n# Temp File Directory: /mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp\\n[2016-05-02T09:35:05.8615460+00:00] INFO  - Minimum event threshold has been set to 0.2\\n[2016-05-02T09:35:05.8811140+00:00] INFO  - Parsed file start date as 2015-09-06T21:00:00.0000000+00:00\\n[2016-05-02T09:35:05.8815530+00:00] WARN  - Neither start nor end segment offsets provided. Therefore ignored\\n[2016-05-02T09:35:06.0472510+00:00] INFO  - Image tiling is enabled\\nSTARTING ANALYSIS ...\\nFATAL ERROR:\\n\\tCould not find exe: /usr/local/bin/ffprobe\\nUsage: AnalysisPrograms <action> options\\n\\nGlobal options:\\n    -d   -debug     [switch]        *    \\n    -n   -nodebug   [switch]        *    \\n    -l   -loglevel  [logverbosity]  *    \\n    -v   -verbose   [switch]        *    \\n    -vv  -vverbose  [switch]        *    \\nAction: \\n  audio2csv - Calls AnalyseLongRecording.Execute(): Outputs acoustic indices and LD false-colour spectrograms.\\n    -so  -source              file         1  The source audio file to operate on                                                                                                                                                                           \\n    -c   -config              file         2  The path to the config file                                                                                                                                                                                   \\n    -o   -output              directory    3  A directory to write output to                                                                                                                                                                                \\n    -t   -tempdir             [directory]  *  A TEMP directory where cut files will be stored. Use this option for efficiency (e.g. write to a RAM Disk).                                                                                                   \\n    -s   -startoffset         [double]     *  The start offset to start analyzing from (in seconds)                                                                                                                                                         \\n    -e   -endoffset           [double]     *  The end offset to stop analyzing (in seconds)                                                                                                                                                                 \\n    -w   -whenexitcopylog     [switch]     *  If true, attempts to copy the executable's log file to output directory. If it can't determine an output directory, it copies to the working directory.                                                       \\n    -wh  -whenexitcopyconfig  [switch]     *  If true, attempts to copy the executable's config file to output directory. If it can't determine an output directory, it copies to the working directory. If it can't find a config file, nothing is copied  \\n\\n\", :stderr=>\"\", :time_taken=>1.516216453164816, :exit_code=>52, :execute_msg=>\"External Program: status=52;killed=false;pid=5864;time_out_sec=7200;time_taken_sec=1.516216453164816;timed_out=false;exceptions=[];command=mono \\\"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/programs/AnalysisPrograms/AnalysisPrograms.exe\\\" audio2csv /source:\\\"/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav\\\" /config:\\\"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config\\\" /tempdir:\\\"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp\\\" /output:\\\"/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2\\\"\\n\\tStandard output: QUT Bioacoustic Analysis Program - version 15.10.3309.0 (RELEASE build, 10/30/2015 06:54) \\nGit branch-version: master-7cd0fa5e863158369c8d7043266a6f36342c1cf5\\nCopyright QUT 2016\\n# PROCESS LONG RECORDING\\n# DATE AND TIME: 05/02/2016 09:35:05\\n# Recording file:      /home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav\\n# Configuration file:  /mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config\\n# Output folder:       /home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2\\n# Temp File Directory: /mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp\\n[2016-05-02T09:35:05.8615460+00:00] INFO  - Minimum event threshold has been set to 0.2\\n[2016-05-02T09:35:05.8811140+00:00] INFO  - Parsed file start date as 2015-09-06T21:00:00.0000000+00:00\\n[2016-05-02T09:35:05.8815530+00:00] WARN  - Neither start nor end segment offsets provided. Therefore ignored\\n[2016-05-02T09:35:06.0472510+00:00] INFO  - Image tiling is enabled\\nSTARTING ANALYSIS ...\\nFATAL ERROR:\\n\\tCould not find exe: /usr/local/bin/ffprobe\\nUsage: AnalysisPrograms <action> options\\n\\nGlobal options:\\n    -d   -debug     [switch]        *    \\n    -n   -nodebug   [switch]        *    \\n    -l   -loglevel  [logverbosity]  *    \\n    -v   -verbose   [switch]        *    \\n    -vv  -vverbose  [switch]        *    \\nAction: \\n  audio2csv - Calls AnalyseLongRecording.Execute(): Outputs acoustic indices and LD false-colour spectrograms.\\n    -so  -source              file         1  The source audio file to operate on                                                                                                                                                                           \\n    -c   -config              file         2  The path to the config file                                                                                                                                                                                   \\n    -o   -output              directory    3  A directory to write output to                                                                                                                                                                                \\n    -t   -tempdir             [directory]  *  A TEMP directory where cut files will be stored. Use this option for efficiency (e.g. write to a RAM Disk).                                                                                                   \\n    -s   -startoffset         [double]     *  The start offset to start analyzing from (in seconds)                                                                                                                                                         \\n    -e   -endoffset           [double]     *  The end offset to stop analyzing (in seconds)                                                                                                                                                                 \\n    -w   -whenexitcopylog     [switch]     *  If true, attempts to copy the executable's log file to output directory. If it can't determine an output directory, it copies to the working directory.                                                       \\n    -wh  -whenexitcopyconfig  [switch]     *  If true, attempts to copy the executable's config file to output directory. If it can't determine an output directory, it copies to the working directory. If it can't find a config file, nothing is copied  \\n\\n\\n\\tStandard Error: \"}.\r\n```"}}