title: Quartz Job executed multiple times simultaneously by each cluster machine, rather than one time by one machine for the entire cluster
body: <p><strong>Goal:</strong>
         * Have Job1 run once for a three-node cluster every 10 minutes, and Job2 run once for the same cluster every 5 minutes. Each job generates an email; so at 10:55am I should receive only one Job2 email from the cluster, and at 11:00am I should receive one Job1 email and one Job2 email from the cluster, at 11:05am I should receive only one Job2 email from the cluster, and so on...</p>
  
  <p><strong>Problem:</strong>
         * Job1 is being run multiple times every 10 minutes on each node in the cluster, and the same for Job2 (except every 5 minutes). This leads to many, many more than one or two emails.</p>
  
  <p><strong>Configuration:</strong>
      * Three-node linux cluster
      * Each machine NTP configured and time-sync'd
      * Oracle DB
      * Quartz v2.2.0 (cluster mode)
          * Jobs configured via CronTrigger
      * Each node has an instance of the same standalone Java application running on it, and the Java application instantiates an instance of the quartz scheduler in cluster-mode.
      * quartz.properties files are identical on each machine. </p>
  
  <p>I have investigated all the obvious potential causes, but nothing explains it or presents a fix. I have even tried inserting an artificial 10-second sleep instruction in the job, to ensure that it doesn't finish in under a second. Please find relevant artifacts below (quartz.properties and log output). Any help would be greatly appreciated!</p>
  
  <p><strong>Artifact #1:</strong></p>
  
  <pre><code>============================================================================
  ============================================================================
  Q U A R T Z   ---   P R O P E R T I E S 
  ==================
  
      #============================================================================
      # Configure Main Scheduler Properties  
      #============================================================================
  
      org.quartz.scheduler.instanceName: MyQrtzScheduler
      org.quartz.scheduler.instanceId: AUTO
  
      org.quartz.scheduler.skipUpdateCheck: true
  
      #============================================================================
      # Configure ThreadPool  
      #============================================================================
  
      org.quartz.threadPool.class: org.quartz.simpl.SimpleThreadPool
      org.quartz.threadPool.threadCount: 1
      org.quartz.threadPool.threadPriority: 5
  
      #============================================================================
      # Configure JobStore  
      #============================================================================
  
      org.quartz.jobStore.misfireThreshold: 2592000000
  
      org.quartz.jobStore.class=org.quartz.impl.jdbcjobstore.JobStoreTX
      org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
      org.quartz.jobStore.useProperties=false
      org.quartz.jobStore.dataSource=myDS
      org.quartz.jobStore.tablePrefix=QRTZ_
      org.quartz.jobStore.isClustered=true
      org.quartz.jobStore.clusterCheckinInterval=60000
  
      #============================================================================
      # Other Example Delegates
      #============================================================================
      #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.DB2v6Delegate
      #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.DB2v7Delegate
      #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.DriverDelegate
      #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.HSQLDBDelegate
      #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.MSSQLDelegate
      #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.PointbaseDelegate
      #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
      #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.StdJDBCDelegate
      #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.WebLogicDelegate
      #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
      #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.oracle.WebLogicOracleDelegate
  
      #============================================================================
      # Configure Datasources  
      #============================================================================
  
      org.quartz.dataSource.myDS.driver: oracle.jdbc.driver.OracleDriver
      org.quartz.dataSource.myDS.URL: jdbc:oracle:thin:@myServer:myPort:blah
      org.quartz.dataSource.myDS.user: myDBUser
      org.quartz.dataSource.myDS.password: myDBPassword
      org.quartz.dataSource.myDS.maxConnections: 2
      org.quartz.dataSource.myDS.validationQuery: select 0
  
      #============================================================================
      # Configure Plugins 
      #============================================================================
  
      org.quartz.plugin.shutdownHook.class: org.quartz.plugins.management.ShutdownHookPlugin
      org.quartz.plugin.shutdownHook.cleanShutdown: true
      org.quartz.plugin.triggerHistory.class=org.quartz.plugins.history.LoggingTriggerHistoryPlugin
      org.quartz.plugin.jobHistory.class=org.quartz.plugins.history.LoggingJobHistoryPlugin
  </code></pre>
  
  <p><strong>Artifact #2:</strong></p>
  
  <pre><code>============================================================================
  ============================================================================
  L O G  ---  O U T P U T
  ==================
      2015-01-29 12:56:16,602 [main]  INFO com.mycompany.myapp.jobs.QuartzHelper - Initializing Quartz scheduler...
      2015-01-29 12:56:16,829 [main]  INFO org.quartz.impl.StdSchedulerFactory - Using default implementation for ThreadExecutor
      2015-01-29 12:56:16,855 [main]  INFO org.quartz.core.SchedulerSignalerImpl - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
      2015-01-29 12:56:16,855 [main]  INFO org.quartz.core.QuartzScheduler - Quartz Scheduler v.2.2.0 created.
      2015-01-29 12:56:16,857 [main]  INFO org.quartz.plugins.management.ShutdownHookPlugin - Registering Quartz shutdown hook.
      2015-01-29 12:56:16,859 [main]  INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Using db table-based data access locking (synchronization).
      2015-01-29 12:56:16,864 [main]  INFO org.quartz.impl.jdbcjobstore.JobStoreTX - JobStoreTX initialized.
      2015-01-29 12:56:16,865 [main]  INFO org.quartz.core.QuartzScheduler - Scheduler meta-data: Quartz Scheduler (v2.2.0) 'MyQrtzScheduler' with instanceId 'node1_1422554176832'
        Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
        NOT STARTED.
        Currently in standby mode.
        Number of jobs executed: 0
        Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 1 threads.
        Using job-store 'org.quartz.impl.jdbcjobstore.JobStoreTX' - which supports persistence. and is clustered.
  
      2015-01-29 12:56:16,865 [main]  INFO org.quartz.impl.StdSchedulerFactory - Quartz scheduler 'MyQrtzScheduler' initialized from specified file: '/my/install/directory/quartz.properties'
      2015-01-29 12:56:16,866 [main]  INFO org.quartz.impl.StdSchedulerFactory - Quartz scheduler version: 2.2.0
      2015-01-29 12:56:16,866 [main]  INFO com.mycompany.myapp.jobs.QuartzHelper - Quartz scheduler initialized successfully.
  
      2015-01-29 12:59:53,450 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.core.QuartzSchedulerThread - batch acquisition of 1 triggers
      2015-01-29 13:00:00,007 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' is desired by: MyQrtzScheduler_QuartzSchedulerThread
      2015-01-29 13:00:00,008 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' is being obtained: MyQrtzScheduler_QuartzSchedulerThread
      2015-01-29 13:00:00,809 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' given to: MyQrtzScheduler_QuartzSchedulerThread
      2015-01-29 13:00:00,836 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' returned by: MyQrtzScheduler_QuartzSchedulerThread
      2015-01-29 13:00:00,839 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.simpl.PropertySettingJobFactory - Producing instance of Job 'node2_1422546730757.Job1', class=com.mycompany.myapp.job.Job1
      2015-01-29 13:00:00,851 [MyQrtzScheduler_Worker-1]  INFO org.quartz.plugins.history.LoggingTriggerHistoryPlugin - Trigger node2_1422546730757.Job1Trigger fired job node2_1422546730757.Job1 at:  13:00:00 01/29/2015
      2015-01-29 13:00:00,852 [MyQrtzScheduler_Worker-1]  INFO org.quartz.plugins.history.LoggingJobHistoryPlugin - Job node2_1422546730757.Job1 fired (by trigger node2_1422546730757.Job1Trigger) at:  13:00:00 01/29/2015
      2015-01-29 13:00:00,852 [MyQrtzScheduler_Worker-1] DEBUG org.quartz.core.JobRunShell - Calling execute on job node2_1422546730757.Job1
      2015-01-29 13:00:00,853 [MyQrtzScheduler_Worker-1]  INFO com.mycompany.myapp.job.Job1 - ***Executing Inbound File SLA Job...
      2015-01-29 13:00:02,054 [MyQrtzScheduler_Worker-1]  INFO com.mycompany.myapp.job.Job1 - ***Inbound File SLA Job: No SLA breaches found...
      2015-01-29 13:00:02,150 [MyQrtzScheduler_Worker-1]  INFO com.mycompany.myapp.job.Job1 - Job1 completed successfully in [1297ms]; sleeping [63703ms] to meet the required minimum runtime for quartz-jobs
      2015-01-29 13:00:24,881 [QuartzScheduler_MyQrtzScheduler-node1_1422554176832_ClusterManager] DEBUG org.quartz.impl.jdbcjobstore.JobStoreTX - ClusterManager: Check-in complete.
      2015-01-29 13:01:05,862 [MyQrtzScheduler_Worker-1]  INFO com.mycompany.myapp.job.Job1 - Job1 sleep-delay completed.
      2015-01-29 13:01:05,864 [MyQrtzScheduler_Worker-1]  INFO org.quartz.plugins.history.LoggingJobHistoryPlugin - Job node2_1422546730757.Job1 execution complete at  13:01:05 01/29/2015 and reports: SUCCESS
      2015-01-29 13:01:05,865 [MyQrtzScheduler_Worker-1]  INFO org.quartz.plugins.history.LoggingTriggerHistoryPlugin - Trigger node2_1422546730757.Job1Trigger completed firing job node2_1422546730757.Job1 at  13:01:05 01/29/2015 with resulting trigger instruction code: DO NOTHING
      2015-01-29 13:01:05,868 [MyQrtzScheduler_Worker-1] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' is desired by: MyQrtzScheduler_Worker-1
      2015-01-29 13:01:05,869 [MyQrtzScheduler_Worker-1] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' is being obtained: MyQrtzScheduler_Worker-1
      2015-01-29 13:01:05,872 [MyQrtzScheduler_Worker-1] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' given to: MyQrtzScheduler_Worker-1
      2015-01-29 13:01:05,880 [MyQrtzScheduler_Worker-1] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' returned by: MyQrtzScheduler_Worker-1
      2015-01-29 13:01:05,915 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.core.QuartzSchedulerThread - batch acquisition of 1 triggers
      2015-01-29 13:01:05,917 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' is desired by: MyQrtzScheduler_QuartzSchedulerThread
      2015-01-29 13:01:05,918 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' is being obtained: MyQrtzScheduler_QuartzSchedulerThread
      2015-01-29 13:01:05,921 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' given to: MyQrtzScheduler_QuartzSchedulerThread
      2015-01-29 13:01:05,954 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' returned by: MyQrtzScheduler_QuartzSchedulerThread
      2015-01-29 13:01:05,955 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.simpl.PropertySettingJobFactory - Producing instance of Job 'node1_1422543657050.Job2', class=com.mycompany.myapp.jobs.Job2
      2015-01-29 13:01:05,961 [MyQrtzScheduler_Worker-1]  INFO org.quartz.plugins.history.LoggingTriggerHistoryPlugin - Trigger node1_1422543657050.Job2Trigger fired job node1_1422543657050.Job2 at:  13:01:05 01/29/2015
      2015-01-29 13:01:05,962 [MyQrtzScheduler_Worker-1]  INFO org.quartz.plugins.history.LoggingJobHistoryPlugin - Job node1_1422543657050.Job2 fired (by trigger node1_1422543657050.Job2Trigger) at:  13:01:05 01/29/2015
      2015-01-29 13:01:05,963 [MyQrtzScheduler_Worker-1] DEBUG org.quartz.core.JobRunShell - Calling execute on job node1_1422543657050.Job2
      2015-01-29 13:01:05,963 [MyQrtzScheduler_Worker-1]  WARN com.mycompany.myapp.jobs.Job2 - No outbound files found; Outbound File SLA Job cannot check for SLA breaches.
      2015-01-29 13:01:05,965 [MyQrtzScheduler_Worker-1]  INFO org.quartz.plugins.history.LoggingJobHistoryPlugin - Job node1_1422543657050.Job2 execution complete at  13:01:05 01/29/2015 and reports: null
      2015-01-29 13:01:05,966 [MyQrtzScheduler_Worker-1]  INFO org.quartz.plugins.history.LoggingTriggerHistoryPlugin - Trigger node1_1422543657050.Job2Trigger completed firing job node1_1422543657050.Job2 at  13:01:05 01/29/2015 with resulting trigger instruction code: DO NOTHING
  </code></pre>