ee/app/workers/geo/scheduler/scheduler_worker.rb
# The scheduling works as the following:
#
# 1. Load a batch of IDs that we need to schedule (DB_RETRIEVE_BATCH_SIZE) into a pending list.
# 2. Schedule them so that at most `max_capacity` are running at once.
# 3. When a slot frees, schedule another job.
# 4. When we have drained the pending list, load another batch into memory, and schedule the
#    remaining jobs, excluding ones in progress.
# 5. Quit when we have scheduled all jobs or exceeded MAX_RUNTIME.
The scheduling works as the following:
1. Load a batch of IDs that we need to schedule (DB_RETRIEVE_BATCH_SIZE) into a pending list.
2. Schedule them so that at most `max_capacity` are running at once.
3. When a slot frees, schedule another job.
4. When we have drained the pending list, load another batch into memory, and schedule the
remaining jobs, excluding ones in progress.
5. Quit when we have scheduled all jobs or exceeded MAX_RUNTIME.
The scheduling works as the following : 1 . Load a batch of IDs that we need to schedule ( DB_RETRIEVE_BATCH_SIZE ) into a pending list . 2 . Schedule them so that at most ` max_capacity ` are running at once . 3 . When a slot frees schedule another job . 4 . When we have drained the pending list load another batch into memory and schedule the remaining jobs excluding ones in progress . 5 . Quit when we have scheduled all jobs or exceeded MAX_RUNTIME .
def perform
        @start_time = Time.now.utc
        @loops = 0
        try_obtain_lease do
          log_info('Started scheduler')
          reason = :unknown
          begin
            reason = loop do
              break :node_disabled unless node_enabled?
              update_jobs_in_progress
              update_pending_resources
              break :over_time if over_time?
              break :complete unless resources_remain?
              last_batch = reload_queue?
              schedule_jobs
              @loops += 1
              break :last_batch if last_batch
              break :lease_lost unless renew_lease!
              sleep(1)
            end
          rescue => err
            reason = :error
            log_error(err.message)
            raise err
          ensure
            duration = Time.now.utc - start_time
            log_info('Finished scheduler', total_loops: loops, duration: duration, reason: reason)
          end
        end
      end
def perform @start_time = Time . now . utc @loops = 0 try_obtain_lease do log_info ( 'Started scheduler' ) reason = :unknown begin reason = loop do break :node_disabled unless node_enabled? update_jobs_in_progress update_pending_resources break :over_time if over_time? break :complete unless resources_remain? last_batch = reload_queue? schedule_jobs @loops += 1 break :last_batch if last_batch break :lease_lost unless renew_lease! sleep ( 1 ) end rescue => err reason = :error log_error ( err . message ) raise err ensure duration = Time . now . utc - start_time log_info ( 'Finished scheduler' , total_loops : loops , duration : duration , reason : reason ) end end end
The scheduling works as the following:
1.
The scheduling works as the following : 1 .
# Prevent multiple Sidekiq workers from attempting to schedule jobs # If we're still under the limit after refreshing from the DB, we # can end after scheduling the remaining transfers.