lib/wyrm/pump.rb
# Use limit / offset, but not for all fields.
# The idea is that large offsets are expensive in the db because the db server has to read
# through the data set to reach the required offset. So make that only ids need to be read,
# and then do the main select from the limited id list.
# select * from massive as full
#   inner join (select id from massive order by whatever limit m, n) limit
#   on full.id = limit.id
# order by full.whatever
# http://www.numerati.com/2012/06/26/reading-large-result-sets-with-hibernate-and-mysql/
Use limit / offset, but not for all fields.
The idea is that large offsets are expensive in the db because the db server has to read
through the data set to reach the required offset. So make that only ids need to be read,
and then do the main select from the limited id list.
Use limit / offset but not for all fields . The idea is that large offsets are expensive in the db because the db server has to read through the data set to reach the required offset . So make that only ids need to be read and then do the main select from the limited id list .
def inner_dump( &encode_block )
      on_conditions = primary_keys.map{|f| [f,f]}.to_h
      (0..table_dataset.count).step(page_size).each do |offset|
        limit_dataset = table_dataset.select( *primary_keys ).limit( page_size, offset ).order( *primary_keys )
        page = table_dataset.join( limit_dataset, on_conditions ).order( *primary_keys ).qualify(table_name)
        logger.info "#{__method__} #{table_name} #{offset}"
        logger.debug page.sql
        page.each &encode_block
      end
    end
def inner_dump ( & encode_block ) on_conditions = primary_keys . map { | f | [ f , f ] } . to_h ( 0 .. table_dataset . count ) . step ( page_size ) . each do | offset | limit_dataset = table_dataset . select ( * primary_keys ) . limit ( page_size , offset ) . order ( * primary_keys ) page = table_dataset . join ( limit_dataset , on_conditions ) . order ( * primary_keys ) . qualify ( table_name ) logger . info "#{__method__} #{table_name} #{offset}" logger . debug page . sql page . each & encode_block end end
Use limit / offset, but not for all fields.
Use limit / offset but not for all fields .
# could possibly overrride Dataset#paginate(page_no, page_size, record_count=nil)
encode_block
<not_specific>