B2G/gaia/apps/email/js/ext/mailapi/composite/configurator.js
/**
   * Whatever synchronization we last triggered has now completed; we should
   * either trigger another sync if we still want more data, or close out the
   * current sync.
   *
   * ## Block Flushing
   *
   * We only cause a call to `ImapAccount.__checkpointSyncCompleted` (via a call
   * to `_doneSync`) to happen and cause dirty blocks to be written to disk when
   * we are done with synchronization.  This is because this method declares
   * victory once a non-trivial amount of work has been done.  In the event that
   * the sync is encountering a lot of deleted messages and so keeps loading
   * blocks, the memory burden is limited because we will be emptying those
   * blocks out so actual memory usage (after GC) is commensurate with the
   * number of (still-)existing messages.  And those are what this method uses
   * to determine when it is done.
   *
   * In the cases where we are synchronizing a ton of messages on a single day,
   * we could perform checkpoints during the process, but realistically any
   * device we are operating on should probably have enough memory to deal with
   * these surges, so we're not doing that yet.
   *
   * @args[
   *   @param[err]
   *   @param[bisectInfo]
   *   @param[messagesSeen Number]
   *   @param[effStartTS DateMS]{
   *     Effective start date in UTC after compensating for server tz offset.
   *   }
   *   @param[effEndTS @oneof[DateMS null]]{
   *     Effective end date in UTC after compensating for server tz offset.
   *     If the end date was open-ended, then null is passed instead.
   *   }
   * ]
   */
Whatever synchronization we last triggered has now completed; we should
either trigger another sync if we still want more data, or close out the
current sync.

Block Flushing

We only cause a call to `ImapAccount.__checkpointSyncCompleted` (via a call
to `_doneSync`) to happen and cause dirty blocks to be written to disk when
we are done with synchronization.  This is because this method declares
victory once a non-trivial amount of work has been done.  In the event that
the sync is encountering a lot of deleted messages and so keeps loading
blocks, the memory burden is limited because we will be emptying those
blocks out so actual memory usage (after GC) is commensurate with the
number of (still-)existing messages.  And those are what this method uses
to determine when it is done.

In the cases where we are synchronizing a ton of messages on a single day,
we could perform checkpoints during the process, but realistically any
device we are operating on should probably have enough memory to deal with
these surges, so we're not doing that yet.


Whatever synchronization we last triggered has now completed ; we should either trigger another sync if we still want more data or close out the current sync . Block Flushing We only cause a call to ` ImapAccount . __checkpointSyncCompleted ` ( via a call to ` _doneSync ` ) to happen and cause dirty blocks to be written to disk when we are done with synchronization . This is because this method declares victory once a non - trivial amount of work has been done . In the event that the sync is encountering a lot of deleted messages and so keeps loading blocks the memory burden is limited because we will be emptying those blocks out so actual memory usage ( after GC ) is commensurate with the number of ( still - ) existing messages . And those are what this method uses to determine when it is done . In the cases where we are synchronizing a ton of messages on a single day we could perform checkpoints during the process but realistically any device we are operating on should probably have enough memory to deal with these surges so we ' re not doing that yet .
function ifs_onSyncCompleted(err, bisectInfo, messagesSeen,
                                                effStartTS, effEndTS) {
    if (err === 'bisect') {
      var curDaysDelta = bisectInfo.curDaysDelta,
          numHeaders = bisectInfo.numHeaders;
      if (this._curSyncDir === FUTUREWARDS && this._fallbackOriginTS) {
        this.folderStorage.clearSyncedToDawnOfTime(this._fallbackOriginTS);
        bisectInfo.oldStartTS = this._fallbackOriginTS;
        this._fallbackOriginTS = null;
        var effOldEndTS = bisectInfo.oldEndTS ||
                          quantizeDate(NOW() + DAY_MILLIS +
                                       this._account.tzOffset);
        curDaysDelta = Math.round((effOldEndTS - bisectInfo.oldStartTS) /
                                  DAY_MILLIS);
        numHeaders = $sync.BISECT_DATE_AT_N_MESSAGES * 1.5;
      }
      else if (curDaysDelta > 1000)
        curDaysDelta = 30;
      var shrinkScale = $sync.BISECT_DATE_AT_N_MESSAGES /
                          (numHeaders * 2),
          dayStep = Math.max(1,
                             Math.min(curDaysDelta - 2,
                                      Math.ceil(shrinkScale * curDaysDelta)));
      this._curSyncDayStep = dayStep;
      if (this._curSyncDir === PASTWARDS) {
        bisectInfo.newEndTS = bisectInfo.oldEndTS;
        this._nextSyncAnchorTS = bisectInfo.newStartTS =
          makeDaysBefore(bisectInfo.newEndTS, dayStep, this._account.tzOffset);
        this._curSyncDoNotGrowBoundary = bisectInfo.oldStartTS;
      }
      else { 
        bisectInfo.newStartTS = bisectInfo.oldStartTS;
        this._nextSyncAnchorTS = bisectInfo.newEndTS =
          makeDaysBefore(bisectInfo.newStartTS, -dayStep, this._account.tzOffset);
        this._curSyncDoNotGrowBoundary = bisectInfo.oldEndTS;
      }
      return;
    }
    else if (err) {
      this._doneSync(err);
      return;
    }
    console.log("Sync Completed!", this._curSyncDayStep, "days",
                messagesSeen, "messages synced");
    if (this._syncSlice.isDead) {
      this._doneSync();
      return;
    }
     NB: If there are any deleted messages, this logic will not save us
     because we ignored those messages.  This is made less horrible by issuing
     a time-date that expands as we go further back in time.
     (I have considered asking to see deleted messages too and ignoring them;
     that might be suitable.  We could also just be a jerk and force an
     expunge.)
    var folderMessageCount = this.folderConn && this.folderConn.box &&
                             this.folderConn.box.messages.total,
        dbCount = this.folderStorage.getKnownMessageCount(),
        syncedThrough =
          ((this._curSyncDir === PASTWARDS) ? effStartTS : effEndTS);
console.log("folder message count", folderMessageCount,
            "dbCount", dbCount,
            "syncedThrough", syncedThrough,
            "oldest known", this.folderStorage.getOldestMessageTimestamp());
    if (this._curSyncDir === PASTWARDS &&
        folderMessageCount === dbCount &&
        (!folderMessageCount ||
         TIME_DIR_AT_OR_BEYOND(this._curSyncDir, syncedThrough,
                               this.folderStorage.getOldestMessageTimestamp()))
       ) {
       expand the accuracy range to cover everybody
      this.folderStorage.markSyncedToDawnOfTime();
      this._doneSync();
      return;
    }
     If we've synchronized to the limits of syncing in the given direction,
     we're done.
    if (!this._nextSyncAnchorTS ||
        TIME_DIR_AT_OR_BEYOND(this._curSyncDir, this._nextSyncAnchorTS,
                              this._syncThroughTS)) {
      this._doneSync();
      return;
    }
     - Done if this is a grow and we don't want/need any more headers.
    if (this._curSyncIsGrow &&
        this._syncSlice.headers.length >= this._syncSlice.desiredHeaders) {
         (limited syncs aren't allowed to expand themselves)
      console.log("SYNCDONE Enough headers retrieved.",
                  "have", this._syncSlice.headers.length,
                  "want", this._syncSlice.desiredHeaders,
                  "conn knows about", this.folderConn.box.messages.total,
                  "sync date", this._curSyncStartTS,
                  "[oldest defined as", $sync.OLDEST_SYNC_DATE, "]");
      this._doneSync();
      return;
    }
    else if (this._syncSlice._accumulating) {
       flush the accumulated results thus far
      this._syncSlice.setStatus('synchronizing', true, true, true);
    }
     - Increase our search window size if we aren't finding anything
     Our goal is that if we are going backwards in time and aren't finding
     anything, we want to keep expanding our window
    var daysToSearch, lastSyncDaysInPast;
     If we saw messages, there is no need to increase the window size.  We
     also should not increase the size if we explicitly shrank the window and
     left a do-not-expand-until marker.
    if (messagesSeen || (this._curSyncDoNotGrowBoundary !== null &&
         !TIME_DIR_AT_OR_BEYOND(this._curSyncDir, this._nextSyncAnchorTS,
                                this._curSyncDoNotGrowBoundary))) {
      daysToSearch = this._curSyncDayStep;
    }
    else {
      this._curSyncDoNotGrowBoundary = null;
       This may be a fractional value because of DST
      lastSyncDaysInPast = ((quantizeDate(NOW() + this._account.tzOffset)) -
                           this._nextSyncAnchorTS) / DAY_MILLIS;
      daysToSearch = Math.ceil(this._curSyncDayStep *
                               $sync.TIME_SCALE_FACTOR_ON_NO_MESSAGES);
       These values used to be more conservative, but the importance of these
       guards was reduced when we switched to only syncing headers.
       At current constants (sync=3, scale=2), our doubling in the face of
       clamping is: 3, 6, 12, 24, 45, ... 90,
      if (lastSyncDaysInPast < 180) {
        if (daysToSearch > 45)
          daysToSearch = 45;
      }
      else if (lastSyncDaysInPast < 365) {  1 year
        if (daysToSearch > 90)
          daysToSearch = 90;
      }
      else if (lastSyncDaysInPast < 730) {  2 years
        if (daysToSearch > 120)
          daysToSearch = 120;
      }
      else if (lastSyncDaysInPast < 1825) {  5 years
        if (daysToSearch > 180)
          daysToSearch = 180;
      }
      else if (lastSyncDaysInPast < 3650) {  10 years
        if (daysToSearch > 365)
          daysToSearch = 365;
      }
      else if (daysToSearch > 730) {
        daysToSearch = 730;
      }
      this._curSyncDayStep = daysToSearch;
    }
     - Move the time range back in time more.
    var startTS, endTS;
    if (this._curSyncDir === PASTWARDS) {
      endTS = this._nextSyncAnchorTS;
      this._nextSyncAnchorTS = startTS = makeDaysBefore(endTS, daysToSearch,
                                                        this._account.tzOffset);
    }
    else { 
      startTS = this._nextSyncAnchorTS;
      this._nextSyncAnchorTS = endTS = makeDaysBefore(startTS, -daysToSearch,
                                                      this._account.tzOffset);
    }
    this.folderConn.syncDateRange(startTS, endTS, this._curSyncAccuracyStamp,
                                  this.onSyncCompleted.bind(this));
  }
function ifs_onSyncCompleted ( err , bisectInfo , messagesSeen , effStartTS , effEndTS ) { if ( err === 'bisect' ) { var curDaysDelta = bisectInfo . curDaysDelta , numHeaders = bisectInfo . numHeaders ; if ( this . _curSyncDir === FUTUREWARDS && this . _fallbackOriginTS ) { this . folderStorage . clearSyncedToDawnOfTime ( this . _fallbackOriginTS ) ; bisectInfo . oldStartTS = this . _fallbackOriginTS ; this . _fallbackOriginTS = null ; var effOldEndTS = bisectInfo . oldEndTS || quantizeDate ( NOW ( ) + DAY_MILLIS + this . _account . tzOffset ) ; curDaysDelta = Math . round ( ( effOldEndTS - bisectInfo . oldStartTS ) / DAY_MILLIS ) ; numHeaders = $sync . BISECT_DATE_AT_N_MESSAGES * 1.5 ; } else if ( curDaysDelta > 1000 ) curDaysDelta = 30 ; var shrinkScale = $sync . BISECT_DATE_AT_N_MESSAGES / ( numHeaders * 2 ) , dayStep = Math . max ( 1 , Math . min ( curDaysDelta - 2 , Math . ceil ( shrinkScale * curDaysDelta ) ) ) ; this . _curSyncDayStep = dayStep ; if ( this . _curSyncDir === PASTWARDS ) { bisectInfo . newEndTS = bisectInfo . oldEndTS ; this . _nextSyncAnchorTS = bisectInfo . newStartTS = makeDaysBefore ( bisectInfo . newEndTS , dayStep , this . _account . tzOffset ) ; this . _curSyncDoNotGrowBoundary = bisectInfo . oldStartTS ; } else { bisectInfo . newStartTS = bisectInfo . oldStartTS ; this . _nextSyncAnchorTS = bisectInfo . newEndTS = makeDaysBefore ( bisectInfo . newStartTS , - dayStep , this . _account . tzOffset ) ; this . _curSyncDoNotGrowBoundary = bisectInfo . oldEndTS ; } return ; } else if ( err ) { this . _doneSync ( err ) ; return ; } console . log ( "Sync Completed!" , this . _curSyncDayStep , "days" , messagesSeen , "messages synced" ) ; if ( this . _syncSlice . isDead ) { this . _doneSync ( ) ; return ; } var folderMessageCount = this . folderConn && this . folderConn . box && this . folderConn . box . messages . total , dbCount = this . folderStorage . getKnownMessageCount ( ) , syncedThrough = ( ( this . _curSyncDir === PASTWARDS ) ? effStartTS : effEndTS ) ; console . log ( "folder message count" , folderMessageCount , "dbCount" , dbCount , "syncedThrough" , syncedThrough , "oldest known" , this . folderStorage . getOldestMessageTimestamp ( ) ) ; if ( this . _curSyncDir === PASTWARDS && folderMessageCount === dbCount && ( ! folderMessageCount || TIME_DIR_AT_OR_BEYOND ( this . _curSyncDir , syncedThrough , this . folderStorage . getOldestMessageTimestamp ( ) ) ) ) { this . folderStorage . markSyncedToDawnOfTime ( ) ; this . _doneSync ( ) ; return ; } if ( ! this . _nextSyncAnchorTS || TIME_DIR_AT_OR_BEYOND ( this . _curSyncDir , this . _nextSyncAnchorTS , this . _syncThroughTS ) ) { this . _doneSync ( ) ; return ; } if ( this . _curSyncIsGrow && this . _syncSlice . headers . length >= this . _syncSlice . desiredHeaders ) { console . log ( "SYNCDONE Enough headers retrieved." , "have" , this . _syncSlice . headers . length , "want" , this . _syncSlice . desiredHeaders , "conn knows about" , this . folderConn . box . messages . total , "sync date" , this . _curSyncStartTS , "[oldest defined as" , $sync . OLDEST_SYNC_DATE , "]" ) ; this . _doneSync ( ) ; return ; } else if ( this . _syncSlice . _accumulating ) { this . _syncSlice . setStatus ( 'synchronizing' , true , true , true ) ; } var daysToSearch , lastSyncDaysInPast ; if ( messagesSeen || ( this . _curSyncDoNotGrowBoundary !== null && ! TIME_DIR_AT_OR_BEYOND ( this . _curSyncDir , this . _nextSyncAnchorTS , this . _curSyncDoNotGrowBoundary ) ) ) { daysToSearch = this . _curSyncDayStep ; } else { this . _curSyncDoNotGrowBoundary = null ; lastSyncDaysInPast = ( ( quantizeDate ( NOW ( ) + this . _account . tzOffset ) ) - this . _nextSyncAnchorTS ) / DAY_MILLIS ; daysToSearch = Math . ceil ( this . _curSyncDayStep * $sync . TIME_SCALE_FACTOR_ON_NO_MESSAGES ) ; if ( lastSyncDaysInPast < 180 ) { if ( daysToSearch > 45 ) daysToSearch = 45 ; } else if ( lastSyncDaysInPast < 365 ) { if ( daysToSearch > 90 ) daysToSearch = 90 ; } else if ( lastSyncDaysInPast < 730 ) { if ( daysToSearch > 120 ) daysToSearch = 120 ; } else if ( lastSyncDaysInPast < 1825 ) { if ( daysToSearch > 180 ) daysToSearch = 180 ; } else if ( lastSyncDaysInPast < 3650 ) { if ( daysToSearch > 365 ) daysToSearch = 365 ; } else if ( daysToSearch > 730 ) { daysToSearch = 730 ; } this . _curSyncDayStep = daysToSearch ; } var startTS , endTS ; if ( this . _curSyncDir === PASTWARDS ) { endTS = this . _nextSyncAnchorTS ; this . _nextSyncAnchorTS = startTS = makeDaysBefore ( endTS , daysToSearch , this . _account . tzOffset ) ; } else { startTS = this . _nextSyncAnchorTS ; this . _nextSyncAnchorTS = endTS = makeDaysBefore ( startTS , - daysToSearch , this . _account . tzOffset ) ; } this . folderConn . syncDateRange ( startTS , endTS , this . _curSyncAccuracyStamp , this . onSyncCompleted . bind ( this ) ) ; }
Whatever synchronization we last triggered has now completed; we should
 either trigger another sync if we still want more data, or close out the
 current sync.
Whatever synchronization we last triggered has now completed ; we should either trigger another sync if we still want more data or close out the current sync .
// In the event the time range had to be bisected, update our info so if // we need to take another step we do the right thing. // If we had a fallback TS because we were synced to the dawn of time, // use that and start by just cutting the range in thirds rather than // doing a weighted bisection since the distribution might include // a number of messages earlier than our fallback startTS. // Sanity check the time delta; if we grew the bounds to the dawn // of time, then our interpolation is useless and it's better for // us to crank things way down, even if it's erroneously so. // - Interpolate better time bounds. // Assume a linear distribution of messages, but overestimated by // a factor of two so we undershoot.  Also make sure that we subtract off // at least 2 days at a time.  This is to ensure that in the case where // endTS is null and we end up using makeDaysAgo that we actually shrink // by at least 1 day (because of how rounding works for makeDaysAgo). // FUTUREWARDS // We return now without calling _doneSync because we are not done; the // caller (syncDateRange) will re-trigger itself and keep going. // - Slice is dead = we are done // If it now appears we know about all the messages in the folder, then we // are done syncing and can mark the entire folder as synchronized.  This // requires that: // - The direction is pastwards. (We check the oldest header, so this //   is important.  We don't really need to do a future-wards variant since //   we always use pastwards for refreshes and the future-wards variant //   really does not need a fast-path since the cost of stepping to 'today' //   is much cheaper thana the cost of walking all the way to 1990.) // - The number of messages we know about is the same as the number the //   server most recently told us are in the folder. // - (There are no messages in the folder at all OR) // - We have synchronized past the oldest known message header.  (This, //   in combination with the fact that we always open from the most recent //   set of messages we know about, that we fully synchronize all time //   intervals (for now!), and our pastwards-direction for refreshes means //   that we can conclude we have synchronized across all messages and //   this is a sane conclusion to draw.) // // NB: If there are any deleted messages, this logic will not save us // because we ignored those messages.  This is made less horrible by issuing // a time-date that expands as we go further back in time. // // (I have considered asking to see deleted messages too and ignoring them; // that might be suitable.  We could also just be a jerk and force an // expunge.) // expand the accuracy range to cover everybody // If we've synchronized to the limits of syncing in the given direction, // we're done. // - Done if this is a grow and we don't want/need any more headers. // (limited syncs aren't allowed to expand themselves) // flush the accumulated results thus far // - Increase our search window size if we aren't finding anything // Our goal is that if we are going backwards in time and aren't finding // anything, we want to keep expanding our window // If we saw messages, there is no need to increase the window size.  We // also should not increase the size if we explicitly shrank the window and // left a do-not-expand-until marker. // This may be a fractional value because of DST // These values used to be more conservative, but the importance of these // guards was reduced when we switched to only syncing headers. // At current constants (sync=3, scale=2), our doubling in the face of // clamping is: 3, 6, 12, 24, 45, ... 90, // 1 year // 2 years // 5 years // 10 years // - Move the time range back in time more. // FUTUREWARDS