path: root/comm/mailnews/imap/src/nsAutoSyncManager.h

/* This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, You can obtain one at http://mozilla.org/MPL/2.0/. */

#ifndef nsAutoSyncManager_h__
#  define nsAutoSyncManager_h__

#  include "nsString.h"
#  include "nsCOMArray.h"
#  include "nsIObserver.h"
#  include "nsIUrlListener.h"
#  include "nsITimer.h"
#  include "nsTObserverArray.h"
#  include "nsIAutoSyncManager.h"
#  include "nsIAutoSyncMsgStrategy.h"
#  include "nsIAutoSyncFolderStrategy.h"
#  include "nsIUserIdleService.h"
#  include "prtime.h"

// clang-format off
/* Auto-Sync
 *
 * Background:
 *  it works only with offline imap folders. "autosync_offline_stores" pref
 *  enables/disables auto-sync mechanism. Note that setting "autosync_offline_stores"
 *  to false, or setting folder to not-offline doesn't stop synchronization
 *  process for already queued folders.
 *
 * Auto-Sync policy:
 *  o It kicks in during system idle time, and tries to download as much messages
 *    as possible based on given folder and message prioritization strategies/rules.
 *    Default folder prioritization strategy dictates to sort the folders based on the
 *    following order:  INBOX > DRAFTS > SUBFOLDERS > TRASH.
 *    Similarly, default message prioritization strategy dictates to download the most
 *    recent and smallest message first. Also, by sorting the messages by size in the
 *    queue, it tries to maximize the number of messages downloaded.
 *  o It downloads the messages in groups. Default groups size is defined by |kDefaultGroupSize|.
 *  o It downloads the messages larger than the group size one-by-one.
 *  o If new messages arrive when not idle, it downloads the messages that do fit into
 *    |kFirstGroupSizeLimit| size limit immediately, without waiting for idle time, unless there is
 *    a sibling (a folder owned by the same imap server) in stDownloadInProgress state in the q
 *  o If new messages arrive when idle, it downloads all the messages without any restriction.
 *  o If new messages arrive into a folder while auto-sync is downloading other messages of the
 *    same folder, it simply puts the new messages into the folder's download queue, and
 *    re-prioritize the messages. That behavior makes sure that the high priority
 *    (defined by the message strategy) get downloaded first always.
 *  o If new messages arrive into a folder while auto-sync is downloading messages of a lower
 *    priority folder, auto-sync switches the folders in the queue and starts downloading the
 *    messages of the higher priority folder next time it downloads a message group.
 *  o Currently there is no way to stop/pause/cancel a message download. The smallest
 *    granularity is the message group size.
 *  o Auto-Sync manager periodically (kAutoSyncFreq) checks folder for existing messages
 *    w/o bodies. It persists the last time the folder is checked in the local database of the
 *    folder. We call this process 'Discovery'. This process is asynchronous and processes
 *    |kNumberOfHeadersToProcess| number of headers at each cycle. Since it works on local data,
 *    it doesn't consume lots of system resources, it does its job fast.
 *  o Discovery is necessary especially when the user makes a transition from not-offline
 *    to offline mode.
 *  o Update frequency is defined by nsMsgIncomingServer::BiffMinutes.
 *
 * Error Handling:
 *  o if the user moves/deletes/filters all messages of a folder already queued, auto-sync
 *    deals with that situation by skipping the folder in question, and continuing with the
 *    next in chain.
 *  o If the message size is zero, auto-sync ignores the message.
 *  o If the download of the message group fails for some reason, auto-sync tries to
 *    download the same group |kGroupRetryCount| times. If it still fails, continues with the
 *    next group of messages.
 *
 * Download Model:
 *  Parallel model should be used with the imap servers that do not have any "max number of sessions
 *  per IP" limit, and when the bandwidth is significantly large.
 *
 * How it really works:
 * The AutoSyncManager gets an idle notification. First it processes any
 * folders in the discovery queue (which means it schedules message download
 * for any messages it previously determined it should download). Then it sets
 * a timer, and in the timer callback, it processes the update q, by calling
 * InitiateAutoSync on the first folder in the update q.
 *
 * See additional info near the bottom of this file.
 */
// clang-format on

/**
 * Default strategy implementation to prioritize messages in the download queue.
 */
class nsDefaultAutoSyncMsgStrategy final : public nsIAutoSyncMsgStrategy {
  static const uint32_t kFirstPassMessageSize = 60U * 1024U;  // 60K

 public:
  NS_DECL_ISUPPORTS
  NS_DECL_NSIAUTOSYNCMSGSTRATEGY

  nsDefaultAutoSyncMsgStrategy();

 private:
  ~nsDefaultAutoSyncMsgStrategy();
};

/**
 * Default strategy implementation to prioritize folders in the download queue.
 */
class nsDefaultAutoSyncFolderStrategy final : public nsIAutoSyncFolderStrategy {
 public:
  NS_DECL_ISUPPORTS
  NS_DECL_NSIAUTOSYNCFOLDERSTRATEGY

  nsDefaultAutoSyncFolderStrategy();

 private:
  ~nsDefaultAutoSyncFolderStrategy();
};

// see the end of the page for auto-sync internals

/**
 * Manages background message download operations for offline imap folders.
 */
class nsAutoSyncManager final : public nsIObserver,
                                public nsIUrlListener,
                                public nsIAutoSyncManager {
  static const PRTime kAutoSyncFreq = 60UL * (PR_USEC_PER_SEC * 60UL);  // 1hr
  static const uint32_t kDefaultUpdateInterval = 10UL;                  // 10min
  static const int32_t kTimerIntervalInMs = 400;
  static const uint32_t kNumberOfHeadersToProcess = 250U;
  // enforced size of the first group that will be downloaded before idle time
  static const uint32_t kFirstGroupSizeLimit = 60U * 1024U /* 60K */;
  static const int32_t kIdleTimeInSec = 10;
  static const uint32_t kGroupRetryCount = 3;

  enum IdleState { systemIdle, appIdle, notIdle };

 public:
  NS_DECL_ISUPPORTS
  NS_DECL_NSIOBSERVER
  NS_DECL_NSIURLLISTENER
  NS_DECL_NSIAUTOSYNCMANAGER

  nsAutoSyncManager();

 private:
  ~nsAutoSyncManager();

  void SetIdleState(IdleState st);
  IdleState GetIdleState() const;
  nsresult StartIdleProcessing();
  nsresult AutoUpdateFolders();
  void ScheduleFolderForOfflineDownload(nsIAutoSyncState* aAutoSyncStateObj);
  nsresult DownloadMessagesForOffline(nsIAutoSyncState* aAutoSyncStateObj,
                                      uint32_t aSizeLimit = 0);
  nsresult HandleDownloadErrorFor(nsIAutoSyncState* aAutoSyncStateObj,
                                  const nsresult error);

  // Helper methods for priority Q operations
  static void ChainFoldersInQ(const nsCOMArray<nsIAutoSyncState>& aQueue,
                              nsCOMArray<nsIAutoSyncState>& aChainedQ);
  static nsIAutoSyncState* SearchQForSibling(
      const nsCOMArray<nsIAutoSyncState>& aQueue,
      nsIAutoSyncState* aAutoSyncStateObj, int32_t aStartIdx,
      int32_t* aIndex = nullptr);
  static bool DoesQContainAnySiblingOf(
      const nsCOMArray<nsIAutoSyncState>& aQueue,
      nsIAutoSyncState* aAutoSyncStateObj, const int32_t aState,
      int32_t* aIndex = nullptr);
  static nsIAutoSyncState* GetNextSibling(
      const nsCOMArray<nsIAutoSyncState>& aQueue,
      nsIAutoSyncState* aAutoSyncStateObj, int32_t* aIndex = nullptr);
  static nsIAutoSyncState* GetHighestPrioSibling(
      const nsCOMArray<nsIAutoSyncState>& aQueue,
      nsIAutoSyncState* aAutoSyncStateObj, int32_t* aIndex = nullptr);

  /// timer to process existing keys and updates
  void InitTimer();
  static void TimerCallback(nsITimer* aTimer, void* aClosure);
  void StopTimer();
  void StartTimerIfNeeded();

 protected:
  nsCOMPtr<nsIAutoSyncMsgStrategy> mMsgStrategyImpl;
  nsCOMPtr<nsIAutoSyncFolderStrategy> mFolderStrategyImpl;
  // contains the folders that will be downloaded on background
  nsCOMArray<nsIAutoSyncState> mPriorityQ;
  // contains the folders that will be examined for existing headers and
  // adds the headers we don't have offline into the autosyncState
  // object's download queue.
  nsCOMArray<nsIAutoSyncState> mDiscoveryQ;
  // contains the folders that will be checked for new messages with STATUS,
  // and if there are any, we'll call UpdateFolder on them.
  nsCOMArray<nsIAutoSyncState> mUpdateQ;
  // this is set true when autosync is initiated for a single folder. Its
  // purpose is ensure that update for a folder finishes before the next one
  // starts.
  bool mUpdateInProgress;

  // This is set if auto sync has been completely paused.
  bool mPaused;
  // This is set if we've finished startup and should start
  // paying attention to idle notifications.
  bool mStartupDone;

 private:
  uint32_t mGroupSize;
  IdleState mIdleState;
  int32_t mDownloadModel;
  nsCOMPtr<nsIUserIdleService> mIdleService;
  nsCOMPtr<nsITimer> mTimer;
  nsTObserverArray<nsCOMPtr<nsIAutoSyncMgrListener> > mListeners;
};

#endif

/*
How queues inter-relate:

nsAutoSyncState has an internal priority queue to store messages waiting to be
downloaded. nsAutoSyncMsgStrategy object determines the order in this queue,
nsAutoSyncManager uses this queue to manage downloads. Two events cause a
change in this queue:

1) nsImapMailFolder::HeaderFetchCompleted: is triggered when TB notices that
there are pending messages on the server -- via IDLE command from the server,
via explicit select from the user, or via automatic Update during idle time. If
it turns out that there are pending messages on the server, it adds them into
nsAutoSyncState's download queue.

2) nsAutoSyncState::ProcessExistingHeaders: is triggered for every imap folder
every hour or so (see kAutoSyncFreq). nsAutoSyncManager uses an internal queue
called Discovery queue to keep track of this task. The purpose of
ProcessExistingHeaders() method is to check existing headers of a given folder
in batches and discover the messages without bodies, in asynchronous fashion.
This process is sequential, one and only one folder at any given time, very
similar to indexing. Again, if it turns out that the folder in hand has messages
w/o bodies, ProcessExistingHeaders adds them into nsAutoSyncState's download
queue.

Any change in nsAutoSyncState's download queue, notifies nsAutoSyncManager and
nsAutoSyncManager puts the requesting  nsAutoSyncState into its internal
priority queue (called mPriorityQ) -- if the folder is not already there.
nsAutoSyncFolderStrategy object determines the order in this queue. This queue
is processed in two modes: chained and parallel.

i) Chained: One folder per imap server any given time. Folders owned by
different imap servers are simultaneous.

ii) Parallel: All folders at the same time, using all cached-connections -
a.k.a 'Folders gone wild' mode.

Note: The "Chained" mode is currently in use: mDownloadModel = dmChained;

The order the folders are added into the mPriorityQ doesn't matter since every
time a batch completed for an imap server, nsAutoSyncManager adjusts the order.
So, lets say that updating a sub-folder starts downloading message immediately,
when an higher priority folder is added into the queue, nsAutoSyncManager
switches to this higher priority folder instead of processing the next group of
messages of the lower priority one. Setting group size too high might delay
this switch at worst.

And finally, Update queue helps nsAutoSyncManager to keep track of folders
waiting to be updated. With the latest change, we update one and only one
folder at any given time. Default frequency of updating is 10 min
(kDefaultUpdateInterval). We add folders into the update queue during idle time,
if they are not in mPriorityQ already.

*/