mswatch

About

mswatch is a command line unix program that keeps two mailboxes synchronized more efficiently and with shorter delays than periodically synchronizing the two mailboxes.

mswatch watches mailboxes to know when to initiate mailbox syncs. Using mswatch, your mail synchronization program can be called on demand instead of through polling, resulting in prompter mail delivery and lower bandwidth usage and server load.

mswatch is designed to work in conjunction with mailbox synchronization programs, currently supports watching Linux (2.4+) hosted Maildirs (including Maildir folders, Maildir++), and is licensed under the GNU GPL. Future support for additional mailbox formats, especially for mbox, is planned.

Download

Current release: 1.1.0
Other releases
Development (in subversion): https://mswatch.svn.sourceforge.net/svnroot/mswatch/trunk/ (Browse)

Why mswatch

Keeping a set of mailboxes on multiple computers means keeping the copies synchronized (eg on one's desktop and laptop or on a personal computer that cannot be reached while on the road and an always-reachable server). While the IMAP protocol is designed to support this and is widely used, IMAP clients can only detect mailbox changes by querying the server. Thus clients generally either sync when the user requests or sync periodically. However, because each mailbox sync uses server and network resources and because users often have a number of mailboxes, many of which receive mail infrequently, one must compromise between prompt message delivery and low server and network resource usage. Typical practice is to sync only when the user requests or to sync periodically with 5-10 minute frequencies. Nonetheless, it is often helpful for users to see new mail fairly soon after its arrival (eg notifications, group meeting changes, or active conversation threads).

mswatch addresses this speed of delivery vs resource usage tradeoff using interrupt-based instead of polling behavior to detect mailbox changes. mswatch asks each of the two mailstores to send mailbox change notifications; when mswatch receives a notification it in turn invokes the user's mailbox synchronization program for the changed mailbox. Thus, idle mailboxes do not use active resources and prompt mailbox synchronization is efficient.

Why mswatch Over Other Interrupt-based Approaches

While the above explains why one might want interrupt- instead of polling-based mailbox syncing, why use mswatch instead of another interrupt-based approach? Other known approaches and their limitations:

• IMAP IDLE command
  IMAP servers and clients may support the "IDLE" command - the server tells the client when a mailbox changes. However, the IDLE command watches only a single mailbox, not a set of mailboxes. One could maintain an IMAP connection for each mailbox, using the IDLE command with each connection. Mark Crispin even lists using multiple IDLE connections, instead of polling, as one of the ten IMAP client commandments. However, each connection often runs within its own process and uses 5+MB of ram. Consider doing this with 100 mailboxes, and then keep in mind that this is per-user on the mail server. This approach does not scale well to many mailboxes and/or users.

  If mail is delivered only to a single folder on the remote host and all other mailbox changes occur only locally then a client and server that support the IDLE command are sufficient to use interrupt-based synchronization.

• Create an IMAP IDLE-MULTIPLE command
  IMAP's IDLE command is almost what is desired, except that it watches only a single mailbox. So why not create an IDLE-MULTIPLE command that watches a set of mailboxes?
  • The author of IMAP, Mark Crispin, does not see this capability as belonging in IMAP. He states that multiple mailbox notification belongs in a separate protocol designed for event notification [citation].
  • Protocol and software simplicity. While an approach usable by a broader client base than mswatch is usable would be great, mswatch is simple enough that it can exist now. When more neutral approaches come to fruition, just move on!
• Notification Protocols
  Part of the IETF "Lemonade" working group's goal is to extend IMAP to support interrupt-based synchronization. However, their design goals are for constrained platforms instead of general computers, their goals include much more than just interrupt-based synchronization, and the project is not yet complete (but the group does appear to be active). See also their protocol Push-IMAP and the "Simple Message Notification Protocol".

How mswatch Works

mswatch watches two sets of mailboxes for changes; when a change is detected mswatch invokes the user's mail synchronizer for the changed mailbox(es).

mswatch

Each set of mailboxes (a "mailstore" in mswatch terminology) is watched through an external program which watches for changes to its mailstore and prints the name of a mailbox when the box is changed. Through this interface mswatch can watch local or remote (using ssh) mailstores. mswatch currently includes the program watch_maildirs to watch Linux hosted Maildirs. Programs to watch other mailstore types or support more platforms are most welcome! (If you are open to helping to test watch_mboxes, please email the mswatch list.)

mswatch reads and coalesces the mailbox names from each of the two mailstores. Following a short delay after a mailbox change (to allow multiple mailboxes to be synced with the same synchronization connection) mswatch invokes the mailbox synchronization program with the names of the changed mailboxes. mswatch was developed with mbsync in mind, but any mail synchronizer that implements the mailbox synchronizer interface can be invoked.

mswatch is designed to robustly interact with other programs and with networks:

• mswatch is conservatively designed to never miss mailbox changes; for example, mailstore watching is started before initial mailstore synchronization to ensure messages delivered during or shortly after synchronization are not missed.
• mswatch safely recovers from lost connections. A failed mailbox synchronization is gracefully restarted. A disconnected watcher is restarted and the mailstores then resynched (to detect any messages delivered between watchings).
• mswatch incrementally backs off when repeated failures are encountered to avoid assaulting your network connection and mail server.

watch_maildirs

watch_maildirs watches a Maildir-based mailstore for and reports changes; Maildir and Maildir folders (Maildir++) are supported. watch_maildirs uses Linux's inotify (2.6.13+) or dnotify (2.4.19+) interfaces to watch the Maildirs for mailbox changes.

inputkill

inputkill is a small program that promptly kills a remote mailstore watcher when its connection to mswatch's dies; without it remote mailstore watchers would only exit much later, running needlessly. inputkill monitors its standard input; when stdin is closed, inputkill kills its child process and exits.

Limitations

Because mswatch abstracts mail changes and synchronization to the mailbox (instead of message) level and because mail watcher communication is one-way, extra mailbox syncs occur. While it would be great to avoid this extra work, it is in the noise compared to the number of empty syncs that polling accumulates.

For example, the case of new mail delivered to the mail server's INBOX:

1. mswatch detects the changed server INBOX
2. mswatch invokes the mail syncer for INBOX, which adds the mail to the local INBOX
3. mswatch detects the changed local INBOX
4. mswatch invokes the mail syncer for INBOX, which detects no changes

Thus two syncs occur in this case instead of the single necessary sync.

Documentation

• README
• Man pages: mswatch(1), watch_maildirs(1), inputkill(1)
• Mailstore watcher and mailbox synchronizer interfaces
• Mailing list: mswatch-list (user and devel discussions)
• SourceForge project page

Related:

• mbsync: synchronizes Maildir and/or IMAP mailboxes (thank you mbsync developers!)
• rpatterson.mailsync: integrates mswatch, the Maildir and/or IMAP mailbox synchronizer OfflineIMAP, and the mail user agent Gnus

People

Chris Frost