Teaching Your Email to Fetch

Mail systems, like the messages they convey, have this irritating habit of going forth and multiplying. The more I use email, the more mail systems I find it necessary to check. Thankfully, some of the systems I use make it convenient for me to forward my mail to another off-site address, leaving the headers relatively untouched. Some systems, however, make all mail forwarded from my account appear to be from me, and not from the original sender. Other systems have any number of administrative or technical reasons why the mail can't be forwarded.

Enter Fetchmail. This glorious little utility is one of those tools you could set up once and forget about, if it weren't so obvious how much easier it has made your life. Fetchmail sits on your favorite UNIX host, periodically acting as an unattended mail-user agent connecting through any number of mail-access protocols to retrieve your messages and relay them on to your preferred address.

If you want to find even more ways to organize your mail, Kevin recently wrote another article, Teaching Your Email to Be Even More Fetching, which covers some of Fetchmail's more sophisticated features, including ways to use encryption in mail retrieval and alternative methods of mail delivery.

Historic and Cultural Significance of Fetchmail

In 1996, Eric S. Raymond, Fetchmail's author, needed some software he could run intermittently to relay email from remote hosts to his home system when it was connected to the Internet. Not only did he need something to consolidate all his mail in one place, but he needed to preserve legitimately formatted headers, and delicately munge those headers that didn't have a fully qualified domain in the address, like "bob" or "sue@bighost" (instead of "bobsue@bighost.com").

For several years, Raymond had seen some of the best software on the Internet, Linux in particular, emerge from a loosely organized peer-review process that free software had engaged in. He describes this process as "the Bazaar," as opposed to "the Cathedral," which describes the closed-source, proprietary commercial development cycle. Raymond's seminal paper, "The Cathedral & The Bazaar," originally published online (and expanded into an O'Reilly book), details his development of Fetchmail as a case study in the Bazaar. It brought industrywide attention to the differences between the two methods and played a central role in the public release of Netscape Communicator and other commercial source code. Whole sectors of the software industry were suddenly galvanized to embrace the Open Source approach.

In an ironic twist to the open source tale, the Cathedral itself was unwittingly one of the sources of the Bazaar: Because a 1956 consent decree prohibited AT&T from selling software, it found itself giving away source code to its new UNIX operating system in the late '60s and early '70s to anyone who could make use of it. Users became contributors, and contributors became coauthors. The culture of development through peer review, interdependency, and user-driven software engineering has been part of the UNIX culture ever since. Back to topic at hand, though. Given a few unpretentious resources, you can build Fetchmail in no time. Once it's built, you'll be able to participate in the movement, and retrieve your mail from disparate mailboxes at the same time.

Building Fetchmail

Fetchmail has modest dependencies. You'll need version 2.5.3, or greater, of the GNU Flex lexical analyzer. If you're installing under Linux, you've probably already got it. If you're installing under Solaris or HP/UX, take a minute and replace your lex with GNU flex. The INSTALL file contained in the Fetchmail distribution advises you on where to get it.

Most of the time, and especially the first time you build it, you'll want to do just a plain-vanilla build of Fetchmail with no special features to get a baseline version working. There are a number of features you can turn on from the configure script, though. Some of these features also require additional software, as specified in the INSTALL file. These features include one-time passwords, IPV6, IPSec, Compuserve RPA authentication, Microsoft NTLM authentication, GSSAPI authentication, SSL, and Kerberos.

As is, Fetchmail will use POP and IMAP to retrieve your mail and relay it using SMTP. There are some options to tunnel these protocols inside more secure, encrypted protocols, but we'll get to that later in this article.

A basic install of Fetchmail is as easy as untarring the source into a directory, and typing ./configure;make;make install. The make install will issue a set of messages telling you where you installed the man pages, the Fetchmail binary, and the Fetchmailconf utility for generating your Fetchmail recipe file.

The default installation for everything is in /usr/local: for Fetchmail, /usr/local/bin; for the man page, /usr/local/man/man1, and so on. You may, of course, change these to suit yourself with command-line switches to the configure command. Issue the command ./configure --help for a listing of all the options available to you. All the really important options, or those that clearly need further explanation are outlined in the INSTALL file.

Running Fetchmail

6,16,26,36,46,56 * * * * /usr/local/bin/fetchmail>/dev/null 2>&1

Fetchmail looks for its recipe file by default in $HOME/.fetchmailrc. Under most circumstances, the .fetchmailrc file should have the permissions set to 600 (by issuing a chmod 600 .fetchmailrc command), closing off all access to "group" and "other." If the recipe file has permissions greater than 0710 (which grants all permissions to the owner, and execute-only to the group), Fetchmail will fail to run, interpreting the condition as a security flaw. Because your mail passwords are stored in cleartext in the recipe file, granting read permission to group or other is not advised.

Eric Raymond argues that encrypting the passwords in the recipe file will not result in increased security. If someone can su to your ID and become you, so to speak, then arguably they could read your mail. Any decryption code contained in Fetchmail that would permit unattended decryption could likewise be used maliciously to decrypt encrypted passwords. One caveat to which you might adhere would be to ensure that any passwords you put in your .fetchmailrc file aren't used for more than one purpose, to minimize your risk should the Fetchmail host be compromised.

A complete description of everything you can do inside your Fetchmail recipe file is beyond the scope of this article. Once you've tried variations on the recipes in this article, however, examine the sample recipes in the Fetchmail distribution and accompanying man pages for more ideas. Now, here's an example of a stripped-down .fetchmailrc file:

defaults
    no fetchall
    keep

poll enterprise.themullets.net protocol IMAP
    user temp1 password SoMeThInG
    smtphost enterprise.themullets.net

poll wtmi.net protocol POP3
    user temp2 password SoMeThInG
    smtphost enterprise.themullets.net

poll enterprise.atomicconsulting.com protocol IMAP
    user temp4 password SoMeThInG
    smtphost enterprise.themullets.net

poll www.atomicconsulting.com protocol IMAP
    user temp4 password SoMeThInG
    smtphost enterprise.themullets.net
    interval 5

skip pop.mail.yahoo.com protocol POP3
    user temporaryrooster password SoMeThInG
    smtphost enterprise.themullets.net

This file starts off with a "defaults" section and follows with five recipes. It is formatted for user readability, but not because Fetchmail needs it that way. Aside from comments (any line or right-hand portion of a line that begins with a "#"), global options and server entries (individual recipes) may be written in free form.

There are mechanisms in both POP and IMAP for determining if a given message has been downloaded before. For POP, a local table is kept so it's possible for a message to be "new" to one user agent or to a copy of Fetchmail, and not to another. IMAP has server-based message flags that track whether a message has been "seen" or not. fetchall, for example, would unconditionally retrieve every message in the inbox during each polling cycle.

The defaults section specifies the settings that apply unless you override them in a given recipe. The hardcoded defaults for each option are given in the fetchmail(1) man page. There are many more default options you can set, but here are two good ones to use to get started:

no fetchall
Results in Fetchmail retrieving only new messages.

keep
Determines whether or not Fetchmail deletes messages from the server once it retrieves them.

The combination of fetchall and keep is a good way to ensure lots of duplicate messages.

The first five recipes start with the keyword poll. Each time Fetchmail is started, with no contravening options, the poll recipes will be performed. The skip recipes are just stored for reference in the recipe file until Fetchmail is started and specifically requested to poll that particular site. For example, to retrieve my Yahoo mail, I'd start Fetchmail with the command line fetchmail -v pop.mail.yahoo.com. The -v gives me verbose output, which is good for debugging. Usually, you wouldn't see these messages.

The skip recipe above for pop.mail.yahoo.com connects with Yahoo mail server. Note that to do this with your Yahoo Mail account, you need to configure it from the user options section in their Webmail system. As you can see in the example below, Fetchmail connects with the Yahoo server and sees that there are eleven messages in my inbox, three of which are new. Fetchmail retrieves those three, leaving all eleven "old" messages on the server.

% fetchmail -v pop.mail.yahoo.com

fetchmail: 5.5.1 querying pop.mail.yahoo.com (protocol POP3)
    at Thu, 07 Sep 2000 18:44:26 -0500 (CDT)
fetchmail: POP3< +OK hello from popgate
fetchmail: POP3> USER temporaryrooster
fetchmail: POP3< +OK password required.
fetchmail: POP3> PASS *
fetchmail: POP3< +OK maildrop ready, 11 messages (12275 octets)
fetchmail: POP3> STAT
fetchmail: POP3< +OK 11 12275
fetchmail: POP3> LAST
fetchmail: POP3< +OK 8
11 messages (8 seen) for temporaryrooster at
     pop.vip.suc.yahoo.com (12275 octets).
fetchmail: POP3> LIST
fetchmail: POP3< +OK 11 messages (12275 octets)
fetchmail: POP3< 1 6895
fetchmail: POP3< 2 538
fetchmail: POP3< 3 538
fetchmail: POP3< 4 538
fetchmail: POP3< 5 538
fetchmail: POP3< 6 538
fetchmail: POP3< 7 538
fetchmail: POP3< 8 538
fetchmail: POP3< 9 538
fetchmail: POP3< 10 538
fetchmail: POP3< 11 538
fetchmail: POP3< .
skipping message 1 not flushed
skipping message 2 not flushed
skipping message 3 not flushed
skipping message 4 not flushed
skipping message 5 not flushed
skipping message 6 not flushed
skipping message 7 not flushed
skipping message 8 not flushed
fetchmail: POP3> RETR 9
fetchmail: POP3< +OK 538 octets
reading message 9 of 11 (538 octets)
fetchmail: SMTP< 220 wtmi.net ESMTP
    Sendmail 8.10.1/8.10.1; Thu, 7 Sep 2000 18:44:30 -0500
fetchmail: SMTP> EHLO localhost
fetchmail: SMTP< 250-wtmi.net Hello IDENT:jtemp@7of9 [166.21.99.172],
    pleased to meet you
fetchmail: SMTP< 250-ENHANCEDSTATUSCODES
fetchmail: SMTP< 250-8BITMIME
fetchmail: SMTP< 250-SIZE
fetchmail: SMTP< 250-DSN
fetchmail: SMTP< 250-ONEX
fetchmail: SMTP< 250-ETRN
fetchmail: SMTP< 250-XUSR
fetchmail: SMTP< 250 HELP
fetchmail: SMTP> MAIL FROM: SIZE=538
fetchmail: SMTP< 250 2.1.0 ... Sender ok
fetchmail: SMTP> RCPT TO:
fetchmail: SMTP< 250 2.1.5 ... Recipient ok
fetchmail: SMTP> DATA
fetchmail: SMTP< 354 Enter mail, end with "." on a line by itself
#fetchmail: SMTP>. (EOM)
fetchmail: SMTP< 250 2.0.0 e87NiUs31355 Message accepted for delivery not flushed
fetchmail: POP3> RETR 10
    [message 10 retrieved]
fetchmail: POP3> RETR 11
    [message 11 retrieved]
fetchmail: POP3> QUIT
fetchmail: POP3< +OK server signing off.
fetchmail: SMTP> QUIT
fetchmail: SMTP< 221 2.0.0 wtmi.net closing connection
fetchmail: normal termination, status 0
%

Again, you normally won't see all this verbosity, but it helps to illustrate what Fetchmail is up to when it's doing your bidding.

Let's walk through the recipes. As I mentioned above, the first four recipes will run when you start Fetchmail without any command-line directives, and the last one will run only when you issue the command fetchmail pop.mail.yahoo.com.

The recipe for www.atomicconsulting.com has the sequence interval 5. This means that recipe will run only once every five times that Fetchmail is run. This would be a good option to use for those recipes pointing to mailboxes you want to watch, but from which you have no expectation of important mail.

Each recipe takes the form:

poll or skip hostname PROTOCOL protocolname
    various options

In fact, a really stripped-down recipe could take the form:

poll mail.myisp.net protocol auto user joeuser password mypass

With this recipe, the host mail.myisp.net would be queried using IMAP, POP3, and various mail retrieval protocols until Fetchmail found one that worked. Once the connection was made, Fetchmail would authenticate with userID joeuser and password mypass, retrieve the mail from the inbox, and send it to the local user by connecting to an SMTP server at localhost.

Sometimes, you may run Fetchmail on a host without a reliable local SMTP server. In those cases, rather than using the default behavior of connection to SMTP at localhost, you can specify an SMTP server for each recipe as I've done here with the smtphost keyword.

Depending on how you look at it, the wealth of Fetchmail features constitute either enough rope to hang yourself, or a fabulous toolkit to manage buckets of unruly mailboxes.

And That's Not All

One of the most useful tricks is to specify a program to use as a wrapper around your upstream TCP connection for POP3, IMAP, or whatever mail-access protocol you use, or for your downstream SMTP connection. Fetchmail calls the software wrapped around your upstream TCP connection a "plugin," and the one wrapped around your downstream SMTP connection a "plugout."

Usually, this is done with Secure Shell (SSH), which has its own particular requirements on the remote server. Some users may find it better to build Secure Sockets Layer (SSL) support into Fetchmail, especially if their IMAP, POP, or SMTP servers natively support it. The advantage of doing either SSH or SSL would be to robustly encrypt and secure the connection between your Fetchmail host and other hosts so that neither your credentials nor the content of your messages is transmitted in the clear.

Another interesting Fetchmail feat is servicing a multidrop mailbox. If you've got a remote mailbox that receives mail for multiple usernames, Fetchmail can retrieve the mail, sort it out by username, and act on each username differently. This particular feature represents a bit of overlap with the venerable Procmail, but it's always nice to have more than one tool to choose from.

With all these features, Fetchmail is, quite simply, gravity for your email. Fire this utility up on your preferred mailhost, and watch all the mail from your other accounts get sucked into one common mailbox. It's really rather cool.

Kevin Mullet is the vice president of Atomic Consulting, a UNIX and network consulting business for small- and medium-size companies in the Dallas area. Kevin can be reached at kwm@themullets.net.

O'Reilly & Associates recently released (September 2000) Managing IMAP.

• Sample Chapter 9, Cyrus System Administration, is available free online.

• You can also look at the Table of Contents, the Index, and the Full Description of the book.

• For more information, or to order the book, click here.