Understanding an email message encoded with
MIME can be very, very, very difficult. It can get frustrating due to
the number of options and different ways to do the actual
encoding. Now add to that the sometimes too-liberal interpretations of
the relevant RFCs by the email client designers and you will begin to
get the idea. This article will show you how this task can be
laughably simple thanks to Perl's extensive bag of tricks, CPAN
.
I started out with a simple and straightforward mission: Fetch an Email from a POP mailbox and display it in a 7-bit, text-only capable device. This article describes the different stages for a simple tool that accomplishes this task, written in Perl with a lot of help from CPAN modules. I hope this to be useful to other Perl folks who might have a similar mission. Let's discuss each part of this task in turn, as we read through mfetch, the script I prepared as an example. Keep in mind that TIMTOWTDI.
Setting up the script
The first thing, as you know, is loading up all of the modules I will be using. I'm sure you already know strict and warnings. We'll see how do we use the rest of the modules a bit later.
1: #!/usr/bin/perl
2:
3: # This script is (c) 2002 Luis E. Mu帽oz, All Rights Reserved
4: # This code can be used under the same terms as Perl itself. It comes
5: # with absolutely NO WARRANTY. Use at your own risk.
6:
7: use strict;
8: use warnings;
9: use IO::File;
10: use Net::POP3;
11: use NetAddr::IP;
12: use Getopt::Std;
13: use MIME::Parser;
14: use HTML::Parser;
15: use Unicode::Map8;
16: use MIME::WordDecoder;
17:
18: use vars qw($opt_s $opt_u $opt_p $opt_m $wd $e $map);
19:
20: getopts('s:u:p:m:');
21:
22: usage_die("-s server is required\n") unless $opt_s;
23: usage_die("-u username is required\n") unless $opt_u;
24: usage_die("-p password is required\n") unless $opt_p;
25: usage_die("-m message is required\n") unless $opt_m;
26:
27: $opt_s = NetAddr::IP->new($opt_s)
28: or die "Cannot make sense of given server\n";
Note the lines 27 and 28, where I use NetAddr::IP to convert whatever the user gave us through the
-s option into an IP address. This is a very common use of
this module, as its new() method will convert many common IP
notations into an object I can later extract an IP address from. It
will even perform a name resolution for us if required. So far,
everything should look familiar, as a lot of scripts start like this
one.
It is worth noting that the error handling in lines 22-25 is not a
brilliant example of good coding or documentation. It is much better
practice to write your scripts' documentation in POD, and use a module
such as Pod::Usage to provide useful error messages to the user. At
the very least, try to provide an informative usage message. You can
see the usage_die() function if you download the complete script.
Fetching a message via POP3
On to deeper waters. The first step in parsing a message, is getting
at the message itself. For this, I'll use Net::POP3, which implements the POP3 protocol described in
RFC-1939. This is all done in the code below.
30: my $pops = Net::POP3->new($opt_s->addr)
31: or die "Failed to connect to POP3 server: $!\n";
32:
33: $pops->login($opt_u, $opt_p)
34: or die "Authentication failed\n";
35:
36: my $fh = new_tmpfile IO::File
37: or die "Cannot create temporary file: $!\n";
38:
39: $pops->get($opt_m, $fh)
40: or die "No such message $opt_m\n";
41:
42: $pops->quit();
43: $pops = undef;
44:
45: $fh->seek(0, SEEK_SET);
At line 30, a connection to the POP server is attempted. This is a TCP connection, in this case to port 110. If this connection succeeds, the USER and PASS commands are issued at line 33, which are the simplest form of authentication supported by the POP protocol. Your username and password are being sent here through the network without the protection of cryptography, so a bit of caution is in order.
Net::POP3 supports many operations defined in the POP
protocol that allow for more complex actions, such as fetching the
list of messages, unseen messages, etc. It can also fetch messages for
us in a variety of ways. Since I want this script to be as lightweight
as possible (i.e., to burn as little memory as possible), I want to
fetch the message to a temporary on-disk file. The temporary file is
nicely provided by the new_tmpfile method of IO::File in line 36, which returns a file handle to a
deleted file. I can work on this file, which will magically disappear
when the script is finished.
Later, I instruct the Net::POP3 object to fetch the required message from the mail server and write it to the supplied filehandle using the get method, on line 39. After this, the connection is terminated gracefully by invoking quit and destroying the object. Destroying the object insures that the TCP connection with the server is terminated, freeing the resources being held in the POP server as soon as possible. This is a good programming practice for network clients.
The interaction required by mfetch with the POP server is really simple, so I'm not making justice to Net::POP3. It provides a very complete implementation of the protocol, allowing for much more sophisticated applications.
Note that in line 45, I rewind the file so that the fetched message can be read back by the code that follows.
For this particular example, we could also have used Net::POP3Client, which provides a somewhat similar interface. The code would have looked more or less like the following fragment.
1: my $pops = new Net::POP3Client(USER => $opt_u,
2: PASSWORD => $opt_p,
3: HOST => $opt_s->addr)
4: or die "Error connecting or logging in: $!\n";
5:
6: my $fh = new_tmpfile IO::File
7: or die "Cannot create temporary file: $!\n";
8:
9: $pops->HeadAndBodyToFile($fh, $opt_m)
10: or die "Cannot fetch message: $!\n";
11:
12: $pops->Close();
Parsing the MIME structure
Just as email travels inside a sort of envelope (the headers), complex messages that include attachments and generally, HTML messages, travel within a collection of MIME entities. You can think of these entities as containers that can transfer any kind of binary information through the Email infrastructure, which in general does not know how to deal with 8-bit data. The code reproduced below, takes care of parsing this MIME structure.
47: my $mp = new MIME::Parser;
48: $mp->ignore_errors(1);
49: $mp->extract_uuencode(1);
50:
51: eval { $e = $mp->parse($fh); };
52: my $error = ($@ || $mp->last_error);
53:
54: if ($error)
55: {
56: $mp->filer->purge; # Get rid of the temp files
57: die "Error parsing the message: $error\n";
58: }
Perl has a wonderful class that provides the ability to understand
this MIME encapsulation, returning a nice hierarchy of objects that
represent the message. You access this facilities through the MIME::Parser class, part of the MIME-Tools bundle. MIME::Parser returns a hierarchy
of MIME::Entity objects representing your message. The parser
is so smart, that if you pass it a non-MIME email, it will be returned
to you as a text/plain entity.
MIME::Parser can be tweaked in many ways, as its documentation will show you. One of the points where this toggling might be important, is the decoding process. Remember that I need to be as light in memory usage as possible. The default behavior of MIME::Parser involves the use of temporary files for decoding of the message. These temporary files can be spared and core memory used instead by invoking output_to_core(). Before doing this, note all the caveats cited in the module's documentation. The most important one is that if a 100 MB file ends up in your inbox, this whole thing needs to be slurped into RAM.
In line 47 I create the parser object. The call to ignore_errors() in line 48 is an attempt to made this parser as tolerant as possible. extract_uuencode() in line 49, takes care of pieces of the email that are uu-encoded automatically, translating them back into a more readable form. The actual request to parse the message, available through reading the $fh filehandle, is in line 51. Note that it is enclosed in an eval block. I have to do this as the parser might throw an exception if certain errors are encountered. The eval allows me to catch this exception and react in a way that is sensible to this application. In this case, I want to be sure that any temporary file created by the parsing process is cleared by a call to purge(), as seen in lines 56 and 57.
Setting up the HTML parser
Parsing HTML can be a tricky and tedious task. Thankfully, Perl has a
number of nice ways to help you do this job. A number of excellent
books such as The Perl Cookbook (from O'Reilly &
Associates) has a couple of recipes that came very close to what I
needed, especially recipe 20.5, "Converting HTML to ASCII",
which I reproduce below.
1: use HTML::TreeBuilder;
2: use HTML::FormatText;
3:
4: $html = HTML::TreeBuilder->new();
5: $html->parse($document);
6:
7: $formatter = HTML::FormatText->new(leftmargin => 0, rightmargin => 50);
8:
9: $ascii = $formatter->format($html);
I did not want to use this recipe because of two reasons: I needed fine-grained control in the HTML to ASCII conversion and I wanted to have as little impact as possible in resources. I did a small benchmark that shows the kind of performance difference among the two options while parsing a copy of one of my web articles. The result below shows that the custom parser explained later runs faster than the Cookbook's recipe. This does not mean that the recipe or the modules it uses are bad. This result simply means that the recipe is actually doing a lot of additional work, which just happens to not be all that useful for this particular task.
bash-2.05a$ ./mbench
Benchmark: timing 100 iterations of Cookbook's, Custom...
Cookbook's: 73 wallclock secs (52.82 usr + 0.00 sys = 52.82 CPU) @ 1.89/s (n=100)
Custom: 1 wallclock secs ( 1.17 usr + 0.00 sys = 1.17 CPU) @ 85.47/s (n=100)
Rate Cookbook's Custom
Cookbook's 1.89/s -- -98%
Custom 85.5/s 4415% --
HTML::FormatText does an awesome job of converting the HTML
to plain text. Unfortunately I have a set of guidelines that I need to
follow in the conversion and that are not compatible with the output
of this module. Additionally, HTML::TreeBuilder does an excellent job of parsing an HTML
document, but produces an intermediate structure - the parse tree -
that in my case, wastes resources.
However, Perl has an excellent HTML parser in the HTML::Parser module. In this case, I chose to use this class
to implement an event-driven parser, where tokens (syntactic elements)
in the source document cause the parser to call functions I
provide. This allowed me complete control on the translation while
sparing the intermediate data structure.
Converting HTML to text is a lossy transformation. This means that what goes out of this transformation is not exactly equivalent to what went in in the first place. Pictures, text layout, style and a few other information elements are lost. My needs required that I noted the existence of images as well as a reasonably accurate rendition of the page's text, but nothing else. Remember that the target device can only display 7-bit text, and this is within a very small and limited display. This piece of code sets up the parser to do what I need.
62: my $parser = HTML::Parser->new
63: (
64: api_version => 3,
65: default_h => [ "" ],
66: start_h => [ sub { print "[IMG ",
67: d($_[1]->{alt}) || $_[1]->{src},"]\n"
68: if $_[0] eq 'img';
69: }, "tagname, attr" ],
70: text_h => [ sub { print d(shift); }, "dtext" ],
71: ) or die "Cannot create HTML parser\n";
72:
73: $parser->ignore_elements(qw(script style));
74: $parser->strict_comment(1);
Starting on line 71, I set up the HTML::Parser object that will help me do this. First, I tell it I want to use the latest (as of this writing) interface style, which provides more flexibility than earlier interfaces. On line 65, I tell the object that by default, no parse events should do anything. There are other ways to say this, but the one shown is the most efficient.
Lines 66 through 69 define a handler for the start events. This handler will be called each time an opening tag such as <a> or <img> is recognized in the source being parsed. Handlers are specified as a reference to an array whose first element tells the parser what to do and its second element, tells the parser what information to pass to the code. In this example, I supply a function that for any img tag, will output a hopefully descriptive text composed with either the alt or the src attributes. I request this handler to be called with the name of the tag as the first argument and the list of attributes as further arguments, through the string "tagname, attr" found in line 69. The d() function will be explained a bit later, but it has to do with decoding its argument.
The text event will be triggered by anything inside tags in the input text. I've set up a simpler handler for this event that merely prints out whatever is recognized. I also request that HTML entities such as € or ñ be decoded for me through the string "dtext" on line 70. HTML entities are used to represent special characters outside the traditional ASCII range. In the interest of document accuracy, you should always use entities instead of directly placing 8-bit characters in the text.
Some syntactic elements are used to enclose information that is not important for this application, such as <style>...</style> and <script>...</script>. I ask the parser to ignore those elements with the call to ignore_elements() at line 73. I also request the parser to follow strict comment syntax through the call to strict_comment() on line 74.
Setting up the Unicode mappings
MIME defines various ways to encode binary data depending on the frequency of octets greater than 127. With relatively few high-bit octets, Quoted-Printable encoding is used. When many high-bit octets are present, Base-64 encoding is used instead. The reason is that Quoted-Printable is slightly more readable but very inefficient in space while Base-64 is completely unreadable by standard humans and adds much less overhead in the size of encoded files. Often, message headers such as the sender's name are encoded using Quoted-Printable when they contain characters such as a '帽'. These headers look like From: =?ISO-8859-1?Q?Luis_Mu=F1oz?= <some@body.org> and should be converted to From: Luis Mu帽oz <some@body.org>. In plain english, Quoted-Printable encoding is being used to make the extended ISO-8859-1 characters acceptable for any 7-bit transport such as email. Many contemporary mail transport agents can properly handle message bodies that contain high-bit octets but will choke on headers with binary data, in case you were wondering why all this fuzz.
Lines 92 through 102 define setup_decoder(), which can use
the headers contained in a MIME::Head object to setup a suitable decoder based on the
MIME::WordDecoder class. This will translate instances of
Quoted-Printable text, to its high-bit equivalent. Note that I
selected ISO-8859-1 as the default in case no proper character set can
be identified. This was a sensible choice for me, as ISO-8859-1
encloses spanish characters, which happen to be my native language.
92: sub setup_decoder
93: {
94: my $head = shift;
95: if ($head->get('Content-Type')
96: and $head->get('Content-Type') =~ m!charset="([^\"]+)"!)
97: {
98: $wd = supported MIME::WordDecoder uc $1;
99: }
100: $wd = supported MIME::WordDecoder "ISO-8859-1" unless $wd;
101: }
But this clever decoding is not enough. Getting at the original
high-bit characters is not enough. I must recode these high characters
into something usable by the 7-bit display device. So in line 76 I set
up a mapping based on Unicode::Map8. This module can convert 8-bit characters such
as ISO-8859-1 or ASCII into wider characters (Unicode) and then back into
our chosen representation, ASCII, which only defines 7-bit
characters. This means that any character that cannot be properly
represented, will be lost, which for our application is acceptable.
76: $map = Unicode::Map8->new('ASCII')
77: or die "Cannot create character map\n";
The decoding and character mapping is then brought together at line 90, where I define the d() function, that simply invokes the adequate MIME decoding method, transforms the resulting string into Unicode via the to16() method and then, transforms it back into ASCII using to8() to insure printable results in our device. Since I am allergic to warnings related to undef values, I make sure that decode() always get a defined string to work with.
90: sub d { $map->to8($map->to16($wd->decode(shift||''))); }
As you might notice if you try this code, the conversion is again lossy because there are characters that does not exist in ASCII. You can experiment with the addpair() method to Unicode::Map8 in order to add custom character transformations (i.e., '鈧? might be 'E'). Another way to achieve this, is through deriving a class from Unicode::Map8 and implementing the unmapped_to8 method to supply your own interpretation of the missing characters. Take a look at the module's documentation for more information.
Starting the decode process
With all the pieces in place, all that's left is to traverse the hierarchy of entities that MIME::Parser provides after parsing a message. I implemented a very simple recursive function decode_entities starting at line 103. This is a recursive function because recursion comes naturally as a way to handle trees such as those produced by MIME::Parser. At least to me.
103: sub decode_entities
104: {
105: my $ent = shift;
106:
107: if (my @parts = $ent->parts)
108: {
109: decode_entities($_) for @parts;
110: }
111: elsif (my $body = $ent->bodyhandle)
112: {
113: my $type = $ent->head->mime_type;
114:
115: setup_decoder($ent->head);
116:
117: if ($type eq 'text/plain')
118: { print d($body->as_string); }
119: elsif ($type eq 'text/html')
120: { $parser->parse($body->as_string); }
121: else
122: { print "[Unhandled part of type $type]"; }
123: }
124: }
The condition at line 107 asks if this part or entity contains other parts. If it does, it extracts them and invokes itself recursively to process each sub-part at line 109.
If this part is a leaf, its body is processed. Line 111 gets it as a
MIME::Body object. On line 155 I setup a decoder for this
part's encoding and based on the type of this part, taken at line 113,
the code on lines 117 to 122 call the proper handlers.
In order to fire the decoding process, I call decode_entities() with the result of the MIME decoding of the message on line 86. This will invoke the HTML parser when needed and in general, produce the output I look for in this example. After this processing is done, I make sure to wipe temporary files created by MIME::Parser on line 88. Note that if the message is not actually encoded with MIME, MIME::Parser will arrange for you to receive a single part of type text/plain that contains the whole message text, which is perfect for our application.
86: decode_entities($e);
87:
88: $mp->filer->purge;
And that's about it
After these less than 130 lines of code, I can easily fetch and decode a message, such as in the following example:
bash-2.05a$ ./mfetch -s pop.foo.bar -u myself \
-p very_secure_password -m 5
Date: Sat, 28 Dec 2002 20:14:37 -0400
From: root <root@foo.bar>
To: myself@foo.bar
Subject: This is the plain subject
This is a boring and plain message.
More complex MIME messages can also be decoded. Look at this example where I dissect a dreaded piece of junk mail, but don't worry. I used head to spare you pages and pages of worthless image links:
bash-2.05a$ ./mfetch -s pop.foo.bar -u myself \
-p very_secure_password -m 2 | head -20
Date: Sun, 22 Dec 2002 23:22:25 -0400
From: Luis Muoz <lem@foo.bar>
To: Myself <myself@foo.bar>
Subject: Fwd: Get $860 Free - Come, Play, Have Fun!
Begin forwarded message:
> From: Cosmic Offers <munged@migada.com.INVALID>;
> Date: Sun Dec 22, 2002 20:59:43 America/Caracas
> To: spam@victim.net
> Subject: Get $860 Free - Come, Play, Have Fun!
>
>
[IMG http://www.migada.com/email/Flc_600_550_liberty_mailer_.gif]
[IMG http://www.migada.com/email/Flc_600_550_liberty_mail-02.gif]
[IMG http://www.migada.com/email/Flc_600_550_liberty_mail-03.gif]
[IMG http://www.migada.com/email/Flc_600_550_liberty_mail-04.gif]
If you're curious, please download the complete script and play with it a bit. I hope this tutorial and its related script to be as helpful for you as it has been for me
