Emailing Content to Django with Webfaction's mail2script

Update: in recent versions of Django, its much better to use custom management commands

Webfaction, my shared-hosting provider of choice, has a nice feature that allows you to pipe emails to a script. This is useful for all sorts of things, but I decided to try it out by building a way for a friend to email photos to the front page of his website, which is built with Django.

To start, create a python script (I called mine EmailPhotoUpload.py). You can place it anywhere really, but I placed mine right in the django app that had the models I would be working with. Make sure the script starts with 

1
/usr/local/bin/python2.5

or whatever Python you're using with webfaction. Also, make sure it's executable:

1
chmod +x EmailPhotoUpload.py

When mail2script executes the script, it doesn't execute with knowledge about your Django project, so you'll need to set the DJANGO_SETTINGS_MODULE environment variable and place your project on your Python path if its not already. Something like this works:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
PROJECT_ROOT = '/home/webfactionuser/webapps/django/yourproject/'
DJANGO_SETTINGS_MODULE = "settings"

def setup_django():
    sys.path.append(PROJECT_ROOT)
    keys = os.environ.keys()
    from re import search
    for key in keys:
        if not search("DJANGO_SETTINGS_MODULE", key):
        os.environ["DJANGO_SETTINGS_MODULE"] = DJANGO_SETTINGS_MODULE     

setup_django()

Fix the project root variable to point at your project.

Once you've done that, you can import the models you want to manipulate. I have a model called 'Photo' in the 'photo' app. So I import it like so:

1
from photo.models import Photo

The email is accessible at sys.stdin, which is a file-like object. Python's email module has a method that takes a file and makes a Message object, so this works out nicely:

1
msg = email.message_from_file(sys.stdin)

Now you can refer to the email module documentation to get your data. I started by writing a few helper functions to get the data I want out of email.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
def get_text(msg):
    "Looks for a part of the message with a text/plain mime type and returns it"
    text = ""
    if msg.is_multipart():
        for part in msg.get_payload():
            if part.get_content_type() == 'text/plain':
                text = part.get_payload()
    else:
        text = msg.get_payload()
    return text

def get_jpegs(msg):
    "Looks for any part of the message with image/jpeg mime type and returns a list."
    jpegs = []
    if msg.is_multipart():
        for part in msg.get_payload():
            if part.get_content_type() == 'image/jpeg':
                data = part.get_payload()
                tempjpeg = tempfile.NamedTemporaryFile('w+b', -1)
                tempjpeg.write(base64.b64decode(data))
                jpegs.append(tempjpeg)
    return jpegs

A few things to note: I'm using Python's tempfile module to hold the jpegs until I can save them in my model. Django has its own NamedTemporaryFile class, but it's for Windows compatibility and since this is a linux environment, we can use the standard library's. All we need to do is take the payload from the message part, base64 decode it and write it into a file. If you expect large files, or many, you'll eat up memory here and you'll have to do something more sophisticated.

The Photo model I'm working with looks something like this:

1
2
3
4
5
6
class Photo(models.Model):
    title = models.CharField("Photo Title", max_length=255)
    original = models.ImageField("Image", upload_to="img/uploaded/original")
    resized = models.ImageField(upload_to="img/uploaded/resized")
    credit = models.CharField("Credit", max_length=255, blank=True)
    caption = models.TextField("Description", blank=True)

I'm going to use the From field to populate the photo credit, the text portion of the email to populate the caption, the subject line of the email to populate the photo's title. 

1
2
3
4
5
sender = re.sub("\s<(.+)>(\s*)$", "", get_header(msg, 'From'))
subject = get_header(msg, 'Subject')
text = get_text(msg) 
jpegs = get_jpegs(msg)
basename = re.sub(r"[\\/ \(\):?;,]", "_", subject)

For any of the header information, just call get_header, pass in the name of the email header and then perform any clean up on the result. For instance, I'm stripping the email address portion from the "From" header. Now just iterate through your list of jpegs and make a Photo object for each:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
for index, jpeg in enumerate(jpegs):
    photo = Photo()
    photo.title = subject
    photo.caption = text
    photo.credit = sender
    filename = "".join([basename, str(index), '.jpg'])
    photo.original.save(filename, File(jpeg))
    photo.promote = True
    photo.save()
    jpeg.close()

Remember, jpegs is a list of temporary files; be sure to close them so the os cleans them up.

Now log in to webfaction, create an email address and put the absolute path to this script in the target area.

Please note, this isn't very secure. Anyone with the address can post content, and if you're not cleaning it, they may be able to do worse--never trust user input.

I plan on releasing a cleaner, more reusale classed-based version with some added features and exception handling, but for now I'll post the full text of what I describe above so you can see everything in context. Remember, you can test your script by downloading the raw email text and redirecting it to your script at the command line like this:

1
./EmailPhotoUpload.py < test-msg.txt

Good luck! 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
:::/usr/bin/env python

import os
import re
import sys
import email
import base64
import tempfile
from django.core.files import File

PROJECT_ROOT = '/home/justin/Documents/sites/workshop/workshop/'
DJANGO_SETTINGS_MODULE = "settings" 

def setup_django():
    sys.path.append(PROJECT_ROOT)
    keys = os.environ.keys()
    from re import search
    for key in keys:
        if not search("DJANGO_SETTINGS_MODULE", key):
            os.environ["DJANGO_SETTINGS_MODULE"] = DJANGO_SETTINGS_MODULE     

def get_text(msg):
    text = ""
    if msg.is_multipart():
        for part in msg.get_payload():
            if part.get_content_type() == 'text/plain':
                text = part.get_payload()
    return text

def get_jpegs(msg):
    jpegs = []
    if msg.is_multipart():
        for part in msg.get_payload():
            if part.get_content_type() == 'image/jpeg':
                data = part.get_payload()
                tempjpeg = tempfile.NamedTemporaryFile('w+b', -1)
                tempjpeg.write(base64.b64decode(data))
                jpegs.append(tempjpeg)
    return jpegs

# Start
setup_django()
from photo.models import Photo
msg = email.message_from_file(sys.stdin)

address = msg.get('Return-Path', '').strip("<>")
sender = re.sub("\s<(.+)>(\s*)$", "", msg.get('From', 'Unknown'))
subject = msg.get('Subject', 'Untitled')
text = get_text(msg) 
jpegs = get_jpegs(msg)
basename = re.sub(r"[\\/ \(\):?;,]", "_", subject)

for index, jpeg in enumerate(jpegs):
    photo = Photo()
    photo.title = subject
    photo.caption = text
    photo.credit = sender
    filename = "".join([basename, str(index), '.jpg'])
    photo.original.save(filename, File(jpeg))
    photo.promote = True
    photo.save()

for jpeg in jpegs:
    jpeg.close()

2009-12-21