Wikipedia has a list of the one hundred most frequent words in English. Do yourself a favor and read them all aloud in order.

• last modified: 08/28/2011

Jason Kottke recently linked to a post on the Boston Globe’s website showing what they call a “webpage-like” news wall from the first-half of the 20th C. Of course, the medium is much older. As Will Durant writes in Ceasar and Christ,

To bring the Senate under public scrutiny, [Julius Ceasar] established the first newspaper by having clerks make a record of Senatorial and other public proceedings and news, and post these Acta Diurna,or "Daily Doings," on the walls of the forums. From these walls the reports were copied and sent by private messengers to all parts of the Empire.

Not only did Ceasar have a webpage, he had an PubSub feed.

It’s interesting that Mark Zuckerberg, who cites his love of classical literature (even though his employees have misinterpreted some of it.) and was called "Our New Ceasar" by Vanity Fair runs a company with a central feature called the “Wall.”

Interesting, but unsurprising. No matter where you look, you can find history repeating (or rhyming.) I used to work for WebMD, who has a set of forums where users with various ailments can exchange information, post about their symptoms and how they’ve dealt with them. It seems to have the potential to be an efficient--and forgetting the other online medical boards before it--revolutionary way for people to exchange information; replacing or modifying the role of a medical professional. But then one might remember when Herodotus writes of the Babylonians:

The following custom seems to me the wisest instituition.... They have no physicians, but when a man is ill, they lay him in the public square, and the passers-by come up to him, and if they have ever had his disease themselves or have known any one who has suffered from it, they give him advice, recommending him to do whatever they found good in their own case, or in the case known to them; an no one is allowed to pass the sick man in silence without asking him what his ailment is.

• last modified: 05/21/2011

These instructions will help you walk through setting up Django in one of the preferred deployments: We'll use Nginx as a proxy to pass off requests to Django. One of the benefits here is that you can set up other sites on your server running on various technologies and nginx will just pass the requests to the correct handler. This way, you can have some static sites, maybe a couchdb instance, a rails app, a Django app etc, all on the same server, which should be fine for low traffic sites. Django will run on Apache under modwsgi and live inside a virtualenv so you can install libraries or upgrade your Python wihout messing with your system Python.

I'm using a Linode instance with Ubuntu 10, but the instructions should be helpful for any Linux. One last thing, I'm not a systems guy, so if you see something wrong here, please leave a comment and remember you're responsible for understanding what these commands and configurations are doing.

Before beginning, I'd just go read through the docs on deploying Django on mod_wsgi and running mod_wsgi on virtualenv. Both are pretty good. If you have some experience configuring a server, they're all you need. If you don't, having them in your head will be handy as you work through the following and they'll be crucial during trouble shooting. Either way, let's get started..

First, install virtualenv with easy_install. Create a directory in /usr/local/ called pythonenv and change into it, so you're in:

1

/usr/local/pythonenv

Then, create a clear virtualenv by running the command:

1

virtualenv --no-site-packages BASELINE

Create a user and under that create a directory sites/ and cd into it, so your pwd is something like this:

1

/home/someuser/sites/

In this folder, create a new virtualenv; this is the virtualenv that you'll install Django into.

1

 virtualenv --no-site-packages example.com.env

Now if you ls example.com.env you should see:

1

bin include site

Create two more directories, site and src.

1

mkdir example.com.env/site; mkdir example.com.env/src;

Checkout Django into the src directory

1

svn co http://code.djangoproject.com/svn/django/trunk/ example.com.env/src/django-trunk

Then add django to the environment's path. You can do this with a symbolic link, or a .pth file. Your choice. Here's the .pth file creation:

1
2

echo $PWD/example.com.env/src/django-trunk >
example.com.env/lib/python2.6/site-packages/django.pth

(Note, check your python version above). If you want, you can set up django-admin on your shell path. Complete details on this can be found on the Django site.

At this point you should be able to activate your virtualenv and import django.

1
2
3

source example.com.env/bin/activate
python
>>> import django

Now, create or move your Django application code into the site directory you created a few steps earlier. You'll need to create a wsgi file if you don't have one already and add it somewhere in this folder. For help creating the WSGI file, follow the docs on djangoproject.com, but skip the Apache configuration part at the beginning. Just make the wsgi file and add it to a directory in your Django project. You'll also have to add the following to the top:

1
2

import site
site.addsitedir('/home/someuser/sites/example.com.env/lib/python2.6/site-packages')

Change the directory above to match the site-packages directory in your new virtualenv.

I personally set up two versions of my Django settings under a common "configs" folder with separate subdirectories for wsgi and settings files which correspond to my development and production environments. This is a technique I think I got from Simon Willison, but you can keep it anywhere in your project. Most people create an "apache" directory and stick it there. Just make sure all the configurations coming up point to it.

Now follow the instructions to set up Apache2. but don't follow the virtual host configuration. When you get there, instead install modwsgi:

1
2
3

apt-get update
apt-get upgrade
apt-get install python-setuptools libapache2-mod-wsgi

Then configure your virtual host with the following instructions. First, put the following in your ports.conf file.

1

NameVirtualHost *:8080

Or whatever port you want to run Django on. Remember Nginx will be listening on 80 and passing off requests to Apache on some other port.

Then go to your apache conf folder (probably /etc/apache2/) and inside the site-available folder, make a file called example.com or whatever your domain is. Inside, put something like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

<VirtualHost *:8080>
    ServerAdmin webmaster@example.com
    ServerName  www.example.com
    ServerAlias example.com

    <Directory /home/someuser/example.com/site/configs/prod/ >
            Order deny,allow
            Allow from all
    </Directory>

    WSGIScriptAlias / /home/someuser/sites/example.com.env/site/configs/prod/apache.wsgi
    WSGIDaemonProcess  example.com user=www-data group=www-data threads=25
    WSGIProcessGroup example.com

    ErrorLog  /var/www/example.com/logs/error.log
    CustomLog /var/www/example.com/logs/access.log combined
</VirtualHost>

The important thing in this configuration is that the Directory directive points to the folder you put your Django wsgi file and the WSGIScriptAlias points directly to it. While you're at it, you can place your error logs whereever you want as well. The above might not be a great place for them. Make sure the directories exist.

Ok, now enable this site by running a2ensite

1

sudo a2ensite example.com

This creates symlink from your available sites to your enabled sites.

Also, edit your main apache2.conf file in /etc/apache2/ and at the bottom of the file add:

1

WSGIPythonHome /usr/local/pythonenv/BASELINE

Now you've installed a virtualenv with Django and Apache with mod_wsgi, so we're getting close.

If you haven't already, you’ll need to create a DNS entry in the Linode manager for your domain and point your domain to the Linode nameservers.

Edit your /etc/hosts file so it has a line like this.

1

127.0.0.1        localhost.localdomain        localhost

You can add your domain to this too. Add it as 127.0.0.1 or your external IP name.

Make up some name for your server and make it your machine's hostname. For our purposes I'm going to use 'paintbrush'.

1
2

echo "paintbrush" > /etc/hostname
hostname -F /etc/hostname

Now install Nginx using the linode instructions

Then edit your nginx config file, which is probably at:

1

/opt/nginx/conf/nginx.conf

Nginx has a straightforward configuration syntax you should be able to figure out just by reading the examples. Follow Linode's guide. But the following is what you want...

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

server {
    listen       80;
    server_name  *.example.com;
    access_log   logs/example.com.access.log;
    error_log   logs/example.com.error.log;

    location / {
        proxy_pass http://example.com:8080;
        include /opt/nginx/conf/proxy.conf;
    }
}

Finally, you can add location configurations to the above to server your static media. Something like this should work, but again, refer to the Nginx documentation.

1
2
3

location /media/  {
   root    /home/someuser/example.com.env/site/static/
}

Now, retstart Apache and Nginx

1
2

sudo /etc/init.d/apache2 restart
sudo /etc/init.d/nginx restart

Fire up a browser and go to your domain. Guess what? it didn't work. This is a complicated process and you're bound to mess something up. The key here is that you need to check your logs. Tail the nginx and apache error logs that you configured above and fix each problem until you have this working.

If you see import errors, that means your Python path isn't correct. Make sure you really understand it, fix it and don't forget to restart Apache when you're testing fixes.

If you have any questions, feel free to post below, but I can't promise I'll be able to help.

• last modified: 05/11/2011

I've been looking at the 3D transforms that the latest webkits have included. Here's a quick proof of concept for something I'm working on. Should work in newish releases of Chrome or Safari.

• last modified: 01/23/2011

Sorry about the poor camera phone quality, but I found this video of a project I'm working on sort of mesmerizing. I'm testing a customizable javascript slideshow for performance. This is Javascript code running in Chrome. There are 64 individual 'frames' polling a service for the next picture to show.

The fact that you can do this in the browser with a web architecture underneath is extraordinary.

• last modified: 12/29/2010

I recently rebuilt this site to scratch some itches I had. As a result there's some missing content and some dead links. I'll put everything back together again with a bit of time. If there's something you're really looking for, please contact me.

• last modified: 11/20/2010

Lately we've been seeing a lot of new ways to provide authentication to your web service. There's OAuth, and OpenID and Facebook connect, but frankly they're all the same boring crap. Just for fun, I thought I would write something a little more interesting.

Prohibition is a "Secret Knock" authentication system for web pages. It allows you to make a DOM Element that a user can click on to record and compare sequences of knocks.

It's completely insecure, but some day one of these trendy speakeasy bars is going to call you up and ask you to build them a webpage, and you're going to hide the directions page behind this thing and they are going to love it. And on that day, you owe me a 12 dollar drink.

Take a look at the crummy looking demo page or fork the code from github.com.

• last modified: 12/13/2010

So I never forget: If you need to use multiprocessing within an app bundled with Py2Exe, you need to use multiprocessing.freeze_support().

• last modified: 11/19/2010

I was looking for a little command line issue tracker to use on small projects. I found Tracker, but received an error when I tried to install it. I work mostly in Python when I can, so I thought I'd hack one together quickly and remove the Ruby dependency for myself.

You can download it from the bitbucket repository. Unpack, cd into the directory and run: sudo python setup.py install

Check out the README for examples

• last modified: 11/19/2010

Recently I was thinking about how much information gets lost between the moment someone begins typing a comment on a webpage and the moment they click "submit." They may have spent a long time composing it, deleting words and phrases as they go only to replace them with better ones. It sounded like an interesting project to somehow capture this information.

The simplest way I could think to index this activity was to provide a text area that counts keystrokes as well as characters.

While obviously not a perfect indicator, a high keystroke-to-character ratio would indicate that the user has spent a lot of time editing. This could indicate that the user is a bad typist, that the user was being careful with her words, that the user didn't have the ideas clear in his head, that new information came to her as she typed....who knows.

I chose to visualize this high ratio by coloring the background of the text area, although a numeric score or icon could work just as easily. By giving high keystroke-to-character -count ratios a red color, I'm implying a warning: this message has been heavily edited. But this could be reversed. Perhaps the writer sees value in editing and time spent composing.

I have two more variations on this in the works, but for now, here's the first proof of concept.

• last modified: 01/02/2011

Update: in recent versions of Django, its much better to use custom management commands

Webfaction, my shared-hosting provider of choice, has a nice feature that allows you to pipe emails to a script. This is useful for all sorts of things, but I decided to try it out by building a way for a friend to email photos to the front page of his website, which is built with Django.

To start, create a python script (I called mine EmailPhotoUpload.py). You can place it anywhere really, but I placed mine right in the django app that had the models I would be working with. Make sure the script starts with #!/usr/local/bin/python2.5, or whatever Python you're using with webfaction. Also, make sure it's executable:

1

chmod +x EmailPhotoUpload.py

When mail2script executes the script, it doesn't execute with knowledge about your Django project, so you'll need to set the DJANGO_SETTINGS_MODULE environment variable and place your project on your Python path if its not already. Something like this works:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

PROJECT_ROOT = '/home/webfactionuser/webapps/django/yourproject/'
DJANGO_SETTINGS_MODULE = "settings"

def setup_django():
    sys.path.append(PROJECT_ROOT)
    keys = os.environ.keys()
    from re import search
    for key in keys:
        if not search("DJANGO_SETTINGS_MODULE", key):
        os.environ["DJANGO_SETTINGS_MODULE"] = DJANGO_SETTINGS_MODULE

setup_django()

Fix the project root variable to point at your project.

Once you've done that, you can import the models you want to manipulate. I have a model called 'Photo' in the 'photo' app. So I import it like so:

1

from photo.models import Photo

The email is accessible at sys.stdin, which is a file-like object. Python's email module has a method that takes a file and makes a Message object, so this works out nicely:

1

msg = email.message_from_file(sys.stdin)

Now you can refer to the email module documentation to get your data. I started by writing a few helper functions to get the data I want out of email.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

def get_text(msg):
    "Looks for a part of the message with a text/plain mime type and returns it"
    text = ""
    if msg.is_multipart():
        for part in msg.get_payload():
            if part.get_content_type() == 'text/plain':
                text = part.get_payload()
    else:
        text = msg.get_payload()
    return text

def get_jpegs(msg):
    "Looks for any part of the message with image/jpeg mime type and returns a list."
    jpegs = []
    if msg.is_multipart():
        for part in msg.get_payload():
            if part.get_content_type() == 'image/jpeg':
                data = part.get_payload()
                tempjpeg = tempfile.NamedTemporaryFile('w+b', -1)
                tempjpeg.write(base64.b64decode(data))
                jpegs.append(tempjpeg)
    return jpegs

A few things to note: I'm using Python's tempfile module to hold the jpegs until I can save them in my model. Django has its own NamedTemporaryFile class, but it's for Windows compatibility and since this is a linux environment, we can use the standard library's. All we need to do is take the payload from the message part, base64 decode it and write it into a file. If you expect large files, or many, you'll eat up memory here and you'll have to do something more sophisticated.

The Photo model I'm working with looks something like this:

1
2
3
4
5
6

class Photo(models.Model):
    title = models.CharField("Photo Title", max_length=255)
    original = models.ImageField("Image", upload_to="img/uploaded/original")
    resized = models.ImageField(upload_to="img/uploaded/resized")
    credit = models.CharField("Credit", max_length=255, blank=True)
    caption = models.TextField("Description", blank=True)

I'm going to use the From field to populate the photo credit, the text portion of the email to populate the caption, the subject line of the email to populate the photo's title.

1
2
3
4
5

sender = re.sub("\s<(.+)>(\s*)$", "", get_header(msg, 'From'))
subject = get_header(msg, 'Subject')
text = get_text(msg) 
jpegs = get_jpegs(msg)
basename = re.sub(r"[\\/ \(\):?;,]", "_", subject)

For any of the header information, just call get_header, pass in the name of the email header and then perform any clean up on the result. For instance, I'm stripping the email address portion from the "From" header. Now just iterate through your list of jpegs and make a Photo object for each:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

for index, jpeg in enumerate(jpegs):
    photo = Photo()
    photo.title = subject
    photo.caption = text
    photo.credit = sender
    filename = "".join([basename, str(index), '.jpg'])
    photo.original.save(filename, File(jpeg))
    photo.promote = True
    photo.save()
    jpeg.close()

Remember, jpegs is a list of temporary files; be sure to close them so the os cleans them up.

Now log in to webfaction, create an email address and put the absolute path to this script in the target area.

Please note, this isn't very secure. Anyone with the address can post content, and if you're not cleaning it, they may be able to do worse--never trust user input.

I plan on releasing a cleaner, more reusale classed-based version with some added features and exception handling, but for now I'll post the full text of what I describe above so you can see everything in context. Remember, you can test your script by downloading the raw email text and redirecting it to your script at the command line like this:

1

./EmailPhotoUpload.py < test-msg.txt

Good luck!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

#!/usr/bin/env python

import os
import re
import sys
import email
import base64
import tempfile
from django.core.files import File

PROJECT_ROOT = '/home/justin/Documents/sites/workshop/workshop/'
DJANGO_SETTINGS_MODULE = "settings"

def setup_django():
    sys.path.append(PROJECT_ROOT)
    keys = os.environ.keys()
    from re import search
    for key in keys:
        if not search("DJANGO_SETTINGS_MODULE", key):
            os.environ["DJANGO_SETTINGS_MODULE"] = DJANGO_SETTINGS_MODULE

def get_text(msg):
    text = ""
    if msg.is_multipart():
        for part in msg.get_payload():
            if part.get_content_type() == 'text/plain':
                text = part.get_payload()
    return text

def get_jpegs(msg):
    jpegs = []
    if msg.is_multipart():
        for part in msg.get_payload():
            if part.get_content_type() == 'image/jpeg':
                data = part.get_payload()
                tempjpeg = tempfile.NamedTemporaryFile('w+b', -1)
                tempjpeg.write(base64.b64decode(data))
                jpegs.append(tempjpeg)
    return jpegs

# Start
setup_django()
from photo.models import Photo
msg = email.message_from_file(sys.stdin)

address = msg.get('Return-Path', '').strip("<>")
sender = re.sub("\s<(.+)>(\s*)$", "", msg.get('From', 'Unknown'))
subject = msg.get('Subject', 'Untitled')
text = get_text(msg) 
jpegs = get_jpegs(msg)
basename = re.sub(r"[\\/ \(\):?;,]", "_", subject)

for index, jpeg in enumerate(jpegs):
    photo = Photo()
    photo.title = subject
    photo.caption = text
    photo.credit = sender
    filename = "".join([basename, str(index), '.jpg'])
    photo.original.save(filename, File(jpeg))
    photo.promote = True
    photo.save()

for jpeg in jpegs:
    jpeg.close()

• last modified: 02/24/2011

Flex has handy default event handlers that handle basic drag and drop between DataGrids. On a recent project, I wanted to enable drag and drop for reordering within a single grid, but I wanted to restrict the user from dragging from one grid into another.

Flex provides a DragManager class to provide full control over these events, and you might be tempted to reach for it, but there’s an easier method. Basically, you enable drag and drop as you normally would:

1
2
3

dragEnabled="true"
dragMoveEnabled="true"
dropEnabled="true"

but also specify a custom drop handler where you check the initiator of event with the event target. Here’s the full source that shows the basic idea. You can override other events, like dragEnter, to provide full functionality.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72

<?xml version="1.0" encoding="utf-8"?>
<mx:Panel xmlns:mx="http://www.adobe.com/2006/mxml"
    layout="vertical" width="100%"
    paddingTop="10" paddingRight="10" paddingBottom="10" paddingLeft="10">
    <mx:DataGrid 
        id="datagrid1"
        dataProvider="{dataProvider1}"
        width="50%"
        dragEnabled="true"
        dragMoveEnabled="true"
        dropEnabled="true"
        dragDrop="dragDropHandler(event);"
        sortableColumns="false"
        draggableColumns="false">
       <mx:columns>
           <mx:DataGridColumn
               headerText="Data"
               dataField="data"
               draggable="false" />
               <mx:DataGridColumn
                   headerText="More Data"
                   dataField="moredata"
                   draggable="false" />
       </mx:columns>
   </mx:DataGrid>
   <mx:DataGrid 
    id="datagrid2"
    dataProvider="{dataProvider2}"      
    width="50%"
    dragEnabled="true"
    dragMoveEnabled="true"
    dropEnabled="true"
    dragDrop="dragDropHandler(event);"
    sortableColumns="false"
    draggableColumns="false">
        <mx:columns>
           <mx:DataGridColumn
               headerText="Data"
               dataField="data"/>
           <mx:DataGridColumn
               headerText="More Data"
               dataField="moredata" />
        </mx:columns>
   </mx:DataGrid>
    <mx:Script>
               <![CDATA[
                       import mx.collections.ArrayCollection;                       
                       import mx.events.DragEvent;

            private var dataProvider1:Array = [
                {'data': 'grid 1 data 1', 'moredata': 'grid 1 more data 1'},
                {'data': 'grid 1 data 2', 'moredata': 'grid 1 more data 2'},
                {'data': 'grid 1 data 3', 'moredata': 'grid 1 more data 3'}
            ]

            private var dataProvider2:Array = [
                {'data': 'grid 2 data 1', 'moredata': 'grid 2 more data 1'},
                {'data': 'grid 2 data 2', 'moredata': 'grid 2 more data 2'},
                {'data': 'grid 2 data 3', 'moredata': 'grid 2 more data 3'}
            ]

            private function dragDropHandler(evt:DragEvent):void{
                       if(evt.dragInitiator != evt.currentTarget){
                                var dragObj:Array= evt.dragSource.dataForFormat("items") as Array;
                                ((evt.dragInitiator as DataGrid).dataProvider as ArrayCollection).addItem(dragObj[0]);
                               evt.preventDefault();
                       }
               }

               ]]>
       </mx:Script>
</mx:Panel>

• last modified: 11/19/2010

Lately I’ve been doing some Wordpress plugin development for a project. I found myself needing a way to get all the tags associated with a particular category. Wordpress doesn’t offer this as a template tag, but its easy enough to query the database directly. I didn't find this anywhere in the Wordpress codex or in a search, so I thought I'd post it here:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

<?php

$sql = "SELECT name, slug
        FROM wp_terms
        JOIN wp_term_relationships
        ON wp_terms.term_id = wp_term_relationships.term_taxonomy_id
        JOIN wp_term_taxonomy
        ON wp_terms.term_id = wp_term_taxonomy.term_id
        WHERE wp_term_relationships.object_id IN (
        SELECT object_id FROM wp_term_relationships WHERE
        wp_term_relationships.term_taxonomy_id = %d)
        AND wp_term_taxonomy.taxonomy = 'post_tag'";

// $cat_number is the number of the category in question.

$sql = $wpdb->prepare($sql, $cat_number); 
$results = $wpdb->get_results($sql);

?>

Now you can iterate through the results and display them however you want. Note that I'm selecting the name and slug so I can print links to the tag page. Alter the query as you see fit.

• last modified: 11/19/2010

The standard HTML list box is a much maligned user interface element. It requires use of both mouse and keyboard to operate, and with any more than a few elements it requires the user to scroll. It's also fairly ugly and difficult to style. I've seen a few nice javascript widgets that improve them, but often they still forced the user to scroll, or simply created lists that pushed page contents down.

I decided to write my own, mostly as an exercise, but also to fit a few requirements I thought were essential to an easy-to-use multi-select element: it should occupy a fixed vertical space, the user should never have to scroll, the user should be able to quickly see the elements she has selected and the elements she has yet to select, and finally, it should allow the user to filter down the elements that are important. Also, I wanted something that worked with MooTools, a great javascript framework.

MTMultiselect is my answer. It's a easy-to-use, paginated, filterable multi-select widget built on MooTools. Take a look at the demo page or download the code from bitbucket.org.

• last modified: 01/03/2011