Installing Python 3 and Django on Dreamhost

I’m working on a side project for a friend that has zero budget and since I’m already paying for Dreamhost (my favorite and only choice for shared hosting) and totally underusing it, I figured I’d use this project as my opportunity to figure out how to install Python 3 and Django since A) Dreamhost lets you do install things even on their shared platform (one of the reasons I love them!), and B) the default Python on Dreamhost is Python 2.7.3.

1. Install Python 3.5.1

This part’s really simple and Dreamhost even documents it nicely on their knowledge base, but for the sake of one-stop shopping I’ll include the steps I performed here. Note that I’m leaving the default Python 2.7.3 in place since I’m not sure what havoc it would wreak to blow it away, so Python 3.5.1 will be installed in such a way that it’ll be run as python3. (For the curious, you can also read more about Python at Dreamhost in general.)

Before going any further, if you haven’t enabled SSH access for your Dreamhost user you need to do that first since if you can’t log in you can’t install anything.

  1. Log into the Dreamhost web panel
  2. Click on “Users” and then “Manage Users”
  3. Find the relevant user and click the edit button
  4. Under “User Type” choose “Shell user,” and I’d recommend also choosing /bin/bash as the shell type
  5. Scroll to the bottom of the page and click “Save Changes”

Next, we’ll install Python 3.5.1.

  1. Go to https://www.python.org/downloads/release/python-351/ and get the URL for the gzipped source tarball.
  2. SSH into your Dreamhost server
  3. In your user’s home directory, wget the tgz file using the URL you grabbed in step 1:
    wget https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz
  4. Extract the Python source:
    tar xvf Python-3.5.1.tgz
  5. cd into the extracted source directory:
    cd Python-3.5.1
  6. Configure Python 3.5.1 with a prefix flag so as not to stomp on anything that’s already installed:
    ./configure –prefix=$HOME/opt/python-3.5.1
  7. Run make and make install:
    make && make install
  8. Add the new Python installation to your path:
    export PATH=$HOME/opt/python-3.5.1/bin:$PATH
  9. Update your bash profile:
    . ~/.bash_profile
  10. Verify that everything’s working (this should output Python 3.5.1):
    python3 -V
If that all looks good at this point, we’re ready to proceed and create a virtualenv in which we’ll install Django.

2. Create a virtualenv

Python 3 ships with pyvenv, but since it works a bit differently than virtualenv which is what I’m used to using, I decided to install virtualenv and use that. Feel free to use pyenv here if that floats your boat.
  1. Use pip3 to install virtualenv:
    pip3 install virtualenv
  2. Create and/or cd into the directory in which you want to put your virtualenv. I tend to put mine in ~/.virtualenvs, so I did this:
    mkdir ~/.virtualenvs
    cd ~/.virtualenvs
  3. Create a virtualenv to use for our Django project (since we installed virtualenv with pip3 the default Python environment will be Python 3.5.1):
    virtualenv my_django_project
  4. Activate your virtualenv:
    source my_django_project/bin/activate
  5. Confirm everything looks correct at this point (this should output /path/to/virtualenv/bin/python3):
    which python3
If we’re still on track we can proceed to install Django.

3. Install Django (and your other requirements)

At this point since we have Python 3.5.1 installed and have successfully created a virtualenv, we can do the usual installation of requirements using pip3, which typically will come from a requirements.txt file. For the sake of illustration we’ll go through installing Django into our virtualenv and making sure it works, but pip3 install -r requirements.txt will be the more typical way of installing all the packages you need.
  1. With your virtualenv activated, install Django:
    pip3 install django
  2. Verify everything is working by starting a Python interpreter:
    python3
  3. Import Django to make sure it’s in your virtualenv (this shouldn’t throw any errors):
    >>> import django
Now that we have verified that our virtualenv is working and sane, we can proceed to the rest of the configuration to get a Django project running successfully in this environment on Dreamhost.

4. Enable Passenger on Your Web Site

Rather than the gunicorn or some of the other WSGI servers you may be familiar with, on Dreamhost you can use either Passenger or FastCGI as the application server for Django applications.

For this post I’ll be using Passenger, a tool with which I was not familiar until I saw information about it on Dreamhost. Passenger is apparently more common in Ruby on Rails setups, but it serves the same purpose and falls in the same place in the stack as any of the other WSGI tools with which you may have experience. You can read more about Dreamhost’s use of Passenger here and here.

Some Notes About Passenger

Worthy to note about using Passenger on Dreamhost is that once Passenger is enabled, all requests that come into your domain will be handed off to Passenger by Apache. Passenger will first look see if there are any files matching the request in your domain’s /public directory (and note that the files must be in the /public directory or subdirectories under /public specifically), and if it can’t find an appropriate file, Passenger then tries to handle the request itself.

To provide a couple of concrete examples, let’s assume you have a file called hello.html in your domain’s /public directory, and a request comes in for http://yourdomain.com/hello.html. Here’s what happens next:

  1. Apache hands the request off to Passenger immediately. This is important to note since Apache is out of the picture entirely other than serving as a proxy to Passenger.
  2. Passenger looks for a file called hello.html in your domain’s /public directory. Since it finds one, it returns that page to the requester.
Now let’s assume a request comes in for http://yourdomain.com/some/django/url. Here’s what happens in that case:
  1. Apache hands the request off to Passenger.
  2. Passenger looks for an appropriate file in /public. This time it doesn’t find one.
  3. Passenger then attempts to handle the request itself, and since later we’ll be configuring Passenger to serve up our Django application using WSGI, at this point Django would be processing the request.

Enable Passenger

In order to run a Django application on Dreamhost we’ll need to enable Passenger on the domain on which we’re going to run our Django application.

  1. Log into the Dreamhost web panel
  2. Click “Domains” and then click “Manage Domains”
  3. Locate the domain on which you’ll be hosting your Django application and click “Edit”
  4. Scroll down and under “Web Options” you’ll see a checkbox labeled “Passenger (Ruby/NodeJS/Python apps only).” Check that box.
  5. Scroll to the bottom of the page and click “Change Settings.”
That’s it for enabling Passenger. Next we’ll throw our Django application at Dreamhost and see what sticks.

5. Create a Django App

For the purposes of this post we’ll be creating and configuring a Django application directly on the Dreamhost server. Obviously if you have an application that’s already written that you want to run on Dreamhost you’d be uploading your application’s files as opposed to doing things from scratch on the server, but this should illustrate some of the nuances of what you need to do with your own Django application to get it running on Dreamhost with Passenger.

  1. ssh into the Dreamhost server
  2. Activate the virtualenv you created above:
    source ~/.virtualenvs/my_django_project/bin/activate
  3. cd into the root directory of the web site on which you’ll be running your Django application. Note that this is not the aforementioned /public directory under your web site’s directory, but the root of the web site itself.
  4. Start a new Django project using django-admin.py:
    python3 django-admin.py startproject my_django_project
  5. Do a quick gut check by firing up the Django development server:
    cd my_django_project
    python3 manage.py runserver
At this point you should see the Django development server start up without any errors. We’re nearly there! The last step is to configure this newly created Django application to work with Passenger. But before we do that, a quick note about static files.

Serving Static Files and the Static Root Setting

It wouldn’t be Django if we didn’t have to stop and say a word about serving static files. Luckily this isn’t anything out of the ordinary, but given how Passenger works you do need to be aware of how to set your STATIC_ROOT to make sure everything comes together smoothly.

  1. Create a static directory under your domain’s public directory, i.e. you’ll wind up with ~/yourdomain.com/public/static
  2. Go into your Django project’s settings file and add a STATIC_ROOT setting:
    STATIC_ROOT = os.path.dirname(BASE_DIR) + ‘/public/static/’
With that you should be all set to run collectstatic and have everything work properly. Now let’s move on to configuring Passenger.

6. Configure Passenger

In order to have Passenger load up our Django application, we need to create a passenger_wsgi.py file in the root of our domain. As with all WSGI files this is what tells the application server, in this case Passenger, how to load up the application so it can handle requests.

In the root of your domain, create a passenger_wsgi.py file with the following contents:

With that file in place, you should be able to hit http://yourdomain.com and see the default “Congratulations” Django page. Congratulations indeed! You now have a Django application running on Python 3.5.1 on Dreamhost.

A common error you might run into is something like “no module named foo” or “no module named foo.settings.” If you get something along these lines it means your paths are incorrect, so make sure that your Django project is in a directory directly inside your domain’s root, not in the /public directory. If that isn’t the issue, double-check all the paths in the passenger_wsgi.py file.

Additional Passenger Considerations

It’s worth noting that if you change your passenger_wsgi.py file you need to run pkill python to reload the Django application since Passenger loads your application into memory (which is one of its advertised advantages since it makes things pretty zippy).

I also saw note of having to run touch tmp/restart.txt but in my experience I didn’t have to do that, so I’m not sure if that’s more of a Rails thing or if I simply didn’t yet run into situations in which that was necessary. I’m noting it here in the event pkill python doesn’t do the trick.

Final Notes

There you have it. You now have the latest Python and Django running on Dreamhost. At this point you can load your own Django project, set up a database (note that Dreamhost only supports MySQL, and even though I much prefer Postgres, MySQL works just fine), and do all the usual Django stuff.

Enjoy!

Python 2.7, Django, Apache, and Gunicorn on CentOS 6.5

I’m working on a little Django project for KPTZ Radio in Port Townsend and since this project has to talk to a serial relay board from a specific server that has other things running on it, I’ve been going through the process of installing Python 2.7 on CentOS 6.5, along with configuring Django, Apache, and Gunicorn.

Since I’m a lot more used to dealing with Nginx and Gunicorn on Ubuntu, getting this all up and running correctly took a lot more trial and error than I thought it would, but I finally got it figured out so figured I’d share since I found a lot of either incomplete or misleading information about this as I searched for solutions.

Installing Python 2.7

Your first question is probably why I’m not installing Python 3. In the case of this particular project, pyserial was not (when I first started the project) Python 3 compatible, so rather than fight that battle I decided to use Python 2.7.

The problem with Python 2.7 is on CentOS 6.5, Python 2.6.6 is the default, and since there’s other Python-related stuff running on the server already I didn’t want to run the risk of screwing anything else up, so I had to install Python 2.7 as an alternate Python installation. Luckily there were a couple of resources from people who had already been through this so it wasn’t an issue. Here’s the steps I took on a fresh CentOS 6.5 VM I was using to do some trial runs before doing everything on the production server (do all these as the root user).

  1. yum -y update
  2. yum -y groupinstall “development tools” –skip-broken
  3. yum -y install wget gcc gcc-c++ make httpd-devel git vim zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel
  4. wget https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tgz
  5. tar xvf Python-2.7.10.tgz
  6. cd Python-2.7.10
  7. ./configure –prefix=/usr/local –enable-shared LDFLAGS=”-Wl,-rpath /usr/local/lib”
  8. make && make altinstall
  9. python2.7 -V (to confirm it’s working)
  10. cd ..
  11. wget –no-check-certificate https://pypi.python.org/packages/source/s/setuptools/setuptools-18.2.tar.gz
  12. tar xvf setuptools-18.2.tar.gz
  13. cd setuptools-18.2
  14. python2.7 setup.py install
  15. cd ..
  16. curl https://raw.githubusercontent.com/pypa/pip/master/contrib/get-pip.py | python2.7 –
  17. pip2.7 install virtualenv

Create a User to Own the Project

Depending on how you want to do things this could be considered optional, but I created a user to own the project files (again as root):
  1. useradd -m -s /bin/bash/someuser

Create a Python virtualenv and Install the Django Project

Next we’ll create a Python 2.7 virtualenv, grab the Django code, install the Django project’s requirements, and do a couple of other configuration things for the Django app.
  1. sudo su – someuser
  2. mkdir ~/.virtualenvs
  3. cd ~/.virtualenvs
  4. virtualenv foo –python=python2.7
  5. cd foo
  6. source bin/activate
  7. cd ~/
  8. git clone foo
  9. cd foo
  10. pip install -r requirements.txt
  11. python manage.py runserver (just to make sure things are working at this point)
  12. python manage.py collectstatic
  13. python manage.py migrate

Configure Upstart to Start the Gunicorn Process When the Server Boots

I suppose on the next version of CentOS this will be done with systemd but thankfully on CentOS 6.5 we can still use Upstart. Note that if you’re familiar with Upstart on Ubuntu the syntax is quite different on CentOS — thanks to my good friend and former coworker Brandon Culpepper for pointing that out before I lost my mind.
First, we’ll do a quick test to make sure everything’s working at this point:
  1. sudo su – 
  2. cd /home/someuser/foo
  3. /home/someuser/.virtualenvs/foo/bin/gunicorn –workers 4 –timeout 60 –bind 0.0.0.0:8000 foo.wsgi:application
  4. Hit Ctrl-C to kill the process if you don’t see any errors.
Next we’ll create the upstart file:
  1. vim /etc/init/foo.conf
  2. Put the following in the foo.conf file and save it:
    description “Gunicorn process for foo app”

    start on started sshd
    stop on shutdown

    script
      cd /home/someuser/foo
      /home/someuser/.virtualenvs/foo/bin/gunicorn –workers 4 –timeout 60 –log-level debug –bind 0.0.0.0:8000 foo.wsgi:application
    endscript

  3. start foo (to make sure the upstart process works)
  4. ps -wef | grep python (you should see some python processes running under your virtualenv)

Create Apache Virtual Host for the App

There’s a bunch of ways to set up Django apps with Apache. In my early days with Django I would have used mod_wsgi but since I’m way more used to Nginx and Gunicorn these days, I figured I’d set up Apache in similar fashion and have it proxy to Gunicorn.
  1. vim /etc/httpd/conf/httpd.conf
  2. Uncomment the NameVirtualHost *.80 line if it isn’t already uncommented
  3. Add a new VirtualHost section at the bottom of the Apache config file:
    <VirtualHost *:80>
      ServerName whatever
      DocumentRoot /home/someuser/foo

      # serve static files from Apache
      RewriteEngine on
      RewriteRule ^/static/.* – [L]

      # proxy everything else to the gunicorn process
      ProxyPreserveHost on

      RewriteRule ^(.*)$ http://127.0.0.1:8000$1 [P]
      ProxyPassReverse / http://127.0.0.1:8000/
    </VirtualHost>

  4. apachectl restart
At that point you should be all set! Hope that helps people who are in this same or a similar boat save some time.

Dynamically Adding Forms To and Removing Forms From Django Formsets

For my latest big project I am running into numerous situations in which I need to dynamically add forms to and remove forms from Django formsets, which turned into an interesting and fun challenge, and led to a lot of learning about how Django formsets work along with some ancillary details (like the behavior of the jQuery UI datepicker when you manipulate the DOM) that I wasn’t necessarily expecting to encounter.

To set the stage for the discussion by way of a concrete example, one aspect of the application in question is the ability to request leave (e.g. vacation, sick leave, etc.), and as you might imagine on the leave request page you can add days to and remove days from your leave request.
In the first iteration of this feature (random aside: we do Kanban and try our darndest to focus on “good enough” features, particularly on early iterations, so that we can deliver the most usable functionality in the least amount of time, looping back around to make improvements as time permits), each form in the leave formset has the following fields:
  • date of leave
  • start time of leave
  • end time of leave
  • hours of leave
  • minutes of leave
Why does the user have to specify the number of hours and minutes they’re taking for their leave if they’re providing the start and end times, you might ask? Basically it boils down to people with access to the leave calendar wanting to know the specific times people are out, which may not always match up with the amount of leave they’re taking.
For example if you take a full day of leave, let’s say you’re out 8 am to 5 pm, but there’s a lunch hour in the mix, so the clock time you’re out is 9 hours, but the amount of leave you have to take for the full day is only 8 hours. And again, “good enough” is the focus on this first iteration. This basic functionality lets people request leave, and in future iterations, time permitting, we’ll make the system smarter and remove some of the burden off the user with things like calculating hours based on start/end times, taking into account weekends and holidays, letting people select date spans, and all sorts of other niceties that are in our backlog.
But back to the discussion at hand, namely how this all happens with Django formsets. Since the last thing I wanted to do was have the leave form code as a variable in a JavaScript function, I decided to go the route of grabbing a new instance of the form via Ajax when the user clicks the “Add Day” button. That way the leave form code lives in one place, in a normal Django template, and is therefore much more maintainable.
This immediately got interesting, however, given how Django formsets work. By default (i.e. if you aren’t using the prefix option in the formset constructor), forms in Django formsets will all have ids along the lines of id_form-0-field_name, and names along the lines of form-0-field_name, where 0 is the index corresponding to the form’s position in the overall formset. So if you have multiple forms the first form will have an index of 0, the second one will have an index of 1, etc.
Also in the mix is the Django formset’s management form, which contains the information necessary for Django to manage and process the formset, such as the total number of forms, the minimum and maximum number of forms allowed, and so on. The key point here is that if you’re going to be dynamically adding and removing forms, you also need to be updating the form-TOTAL_FORMS value in the management form in order for things to stay in sync.
As you might be guessing by now, the syntax of the form fields in a formset doesn’t quite mix with making an Ajax call out of the blue to an otherwise oblivious Django template to get a new form. Thankfully that was pretty easy to resolve by making sure to get the next form number and including that in the Ajax call:

And the form template (small snippet here) simply includes the form number in all the necessary places to make the form gibe with what the formset expects:

So when you get the form back from the Ajax call it has the correct number in all the fields, and all is right with the world.

Or so it would seem. [cue dramatic music]

At this point if all you ever want to do is ADD forms to a formset, this approach works perfectly. Where things get weird is when you start removing forms from your formset, since the aforementioned form indices get all kinds of screwed up (to use a technical term) if you don’t adjust them as you go.

When I first started adding the remove functionality I was optimistically thinking, “Oh Django probably doesn’t care about the form indices so much. I’m sure as long as the total number of forms it has in the management form matches the number of forms in the formset, it won’t care if the indices go 0, 1, 4, 7, or whatever.”

And that’s actually partially true. I’ve gone through so many iterations of everything prior to writing this blog post that you’ll have to forgive me for slightly fuzzy details here, but my recollection of the first incarnation of this feature is that yes, indeed, I could add and remove rows to my heart’s content, and as long as I was ensuring the next form number was incremented, and the total forms value in the management form was right, I could post the formset, validate the formset, route back with errors, and that all worked.

Where things proved interesting is when our QA folks took a crack at this, they (as QA folks tend to do) did something I hadn’t:

  1. Add a few rows to the form, and set the dates (N.B. we’re using jQuery UI’s datepicker for this; that fact becomes important in a moment)
  2. Add another new row and don’t set the date
  3. Delete one of the rows in the middle of the set of rows
  4. Set the date using the datepicker in the last row that was added
At that point what would happen is that the date selected in the final row would change the value of the date in one of the rows above it. Also if you submitted the form that way with errors, when it came back and rendered the populated forms, some of the forms were blank.
This turned out to be a great find that set me down the path of learning a whole lot of stuff I’m glad I learned now before it bit me later.
Attacking the two problems (1. Django forms don’t re-render correctly in an error state, and 2. datepicker changes the wrong date) in sequence, my first thought went back to the Django formset indices and my optimistic assumption that the indices didn’t matter as long as the number of forms in the formset matched the total forms value in the management form.
This, not surprisingly, turned out to not be the case. As I said earlier if you only add forms things are incremented nicely and nothing gets out of whack, but I was now finding that when a form was removed, what needed to happen is that all the existing form fields needed to be reindexed so they started with 0 and incremented sequentially. A bit of JavaScript handled that without too much trouble. (I’ll put the solution below since it involves problem #2 as well.)
Even after I got the form indices back into a sequential state as forms were removed, however, the datepicker was still behaving badly. I’d add and remove forms in random order and in some cases the datepicker would still change the wrong date field (in a predictable fashion with respect to the index), or throw an error that it couldn’t find the form field to which it was attached.
That latter error led me to the solution for this problem. The long and short of it is that when you remove something from the DOM that has a datepicker attached to it, you have to destroy and reattach all the datepicker elements because each datepicker retains a reference to the field to which it was originally attached.
To sum up what has to happen when a form is removed for everything to behave properly:
  1. Resequence all the form indices so they start at 0 and increment sequentially; and
  2. Destroy all remaining datepickers and recreate them
To keep things simple the remove link on each form looks like this:

That way I didn’t have to keep track of the index on the delete button too, I could simply reference its form container parent and go from there.

With that in place the delete row function wound up looking like this:

Main things of interest there are getting a handle on the delete link’s parent form via the form class, and lines 13-16 where the datepickers are destroyed and rebound so they get attached to the correct form field.

And of course the call to updateFormElementIndices(), the bit that resequences the form indices, which was inspired by various snippets and other solutions I found to address this issue and looks like this:

Pretty straight-forward, but to go over what’s happening there:

  1. Get a handle on all the forms of the class passed to the function and loop over them
  2. Get the current index number of the form being updated
  3. Update the form index to the loop increment (0-based, sequential)
  4. Get all the inputs with the class matching the form class passed in, plus ‘Input’ at the end
  5. Loop over the inputs, changing the IDs and names in similar fashion to how we updated the form ID
With all that in place Django was happy, the datepicker was happy, and (hopefully) our QA folks will be happy as well.
As I said this was a great find not only because I learned a ton slogging through this today, but also because this was the first incarnation of a pattern that will occur in numerous areas throughout this application (adding multiple phone numbers being the next example), so it was nice to catch this early and fix it in such a way that I can do it right on the future instances of similar functionality.
As far as the code goes, it works for the leave form I’ve been describing here, but I’m sure as I work on the future similar functionality I’ll find ways to make it more generic and flexible for more generalized use. The reliance on having to put related classes on the forms and inputs makes things handy for the JavaScript, but is an additional requirement as far as the development goes that may not ultimately be necessary.
But there you have it. It works, I learned a ton, and I share it here both so it might help someone else having to do this, and also so someone who knows more than I do can point out that I may be doing it in a more complicated way than necessary.

Generating CSV Files in Django

This is a very quick tip since it’s so simple but I did run into one little wrinkle with string encoding while doing this so I thought I’d share.

I had a request to generate a CSV of all the Old Dog Haven “Walk for Old Dogs” registrants so the fine folks managing the event can do email blasts, print registration sheets, and the like.

Since Python has CSV functionality as part of the standard library the generation of the CSV data was very easy, and since the Django HTTP Response object was quite wisely designed to behave as a file-like object, writing the CSV data to the HTTP response was dead simple as well.

The only thing I ran into while writing the data to the response object was there were some non-ASCII characters in the database which threw this error:
‘ascii’ codec can’t encode character u’xe9′ in position 4: ordinal not in range(128)
 
To get around this it was just a matter of tacking .encode(‘utf-8’) to the end of the string objects when writing out the CSV data.

Here’s the final result.

Custom Managers in Django

I had to add a bit of functionality to the Old Dog Haven “Walk For Old Dogs” registration and sponsorship site today (feel free to sponsor me!), specifically a way for the good folks running the event to see an aggregate total of the sponsorship dollars by registrant.

Here’s the Django models for Registrant and Sponsor:
Pretty straight-forward — main thing to notice is that Sponsor has a foreign key relationship with Registrant as opposed to the other way around. And yes, in its current incarnation sponsors can only sponsor one registrant; if they want to sponsor multiple registrants they simply create another sponsor record. (It’s agile baby — release early and often! Multiple sponsorships can happen next year!)

While calculating the total sponsorship dollars by each registrant could be handled via the Django ORM, I decided to use the opportunity to execute SQL directly in a function in a custom manager.

You can read more about custom managers in the Django docs, but the short version is that when you call something like Foo.objects.all() the objects bit refers to the model’s Manager class, and it has all sorts of default functionality built in allowing you to easily interact with your model.

Referring back to the Sponsor and Registrant model code above, notice that since we’re using a custom model manager, in the Sponsor model I added the line objects = SponsorManager() to override the default model manager with the custom one for which we’ll see the code below.

As with most things in Django, model managers are easily extended and overridden, and in this case I wanted to add a new function to the Sponsor manager to return a list of registrants with a sum of their sponsorship amounts. This was as simple as adding a SponsorManager model that extends model.Manager and adding a function called with_sponsorship_totals to return the information I need.
Again, pretty straight-forward. SponsorManager extends models.Manager, we add the code to run the SQL query we want to execute, loop over the results and add instances of the Registrant model to the results list, and return the results.

One of the other great things I should point out about Python is the ability to add fields to classes on the fly. The Registrant class doesn’t have a total_sponsorship_amount or team_name field defined, but I can add those as I loop over my results so they’re available in the list returned by the custom manager function.

Another approach to solving this problem might be to add a function to the Registrant model itself that would calculate the total sponsorship amount for each registrant, but I opted to create a custom manager in this case to experiment a little, and I also thought it taking this approach would be more efficient than calculating individually for each registrant.

As you can see from this basic example, creating a custom manager for your Django models is an extremely simple, powerful way to add functionality to your application and is nothing to shy away from. By creating fat models and extending model managers where needed you can keep the business logic where it belongs and avoid a lot of unnecessary complexity in views and other areas of your application.

django-stdimage Custom Fields and South Migrations

I’m putting the finishing touches on my latest Django site (a registration site for a dog rescue charity walk) and part of the functionality is the ability for users to upload photos. Since most people will simply snap a photo on their cell phones and upload it, and since cell phones these days take some pretty large pictures, I want to resize the images to a standard size of 800×600 when they’re uploaded and also create a thumbnail.

As with most things in the Python/Django world there’s a library that does exactly what I need, specifically django-stdimage, which provides a custom model field to handle resizing photos and creating thumbnails.

After installing django-stdimage via pip, updating my model class to use the new form field, and then doing a South migration, I ran into this error:

 ! Cannot freeze field ‘supporter.registrant.photo’
 ! (this field has class stdimage.fields.StdImageField)

 ! South cannot introspect some fields; this is probably because they are custom
 ! fields. If they worked in 0.6 or below, this is because we have removed the
 ! models parser (it often broke things).
 ! To fix this, read http://south.aeracode.org/wiki/MyFieldsDontWork

To give a little background, South used to handle automatic introspection of custom fields, but according to the explanation of why it no longer does that in newer versions of South, “… when it broke, it broke spectacularly.” Fair enough! So the expectation on newer versions of South is that you’ll provide the necessary information to allow South to handle the custom fields.
Giving South what it needs to do migrations involving custom fields is simple enough but since this is the first time I’ve had to deal with it I thought I’d help my future self and potentially others by sharing how to handle it.

There are a couple of different scenarios with South and custom fields that you can read more about in the South docs, but in the case of django-stdimage you simply have to add the location of the custom field to South’s introspection rules.

In the models.py that contains the model in which you’re using django-stdimage, add the following somewhere at the top of the file:

from south.modelsinspector import add_introspection_rules

add_introspection_rules([], [‘^stdimage.fields.StdImageField’])

Now when you run the South schema migration South will know where to find StdImageField and will be able to handle the migration.

If you have more complex custom fields be sure and read the South docs since if your custom field adds new attributes or doesn’t extend another class for which there are already introspection rules, you’ll have to provide the introspection rules yourself as opposed to simply providing the location of the custom field class.

I’ll share the link to the app in which I’m using django-stdimage and put the code for the entire app up on GitHub soon.

Generating and Sorting on a Transient Property in a Django Model Class

I ran into an interesting little issue in a Django application today that led to what I thought was some pretty powerful stuff in Python and Django that I hadn’t had to use before, so I thought I’d share.

I’m working on an application that to keep it generic I’ll simply say deals with requests from users, and these requests require various levels of approval depending on the type and severity of the request.

Here’s a basic Django model class that contains the fields relative to this example:

Basic stuff so far.

Where the wrinkle comes in is there is a page in the application that lists the requests ordered by date, but the date by which each request will be displayed and sorted depends upon the impact of the request as follows:

  • minor: use datetime_supervisor_approved
  • major: use datetime_admin_approved
  • severe: use datetime_president_approved

To state the issues succinctly:

  • The date on which I want to sort varies for each request
  • I don’t want to store an additional, redundant field in the database simply to have a consistent name to use for sorting
  • Because the sort date isn’t stored in the database I can’t use order_by on a QuerySet

That last bullet was the killer. In Python it’s simple enough to add attributes to a class on the fly, so I could loop over the QuerySet and use conditional logic around the change impact to add a new calendar_date field to each instance of the UserRequest class, but that’s kind of ugly because the business logic winds up in a view function as opposed to being in the model, and still doesn’t solve the inability to use order_by in the QuerySet.

Because I want to keep the business logic in the model (fat models FTW!), I looked into using a Python property to add a transient attribute to my UserRequest class that is the result of a function call.

Basically what this means in concrete terms is I’m adding a calendar_date attribute to my model class and the value of calendar_date is set by calling a function in the class itself that contains the aforementioned conditional logic around the request impact.

Here’s the modified model class:

The big addition here is the _get_calendar_date function that returns a calendar date based on the impact of the request. This is added as an attribute to the class, but it’s not saved to the database which is exactly what I wanted in this instance.

That solves one piece of the puzzle, but the other piece is sorting the requests based on this new transient property, which again can’t be done by using order_by on the QuerySet since the field on which we want to order isn’t in the database.

This is where Python’s sorted() function comes in. sorted() can take any iterable (which the Django QuerySet is) and sort it based on a provided comparator function, and this is where I can leverage the transient calendar_date property to sort the requests.

Putting all the pieces together in the view function, here’s how it looks:

Python’s sorted() function is pretty powerful and straight-forward — throw it an iterable and what to compare on, and it does the heavy lifting for you.

The only potentially tricky part of this is setting the key since it uses Python’s lambda statement (great explanation here), which is Python’s way of kinda sorta dipping a toe into the functional programming waters by letting you define an anonymous inline function.

In this case how the lambda plays out is to sort the QuerySet based on two attributes: first the transient calendar_date property that isn’t stored in the database, and subsequently the request_impact attribute that is stored in the database. Perfect!

I’m sure none of this is rocket science to seasoned Python and Django veterans but since I hadn’t run into the need to do this before I thought I’d document what I came up with both for my own future reference and hopefully for the benefit of others who made need to so something similar.

Setting Up Django On a Raspberry Pi

This past weekend I finally got a chance to set up one of my two Raspberry Pis to use as a Django server so I thought I’d share the steps I went through both to save someone else attempting to do this some time as well as get any feedback in case there are different/better ways to do any of this.

I’m running this from my house (URL forthcoming once I get the real Django app finalized and put on the Raspberry Pi) using dyndns.org. I don’t cover that aspect of things in this post but I’m happy to write that up as well if people are interested.

General Comments and Assumptions



Reconfigure the Keyboard Mapping

If you already did this at some other point with your Raspberry Pi or your keyboard is working properly you can skip this step. After the base install of Raspbian “wheezy” in my case I noticed when in vim I couldn’t type things like #, |, and @ so I had to do the following:

  1. sudo dpkg-reconfigure keyboard-configuration
  2. Follow the prompts to set things appropriate for your keyboard (US English in my case)
  3. Restart the Raspberry Pi (sudo reboot)


Install Required Server-Wide Tools and Libraries


  1. sudo apt-get install vim python-dev python-setuptools nginx supervisor
  2. sudo easy_install pip
  3. sudo pip install virtualenv virtualenvwrapper
  4. Configure virtualenvwrapper
    1. sudo vim /etc/bash.bashrc
    2. Add this line at the bottom of the file:
      source /usr/local/bin/virtualenvwrapper.sh
    3. Save the file and exit vim
    4. Log out and log back in


Create and Configure a virtualenv for Your Application

tl;dr version of the following paragraph: Run the following commands from your home directory unless you have a reason not to and know what you’re doing. For the purposes of this tutorial I’m using the default ‘pi’ user on the Raspberry Pi.

You can create a virtualenv for your application in your home directory or wherever you prefer your files to be located, just be aware that depending on where you put your files and whether or not you have to execute virtualenv-specific tasks (pip install, etc.) you may have to explicitly specify which python or pip command to use for it to work properly.

For the purposes of this example we’ll create a ‘foo’ application. The steps below where you’re running django-admin.py startproject and firing up the development server are simply to test to make sure everything’s working properly. If you’re doing this all for the first time, I’d suggest doing this both to get familiar with it as well as to double-check that everything with the environment is working as expected.

  1. mkvirtualenv foo
    1. Note that this will activate the virtualenv. You MUST have the virtualenv activated when performing the steps that follow for everything to work properly.
    2. If you’re used to using virtualenv directly as opposed to virtualenvwrapper, note that when you create a virtualenv using  mkvirtualenv, it will put the virtualenv files (again, if you’re doing this from your home directory) in ~/.virtualenvs/foo
  2. pip install django docutils south gunicorn
  3. django-admin.py startproject foo
  4. cd foo
  5. python manage.py runserver
    1. The first time you run this you may see a ton of blank lines output on the Raspberry Pi. It’ll eventually finish, or if you hit Ctrl-C when this happens and try again a couple of times it’ll eventually calm down and the next time you run things this won’t happen. I’m not sure what’s going on here since the Raspberry Pi is the only platform on which I’ve seen this happen.
    2. Once the development server starts up you should see “0 errors found [other stuff here] Development server is running at http://127.0.0.1:8000” If you don’t see that, double-check everything and try again.
    3. You can make a call to the development server by sshing into the Raspberry Pi from another machine and doing curl localhost:8000


Enable Gunicorn and South and Enable a Database in Your Application


  1. cd ~/foo/foo
  2. vim settings.py
  3. In the DATABASES section make the following changes:
    1. ‘ENGINE’: ‘django.db.backends.sqlite3’
    2. ‘NAME’: ‘/home/pi/foo/foo.db’
  4. In the INSTALLED_APPS section, enable the admin and admindocs (optional):
    1. Uncomment # ‘django.contrib.admin’
    2. Uncomment # ‘django/contribu.admindocs’
  5. Also in the INSTALLED_APPS section, add ‘south’ and ‘gunicorn’ to the list of installed apps.
  6. Save the file and exit vim
  7. Test Gunicorn:
    1. python manage.py run_gunicorn (you can hit Ctrl-C to kill it once you see it’s working)


A note on databases: I’m not addressing the broader topic of which database server you should use simply because it’s a separate discussion that’s potentially fraught with zealous opinions and general peril, and is a bit tangential to the present discussion.

If you’re doing all this just to learn, or for your own personal low-use “screwing around” type applications, SQLite (http://www.sqlite.org/), which is what comes with Django, will work just fine, so that’s what I outlined above.

I’m writing this up as part of setting up a site for alt.* Radio (http://codebassradio.net/shows/alt/), which is a weekly show I do on the super-awesome CodeBass Radio (http://codebassradio.net), so for that I’ll be running PostgreSQL on a separate machine. For low-traffic stuff you might even be able to get away with running MySQL or Postgres directly on the Raspberry Pi but I have a couple of unused machines laying around so I figured I might as well keep the Raspberry Pi focused on the Django side of things.

Note that if you DO want to use PostgreSQL, you’ll have to do this on the Raspberry Pi:

  1. sudo apt-get install libpq-dev
  2. pip install psycopg2 (with your virtualenv active)


Configure Supervisor

Supervisor (http://supervisord.org/), which we installed earlier, is a tool that lets you easily create startup/restart scripts for services that aren’t installed and managed through apt-get. In our case we’ll need this for Gunicorn so it’ll fire up when the machine boots and automatically restart if it crashes.

  1. cd /etc/supervisor/conf.d
  2. sudo vim gunicorn.conf
  3. Put the following in the new file:
    [program:gunicorn]
    command = /home/pi/.virtualenvs/foo/bin/python /home/pi/foo/manage.py run_gunicorn -w 4
    directory = /home/pi/foo
    user = pi
    autostart = true
    autorestart = true
    stdout_logfile = /var/log/supervisor/gunicorn.log
    stderr_logfile = /var/log/supervisor/gunicorn_err.log
  4. Save the file and exit vim
  5. sudo service supervisor restart
  6. sudo supervisorctl start gunicorn
    1. Note that you may get an “already started” error here. If you get a “no such process” error, that means supervisor didn’t load the new configuration file. If that happens:
      1. sudo ps -wef | grep supervisor
      2. sudo kill -9 SUPERVISOR_PROCESS_ID (use the real process ID)
      3. sudo service supervisor start


Configure Nginx


  1. sudo rm -f /etc/nginx/sites-enabled/default
  2. sudo vim /etc/nginx/sites-available/foo
  3. Put the following in the foo file:

# upstream server for gunicorn
upstream gunicorn {
  server localhost:8000;
}

# nginx server for the host
server {
  listen 80;

  server_name foo.com www.foo.com;

  root /home/pi/foo;

  access_log /var/log/nginx/foo_access.log;
  error_log /var/log/nginx/foo_error.log;

  # try to serve a static file and if it doesn’t exist, pass to gunicorn
  try_files $uri @gunicorn;

  # rules for gunicorn
  location @gunicorn {
    proxy_pass http://gunicorn;
    proxy_redirect off;
    proxy_read_timeout 5m;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Fowarded-For $proxy_add_x_forwarded_for;
  }
}

  1. Save the file and exit vim
  2. sudo ln -s /etc/nginx/sites-available/foo /etc/nginx/sites-enabled/foo
  3. sudo service nginx restart


Verify Everything’s Working

If you’re not using a real hostname/URL, make sure and add it to /etc/hosts on the Raspberry Pi so you can easily test. In this case we used foo.com as the hostname, so do the following:

  1. sudo vim /etc/hosts
  2. Add this line:
    127.0.0.1 foo.com
    www.foo.com
  3. Save the file and exit vim


If you want to hit the Raspberry Pi from a browser on another machine you’ll have to also add the IP and host names to /etc/hosts on any machines from which you want to hit that host, but use the IP of the Raspberry Pi instead of 127.0.0.1.

With that modification in /etc/hosts locally on the Raspberry Pi, you can do a curl foo.com to see if you get the default Django page (or your application if you took things a bit further than are outlined in this tutorial). If you do, everything’s working! If not, start over and don’t screw up this time!

Seriously though if you need any help just comment here or email/Google+/Twitter me and I’ll be happy to assist.

Three Approaches to Handling Static Files in Django

I had a really great (and lengthy) pair programming session today with a coworker during which we spent a bit of time going over a couple of different approaches for dealing with static files in Django, so I thought I’d document and share this information while it’s fresh in my mind.

First, a little background. If you’re not familiar with Django it was originally created for a newspaper web site, specifically the Lawrence Journal-World, so the approach to handling what in the Django world are called “static files” — meaning things like images, JavaScript, CSS, etc. — is based on the notion that you might be using a CDN so you should have maximum flexibility as to where these files are located.

While the flexibility is indeed nice, if you’re used to a more self-contained approach it takes a little getting used to, and there are a few different ways to configure your Django app to handle static files. I’m going to outline three approaches, but using different combinations of things and other solutions of which I may be unaware there are certainly more ways to handle static files than what I’ll outline here. (And as I’m still relatively new to Django, if I’m misunderstanding any of this I’d love to hear where I could improve any of what I’m sharing here!)

One other caveat — I’m focusing here on STATIC_URL and ignoring MEDIA_URL but the approach would be the same.

Commonalities Across All Approaches

First, even though it may not strictly be required depending on which approach you take for handling static files, since you wind up needing to use this for other reasons, we’ll use django.template.RequestContext in render_to_response as opposed to the raw request object. This is required if you want access to settings like MEDIA_URL and STATIC_URL in your Django templates. For more details about RequestContext, the TEMPLATE_CONTEXT_PROCESSORS that are involved behind the scenes, and the variables this approach puts into your context, check the Django docs.

I’m also operating under the assumption that the static files will live in a directory called static that’s under the main application directory inside your project directory (i.e. the directory that has your main settings.py file in it). Depending on the approach you use you may be able to put the static directory elsewhere, but unless stated otherwise, that’s where the directory is assumed to be. (Note that if you store static files on another server entirely, such as using a CDN, STATIC_URL can be a full URL as opposed to a root-relative URL like /static/)

Also in all examples it’s assumed that the STATIC_URL setting in the main settings.py file is set to ‘/static/’

Approach One (Basic): Use STATIC_URL Directly in Django Templates

This is the simplest approach and may be all you need. With STATIC_URL set to ‘/static/’ in the main settings.py file, all you really have to worry about is using RequestContext in your view functions and then referencing {{ STATIC_URL }} in your Django templates.

Here’s a sample views.py file:

from django.shortcuts import render_to_response
from django.template import RequestContext

def index(request, template_name=’index.html’):
    return render_to_response(template_name, context_instance=RequestContext(request))

By using RequestContext the STATIC_URL variable will then be available to use in your Django templates like so:

<html>
<head>
    <script src=”{{ STATIC_URL }}scripts/jquery/jquery-1.8.1.min.js”></script>
</head>
<body>
    <img src=”{{ STATIC_URL }}images/header.jpg” />
</body>
</html>

That’s all there is to it. Again, since /static/ will be relative to the root of the main application directory in your project it’s assumed that the static directory is underneath your main application directory for this example, and obviously in the case of the example above that means that underneath the static directory you’d have a scripts and images directory.

Approach Two: Use a URL Pattern, django.views.static.serve, and STATICFILES_DIRS

In this approach you leverage Django’s excellent and hugely flexible URL routing to set a URL pattern that will be matched for your static files, have that URL pattern call the django.views.static.serve view function, and set the document_root that will be passed to the view function to a STATICFILES_DIRS setting from settings.py. This is a little bit more involved than the first approach but gives you a bit more flexibility since you can place your static directory anywhere you want.

The approach I took with this method was to set a CURRENT_PATH variable in settings.py (created by using os.path.abspath since we need a physical path for the document root) and leverage that to create the STATICFILES_DIRS setting. Here’s the relevant chunks from settings.py:

import os
CURRENT_PATH = os.path.abspath(os.path.dirname(__file__).decode(‘utf-8’)).replace(‘\’, ‘/’)

STATICFILES_DIRS = (
    os.path.join(CURRENT_PATH, ‘static’),
)

Note that the replace(‘\’, ‘/’) bit at the end of the CURRENT_PATH setting is to make sure things work on Windows as well as Linux.

Next, set a URL pattern in your main urls.py file:

from django.conf.global_settings import STATICFILES_DIRS

urlpatterns = patterns(”,
    url(r’^static/(?P<path>.*)$’, ‘django.views.static.serve’, {‘document_root’: STATICFILES_DIRS}),
)

And then in your Django templates you simply prefix all your static assets with /static/ as opposed to using {{ STATIC_URL }} as a template variable. Even though you’re specifying /static/ explicitly in your templates, you still have flexibility to put these files wherever you want since the URL pattern acts as an alias to the actual location of the static files.

Approach Three: Use staticfiles_urlpatterns and {% get_static_prefix %} Template Tag

django.contrib.staticfiles was first introduced in Django 1.3 and was designed to clean up, simplify, and create a bit more power for static file management. This approach gives you the most flexibility and employs a template tag instead of a simple template variable when rendering templates.

First, in settings.py we’ll do the same thing we did in the previous approach, namely setting a CURRENT_PATH variable and then use that to set the STATICFILES_DIRS variable:

import os
CURRENT_PATH = os.path.abspath(os.path.dirname(__file__).decode(‘utf-8’)).replace(‘\’, ‘/’)

STATICFILES_DIRS = (
    os.path.join(CURRENT_PATH, ‘static’),
)

Next, in urls.py we’ll import staticfiles_urlpatterns from django.contrib.staticfiles.urls and call that function to add the static file URL patterns to the application’s URL patterns:

from django.contrib.staticfiles.urls import staticfiles_urlpatterns

urlpatterns = patterns(
    # your app’s url patterns here
)

urlpatterns += staticfiles_urlpatterns()

The final line there is what adds the static file URL patterns into the mix. If you output staticfiles_urlpatterns() you’ll see it’s something like so:

[<RegexURLPattern None ^static/(?P<path>.*)$>]

And finally, at the very top of your templates you load the static template tags and then simply use the {% get_static_prefix %} tag to render the static URL:

{% load static %}
<html>
<head>
    <script src=”{% get_static_prefix %}scripts/jquery/jquery-1.8.1.min.js”></script>
</head>
<body>
    <img src=”{% get_static_prefix %}images/header.jpg” />
</body>
</html>

Conclusion

So there you have it, three approaches that more or less accomplish the same thing, but depending on the specific needs of your application or environment one approach may work better for you than another.

For our purposes on our current application we’re using the first approach outlined above since it’s simple and meets our needs, but it’s great to know there’s so much flexibility around static file handling in Django when you need it. As always read the docs for more information and yet more options for managing static files in your Django apps.