For my latest big project I am running into numerous situations in which I need to dynamically add forms to and remove forms from Django formsets, which turned into an interesting and fun challenge, and led to a lot of learning about how Django formsets work along with some ancillary details (like the behavior of the jQuery UI datepicker when you manipulate the DOM) that I wasn’t necessarily expecting to encounter.
To set the stage for the discussion by way of a concrete example, one aspect of the application in question is the ability to request leave (e.g. vacation, sick leave, etc.), and as you might imagine on the leave request page you can add days to and remove days from your leave request.
In the first iteration of this feature (random aside: we do
Kanban and try our darndest to focus on “good enough” features, particularly on early iterations, so that we can deliver the most usable functionality in the least amount of time, looping back around to make improvements as time permits), each form in the leave formset has the following fields:
- date of leave
- start time of leave
- end time of leave
- hours of leave
- minutes of leave
Why does the user have to specify the number of hours and minutes they’re taking for their leave if they’re providing the start and end times, you might ask? Basically it boils down to people with access to the leave calendar wanting to know the specific times people are out, which may not always match up with the amount of leave they’re taking.
For example if you take a full day of leave, let’s say you’re out 8 am to 5 pm, but there’s a lunch hour in the mix, so the clock time you’re out is 9 hours, but the amount of leave you have to take for the full day is only 8 hours. And again, “good enough” is the focus on this first iteration. This basic functionality lets people request leave, and in future iterations, time permitting, we’ll make the system smarter and remove some of the burden off the user with things like calculating hours based on start/end times, taking into account weekends and holidays, letting people select date spans, and all sorts of other niceties that are in our backlog.
But back to the discussion at hand, namely how this all happens with Django formsets. Since the last thing I wanted to do was have the leave form code as a variable in a JavaScript function, I decided to go the route of grabbing a new instance of the form via Ajax when the user clicks the “Add Day” button. That way the leave form code lives in one place, in a normal Django template, and is therefore much more maintainable.
This immediately got interesting, however, given how Django formsets work. By default (i.e. if you aren’t using the prefix option in the formset constructor), forms in Django formsets will all have ids along the lines of id_form-0-field_name, and names along the lines of form-0-field_name, where 0 is the index corresponding to the form’s position in the overall formset. So if you have multiple forms the first form will have an index of 0, the second one will have an index of 1, etc.
Also in the mix is the Django formset’s
management form, which contains the information necessary for Django to manage and process the formset, such as the total number of forms, the minimum and maximum number of forms allowed, and so on. The key point here is that if you’re going to be dynamically adding and removing forms, you also need to be updating the
form-TOTAL_FORMS value in the management form in order for things to stay in sync.
As you might be guessing by now, the syntax of the form fields in a formset doesn’t quite mix with making an Ajax call out of the blue to an otherwise oblivious Django template to get a new form. Thankfully that was pretty easy to resolve by making sure to get the next form number and including that in the Ajax call:
And the form template (small snippet here) simply includes the form number in all the necessary places to make the form gibe with what the formset expects:
So when you get the form back from the Ajax call it has the correct number in all the fields, and all is right with the world.
Or so it would seem. [cue dramatic music]
At this point if all you ever want to do is ADD forms to a formset, this approach works perfectly. Where things get weird is when you start removing forms from your formset, since the aforementioned form indices get all kinds of screwed up (to use a technical term) if you don’t adjust them as you go.
When I first started adding the remove functionality I was optimistically thinking, “Oh Django probably doesn’t care about the form indices so much. I’m sure as long as the total number of forms it has in the management form matches the number of forms in the formset, it won’t care if the indices go 0, 1, 4, 7, or whatever.”
And that’s actually partially true. I’ve gone through so many iterations of everything prior to writing this blog post that you’ll have to forgive me for slightly fuzzy details here, but my recollection of the first incarnation of this feature is that yes, indeed, I could add and remove rows to my heart’s content, and as long as I was ensuring the next form number was incremented, and the total forms value in the management form was right, I could post the formset, validate the formset, route back with errors, and that all worked.
Where things proved interesting is when our QA folks took a crack at this, they (as QA folks tend to do) did something I hadn’t:
- Add a few rows to the form, and set the dates (N.B. we’re using jQuery UI’s datepicker for this; that fact becomes important in a moment)
- Add another new row and don’t set the date
- Delete one of the rows in the middle of the set of rows
- Set the date using the datepicker in the last row that was added
At that point what would happen is that the date selected in the final row would change the value of the date in one of the rows above it. Also if you submitted the form that way with errors, when it came back and rendered the populated forms, some of the forms were blank.
This turned out to be a great find that set me down the path of learning a whole lot of stuff I’m glad I learned now before it bit me later.
Attacking the two problems (1. Django forms don’t re-render correctly in an error state, and 2. datepicker changes the wrong date) in sequence, my first thought went back to the Django formset indices and my optimistic assumption that the indices didn’t matter as long as the number of forms in the formset matched the total forms value in the management form.
This, not surprisingly, turned out to not be the case. As I said earlier if you only add forms things are incremented nicely and nothing gets out of whack, but I was now finding that when a form was removed, what needed to happen is that all the existing form fields needed to be reindexed so they started with 0 and incremented sequentially. A bit of JavaScript handled that without too much trouble. (I’ll put the solution below since it involves problem #2 as well.)
Even after I got the form indices back into a sequential state as forms were removed, however, the datepicker was still behaving badly. I’d add and remove forms in random order and in some cases the datepicker would still change the wrong date field (in a predictable fashion with respect to the index), or throw an error that it couldn’t find the form field to which it was attached.
That latter error led me to the solution for this problem. The long and short of it is that when you remove something from the DOM that has a datepicker attached to it, you have to destroy and reattach all the datepicker elements because each datepicker retains a reference to the field to which it was originally attached.
To sum up what has to happen when a form is removed for everything to behave properly:
- Resequence all the form indices so they start at 0 and increment sequentially; and
- Destroy all remaining datepickers and recreate them
To keep things simple the remove link on each form looks like this:
That way I didn’t have to keep track of the index on the delete button too, I could simply reference its form container parent and go from there.
With that in place the delete row function wound up looking like this:
Main things of interest there are getting a handle on the delete link’s parent form via the form class, and lines 13-16 where the datepickers are destroyed and rebound so they get attached to the correct form field.
And of course the call to updateFormElementIndices(), the bit that resequences the form indices, which was inspired by various snippets and other solutions I found to address this issue and looks like this:
Pretty straight-forward, but to go over what’s happening there:
- Get a handle on all the forms of the class passed to the function and loop over them
- Get the current index number of the form being updated
- Update the form index to the loop increment (0-based, sequential)
- Get all the inputs with the class matching the form class passed in, plus ‘Input’ at the end
- Loop over the inputs, changing the IDs and names in similar fashion to how we updated the form ID
With all that in place Django was happy, the datepicker was happy, and (hopefully) our QA folks will be happy as well.
As I said this was a great find not only because I learned a ton slogging through this today, but also because this was the first incarnation of a pattern that will occur in numerous areas throughout this application (adding multiple phone numbers being the next example), so it was nice to catch this early and fix it in such a way that I can do it right on the future instances of similar functionality.
As far as the code goes, it works for the leave form I’ve been describing here, but I’m sure as I work on the future similar functionality I’ll find ways to make it more generic and flexible for more generalized use. The reliance on having to put related classes on the forms and inputs makes things handy for the JavaScript, but is an additional requirement as far as the development goes that may not ultimately be necessary.
But there you have it. It works, I learned a ton, and I share it here both so it might help someone else having to do this, and also so someone who knows more than I do can point out that I may be doing it in a more complicated way than necessary.