The second hardest problem in computer science.
Caching values is easy. Figuring out when to invalidate these values, triggering recalculations, solving mutex lock while still having your code look presentable is hard. Django and Python offer an almost complete solution through computed properties to abstract complex methods away from readable code.
We’ve created django-cached-fields to bridge the gap between Django’s legendary DBMS integration and Python’s sugary sweet syntax.
Meet our model.
Our model is an Order, consisting of one item with attributes representing its name, price and quantity.
We also have a computed property called total which multiplies the price and quantity of the item to return the total price for the order.
class Order(models.Model):
item = models.CharField(max_length=20)
price = models.IntegerField()
quantity = models.IntegerField()
@property
def total(self):
return self.price * self.quantity
Right now, if we wanted to cache this value, we would have to create a field for it, decide when we should update this value and also have the method that does the actual computation on the class. (Because it wouldn’t really make sense to put it elsewhere right?)
Just saying that tastes like endless nights sipping Red Bull trying to untangle spaghetti code a once more lucid me dreamt up. The solution to this: django-cached-fields.
from cached_fields import fields
def calc_total(instance):
return instance.quantity * instance.price
class Order(models.Model):
item = models.CharField(max_length=20)
price = models.IntegerField()
quantity = models.IntegerField()
total = fields.CachedIntegerField(
calc_total,
field_triggers=['price', 'quantity']
)
A simple example.
We extracted our calculation logic into a pure function that returns the expected value of the cached field.
We then replaced the computed property on our Order model and have replaced it with an instance of django-cached-fields’ CachedIntegerField.
Finally, we specified two parameters for this CachedIntegerField: calc_total and field_triggers.
The first argument specifies the method that performs the calculation for this field. This method will be passed an instance of the Order object and therefore will have full access to the object.
Above, the calc_total method is a pure function and does not do any modification to the object itself. This is because django-cached-fields forbids direct modification of the data in the cached field. This prevents completely arbitrary values from being passed in by mistake. Calculation methods should return a value that is of a compatible or castable type for that field. For example, Django will cast a string to an integer for an IntegerField.
The second argument specifies the fields on the model that if changed, will trigger the recalculation.
In this simple example, the calculation is done synchronously on model save, but later in this article we will outline how to offload this processing to Celery or Django Q, both of which have direct integration with django-cached-fields.
What if a change in another model should trigger this recalculation?
The other primary use case of this package is to update cached fields when related (or unrelated, even) models are modified.
This could be useful, in our demo app, if we wanted to separate Item and all its attributes into a separate model and recalculate the value of the invoices each time the price of the product is changed.
Below, we’ve done just that. Item is now a ForeignKey on the Order model which itself has attributes of price and name. To keep our implementation sane, we would want to update the total of the Order whenever the price of the Item changes OR whenever the quantity of the Order changes.
Our updated models look like this.
from cached_fields import fields
class Item(models.Model):
name = models.CharField(max_length=20)
price = models.IntegerField()
class Order(models.Model):
item = models.ForeignKey(Item)
quantity = models.IntegerField()
total = fields.CachedIntegerField(
calc_total,
field_triggers=['quantity']
)
We’re in a bit of a sticky situation now, because we’re suddenly asking a lot more from our code.
We are asking our code to:
- Trigger a calculation when another model is saved (handle receiving signals).
- Trigger a calculation when this model is saved (handle a model save call).
- Not confuse the hell out of the developers implementing this behaviour.
Any sufficiently advanced technology is indistinguishable from magic.
Arthur C. Clarke
Wouldn’t it be great if we could handle all of these concerns in one place?
Well, that’s exactly what we’re going to do. Have a look at this:
from cached_fields import CachedFieldSignalHandler, for_class
class OrderHandler(CachedFieldSignalHandler):
@for_class("Order")
def calculate_total(instance):
return instance.quantity * instance.item.price
We have defined a handler for our Order class called OrderHandler. Inside this class is a method called calculate_total which accepts an instance of an Order and returns its total, which is calculated by multiplying its quantity and the item’s price.
What’s the for_class decorator for?
Because we are now handling recalculation for cached fields on a number of different models now, we have to have a way for this one single handler to differentiate how to handle the instances that are passed to it. For example, our handler needs to perform slightly different steps to calculate the total of an Order if the quantity was changed versus if the cost of the actual Item was changed.
You can pass for_class either the actual class itself, just the name of the class or the fully qualified name of the class. Note that if you just pass the name of the class, if you have multiple models with the same name, each of them will trigger the same method.
Now that we have the basic layout of our handler set up, let’s make our field update when Item is updated.
To do this, we simply need to add another method in the handler class and inform the decorator we wish that this method should be activated when that particular model is saved.
from cached_fields import CachedFieldSignalHandler, for_class
class OrderHandler(CachedFieldSignalHandler):
@for_class("Order", prefetch=['item'])
def calculate_order_total(self, instance):
return instance.quantity * instance.item.price
@for_class("Item")
def calculate_item_total(self, instance):
self.dispatch('calculate_total', instance.orders.all())
There are a few new things of note here.
First, you’ve just witnessed the first object dispatch.
The dispatch function inherited from CachedFieldSignalHandler allows you to pass objects of a compatible type to another method on the same class.
Second, you’ve just witnessed automatic iterable handling.
The for_class decorator does more than it looks on the surface and will actually handle iterables for you. If an iterable containing compatible types is sent to the method, django-cached-fields will automatically loop through the iterable and run the calculation on all contained elements.
Third, you’ve just witnessed the first parameter you can specify to for_class.
prefetch allows you to specify which fields should be prefetched when objects are selected. This doesn’t seem like a big deal when you are just updating a single object, but when a QuerySet of objects are passed to this method, it will automatically run prefetch_related or select_related (depending on context) to prefetch data to speed up your SQL calls.
What if you wanted to use custom signals?
One of the best ways of notifying the application of a change in value of a related object is through custom signals. Maybe there is another Order that affects the value of this Order or a change in the availability of an Item which nullifies this particular order. Your app could be notified of these events through custom signals.
from cached_fields import CachedFieldSignalHandler, for_class
from item.signals import item_availability_change
class OrderHandler(CachedFieldSignalHandler):
@for_class("Order", prefetch=['item'])
def calculate_order_total(self, instance):
return instance.quantity * instance.item.price
@for_class("Item", signals=[item_availability_change])
def calculate_item_total(self, instance):
self.dispatch('calculate_total', instance.orders.all())
You can pass in a signal to the for_class decorator using the receivers keyword argument. That’s all you have to do, django-cached-fields will create the receiver completely automatically.
Adding your handler to your model.
Now all the hard work is done, how much plumbing do we have to do to get this to work in our model?
from cached_fields import fields
from .handlers import OrderHandler
class Order(models.Model):
item = models.CharField(max_length=20)
price = models.IntegerField()
quantity = models.IntegerField()
total = fields.CachedIntegerField(
OrderHandler.as_handler()
)
That’s it. That’s literally it. All you need to do is just import the handler you just wrote, add it as the first positional argument in the field definition and you’re good to go.
Literally. That’s all there is to it.
Hooking receivers into signals for all the apps you’ve defined in your handler is automatically done for you. There is no more configuration required beyond this point and these two examples should cover 80% of use cases.
We will explore offloading calculations to Celery and Django Q and performing them both as required and as part of a schedule. Subscribe to notifications if you would like to be notified when this post comes out.
Send me a message here if you’d like some help or if you would like to get in touch.
