Auditing of changes to Models

There are 2 services provided by Maymyo for auditing changes to model instances.

• recording who and when an instance was created and last updated, plus incrementing a Last Version number which is used for concurrency control. We call this the RecordOwner pattern.
• recording the actual changes to each field in the model instance since creation.

If your models uses these services, then their admin (ie maintenance programs) will automatically have links that show the history of changes to particular instances.

How to use these services in your models

We shall use an existing model in Maymyo to illustrate:

# From Django
from django.utils.translation import ugettext_lazy as _ # To mark strings for translations
from django.db import models
from django.db.models.signals import pre_save, post_save, pre_delete

# Our modules
from infra.models.record_owner import RecordOwner
from infra.custom.fields import CodeField, CharNullField, DescriptionField
# Too many circular import errors when we use Filter Manager with Value Set
from infra.custom.filter_manager import FilterManager
from infra.custom.audit_handlers import audit_update_handler, audit_add_handler, audit_delete_handler
from bin.constants import RANGE_SEPARATOR

class ValueSet(RecordOwner):
    """
    Application specific List of Values.

    A value set is a List of Values used in this application.
    Instead of hardcoding these lists, we create a set of values
    here. Each ValueSet has as children one or more ValueSetMembers
    whose Value Code and Description are used to
    build the drop-down list (for the HTML Select Input).
    Use the class method get_choices() for this.

    Whenever you want to create a model that has just a Code and
    a Value, avoid doing so and just create a new ValueSet. Use
    your model name (in uppercase) as the Value Set Code. In
    addition, we allow up to 5 attributes per Value, which your
    own program can use in anyway it likes.
    """
    # Each Value Set has a Unique Code
    value_set_code = CodeField(verbose_name=_("Value Set Code"), unique=True)
    # and a helpful description
    value_set_description = DescriptionField(verbose_name=_("Description"),
        help_text=_("A helpful description of this Value Set"))
    # Optionally restrict Value Set Members' Value Code to this size (1-200)
    maximum_length = models.PositiveSmallIntegerField(verbose_name=_("Maximum Value Code Length"),
        blank=True, null=True,
        help_text=_("Optionally constrain its Value Code length (must be 1 to 200)"))
    # Is this an Application Constant, when True, only Administrator can change it
    is_app_constant = models.BooleanField(verbose_name=_("Is this an Application Constant?"), default=False,
        help_text=_("Application Constants can only be changed by Administrators"))

    # Override Manager to our own FilterManager
    objects = FilterManager()
    # Need another Manager without filtering for use in basic system functions
    all_objects = models.Manager()

    class Meta:
        ordering = ['value_set_code']
        verbose_name = _("Value Set")
        app_label = 'infra'
        db_table = 'if_value_set'

    def __unicode__(self):
        # we prefer to describe ourself using code then description
        return self.value_set_code

    def get_choices(self):
        choices = []
        if self.id:
            # Must prepend an empty selection for Fields that allows blank=True
            choices = [(u'', _("No Selection"))] + [(row.value_code, row.value_description)
                for row in self.valuesetmember_set.all().order_by('value_code')]
        # Return a List of 2 value Tuples as choices
        return choices

# Register the audit update handler
pre_save.connect(audit_update_handler, sender=ValueSet)
# Register the audit add handler
post_save.connect(audit_add_handler, sender=ValueSet)
# Register the audit delete handler
pre_delete.connect(audit_delete_handler, sender=ValueSet)

class ValueSetMember(RecordOwner):
    """
    Children of a Value Set.
    """
    # Parent Value Set
    value_set = models.ForeignKey(ValueSet, on_delete=models.PROTECT, verbose_name=_("Value Set"))
    # A unique Value Code
    value_code = models.CharField(verbose_name=_("Value Code"), max_length=200,
        help_text=_("A Unique Value Code within thapp_labelis Value Set"))
    # with a helpful description
    value_description = DescriptionField(verbose_name=_("Description"),
        help_text=_("A helpful description of this Value Code"))
    # up to 5 optional Attributes, with dynamic Choices if Value Set defined a choice
    attribute_1 = CharNullField(verbose_name=_("Attribute 1"), max_length=200,
        help_text=_("Optional Attribute 1 value"))
    attribute_2 = CharNullField(verbose_name=_("Attribute 2"), max_length=200,
        help_text=_("Optional Attribute 2 value"))
    attribute_3 = CharNullField(verbose_name=_("Attribute 3"), max_length=200,
        help_text=_("Optional Attribute 3 value"))
    attribute_4 = CharNullField(verbose_name=_("Attribute 4"), max_length=200,
        help_text=_("Optional Attribute 4 value"))
    attribute_5 = CharNullField(verbose_name=_("Attribute 5"), max_length=200,
        help_text=_("Optional Attribute 5 value"))

    class Meta:
        unique_together = ('value_set', 'value_code')
        ordering = ['value_set', 'value_code']
        verbose_name = _("Value Set Member")
        app_label = 'infra'
        db_table = 'if_value_set_member'

    def __unicode__(self):
        return self.value_description

# Register the audit update handler
pre_save.connect(audit_update_handler, sender=ValueSetMember)
# Register the audit add handler
post_save.connect(audit_add_handler, sender=ValueSetMember)
# Register the audit delete handler
pre_delete.connect(audit_delete_handler, sender=ValueSetMember)

The above defines the models ValueSet and its child model ValueSetMember. It can be found in infra/models/value_set.py. Whenever you find yourself needing to create a new model with just a code and description, resist it and just create a new ValueSet entry. The value_set_code in ValueSet becomes the name of your model while its code-description pairs will reside in ValueSetMember’s value_code and value_description. As an added bonus, each ValueSetMember allows you to define up to 5 attributes, which you can use as you please. A common use of ValueSets is for choices of drop-down lists.

Now if you look at the definition above, you will see the necessary imports to use auditing:

from django.db.models.signals import pre_save, post_save, pre_delete
...
from infra.models.record_owner import RecordOwner
...
from infra.custom.audit_handlers import audit_update_handler, audit_add_handler, audit_delete_handler

These 3 imports will allow you to use the 2 services mentioned above. The 1st and 3rd imports are for auditing while the 2nd is for RecordOwner.

RecordOwner

To use the RecordOwner service, just inherit your model from RecordOwner. Then every time your model instance is saved, who and when it was created and last updated is recorded automatically. We do not normally show these fields in our admin programs. You will have to use a Query Browser tool against your database to view them. You can include these fields in your reports.

Note

How do we know who is the user updating the instance? We cheat by using a MiddleWare class called ThreadLocals. We know that you can get this from request.user but we want to centralise the code with the model so that batch processing programs that execute in the Task Queue will also work. Please have a look at infra/custom/threadlocals.py

Concurrency Control

Besides this we will also increment the Last Version number on updates. What do we use this for? Well, if your model admin form and program inherits from our VersionModelForm and VersionModelAdmin (and VersionTabularInline for inline models) respectively, we will ensure that no 2 users can concurrently update the same model instance.

How can 2 (or more) users update the same instance at the same time? When a model is queried (ie read) and displayed for users to select for update, no database locks is placed on the row (in the table that stores your model instances). So 2 users can view the same set of instances. If they were to select the same instance to update, let’s say the same field to different values, then both of them will be successful (if our concurrency control is not used). Because django admin uses auto-commit, a database lock will be placed on the row for a very short time. The second user need not hit the Save button at the same time as the first user to be able to save her changes, thereby over-writing the updates of the first user.

When you use our VersionModelForm, we will automatically add a validation method that will compare the Last Version number (as queried earlier) with its current value (we perform a quick select against the database). If there are different, we will raise a validation error. When the first user successfully updates the same instance, she would have incremented the Last Version. So when the second user hit Save, the validation would have read the latest version number, which will be different.

You will have noticed that there is a small performance hit taken to requery the model instance on every form validation. From a business point of view, for example, preventing a Bank Account balance from over withdrawn, means that this is a price worth paying.

This works also for inline models (ie child models in a parent-child admin program) if it inherits from VersionTabularInline or VersionStackedInline.

Auditing

Besides the 1st and 3rd imports above, you will need to place the block of code below:

# Register the audit update handler
pre_save.connect(audit_update_handler, sender=ValueSet)
# Register the audit add handler
post_save.connect(audit_add_handler, sender=ValueSet)
# Register the audit delete handler
pre_delete.connect(audit_delete_handler, sender=ValueSet)

right after your model’s definition. You must change the sender parameter to the name of your model class.

It must be mentioned that auditing works independently from RecordOwner, ie your model need not inherit from RecordOwner to use auditing.

We use django’s pre and post save signals to implement our auditing. What the above 3 lines of code does is to tell django to call our handler functions before saving (pre_save), after saving (post_save) and before deleting (pre_delete) a model instance. This works at the model level so it does not matter if the updates happens in an admin program, your own views or batch processing task.

We use 2 generic models to save your audit data. They are infra.AuditHeader and infra.AuditLog. AuditHeader will save the instance primary key and who and when information, while AuditLog will store its changed field values.

When adding a new model instance, we will save only non-null field values in AuditLog. When updating a model instance, we will save only the newly changed field values. When deleting a model instance, we will save only the AuditHeader information.

Viewing Audit History

If you were to run any of our admin programs, you will notice the Deletions button in the Change List. Change List is the page where you see a list of model instances that you can select to change or delete. This is shown below for ValueSet. Notice where the mouse pointer is.

ValueSet Change List page

When you click on Deletions, you should be able to see all the model instances that has been deleted.

ValueSet deletions history

If you wanted to now the entire history of this deleted instance since creation, click on the Show link.

ValueSet instance change history

The same audit history is available for all Maymyo’s models except transactional or log models, which by nature is already an audit model.