Background Task Queue for tasks and reports

The Task Queue will execute tasks submitted by users in the background. These tasks include Application Commands (anything that can be run on the command line) and Reports. After completing each task, it will send a message to the user. All output generated by the task is saved and can be downloaded or viewed by users from their Dashboard’s “Check your Messages” button (on the Toolbar).

The Task Queue is a daemon that can be started automatically during Operating System boot-up. Each Maymyo’s installation will by default start up 2 Task Queue daemons, for Immediate and Queued tasks. The former will execute each task submitted immediately without any queueing while the latter queues task (based on priority and FIFO basis for those with same priority) subject to a maximum number of active tasks.

How many Task Queue daemons to startup is controlled by the configuration file snippets/daemons.tab. You would not normally need to add any more daemons. Adding more daemons on the same server will not improve performance or throughput.

We allow users to submit reports to the Task Queue. Whenever a report is run, we will prompt for the report parameters and allow the user to click on the Run or Queue buttons. The former will run the report interactively (a HTTP Response will be returned to the browser) while the latter will submit the report to a Task Queue and return control back to the user immediately so that she can continue with other transactions. A message will be returned to inform the user which task id is allocated to her report task. Later, when the report is completed, the Task Queue will send her a Message which can be read using the “Check your Messages” button in the Dashboard’s Toolbar. She can also monitor the completion of the report task using the “View your Tasks” button on the same Toolbar.

The Task Queue is also used by the Scheduler daemon for automatically scheduled jobs. Each task of that job will be submitted for execution to the Task Queue(s).

Defining a Task Queue

Below is the definition of the Default Task Queue as loaded by our fixtures.

Maintain Task Queues

The fields of interest are :-

1. Maximum Active Tasks which controls how many task can be active at any one time. Other tasks will queue until an active task completes.
2. Is Disable allows you to temporarily disable the Task Queue from starting.
3. Environment Parameter Set defines which Parameter Set to use for the Task Queue daemon. This is where you would set the PATH, PYTHONPATH and DJANGO_SETTINGS_MODULE variables. Normally during installation, we will change this according to your operating system (ie either Windows or Posix-based).

If you change any of the above fields, it will only take effect the next time the Task Queue daemons are restarted.

Configuring the daemons table

The configuration file that controls which daemons to start up (and shut down) is in snippets/daemons.tab. We show below the section to control the Task Queue:

# Task Queues. Parameters: P1 = Queue Code in TaskQueue, P2 = Execute Mode (Queued or Immediate)
# Should be run in tandem with Scheduler, ie if scheduler runs 7 days a week, Task Queue
# should also run 7 days a week.
tkqq|Y|task_queue.py|N|06:00:00|01:00:00|1|DEF-QUEUE|Q
tkqi|Y|task_queue.py|N|06:00:00|01:00:00|1|DEF-QUEUE|I

Those lines that starts with the # are comments. The tkqq line has | character as separators between fields. Each line will start one daemon instance. There are 9 fields per line :-

1. Unique entry name field, ie tkqq. This entry name is used to name the log and pid files saved in the runtime directory.
2. Start Flag, Y/N to start this daemon.
3. python program to run for this daemon. For Task Queue, this must be as above (task_queue.py). This program can be found in the bin directory.
4. Business Day flag, Y/N to run this daemon only on business days (according to the Application default Calendar). When Y, it will not run on rest days or holidays. (Actually it will run but will be sleeping).
5. Start Time, when this daemon will start working. Above shows that we want the Task Queue to start working at 6am in the morning. This is in 24 hour format.
6. End Time, when this daemon stops working, ie until 1am. So our daemon works from 6am to 1am the next morning, leaving a window of 5 hours (from 1am - 5:59am) which should be used for database backups.
7. Debug mode, 1 (on) or 0 (off). When 1, the log messages will be more verbose. Even security sensitive information will be logged. You may want to turn this off in Production.
8. Task Queue code, the unique code to use, as defined earlier. We use this to get the maximum task and Environment Parameter Set.
9. Queued or Immediate mode for this Task Queue.

The first 7 fields are the same for the other types of daemons. The 8th field onwards are daemon specific. The above shows 2 Task Queue daemons are started, in Queue and Immediate modes respectively.

In the Installation guide, we have already shown you how to start up and shut down the daemons manually and automatically.

Using the Task Queue in your code

You can submit Application Commands or Reports (we call it a Resource) to a Task Queue. Which Task Queue to use depends on the Resource. Each Command may define its own default Task Queue. A Report will get its default Task Queue from its Reporting Engine. When this default is not defined, then you should use the Application default. Your code would probably look like this for a Report:

from infra.objects.task_queue import get_default_task_queue, submit_task
....

# Get Task Queue to submit into
task_queue = app_rep.reporting_engine.default_task_queue or get_default_task_queue()

app_rep is the Application Report instance. You should have prepared the positional and named parameters, plus the conditions dict using infra.objects.app_parameter_set.prepare_parameters after prompting the users to select her report criteria. See infra.views.run_report for an example.

Then submit the report to the Task Queue by:

queued_task = submit_task(task_queue=task_queue, app_resource=app_rep, app_session=app_session,
    pos_params=pos_params, named_params=named_params, conditions=conditions,
    output_format_code=output_format.value_code)

The full parameter list is:

submit_task(task_queue, app_resource, app_session, pos_params=None, named_params=None,
    conditions=None, output_format_code=None, submitter_model=None, instance_id=None)

where task_queue, app_resource and app_session is mandatory. The app_resource should either be an Application Command or Report instance. The app_session is the submitting user’s current Application Session instance. You can always get it by

from infra.custom.threadlocals import get_app_session
....
app_ssn = get_app_session()

Of course, this only works in your views (which can only be run when logged into Maymyo) or for tasks executed using our Task Queue (where we use the app_session of submit_task to simulate a login).

The output_format_code should be passed in for Reports. It should be a Value Code from the Value Set REPORT-OUTPUT-FORMATS.

The submitter_model and instance_id are used only for Scheduled Jobs, ignore them.

Viewing your tasks

If you are a user who has submitted a task, naturally you would want to know where to view the output of the task.

When the task has completed, the Task Queue will send you a completion message. You can view the message using the “Check your Messages” button on your Dashboard’s Toolbar. You can then use the “View your Tasks” button to view its output.

View your Tasks

Just click on the file name link (value-set-list2.pdf in above screen for the 1st task) and then the browser will prompt you on what to do with the download. You can either open or save it. If you open it with the appropriate program, you can send it to your own printer or save it to your own disk. The output file is saved as a Spool Item on the Maymyo server until it is purged (which depends on the configuration in Archiving and Purging). You can download it multiple times (until it is purged).

If the link is not there, it may mean that the task is not completed yet or there has been an error. Scroll further to the right to view the status and error message.