Drupal Queue API example: enqueue in form submit handler, dequeue when cron runs Drupal Queue API example: enqueue in form submit handler, dequeue when cron runs

Posted by jstrecker on 2012.03.14 @ 21:39

Filed under:

Need to store up some items and process them en masse in a cron job? That’s a perfect use case for the Queue API (for Drupal 7+) or the drupal_queue module (the backport for Drupal 6).

Concrete example: One of our consulting clients was using a synchronous webservice for their retail store locator. We needed to create a form where store owners could change whether they’re listed in the store locator. When a store owner submits the form, should she have to wait around while our site sends her change to the store locator webservice? We didn’t think so. So we decided to use Drupal’s Queue API, which lets us save up all the store owners’ changes and send them to the webservice in a batch.

Here’s a simple example of adding items to a queue in a form submit handler and processing them in the next cron run.

A simple example

First, we tell Drupal that we’ll be using a queue called example_queue and that, when cron runs, the _example_print_item function should be called for each item in example_queue.

/*
 * Implements hook_cron_queue_info().
 */
function example_cron_queue_info() {
  $queues['example_queue'] = array(
    'worker callback' => '_example_print_item',
  );
  return $queues;
}
 
function _example_print_item($dequeued_item) {
  drupal_set_message(t('Dequeued: %item', array('%item' => $dequeued_item)));
}

Now, we just have to create a form that lets the user put items into the queue.

/*
 * Implements hook_menu().
 */
function example_menu() {
  $items['enqueue'] = array(
    'title' => t('Enqueue'),
    'type' => MENU_CALLBACK,
    'page callback' => 'drupal_get_form',
    'page arguments' => array('_example_form'),
    'access callback' => TRUE,
  );
  return $items;
}
 
function _example_form($form_state) {
  $form['item_to_enqueue'] = array(
    '#type' => 'textfield',
    '#title' => t('Item to enqueue'),
    '#default_value' => date('H:i:s'),
    '#required' => TRUE,
  );
  $form['submit'] = array(
    '#type' => 'submit',
    '#value' => t('Submit'),
  );
  return $form;
}
 
function _example_form_submit($form, &$form_state) {
  $item_to_enqueue = $form_state['values']['item_to_enqueue'];
  $queue = DrupalQueue::get('example_queue');  // Drupal 7
//$queue = drupal_queue_get('example_queue');  // Drupal 6
  $queue->createQueue();
  $queue->createItem($item_to_enqueue);
}

A complex example

For an example that exercises more of the features of queues, check out the excellent Examples For Developers module.

A hook that gets executed just before queues are processed

For the store locator webservice I mentioned earlier, each call to the worker callback would send a message to the webservice with one store owner’s change. But before sending these messages, we needed to log in to the webservice.

The Queue API provided just the hook we needed: hook_cron_queue_info_alter(). It gets called during a cron run, just before Drupal starts calling the worker callbacks on the enqueued items.

An enqueued item may get processed more than once

From the Queue API documentation:

There are two kinds of queue backends available: reliable, which preserves the order of messages and guarantees that every item will be executed at least once. The non-reliable kind only does a best effort to preserve order in messages and to execute them at least once but there is a small chance that some items get lost… regardless of the queue being reliable or not, the processing code should be aware that an item might be handed over for processing more than once (because the processing code might time out before it finishes).

Unless you’ve gone out of your way to change the type of queue your system is using (by setting the variable queue_class_$name, where $name is the name of your custom queue implementation), you don’t have to worry about the first part; the default queue is reliable. But you do need to be aware that your worker callback might get called more than once per item.

There’s some confusion in the docs about createQueue. In your example, you call it unconditionally every time you’re about to add an item to the queue. This is probably because you’re using a default SystemQueue rather than a custom queue. Other examples recommend only calling createQueue once in your module’s install (or update) hook. Do you think that is a better approach or is it perfectly valid to call every time since it’s probably just a no-op if you’re using the default SystemQueue? Thanks!

DrupalQueue::get is a factory method, which means that it’s purpose to return an object of an arbitrary class. It is correct that you do not need to call DrupalQueueInterface::createQueue() every time you create an item unless you’re using your own implementation (or SimpleTest).

If you do not define the Drupal variable queue_class_QUEUENAME, then Drupal will use the default. This will be SystemQueue.

Actual Answer

  • SystemQueue::createQueue() and MemoryQueue::createQueue() do nothing so it may be called as much or as little as needed.
  • It does not make sense to call createQueue on enable or install because the Queue backend could be changed arbitrarily by another module that is installed later.
  • Most developers say that it IS safe to call createQueue whenever even if there are other implementations like MongoDBQueue::createQueue() that do something.
  • Very few people implement DrupalQueueInterface so no standard practice has developed. I recommend that those who do implement that class make sure that DrupalQueueInterface::__construct() creates the queue if it does not exist.
  • So in all likelihood you will not run into any issue calling createQueue() or not calling createQueue().