How-to library · Developer docs

How-to library

Task-shaped pages: a title that starts with a verb, the situation in a few lines, the steps, the working code in both engines, and why this way rather than another. Found from a search, done, and left.

How-to Jobs and automation

Loop over a set of records without timing out

Published

Run the work in the background, in bounded chunks selected by what is still to do, in a parallel loop that contains its failures, on a schedule that repeats until nothing is left.

Context

Context. A job needs to do something to many records: update every product in a category, send each of a thousand members to a CRM, expire last year's offers. Done in one batch, it runs into limits: a synchronous job runs inside a web request, which has one; a request per element at up to 60 seconds each adds up; and a batch waits for its loop for at most eight hours. Handle it the same way every time: run the work in the background, in bounded chunks, and let each run pick up what the last one left.

Steps

  1. Make the job asynchronous. Only a job that answers a web request needs to be synchronous, and in one of those a loop runs its elements one after another inside the request. A long loop in an endpoint is the most common timeout; move the work to a job of its own, started by Trigger job, and answer the request at once.
  2. Select a bounded chunk. A Retrieve records action with a limit (and a criterion that excludes what is already done), or a Loop whose template source adds at most that many elements. Choose the size from a measurement, as on Scheduled work.
  3. Loop over it in parallel, with wait done and stop on error off, so each element runs as its own background job and one failure does not stop the rest. Use sequential processing only when the elements depend on each other, or when the other system has a strict rate limit.
  4. Mark each element done as the loop's last action: an Update record on the loop's record that sets whatever the selection excludes (a status, an externalId, a date).
  5. Run it on a schedule until nothing is left: a job every 15 minutes that selects the next chunk finishes a backlog of any size without a single run growing.
  6. Test with a limit of 5, read the batch log (one row per loop element, each with its own status), then raise the limit.

Code

A loop source template that selects the next 200 products not yet sent, using query so it can use any criteria, and adds them as ids:

Template code 
{$pending = $query->findBy('Products\Entities\Product', ['externalId' => null], ['id' => 'ASC'], 200)}
{foreach $pending as $product}
  {$loopHelper->addElement(['id' => $product->getId()])}
{/foreach}
{$customData->set('chunk-count', count($pending))}
{% set pending = query.findBy('Products\\Entities\\Product', {'externalId': null}, {'id': 'ASC'}, 200) %}
{% for product in pending %}
  {% do loopHelper.addElement({'id': product.getId()}) %}
{% endfor %}
{% do customData.set('chunk-count', pending|length) %}

Inside the loop, the actions reach the product with source record id and {$loopRecord.id}: the HTTP request that sends it, then an Update record that sets its externalId from the response, which is what takes it out of the next selection.

Why this and not that

Ids as elements, not records. A template source adds elements through loopHelper and ignores what it prints, and an element must be a plain value or array: a record added directly does not arrive as a record. Select by state, not by position. An offset (records 200 to 400) shifts when records change between runs and skips or repeats some; "not yet sent" cannot. No single enormous loop. There is no limit on the number of elements, but a batch waits for its loop for eight hours at most and then never reaches the actions after it, and a failed batch re-runs from the start. Two hundred elements that finish in minutes and repeat every quarter hour recover from anything; twenty thousand in one batch recover from nothing.

Go deeper
GuideConditions: if, else if and else GuideScheduler tasks and the job queue

Was this page helpful?

Your answer goes to the docs team.