A job is a trigger, conditions and an ordered list of actions, set up under Customization › Orchestration in the admin; jobs were called hooks before the module was renamed, and the API's routes keep that name. What starts a job, what it can do, branches and loops, email, scheduled work, integrations with other systems, and every event and action.
L2–L3
Unit 06 · Scheduled jobs
Watermarks and chunking
Scheduled jobsPage 24 of 31Updated
Process only what changed since the last run with a watermark, select the records in a template, keep the watermark consistent across timezones so it does not reprocess the same records on every run, and size chunks so each run finishes well inside its interval.
Who this is for. Developers writing a scheduled job that works through records incrementally: a sync, an export, a clean-up. After this page you can keep a watermark that survives a failed run and a timezone other than UTC, and choose a chunk size from a measurement.
The watermark
A job that handles only what changed since its last run keeps a watermark: the modification time of the last record it finished. Each run selects records modified after the watermark, oldest first, handles a chunk of them, and moves the watermark to the last one it handled. Keep the watermark in site data, under a key named for the job, and store it as a time and an id, for the reason given under Moving the watermark.
Select in a template, not in Retrieve Records
Retrieve Records criteria are compared as text, so a date criterion such as ">modified": "2026-09-30 02:00:00" cannot be relied on against a date field. A template can build a real date and select with it through query, the query helper a job's templates receive (findBy(entity, criteria, orderBy, limit, offset)), and the loop that handles the chunk can use that template as its source:
{# loop source template: the next 200 products changed since the watermark #}
{% set wm = siteData.get('product-sync-watermark')|default({'at': '2000-01-01T00:00:00+00:00', 'id': 0}) %}
{% set products = query.findBy('Products\\Entities\\Product', {'modified_gte': date(wm.at)}, {'modified': 'ASC', 'id': 'ASC'}, 200) %}
{% set count = 0 %}
{% for product in products %}
{% set at = product.getModified().format('c') %}
{% if not (at == wm.at and product.getId() <= wm.id) %}
{% do loopHelper.addRecordById('Products', 'Product', product.getId()) %}
{% do customData.set('next-watermark', {'at': at, 'id': product.getId()}) %}
{% set count = count + 1 %}
{% endif %}
{% endfor %}
{% do customData.set('chunk-count', count) %}
A template source adds its elements by calling loopHelper and ignores what it prints; each render gets a fresh helper. addRecordById('Products', 'Product', id) loads the product and adds the record itself, so every action inside the loop reads $loopRecord (Twig loopRecord) as the product, the same kind of record Record actions return (The records a job sees). It accepts only the entities Record actions support, and addRecordsByIds() adds several records from a list of ids in one call. Plain values, such as an array of fields, a string or a number, go in with addElement().
Moving the watermark
After the loop, with its wait setting on success, a last Render Template moves the watermark:
{% if customData.get('chunk-count') %}{% do siteData.set('product-sync-watermark', customData.get('next-watermark')) %}{% endif %}
Taking the watermark from the last record handled rather than from "now" also handles a subtler case: a record saved while the job was running has a modification time before the run finished, and a watermark of "now" would step over it.
Timezones and the watermark
The platform stores every date in UTC, and a record's dates come to a template as UTC: getModified()->format('c') gives 2026-09-30T05:22:58+00:00. But the template's clock is the site's timezone: date(), Smarty's date_format, Twig's date filter and a date_create() of a string with no zone all read and write the client's Globals.timezone, in a web request and in a background worker alike. A watermark written in one of those zones and read in the other is off by the difference, on every run. With the site on Sydney time (UTC+10):
Written as UTC wall-clock time, read as site time. The job stores getModified()->format('Y-m-d H:i:s'), 2026-09-30 05:22:58, and the next run reads it with date_create() as 05:22:58 in Sydney, which is 19:22:58 UTC the day before. Every run selects the last ten hours again, reprocessing the same records and sending them to the other system every night.
Written as site time, read as UTC. The job stores date('Y-m-d H:i:s') at 02:00 UTC, which is 12:00 in Sydney, and reads it as 12:00 UTC. The watermark is ten hours in the future, so every record changed in those ten hours is skipped.
On a site whose timezone is UTC neither happens, which is how the mistake passes testing on a development site and appears on the client's.
Store the watermark with its offset
Write it with format('c'), take it from the data rather than the clock, and parse it with date_create(), which honours an offset in the string. A date built this way is converted correctly whatever the site's zone: a bound written as 2026-09-30T15:22:57+10:00 selects the same records as one written in UTC.
Chunking
A run should finish well inside its interval: an hourly job in minutes, a nightly one in well under an hour. A run that handles its records in a parallel loop has a ceiling as well: it checks every ten seconds whether the loop's elements have finished, and waits for them for at most eight hours (The limit is time, not count). Choose the chunk size from a measurement, not a number from elsewhere:
Run the job by hand on a chunk of 20 and read the run in Batch Logs: the loop elements' start and end times give the time per record.
Multiply by the number of records a normal run has to handle. If that fits comfortably inside the interval, one chunk per run is enough; if it does not, the job needs more runs (a shorter interval) or cheaper records, not a bigger chunk.
Size the chunk so that one run takes about a quarter of its interval, which leaves room for a slow day and for the backlog after an outage.
Typical ranges, to check the measurement against: records that only change data on the site handle hundreds per minute, so a few thousand per chunk is reasonable; records that each make an HTTP request are bounded by the other system, often a few per second, and by its rate limit, so chunks of 100 to 500 are usual. A parallel loop is faster only as far as the other system allows; against a rate-limited API use sequential processing. A parallel loop can retry a failed element up to five times, if it is set to.
Committing progress between chunks
Progress is committed between chunks by the watermark. Each run handles one chunk and moves the watermark; the next run starts after it. A backlog therefore drains over several runs by itself, and a run that fails costs one chunk, which the next run takes again. Taking a chunk again is harmless only because each piece of work in it is safe to do twice, which is the rule on Scheduler tasks and the job queue.