Guide 09 · Developer docs

Automation, jobs and integration

A job is a trigger, conditions and an ordered list of actions, set up under Customization › Orchestration in the admin; jobs were called hooks before the module was renamed, and the API's routes keep that name. What starts a job, what it can do, branches and loops, email, scheduled work, integrations with other systems, and every event and action.

L2–L3
Unit 06 · Scheduled jobs

Finding out a scheduled job stopped

A scheduled job that stops needs its own alert: nothing built in reports a stopped or missed run. The ways it can stop, and two patterns to catch it: an outbound heartbeat and an inbound health check.

Who this is for. Whoever is responsible for a scheduled job a client relies on. After this page you can say exactly how your team would find out that a nightly job stopped, and have an alert that proves it.

How a scheduled job can stop

A scheduled job can stop in several ordinary ways:

  • it was imported to production without its schedule or its Run on site, which do not travel with an export, or was never published after the import (What travels and what does not);
  • its Run on site was cleared, or its Recurring until date passed, after which its task is never due again;
  • it fails every night, and its batch persistence is never or success, so the failures leave no trace in Batch Logs;
  • the scheduler was not running in its minute, which is not caught up for a recurring schedule (Missed runs);
  • it runs and succeeds but does nothing, because the watermark is ahead of the data or the selection is wrong (Timezones and the watermark).

Watch it from outside the thing that stopped, with something that checks that work happened, not only that a run did. Two patterns, both small; use at least one for every job a client relies on.

An outbound heartbeat

The job's last action is an HTTP Request to a heartbeat monitor, a service that expects a request at an interval and alerts when one does not arrive. Set its expected interval to the schedule plus a margin (25 hours for a nightly job). A run that fails before the end, a run that never started and a schedule that disappeared all end the same way: no request, an alert. Put the request after the loop, with the loop's wait setting on success, and include the count of records handled so the monitor can alert on zero too.

URL https://heartbeat.example.com/ping/{$siteData->get('heartbeat-product-sync')}, method POST, body:

Template code
{$ping = ['handled' => $customData->get('chunk-count')]}
{$ping|json_encode}
{{ {'handled': customData.get('chunk-count')}|json_encode }}

The count is the chunk-count the loop source template stores (Watermarks and chunking). The body is built as a value and encoded, so it is valid JSON whatever the value holds.

An inbound health check

The job's last action records its success in site data; a small synchronous web-event job reports on it; an uptime monitor calls that web event every few minutes and alerts on anything but 200. This one works even when the queue is stuck: a synchronous web event runs inside the web request that calls it, while an asynchronous run waits for a background worker. The job's last action, a Render Template:

Template code
{$now = date_create('now', timezone_open('UTC'))}
{$siteData->set('product-sync-last-success', $now->format('c'))}
{% do siteData.set('product-sync-last-success', 'now'|date('c', 'UTC')) %}

The health job is a GET web event with synchronous on, and one HTTP Response action. Its status template:

Template code
{$last = $siteData->get('product-sync-last-success')}{if $last && date_create($last) > date_create('-26 hours')}200{else}503{/if}
{% set last = siteData.get('product-sync-last-success') %}{% if last and date(last) > date('-26 hours') %}200{% else %}503{% endif %}

Keep the status template on one line, so it renders to the bare code (HTTP Response). Its body template:

Template code
{$health = ['lastSuccess' => $siteData->get('product-sync-last-success')]}
{$health|json_encode}
{{ {'lastSuccess': siteData.get('product-sync-last-success')}|json_encode }}

The response says only when the job last succeeded. Give the web event a long, random slug, as for any web event called from outside the site.

Where the alert goes, and testing it

Whichever pattern you use, send the alert to a shared channel or a team inbox, never to one person, and write in the job's Description which monitor watches it. Then test the monitor once: unpublish the job for a night and confirm the alert arrives.

Go deeper

Was this page helpful?

Your answer goes to the docs team.