Guide 09 · Developer docs

Automation, jobs and integration

A job is a trigger, conditions and an ordered list of actions, set up under Customization › Orchestration in the admin; jobs were called hooks before the module was renamed, and the API's routes keep that name. What starts a job, what it can do, branches and loops, email, scheduled work, integrations with other systems, and every event and action.

L2–L3
Unit 04 · Branches, loops and re-runs

Idempotency and replay

Every job that writes eventually runs twice: a re-run after a fix, a partner sending the same call again, a retried loop element, an overlapping schedule. The external-id pattern that makes the second run change nothing, as job actions, and the test that proves it.

Who this is for. Anyone whose job creates records or writes to another system. Afterwards your jobs can be re-run from the log without duplicating anything, here or elsewhere.

Why every job runs twice

A job is idempotent when a second run with the same input leaves everything as the first run did. On this platform the second run is not hypothetical:

  • Rerun Batch starts again from the first action, with the same event (Reading a job run);
  • a retried parallel loop element runs all its actions again;
  • a partner that did not see your answer sends the same web event again;
  • a schedule fires while the previous run is still going.

Without idempotency each of these duplicates records on the site and in every system the job writes to.

The external-id pattern, as job actions

The pattern itself belongs to the APIs guide (Writing the client layer): give every record a job creates an externalId built from what started it, and look it up before writing. In a job it is three actions:

  1. Retrieve Records find-existing: the entity, criteria externalId equal to the key, limit 1.
  2. Condition exists, template type, on runData.get('find-existing').record: if it matches, Update Record with source Action, find-existing, record index 0; else Create Record with that externalId in its data.
  3. For a write to another system, send your own stable id with it (the record's id, the order's uuid) as the other system's idempotency key, and accept its "already exists" answer as success.

The key must be the same on every run for the same input, and different for different inputs. Build it from the source and its id:

Template code
{* the key, for a record made from a CRM's web event *}
crm-contact:{$event->getBodyData()->id}

{* the key, for a record made from a form submission *}
form-submission:{$event->getRecordId()}
{# the key, for a record made from a CRM's web event #}
crm-contact:{{ event.getBodyData().id }}

{# the key, for a record made from a form submission #}
form-submission:{{ event.getRecordId() }}

The full find-or-create, with the record index and the to-many merge that catch people, is worked through on Creating records from a submission; the same rules for API clients are on Retries and writes that may have applied.

Writes that are idempotent already

  • Setting a value is idempotent; adding to one is not. Write "status": "synced", not "one more".
  • A file at a fixed path is replaced on every run, so a nightly export does not pile up.
  • A watermark that moves only after success makes a failed run take the same chunk next time (Watermarks and chunking).
  • Email is never idempotent. A re-run sends the message again. Put a condition before Send Email that checks a marker the job wrote when it sent the first one, or accept the duplicate knowingly.

The test

Run the job twice against the same input, from the admin or by sending the same request twice, and compare:

  1. The second run creates nothing: no new records on the site, nothing new in the other system.
  2. Every record is as the first run left it.
  3. The second run's log shows the existing record found and updated, or the other system's "already exists" accepted.

A job that fails the test must not be published on an event that can repeat, which is every event.

Go deeper

Was this page helpful?

Your answer goes to the docs team.