How-to library · Developer docs

How-to library

Task-shaped pages: a title that starts with a verb, the situation in a few lines, the steps, the working code in both engines, and why this way rather than another. Found from a search, done, and left.

How-to APIs and JavaScript

Import records so that running the import twice changes nothing

Published

Keep the source system's key in externalId, look each record up by it, update when it is found and create when it is not, and build the content from the source the same way every time. A rerun on unchanged data finds every record and adds no revisions.

Context

Context. Imports fail halfway and get run again, or run every night. Without a key that ties each site record to its source, a rerun duplicates everything it had already done. Most record types carry an externalId for exactly this: a text field of up to 190 characters that the platform stores and filters on and otherwise leaves alone. Pages, blog posts, custom entity records, products, product and content categories, menus and menu elements, users, people, forms, files, orders and invoices have one.

Before you start

A key for each source record that never changes (the source's id, not its name), credentials for a backend user who may create and edit the record type, and a way to make sure only one run of the import happens at a time.

Steps

  1. Give each key a prefix for its system, crm-2001, so keys from two systems never collide and the import can find its own records later.
  2. Look each source record up: GET /cdna-api/custom-entities/apiSandboxes?externalId=crm-2001&properties={id}. A totalCount of 0 means create, 1 means update. More than 1 means stop and report it: externalId is not enforced as unique, so more than one record can hold the same key.
  3. Create with POST, sending externalId with the content.
  4. Update with PUT on the record's id, with the content built from the source by the same function every time.
  5. Do nothing special for records that have not changed. When a write's new revision is identical to the record's latest one, the platform drops it, so an unchanged record keeps its revision and its history stays short.
  6. Run one import at a time. Two runs that look up the same new key at the same moment both find nothing, and both create.
  7. To handle records removed at the source, list the site's records with %externalId=crm- (contains), compare with the keys in this run, and unpublish or delete the rest.

Code

HTTP A real run on this documentation site's sandbox custom entity 
GET /cdna-api/custom-entities/apiSandboxes?externalId=crm-2001&properties={id}
# 200 {"totalCount": 0, "data": []}                                   -> create

POST /cdna-api/custom-entities/apiSandbox?properties={id,externalId,activeRevision{id}}
{"externalId": "crm-2001", "revisions": {"name": "Imported from the CRM", "activeRevision": true, "areas": {"summary": "First import."}}}
# 200 {"id": 7, "externalId": "crm-2001", "activeRevision": {"id": 19}}

GET /cdna-api/custom-entities/apiSandboxes?externalId=crm-2001&properties={id}
# 200 {"totalCount": 1, "data": [{"id": 7}]}                         -> update

PUT /cdna-api/custom-entities/apiSandbox/7?properties={id,externalId,activeRevision{id}}
{"externalId": "crm-2001", "revisions": {"name": "Imported from the CRM", "activeRevision": true, "areas": {"summary": "Second import, with a change."}}}
# 200 {"id": 7, "activeRevision": {"id": 20}}                        a change: a new revision

# the same PUT again
# 200 {"id": 7, "activeRevision": {"id": 20}}                        no change: still revision 20

GET /cdna-api/custom-entities/apiSandbox/7?properties={id,revisions{id}}
# 200 {"id": 7, "revisions": [{"id": 19}, {"id": 20}]}
JavaScript import.js 
// import.js: create or update by external id (uses the coredna() helper from the OAuth how-to)
import { coredna } from './coredna.js';

const ONE = 'custom-entities/apiSandbox';     // a record
const MANY = 'custom-entities/apiSandboxes';  // the list

function revisionFrom(source) {               // the same source always builds the same revision
  return {
    name: source.name,
    activeRevision: true,
    areas: { summary: source.summary ?? null },
  };
}

export async function importOne(source) {
  const externalId = 'crm-' + source.id;
  const found = await coredna('GET', MANY, { query: { externalId, properties: '{id}' } });
  if (found.totalCount > 1) throw new Error(externalId + ' is on ' + found.totalCount + ' records');
  const body = { externalId, revisions: revisionFrom(source) };
  if (found.totalCount === 0) {
    const created = await coredna('POST', ONE, { query: { properties: '{id}' }, body });
    return ['created', created.data.id];
  }
  const id = found.data[0].id;
  await coredna('PUT', ONE + '/' + id, { query: { properties: '{id}' }, body });
  return ['updated', id];
}

// one run at a time, one record after another:
// for (const source of rows) console.log(source.id, ...(await importOne(source)));

Why this and not that

Not a match on the name or the slug: names get edited on the site and slugs are made unique, so a rerun stops finding records and creates copies. Not a patch that applies a change, such as an append: a rerun after a partial failure applies it twice. Build the content from the source instead, and the result is the same however often it runs. Bear in mind that a whole revision replaces every layout area: the layout areas it does not send are emptied. If editors own some areas of an imported record, send a patchRevision that sets only the import's areas with areaContent(name); it repeats just as safely, and an identical one is dropped in the same way (tested on a page). And not parallel workers on one import: the lookup and the create are two requests, and nothing on the platform ties them together.

Go deeper
GuideRetries and writes that may have applied GuideWriting the client layer GuideWhen a custom entity beats a page How-toChange one area of a page without touching the rest ReferenceCreate a record: API reference

Was this page helpful?

Your answer goes to the docs team.