Site search on Core dna is an Elasticsearch index that the platform fills and queries, and a page you build. What reaches the index, search views, the search page, product facets, filtered listings that need no index, and when to go beyond the built-in index.
L2
Unit 05 · Relevance, mixed results and external engines
Keeping an external index in sync
Relevance, mixed results and external enginesPage 20 of 20Updated
An external search engine holds a copy of the site's content, and is only as good as how current that copy is. Push each change from record events with a job, catch what the push missed with a scheduled reconciliation job, and keep a full resend for recovery.
Who this is for. The integrations engineer connecting a site to an external search engine. Afterwards you can build the job that keeps the engine current within seconds of each change, the scheduled job that corrects what it missed, and the checks that notice when either stops.
Push, reconcile, resend
The instinct is a nightly full resend of every record, and it is the wrong default: it is slow, it leaves the engine up to a day behind, and it sends everything again whether it changed or not. Instead:
Push for currency. A job sends each change to the engine as it happens, so the engine trails the site by seconds.
Reconcile for correctness. A scheduled job sends whatever changed since its last run, so a push that failed or never fired is corrected within one cycle.
Resend only to recover: for a new engine, a new environment or a corrupted copy.
Jobs live under Customization › Orchestration. Before the module was renamed they were called hooks, which is why the API routes (/cdna-api/hooks/…) and the code still use that name. How a job is built is the subject of the jobs guide (Events, what starts a job).
Push from record events
Create one job with an entity event per searchable record type, for example EntityPages\Entities\Page, with the Method set to Created, Updated and Deleted in turn. A scheduled go-live and a scheduled expiry reach the job as Updated events too, sent by the platform's publishing task, so content that goes live on a schedule is pushed without a special case.
The job decides between two HTTP request actions with a condition: send the document when the record exists and is published, remove it otherwise. The condition's template sets the result:
{% do result.setMatching(event.getMethod() != 'DeleteRecord' and event.getRecord().isPublished()) %}
When it matches, an HTTP request action sends the document. Its URL and headers are templates; its method is fixed on the action. Here the method is PUT, the URL https://search.example.com/indexes/site/documents/{$event->getRecordId()}, a header Authorization: Bearer {$siteData->get('search-engine-key')}, and the body this template:
{% set record = event.getRecord() %}
{% set body = record.getActiveRevision().getAreaText('body')|default('') %}
{% set text = body|striptags|replace({' ': ' ', '<': '<', '>': '>', '"': '"', ''': "'", '&': '&'}) %}
{{ {'id': event.getRecordId(), 'type': event.getEntityName(), 'title': record.getName(), 'url': record.getFullPath(), 'body': text|trim}|json_encode }}
The body is built as an array and encoded once, so every value is escaped for JSON however it reads. The text is taken from the record's active revision with its tags removed and its character references turned back into characters, for the reason given under Character references in indexed text. Send what a visitor may see: if the site restricts some pages to members, add that to the condition, or the engine will show them to everyone.
Otherwise, a second HTTP request action removes the document, with the method DELETE and the same URL. The record is gone by the time a Deleted event's job runs, so $event->getRecord() has nothing to give, but $event->getRecordId() still holds its id. An engine that keys documents by something else, such as the URL, needs that value captured before the delete: list it under the event's Pre-Event properties, and read it in the action from $event->getPreEventData().
A push is not a guarantee. A request fails while the engine is down and the retries run out. A record is written by an asynchronous or scheduled job, an import or a script on the server: saves made outside a web request fire no record events, so no job runs for them. A global component, a tag or a category changes, and every page that shows it is out of date without an event of its own. Each leaves a document stale until the reconciliation job catches it up.
The same rule makes a job that writes back to the site safe from triggering itself, as long as it stays asynchronous; a synchronous job does not have that protection (Stop a job from triggering itself).
The reconciliation job
A scheduled job catches up whatever the push missed. Every so often it sends again every record that changed since its last run, so a missed push is corrected within one cycle instead of lasting until a full resend. It is the chunked, watermarked loop of the jobs guide (Watermarks and incremental processing): it keeps the modification time of the last record it sent in site data, selects the next chunk modified after it in a template, sends each record with the same body as the push, and moves the watermark only when the chunk succeeded. Two details are particular to search:
Select by publish date as well. A scheduled go-live changes no modification time, because nothing is saved. Select records whose published date fell since the last run too, or a page that went live on a schedule while the push was failing is never corrected.
Find deletes on a slower cycle. A deleted record cannot be selected by date. Once a day or once a week, compare the engine's list of ids with the site's, and remove what the site no longer has.
Run it every fifteen minutes for near-real-time safety, or nightly if a few hours' lag after a missed push is acceptable. A full resend is the same job with its watermark set back to the beginning: it then works through every record a chunk at a time, which is the safe way to fill a new engine or recover a corrupted one.
Noticing when it stops
A sync that stops is easy to miss: the engine keeps answering, just further behind the site each day it runs unwatched. Monitor both jobs the way any unattended work is monitored, with a heartbeat or a health check that alerts when they stop succeeding (Monitoring: finding out a nightly job stopped). And because the engine can fail or lag, the search page that reads it should show "search is unavailable" when the engine does not answer, not "no results": this page can tell the difference between an outage and a genuine zero, so it should.