Site search on Core dna is an Elasticsearch index that the platform fills and queries, and a page you build. What reaches the index, search views, the search page, product facets, filtered listings that need no index, and when to go beyond the built-in index.
L2
Unit 02 · The search page
Queries, ranking and highlighting
The search pagePage 7 of 20Updated
How a query is broken into words and matched, how results are ranked and what changes their order, and how to highlight matches without opening an escaping hole, including the character references that rich text leaves in the index.
Who this is for. The theme developer answering "why did that match?" and "why did that not match?", and printing highlighted results safely. Afterwards you can predict what a query finds, choose the view settings that rank it well, and print highlights that cannot inject markup.
How text is broken into words
Text fields (names, paths, a post's body, tag and category names, and every search field of type Text) are broken into words on the way into the index, and a query is broken up the same way:
Case and accents are ignored.Café, cafe and CAFE are the same word.
Words are reduced to a stem, with English rules: running and runs match run, and on this site client and clients find the same 15 records. The rules are English whatever the site's language, so a French or German site gets little from them.
English stop words are dropped: the, and, of, to, it and the like. A query made only of them finds nothing, which on this site is what ?q=the and of returns.
Everything from an apostrophe on is dropped from a word: client's is indexed as client, and O'Brien as o.
HTML is stripped.
Filter values are not broken into words. A filter compares the whole stored value, with only case and accents folded, which is why tags.name=News needs the tag's full name.
How the query is matched
The query in q is matched against the view's full text search fields, or against every field when the view lists none. The view's Full text query type decides how:
Match words, the default, finds records that contain any of the words, and scores each by its best-matching field; the Tie breaker adds a share of the other fields' scores. "red wool jumper" therefore returns everything red. Set Minimum should match to 2 or 75% when multi-word queries return too much: on this site "search view" finds 189 records, and 36 with minimumShouldMatch=100%.
Match phrase needs the words in order, with the last one taken as the start of a word: on this site "template reso" finds 6 records and "resolution template" none. It suits search as you type, and is too strict for a full results page.
Fuzzy search, on by default, accepts one wrong letter in a word of three to five letters and two in a longer one; words of one or two letters must match exactly. On this site "templte" finds the same 213 records as "template", and nothing with fuzzy=0. Fuzziness is what keeps typing errors from reaching the zero-result page; switch it off only for views that search codes or part numbers, where a near match is a wrong match.
Match phrase and fuzzy search cannot be combined. Elasticsearch refuses a phrase query with fuzziness, so keep Fuzzy search off on every Match phrase view, and its Changeable off too, so ?fuzzy=1 cannot turn it back on.
Filters, finally, narrow the results without affecting their order: a record either passes a filter or is left out.
How results are ranked
With a query and no sort, results come in order of relevance, which Elasticsearch scores per field: a word that is rare across the index counts for more than a common one, a match in a short field counts for more than one in a long field, and repeated matches add up with diminishing returns. The view's field weights multiply those scores, and weighting the title well above the body is the most useful single setting:
HTMLFull text search fields, as a searchFields value
A sort replaces relevance. Sorting by published:desc orders by date and ignores how well each result matched. Offer "newest first" next to "best match", not instead of it.
Relevance can be sorted on._score is a sort field, so orderBy=_score,published:desc keeps relevance and breaks ties by date; a _score with no direction sorts highest first.
Without a query there is no relevance. Every result scores the same, so a view used without q, for a list or a browse page, needs a sort, or its order is arbitrary.
Tune weights against the searches people make, not against a guess (Tuning relevance).
Highlighting
List the fields to highlight on the view, or pass highlightFields if it is changeable (a value in the URL replaces the view's list). For each of those fields, a result then carries up to five short fragments of text around the matches, with each matched word wrapped in <em>. Two rules decide whether a fragment comes back at all:
Only a field the query searched is highlighted. On this site nativeText is in the view's highlight list but not among its search fields, so it never comes back highlighted; add it to searchFields and it does.
Only a Text field has words to highlight.
getHighlightText('name') returns the fragments joined with a comma and a space, and falls back to the plain field when there was no highlight; getHighlightTexts() returns the list. Fragments are excerpts, so a highlighted body field doubles as a result snippet.
Escaping highlighted fragments safely
A fragment is the stored text with <em> tags put into it, and nothing in it is escaped. That leaves two wrong ways to print it:
Raw: the <em> works, and so does any other markup in the text. Record names are indexed exactly as typed, so a product called <b>Sale</b>, or any name containing < or &, breaks the page or injects into it.
Escaped: the page is safe, and every highlight shows as a literal <em>.
The right way is both: escape everything, then turn the escaped highlight tags back into markup. Printing them as <mark> also gives the highlight its proper meaning:
{% set title = item.getHighlightText('name') %}
<a href="{{ item.getText('fullPath')|e }}">{{ title|e|replace({'<em>': '<mark>', '</em>': '</mark>'}) }}</a>
Neither engine escapes output on Core dna, so the order of the filters is what makes this safe: escape first, then replace the two exact strings that escaping made of the highlight tags. Nothing typed into a record can produce an <em> that survives, because escaping has already turned every < into < before the replacement looks for one.
The query printed back on the page, in "results for…" or in the search box, is visitor input from the URL and must be escaped too. A search box whose value prints the query raw lets anyone craft a link that breaks out of the attribute and adds markup to your page.
Character references in indexed text
Text from a rich-text area is indexed with its tags removed but with the character references the editor wrote left as they are: &, <, . This fragment comes from this site's index, from a page that mentions code:
JSON
"Each <em>facet's</em> parameter name is fields.<identifier>, which $<em>facet</em>->getFilter() returns."
Escaped as it stands, it prints fields.<identifier> on the page. For a snippet from a rich-text field, turn the references back into characters first, then escape, then restore the highlight:
{% set snippets = item.getHighlightTexts('fields.content') %}
{% if snippets %}
<p class="snippet">
{% for snippet in snippets %}
{{ loop.first ? '' : ' … ' }}
{% set text = snippet|replace({' ': ' ', '<': '<', '>': '>', '"': '"', ''': "'", '&': '&'}) %}
{{ text|e|replace({'<em>': '<mark>', '</em>': '</mark>'}) }}
{% endfor %}
</p>
{% endif %}
Smarty uses PHP's htmlspecialchars_decode(), which the template security policy allows; Twig has no such filter, so the template replaces the same references itself. Both leave the text plain, and the escape that follows makes it safe again. Fields filled from text and textarea areas, and the record's name, hold characters as typed and need only the escape. That is one more reason to fill the field you print whole, such as a summary, from a plain-text area (Two record types, one field).