Guide 03 · Developer docs

The theme repo and the asset pipeline

Everything a developer changes on a Core dna site lives in its theme repository. This guide covers the repository and its build, printing records in templates, stylesheets and client-side code, and the parts of a site around the main content.

L1–L2
Unit 05 · Around the main content

Robots and the sitemap

Around the main content Page 34 of 35 Updated

What a theme repo's robots.txt is, how each record's robots setting becomes a meta tag, how the platform builds /sitemap.xml, and which settings and records decide what it lists.

Who this is for. Developers preparing a site for search engines at launch. After this page you can ship the right robots.txt, print each record's robots setting as a meta tag, and decide which content the sitemap lists.

robots.txt

/robots.txt is not a platform route. It is a plain file at the root of the theme repo, served as it is like any other static file. Because it is a file, it is the same on every hostname and environment that serves the repo: keep staging copies out of search results with a noindex meta tag (Markup that differs by environment), not with the file.

A live site that allows crawling, with the address of its sitemap:

robots.txt 
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

A theme repo built before launch often carries Disallow: / to keep an unfinished site out of search results. Check the file in the release that goes live.

The robots setting on each record

Every versioned record (page, blog post, book page, FAQ page and question, product and category description) has SEO settings, which a template reads with getSeoSettings(). Their robots select stores one of these keys, which getRobots() returns:

KeyLabel in the adminDirective to print
(empty)Default (recommended)No tag
allIndex & followindex, follow
indexIndex, but don't followindex, nofollow
followDon't index, but follownoindex, follow
noneDon't index and don't follownoindex, nofollow

Map each key to the directive its label describes, and let the environment check from Markup that differs by environment take precedence off production:

Template code 
{$robotsMap = ['all' => 'index, follow', 'index' => 'index, nofollow', 'follow' => 'noindex, follow', 'none' => 'noindex, nofollow']}
{$robots = $page->getSeoSettings()->getRobots()}
{if $environment neq 'production'}
    <meta name="robots" content="noindex, nofollow">
{elseif isset($robotsMap[$robots])}
    <meta name="robots" content="{$robotsMap[$robots]}">
{/if}
{% set robotsMap = {'all': 'index, follow', 'index': 'index, nofollow', 'follow': 'noindex, follow', 'none': 'noindex, nofollow'} %}
{% set robots = page.getSeoSettings().getRobots()|default('') %}
{% if environment != 'production' %}
    <meta name="robots" content="noindex, nofollow">
{% elseif robots in robotsMap|keys %}
    <meta name="robots" content="{{ robotsMap[robots] }}">
{% endif %}

The sitemap route

/sitemap.xml is a platform route; a theme repo does not write it. The platform builds the sitemap in the background and serves the built file:

  • A built sitemap is served for up to an hour. After that, the next request starts a fresh build.
  • An hourly task rebuilds the sitemap of each site's primary environment, so the live sitemap stays current on its own.
  • On any other environment, such as staging, the sitemap is rebuilt when /sitemap.xml is requested and the built copy is more than an hour old. To check a staging sitemap, request it, then request it again once the build has run.

What the sitemap lists

Each content module has a setting, named generateInSitemap in every module, that decides whether its records are listed:

ModuleRecordsDefault
PagesPagesOn
BlogsBlogs and posts, one setting eachOn
FAQFAQ pages and questions, one setting eachOn
ProductsProductsOn
BooksBooks, units and pages, one setting each ("Generate books in sitemap", "Generate units in sitemap" and "Generate pages in sitemap")Off

Book content is listed once the three Books settings are turned on. Whatever the module setting, a record is left out when:

  • its robots setting is follow ("Don't index, but follow") or none ("Don't index and don't follow");
  • it has no path;
  • its canonical URL points to another site.

So the sitemap and the meta tag agree: a record an editor marks "Don't index" is neither listed nor offered for indexing.

At go-live

Check that the released robots.txt allows crawling, turn on the Books sitemap settings if book content should be found, and request /sitemap.xml on the live hostname to see what it lists.

Go deeper
GuideCanonical URLs and structured data GuideMarkup that differs by environment GuideError templates

Was this page helpful?

Your answer goes to the docs team.