Crawl a docs site

The crawler is one of three ways in — it reads sitemaps and server-rendered HTML, and turns pages into hierarchy records. It does not run JavaScript, so a page's searchable content must be present in the server-rendered HTML.

Starting a crawl

Paste your URL in the dashboard, or run:

npx @suo/cli crawl --url https://docs.example.com

suo looks for sitemap.xml at your domain root first. If it finds one, every URL listed is a candidate page. If it does not, the crawler follows links starting from the URL you gave it, staying on the same domain.

Include and exclude paths

Adjust which paths are crawled with chips in the dashboard, or in suo.config.ts:

export default defineConfig({
	index: 'docs',
	source: {
		type: 'crawl',
		url: 'https://docs.example.com',
		include: ['/docs/**'],
		exclude: ['/docs/internal/**', '/docs/*/changelog'],
	},
});

A path excluded here is skipped before a request is even made.

suo saves the include and exclude paths of your last crawl on the index and uses them again for the daily re-crawl and for retries, so a scheduled crawl never reaches pages you excluded by hand. Start a crawl with new paths to replace them, or with empty lists to clear them.

What becomes a record

For each page, the crawler reads the <title>, the heading structure (h1–h6) and the body text in between, and produces one record per heading section, each carrying its place in the page's hierarchy (lvl0–lvl6). Navigation chrome, footers and anything marked data-suo-ignore are skipped.

<main data-suo-content>
	<h1>Crawl a docs site</h1>
	<p>The crawler is one of three ways in…</p>
	<h2 id="include-and-exclude-paths">Include and exclude paths</h2>
	<p>Adjust which paths are crawled…</p>
</main>

Wrap your main content region in data-suo-content to tell the crawler exactly where the real content starts, instead of guessing from the page's layout.

Per-URL states

While a crawl runs, each URL is one of:

• Queued — not yet fetched.
• Indexed — fetched and its records written.
• Skipped (with a reason) — matched an exclude rule, blocked by robots.txt, or returned a non-indexable response (a redirect chain, a noindex tag).
• Failed (with a reason) — a network error, a timeout, or a non-2xx status.

The dashboard's page counter is always the live count of pages actually indexed so far, never a percentage estimate.

Re-crawling

suo re-crawls every verified site once a day, at 03:17 UTC. To pick up a deploy sooner, choose Start crawl on the Sources page, or trigger a crawl from CI after a deploy with suo crawl. A re-crawl only rewrites records whose content actually changed; unchanged pages keep their existing version and are not re-ranked.

If a crawl has failed pages, Retry failed starts a new crawl run for just those URLs. It follows robots.txt, your include and exclude paths and crawl-delay like any other crawl, and the new run has its own page in the dashboard.

Crawl limits in the beta

The daily allowances count from 00:00 UTC. A start that would go over answers quota_exceeded with the seconds until the allowance resets. Starting a crawl while another crawl of the same domain is running answers crawl_running, whichever index it belongs to.

A sitemap that lists more than 10,000 pages is crawled up to the first 10,000, and the run is marked as capped. List the sections you care about most first, or use include paths to narrow the crawl.

What the crawler will fetch

Everything the crawler requests has to be on your verified host or one of its subdomains: robots.txt, every sitemap (including the entries of a sitemap index), every page, and every redirect along the way. A sitemap or a redirect that points to another host is ignored. In production each request is https on port 443, so a redirect to plain http or to another port is not followed either.

The crawler also bounds what it reads:

• robots.txt: the first 500 kB.
• A sitemap: 50 MB after decompression, read within 60 seconds, with at most 200 sitemaps and three levels of sitemap index in one crawl.
• A page: 2 MB, read within 10 seconds.

Being polite

The crawler honours crawl-delay from robots.txt up to 30 seconds and waits that long between requests, including between one group of pages and the next. A longer value is treated as 30 seconds. To keep a run from outlasting its day, a site that asks for a long delay is crawled only as far as fits in 20 hours: 2,400 pages at the full 30 seconds.

Stored copies of your pages

suo keeps one copy of each crawled page, so an index can be rebuilt without fetching your site again. A page's copy is written again only when its content changes. A copy is removed after the next complete crawl in which the page no longer appears in your sitemap or is excluded by robots.txt or your paths, and when you delete the index. The newest copy of every page that is still listed is always kept, however long ago it was fetched. Per-crawl page lists older than 90 days are dropped, except those of the newest crawl of each index.

Crawler identity

The crawler only reads domains you have verified, identifies itself with its own user agent, obeys robots.txt and crawl-delay, and never sends credentials. suo checks about once a week that the proof is still in place. See Security for the full crawler policy.