AI indexing strategy

This site is written to be retrieved from, not only read. That is why every page carries structured frontmatter and why headings are required to be meaningful: both become part of the retrieval unit.

Nothing here is speculative about the retrieval layer itself — Honcho already holds board memory, and documentation is a second corpus with the same shape.

Chunking

Split on ## headings, not on a fixed character count. A heading is a statement about where one idea ends, made by the person who understood it. A 600-character window is a statement about nothing, and it routinely cuts a procedure in half so that retrieval returns steps 1–3 and the answer is in step 4.

	value	why
Target chunk	400–800 tokens	Roughly one ## section. Large enough to carry a whole procedure; small enough that a match is specific
Hard maximum	1,200 tokens	Split a longer section at ###, then at paragraphs. Beyond this the embedding averages away the specific thing that would have matched
Minimum	80 tokens	Merge upward into the parent section. A four-line chunk retrieves on a keyword and answers nothing
Overlap	1 sentence	Enough to carry a pronoun's antecedent across the seam. Larger overlaps inflate the index and return near-duplicate chunks that crowd out genuine matches

Never split a table or a code block. A row without its header is worse than absent — it retrieves confidently and means something different.

Prepend the breadcrumb to every chunk's embedded text: Meetings > Board packs > Publishing. A chunk reading "Select the papers you want, then choose Publish" is ambiguous alone and unambiguous with its path.

Metadata stored with every chunk

Straight from frontmatter, plus what the chunker knows:

{
  "url": "https://docs.sanctumboard.com/meetings/board-packs/overview#publishing",
  "title": "Board packs",
  "heading_path": ["Board packs", "Publishing"],
  "category": "help",
  "product_area": "meetings",
  "audience": "customer",
  "last_reviewed": "2026-08-04",
  "version": "1.0",
  "content_sha256": "…",
  "chunk_index": 3
}

audience is the important one. It is a filter, not a label. Any customer-facing retrieval must exclude internal, and that exclusion happens in the query rather than by maintaining a second index — one corpus with a filter cannot drift out of step with itself, whereas two indexes silently do.

content_sha256 makes re-indexing incremental: unchanged chunks are skipped, so a typo fix does not re-embed the site. It is also how you detect a page that changed without its last_reviewed moving.

Retrieval

Filter first, then rank. A support agent asking about credits should search product_area: billing OR pam rather than the whole corpus — filtering is free and ranking is not, and a corpus-wide search returns the plausible before the correct.

Cite the URL and the heading, always. A documentation answer that cannot be checked is the same failure as an unsourced legal citation, and this product already refuses to do that with law. Pam citing docs must be held to the standard Pam citing legislation is held to.

Re-indexing

Triggered by merge, not by a schedule. A cron job re-indexing nightly means documentation is wrong for up to a day after it was fixed, which is precisely the window when someone is reading it because it was wrong.

PR merged to main
  └─ touched apps/docs/**
       ├─ build (broken links fail here, before anything is indexed)
       ├─ extract chunks + frontmatter
       ├─ skip chunks whose content_sha256 is unchanged
       ├─ embed and upsert the remainder
       └─ delete chunks for pages that no longer exist

Deletion matters as much as insertion. A page removed from the site but left in the index is the worst possible state: retrievable, confident, and describing something that no longer exists. Reconcile by URL on every run.

What this is not, yet

Local keyword search ships first. It is honest about being a keyword index, and it has no infrastructure to keep alive. Semantic search replaces it when the corpus is large enough that keyword search visibly fails — which is a judgement to make from search logs, not in advance.

The pages do not change when that happens. That is the point of putting the metadata in from the start.