Knowledge
@aidy-dev/knowledge is aidy’s knowledge index. It stores Markdown documents in collections, chunks them by heading, and searches them with BM25 (plus vectors when you give it an embedder). It runs on any SqlDriver from @aidy-dev/sql: Postgres, SQLite or Cloudflare D1. Plugged into aidy, it is one more search source, named knowledge.
npm install @aidy-dev/knowledge @aidy-dev/sqlCreate the index
Section titled “Create the index”import { createKnowledge } from '@aidy-dev/knowledge';import { postgresDriver } from '@aidy-dev/sql/postgres';
const knowledge = createKnowledge({ driver: postgresDriver(process.env.DATABASE_URL!) });await knowledge.migrate(); // creates aidy_knowledge_* tables| Driver | Import | Notes |
|---|---|---|
| Postgres | postgresDriver(url) from @aidy-dev/sql/postgres |
Needs the postgres package (v4). Ranks with ts_rank. |
| SQLite | sqliteDriver(path) from @aidy-dev/sql/sqlite |
Node’s built-in node:sqlite. FTS5 with BM25. |
| D1 | d1Driver(env.DB) from @aidy-dev/sql/d1 |
Cloudflare Workers. FTS5 with BM25. |
The index can share a database with aidy’s store; its tables have their own prefix.
Ingest documents
Section titled “Ingest documents”ingest(collection, documents) adds or replaces documents and returns { updated, unchanged }.
import { glob, readFile } from 'node:fs/promises';
const documents = [];for await (const file of glob('docs/**/*.md')) { const markdown = await readFile(file, 'utf8'); const slug = file.replace(/^docs\//, '').replace(/\.md$/, ''); documents.push({ id: slug, title: markdown.match(/^# (.+)$/m)?.[1] ?? slug, url: `/help/${slug}`, markdown, });}
const { updated, unchanged } = await knowledge.ingest('docs', documents);- A document is
{ id, title, url, markdown, updatedAt? }.urlis a product path or an absolute http(s) URL. - A document whose content hash is unchanged is skipped. A changed one has its chunks replaced, and chunks whose text did not change keep their embeddings, so re-ingesting everything on each deploy is cheap.
remove(collection, ids)deletes documents and their chunks. Remove what your source no longer has.
Run ingestion from CI after a docs build, or from a job in your app.
Embeddings
Section titled “Embeddings”Lexical search works without them. With an embedder, vector scores are fused with lexical ranks:
import OpenAI from 'openai';
const openai = new OpenAI();const knowledge = createKnowledge({ driver, embed: async (texts) => (await openai.embeddings.create({ model: 'text-embedding-3-small', input: texts })).data.map((d) => d.embedding), dimensions: 1536,});dimensions is checked against every vector embed returns. Any provider works, including an OpenAI-compatible endpoint such as DashScope.
Plug it into aidy
Section titled “Plug it into aidy”source() turns the index into a SearchSource. collections picks, per user, which collections they may search:
const aidy = createAidy({ // ... sources: [ knowledge.source({ collections: (identity) => ['docs', `manuals/${identity.tenant}`], }), ],});The function receives the caller’s identity, so tenant-private manuals stay with their tenant. The source name defaults to knowledge; pass name to change it.
Knowledge hits then appear everywhere search does: in the ⌘K palette, the agent’s search tool, the MCP search tool and aidy search. They are fused with your product’s own search operations by reciprocal rank fusion. See Search.
With aidy-server, set AIDY_KNOWLEDGE_DIR and AIDY_KNOWLEDGE_BASE_URL instead of writing code. At start the server ingests every .md and .mdx file under the directory into the collection docs, in the same database as aidy, and searches it for every user. A document’s URL is the base URL plus its path without the extension (index maps to its directory), and its title is the frontmatter title, else the first heading.
Citations
Section titled “Citations”When the agent answers from search hits, the run emits an aidy.citation event with {messageId, citations: [{n, hit}]}: each cited hit under the number n the message wrote as [n]. @aidy-dev/react renders them as a source list under the message; useThread(...).state.citations holds them by message id for your own UI. A hit’s url points at the chunk’s section: the document URL plus the heading’s anchor.
Set a stable anchor with an explicit heading id, which is stripped from the title:
## Rotate keys {#rotate}Without one, the anchor is the heading’s GitHub-style slug.
How it searches
Section titled “How it searches”- Chunks. Markdown is split at h1–h3 headings. Sections over about 800 tokens (one per CJK character, one per four other characters) are cut at paragraphs, then sentences. A chunk’s title is
document title › heading path. - Lexical. SQLite and D1 use FTS5 (
unicode61) ranked bybm25(). Postgres uses atsvectorwith thesimpleconfiguration, ranked byts_rank. CJK runs are indexed as overlapping bigrams, so Chinese queries match without a segmenter. Query terms are OR-ed. - Vectors. Embeddings are scored by cosine similarity in JavaScript over the top 200 lexical hits plus the newest 20,000 chunks of the searched collections, then fused with the lexical ranking (k = 60).
- Excerpts are at most 300 characters around the densest cluster of matches, with matches wrapped in
«…», for people. - Content. Each hit also carries the chunk’s text as
content, up to 4000 characters. Only the model reads it, through thesearchtool and MCP; the palette and citations show the excerpt.
You can query the index directly as well:
const hits = await knowledge.search('rotate certificates', { collections: ['docs'], limit: 5 });Limits
Section titled “Limits”- Vector search reads every embedding of the searched collections, up to 20,000 chunks per query; older chunks beyond that are found only lexically. At 1,536 dimensions that is a few hundred MB read per query at the cap. Keep embedded collections well under it, or search fewer at once. pgvector is not used.
- On D1, statements are atomic one at a time. A document whose ingest fails halfway may be left without chunks; ingesting it again repairs it.
- Postgres ranking is
ts_rank, close to BM25 but not identical. searchreturns at most 50 hits.