Skip to content

Knowledge

@aidy-dev/knowledge is aidy’s knowledge index. It stores Markdown documents in collections, chunks them by heading, and searches them with BM25 (plus vectors when you give it an embedder). It runs on any SqlDriver from @aidy-dev/sql: Postgres, SQLite or Cloudflare D1. Plugged into aidy, it is one more search source, named knowledge.

Terminal window
npm install @aidy-dev/knowledge @aidy-dev/sql
import { createKnowledge } from '@aidy-dev/knowledge';
import { postgresDriver } from '@aidy-dev/sql/postgres';
const knowledge = createKnowledge({ driver: postgresDriver(process.env.DATABASE_URL!) });
await knowledge.migrate(); // creates aidy_knowledge_* tables
Driver Import Notes
Postgres postgresDriver(url) from @aidy-dev/sql/postgres Needs the postgres package (v4). Ranks with ts_rank.
SQLite sqliteDriver(path) from @aidy-dev/sql/sqlite Node’s built-in node:sqlite. FTS5 with BM25.
D1 d1Driver(env.DB) from @aidy-dev/sql/d1 Cloudflare Workers. FTS5 with BM25.

The index can share a database with aidy’s store; its tables have their own prefix.

ingest(collection, documents) adds or replaces documents and returns { updated, unchanged }.

import { glob, readFile } from 'node:fs/promises';
const documents = [];
for await (const file of glob('docs/**/*.md')) {
const markdown = await readFile(file, 'utf8');
const slug = file.replace(/^docs\//, '').replace(/\.md$/, '');
documents.push({
id: slug,
title: markdown.match(/^# (.+)$/m)?.[1] ?? slug,
url: `/help/${slug}`,
markdown,
});
}
const { updated, unchanged } = await knowledge.ingest('docs', documents);
  • A document is { id, title, url, markdown, updatedAt? }. url is a product path or an absolute http(s) URL.
  • A document whose content hash is unchanged is skipped. A changed one has its chunks replaced, and chunks whose text did not change keep their embeddings, so re-ingesting everything on each deploy is cheap.
  • remove(collection, ids) deletes documents and their chunks. Remove what your source no longer has.

Run ingestion from CI after a docs build, or from a job in your app.

Lexical search works without them. With an embedder, vector scores are fused with lexical ranks:

import OpenAI from 'openai';
const openai = new OpenAI();
const knowledge = createKnowledge({
driver,
embed: async (texts) =>
(await openai.embeddings.create({ model: 'text-embedding-3-small', input: texts })).data.map((d) => d.embedding),
dimensions: 1536,
});

dimensions is checked against every vector embed returns. Any provider works, including an OpenAI-compatible endpoint such as DashScope.

source() turns the index into a SearchSource. collections picks, per user, which collections they may search:

const aidy = createAidy({
// ...
sources: [
knowledge.source({
collections: (identity) => ['docs', `manuals/${identity.tenant}`],
}),
],
});

The function receives the caller’s identity, so tenant-private manuals stay with their tenant. The source name defaults to knowledge; pass name to change it.

Knowledge hits then appear everywhere search does: in the ⌘K palette, the agent’s search tool, the MCP search tool and aidy search. They are fused with your product’s own search operations by reciprocal rank fusion. See Search.

With aidy-server, set AIDY_KNOWLEDGE_DIR and AIDY_KNOWLEDGE_BASE_URL instead of writing code. At start the server ingests every .md and .mdx file under the directory into the collection docs, in the same database as aidy, and searches it for every user. A document’s URL is the base URL plus its path without the extension (index maps to its directory), and its title is the frontmatter title, else the first heading.

When the agent answers from search hits, the run emits an aidy.citation event with {messageId, citations: [{n, hit}]}: each cited hit under the number n the message wrote as [n]. @aidy-dev/react renders them as a source list under the message; useThread(...).state.citations holds them by message id for your own UI. A hit’s url points at the chunk’s section: the document URL plus the heading’s anchor.

Set a stable anchor with an explicit heading id, which is stripped from the title:

## Rotate keys {#rotate}

Without one, the anchor is the heading’s GitHub-style slug.

  • Chunks. Markdown is split at h1–h3 headings. Sections over about 800 tokens (one per CJK character, one per four other characters) are cut at paragraphs, then sentences. A chunk’s title is document title › heading path.
  • Lexical. SQLite and D1 use FTS5 (unicode61) ranked by bm25(). Postgres uses a tsvector with the simple configuration, ranked by ts_rank. CJK runs are indexed as overlapping bigrams, so Chinese queries match without a segmenter. Query terms are OR-ed.
  • Vectors. Embeddings are scored by cosine similarity in JavaScript over the top 200 lexical hits plus the newest 20,000 chunks of the searched collections, then fused with the lexical ranking (k = 60).
  • Excerpts are at most 300 characters around the densest cluster of matches, with matches wrapped in «…», for people.
  • Content. Each hit also carries the chunk’s text as content, up to 4000 characters. Only the model reads it, through the search tool and MCP; the palette and citations show the excerpt.

You can query the index directly as well:

const hits = await knowledge.search('rotate certificates', { collections: ['docs'], limit: 5 });
  • Vector search reads every embedding of the searched collections, up to 20,000 chunks per query; older chunks beyond that are found only lexically. At 1,536 dimensions that is a few hundred MB read per query at the cap. Keep embedded collections well under it, or search fewer at once. pgvector is not used.
  • On D1, statements are atomic one at a time. A document whose ingest fails halfway may be left without chunks; ingesting it again repairs it.
  • Postgres ranking is ts_rank, close to BM25 but not identical.
  • search returns at most 50 hits.