Skip to Content
Developer’s Eye

Architecture

Follow the write path from a seed URL and the read path from a browser request.

One page, three jobs

Imagine a source registry entry for a documentation site. Bootstrap puts its start URL in the frontier. A crawler worker leases the target, applies the host policies, fetches it and extracts a document. The same fetch may discover more URLs for a future pass.

An indexer picks up the new document when its stored index is missing or stale. It analyzes the fields, builds PostgreSQL search vectors and records which content version it indexed. You can stop the crawler at this point and still search the material it collected.

When a visitor searches, the Next.js route forwards the query to FastAPI. FastAPI reads candidate documents from the index, ranks them and builds snippets for the requested page. A search never waits for the crawler to visit a website.

Where state lives

StateOwnerWhy it matters
Sources, frontier, leases and attemptsCrawlerA worker can resume eligible work after another exits.
Document content and availabilityCrawler writerA recrawl can update an existing page or record that it disappeared.
Vectors, index version and vocabulary updatesIndexerSearch and spelling suggestions can follow changes to a document.
Search telemetry, quality judgments and AI usageBackendThe dashboard can inspect outcomes across requests.
Saved resultsBrowserA reader can revisit a result on the same browser.

Where to change behavior

Start in the crawler for missing pages, the indexer for stale searchable content, and search and ranking for unexpected ordering. Backend database models live under backend/src/backend/models. Schema changes go through backend/alembic/versions so workers and the API agree on the same tables.