From pages to answers: How Sulu's Intelligent Search works

Classic site search is keyword or full-text. The visitor has to guess the words the author used. Synonyms break it, phrasing breaks it, and "I just want the answer, not ten blue links" breaks it entirely.

An LLM on its own has the opposite problem. It sounds confident, but it invents facts and knows nothing about your content.

RAG, or Retrieval-Augmented Generation, is the bridge between those two failures. Retrieve the most relevant passages from your content first, then have the model answer using only those passages, with citations back to the source pages. The model supplies the language. Your content supplies the facts.

The whole system splits into two parts, and almost everything else hangs off that split.

The Sulu.ai Platform bundle runs inside your application and handles everything Sulu-specific: extracting content from pages and articles, chunking it into passages, exposing the public search endpoint your frontend calls, and persisting visitor feedback.

The Sulu.ai platform is the managed service behind it and handles everything AI-specific: generating embeddings, storing vectors, semantic retrieval, recency scoring, and answer generation.

This separation is intentional. While Sulu.ai handles the AI-specific infrastructure like embedding generation and retrieval, you retain flexibility regarding the vector database. You can either use a managed service like Qdrant Cloud or self-host your own vector database infrastructure. The platform seamlessly integrates with these options to power the search functionality. This setup enables production-grade RAG without the need for a dedicated ML team, with the core AI services included in your Sulu.ai subscription.

An embedding is a numerical representation of meaning. Passages that discuss similar topics end up close together in that numeric space, even when they don't share the same words.

That's why a search for "How do I recover my account?" can still find content about "Resetting your password," even though the wording differs. This semantic matching is the foundation of modern retrieval, and it's exactly what keyword search can't do.

Once we have these numerical fingerprints of your content, we need a place to store and efficiently search them. A vector database is a specialized storage engine designed to handle these embeddings. Unlike traditional databases that search for exact text matches, a vector database stores these numerical representations to perform "similarity searches." This allows the system to instantly find and compare information based on its underlying meaning and context rather than just matching keywords.

The indexing pipeline begins with your published content. This is where structured content becomes valuable beyond the website itself. Most organizations have already invested years into creating content models, articles, snippets, and documentation. Intelligent Search doesn't require rebuilding that knowledge. It simply makes existing content accessible through a different interface.

Pages and articles are converted into clean, searchable text using version-specific implementations for Sulu 2.6 (PHPCR) and Sulu 3.0 (Doctrine). Along the way, the system captures the public URL, template information, and other metadata needed to provide accurate, contextual responses.

The text is then split into smaller chunks. Chunking matters because retrieval works on passages rather than entire documents: chunk too big and relevance blurs, chunk too small and context is lost. The default chunker is heading-aware and targets roughly 200 words per passage. Each chunk is turned into an embedding and stored in the vector database under a stable identifier, so re-indexing updates content in place rather than duplicating it.

Indexing happens two ways. Publishing or updating content automatically triggers re-indexing, so answers stay aligned with the latest version of the site and editors never think about the index. A command-line tool covers the initial index, full re-indexing, and scheduled refreshes. On the platform side, ingestion is asynchronous, a freshly published page becomes searchable a moment later, not instantly, while its embeddings are generated in the background.

When a visitor submits a question, the pipeline runs in a predictable sequence:

1. The question is converted into an embedding. 
2. The most similar content chunks are retrieved from the vector database.
3. Optional recency scoring is applied, so newer content can be favored when it matters.
4. The best-matching passages are passed to the language model as context.
5. The model generates an answer using only those passages.
6. Citations are attached, linking back to the original source pages.

The answer is then streamed back to the frontend token by token using Server-Sent Events (SSE), so text appears as it's generated. A one-shot JSON mode is also available when you'd rather receive the whole answer at once.

Recency scoring is implemented as an exponential decay function applied over a configurable time window and combined with semantic similarity. The weighting can be tuned to match the content domain: news-heavy sites typically favor recency, while documentation and knowledge bases prioritize semantic relevance.

Each result includes both its raw similarity score and its final score after recency weighting. This makes ranking decisions transparent and helps explain why a particular page appears above another.

Citations are not a cosmetic feature. They're the mechanism that keeps answers grounded. Every generated answer links back to the source content used to produce it.

That single design decision delivers three benefits at once.

First, visitors can verify information for themselves by following the cited sources. Second, editors remain in control of the knowledge base, because the system can only answer questions using content that has actually been published. Third, developers gain transparency into how answers are generated and why a particular response was returned.

When users ask about something that doesn't exist in the published content, a grounded system declines to answer rather than inventing one. The citations provide a clear audit trail, showing exactly where the answer came from.

The objective is not simply to answer questions. It is to answer them responsibly.

Every search interaction creates valuable signals. Questions, answers, sources, locale, and whether the search succeeded are all persisted and fair enough failures included, which is exactly the data you want when debugging "why didn't it answer that?"

Visitors can rate a response with a thumbs up or down and leave an optional comment. Editors review this feedback directly inside the Sulu administration interface, protected by a dedicated permission that integrates with Sulu's existing access control model.

The immediate benefit is practical: recurring negative feedback on a topic often highlights missing content, outdated information, or gaps in the knowledge base. Search stops being a passive utility and becomes a feedback loop for content strategy.

More importantly, this feedback creates the foundation for reporting and continuous improvement. Over time, teams can identify trends, measure content quality, prioritize optimization efforts, and discover opportunities for new content based on what users are actually searching for and where the system struggles to provide helpful answers.

Intelligent Search is only the first step. We're already exploring additional embedding models, alternative vector storage backends such as Elasticsearch, query rewriting to handle short or vague questions, generating the questions a passage could answer during indexing, and multi-turn chat that carries context across follow-ups.

Some of these ideas will become features; others may not. The direction, however, stays the same: structured content shouldn't only power websites — it should become the knowledge layer behind AI-powered experiences. Intelligent Search is the first public step toward that future.