From pages to answers: Setting up Sulu's Intelligent Search

You don't need the earlier posts to follow along, but the one idea to carry with you is the split the whole system is built on:

Intelligent Search consists of two parts. The bundle runs inside your Sulu application and handles everything Sulu-specific: extracting content, chunking it, exposing the search API, and collecting user feedback. The Sulu.ai platform is the managed service responsible for the AI layer: generating embeddings, managing the connection to your Qdrant instance for vector storage, retrieving relevant content, and generating grounded answers.

The result is a production-ready AI search experience without the operational burden of managing complex ML infrastructure, while you maintain full control over your vector data.

What makes this interesting is that you're not building a separate AI knowledge base. You're putting an existing content foundation to work. Pages, articles, documentation, and any additional content source you choose to index become part of the same knowledge layer.

The whole setup in four steps

Configure the project on the platform (a guided wizard), flip one flag in the bundle, import the public routes, and run the first index. Everything below expands those four steps and then shows you how to verify and extend the result.

On the Sulu.ai platform, Intelligent Search is configured per project through a guided wizard with three steps.

First, connect a data store: the Qdrant instance that holds your vectors, including its URL, API key, and collection name. (Qdrant is the vector database used to find the passages most relevant to a question.) Managing this instance is the responsibility of the agency or customer. It can be self-hosted or provided through Qdrant Cloud at cloud.qdrant.io.

This separation is intentional. By design, Sulu.ai does not store your content or vectors. The platform processes requests and provides the AI capabilities, while your data remains under your control in the infrastructure you manage.

Second, define the answer style and tone. This is the setting most teams will adjust first, because it controls how answers are presented to visitors. You can make responses more formal or conversational, concise or detailed, without changing the retrieval pipeline or writing any code.

Third, set recency behavior: a recency-vs-similarity weight, a decay window in days, and a results limit. These control how aggressively newer content is favored over the closest semantic match, which matters far more for a news site than for evergreen documentation.

In the Sulu app, the feature is opt-in. One flag wires up everything.

That single flag conditionally loads the intelligent-search services, registers the public routes and CLI commands, the auto-ingest event subscriber, and the admin feedback views. Flip it off and none of it loads — the routes are even guarded by a Symfony route condition tied to the flag, so the feature truly disappears when off, rather than lingering as dead endpoints.

Two operational steps remain. Import the public routes into your website routing:

Then create the feedback table and run the first index of your content:

That's the entire setup. The rest of this guide explains what just happened and how to make it your own.

Think of ingestion as a journey: a published page becomes a set of searchable vectors.

Ingest providers 

Decide what gets indexed. Out of the box that's pages and articles, with separate implementations per Sulu version (2.6 on PHPCR, 3.0 on Doctrine). The content transformer turns a Sulu structure into clean text, builds the public URL, reads template metadata, and skips fields tagged to be ignored. The chunker splits that text into passages. The default markdown chunker is heading-aware and targets roughly 200 words per chunk.

Chunking matters more than it looks. Retrieval works on passages, not whole documents: chunk too big and relevance blurs, chunk too small and context is lost. The result of the pipeline is an IngestDocument with a stable ID (page-{uuid}-{locale} or article-{uuid}-{locale}), plus title, URL, bucket, locale, updatedAt, and the chunks themselves. That stable ID pattern is what keeps re-indexing consistent and deletes reliably: re-ingest the same page and it updates in place instead of duplicating.

On the platform, ingestion is asynchronous. The ingest endpoint accepts the batch with a 202 Accepted, then a queued handler generates the embeddings and writes them to Qdrant. That's why a freshly published page becomes searchable a moment later, not instantly: the embeddings are being generated in the background.

The one code element worth showing is the provider contract, because it's also the main extension point:

Content gets indexed two ways. 

Automatically on publish: an event subscriber re-ingests a page or article when it transitions to published, and removes it on unpublish or delete. Editors do nothing and never think about the index.

In bulk via CLI, for the initial index, a full re-index, or scheduled refreshes:

Use `--dry-run` to preview without writing, `--purge` to clear and reindex from scratch, and omit the filters to index everything.

When a visitor searches, their frontend calls the public search endpoint with a question and a locale. The default response is an SSE stream; pass format=json for a single response instead. Optional parameters let you scope the search: buckets (limit to articles or pages), webspace, prefer_recent, and since.

Because this is a public endpoint — and therefore an abuse surface — the bundle ships an IP rate limiter that's on by default (10 requests / 60s, configurable). It's worth keeping in mind before you point a load test at it.

The stream shape tells the story better than any prose:

Notice that citations are returned as structured data alongside the answer, not embedded into the generated text. This allows your UI to render sources independently of the answer and gives developers full control over how citations are displayed. The JSON mode response makes the same point: citations are a first-class part of the API, not an afterthought.

Each source carries two numbers: similarity, the raw semantic match, and score, the final ranking after recency adjustment. Exposing both is how you answer "why did this page rank first?" when an editor asks.

Keep this thin. It's a normal HTTP endpoint, so build any UI you like. Most production sites wire the SSE stream into their own search component and style it to match the site.

To see it working before you write a single line of JavaScript, the bundle ships an example UI: one self-contained HTML page (Tailwind plus `marked` for markdown, both via CDN, no build step). It calls the exact same public endpoint a real frontend would, so what you see is a faithful preview of production behavior, not a mock.

It's gated behind two flags. Both must be on, or you get a 404:

Then visit `https://your-site/ai-search`. Keep `example_ui.enabled: false` in production. It's a verification and demo tool, not a finished search experience.

What makes this more than a static diagnostics page is that it builds its form dynamically from your system configuration. The locale dropdown lists every locale configured across your webspaces. The webspace dropdown is populated through the WebspaceManager and includes an additional All Webspaces option.

The buckets dropdown is generated from the registered ingest providers. Any custom provider you add will appear automatically, making this page one of the fastest ways to verify that your provider has been registered and discovered correctly. No additional configuration is required; if the provider is available to the ingestion pipeline, it will be available here as well.

Because it's faithful to production, the demo doubles as a sanity check for the whole pipeline. Ingest first, then ask about a page you know is published: a good answer with a citation to the right page means ingestion, retrieval, and generation all work.

Every search is persisted in the ai_search_feedback table — question, answer, sources, locale, and whether it succeeded. Storing failures too is deliberate: an unanswered question is exactly the thing you want a record of.

Visitors submit a thumbs up or down with an optional comment, keyed to the answer's uuid, through a public feedback endpoint. Editors review it all inside the Sulu admin. The payoff is practical: a topic that keeps collecting thumbs-down is a content gap with a built-in instruction to write that page.

The deep payoff for developers is the extension points.

Index anything, not just Sulu content. Implement an Ingest Provider, and a database table, an external API, or a filesystem becomes searchable right alongside pages and articles. This is the headline customization, and we use it ourselves: on sulu.io a custom provider pulls in our documentation and the README files from our GitHub repositories, so a single question can be answered from blog content, docs, and source repos at once.

For full implementations, see the bundle docs under docs/intelligent-search/ in the bundle.