The Retrieval Loop

Ask a large language model a question about your product, your docs, or your pricing page, and it will answer confidently — sometimes correctly, often not. The model was trained on a snapshot of the internet, not on your changelog from last Tuesday. Retrieval-Augmented Generation (RAG) is the standard fix: instead of trusting the model's memory, you hand it the actual source material at the moment it answers, and instruct it to work from that material alone.

The mechanism is simpler than the acronym suggests. It's six ordinary engineering steps — most of them decades old on their own: crawling, search, string formatting — rearranged around a language model. This post traces all six through a real implementation: ChatBotAi, a Laravel API that answers visitor questions on a company website by crawling it, indexing it, and routing each question through Claude with the right paragraphs attached.

Six stages, one loop

Every RAG system, regardless of vendor or vector database, breaks down the same way. Four stages run once per document, offline, whenever the source material changes. Two stages run once per question, live, on every request.

Offline — indexing a document

1. Crawl — collect the source pages
2. Chunk — split into retrievable units
3. Embed — turn text into vectors
4. Store — index the vectors for search

— a visitor asks a question —

Online — answering a question

5. Retrieve — find the nearest chunks
6. Generate — answer from those chunks only

01 · Crawl

ChatBotAi's ingestion starts with php artisan chatbot:crawl (app/Console/Commands/CrawlWebsite.php), which walks the target site's sitemap — falling back to a robots.txt-aware breadth-first link crawl if no sitemap exists — and saves each page as a CrawledPage row before handing it to the chunker.

Re-crawls are idempotent: old chunks for a URL are deleted and replaced, so the index never accumulates stale duplicates when a page changes.

02 · Chunk

Most RAG tutorials chunk by character count — split every 500 characters, maybe with 50 characters of overlap. ChatBotAi does something more deliberate: semantic chunking (app/Services/SemanticChunker.php), a from-scratch port of LangChain's percentile-breakpoint algorithm.

The idea: split the page into sentences, embed each one together with its neighbors for context, then measure the cosine distance between consecutive sentence embeddings. A sharp jump in distance means the topic shifted — that's where a chunk boundary goes, not at some arbitrary character count.

if ($distance > $threshold && mb_strlen($current) >= $minChunkChars) {    $chunks[] = trim($current);    $current = $nextSentence;} else {    $current .= ' '.$nextSentence;}

03 · Embed

Both the chunker and the retriever call the same embedding provider: Voyage AI's voyage-3.5 model, at 1024 dimensions, through the Laravel AI SDK (app/Services/VoyageEmbedder.php). One detail worth calling out — Voyage's API distinguishes how an embedding will be used:

public function embedDocuments(array $texts): array{    return $this->embed($texts, 'document');} public function embedQuery(string $text): array{    return $this->embed([$text], 'query')[0];}

Passing input_type: document vs input_type: query nudges the same model to place stored chunks and incoming questions in the geometry it was actually trained to be searched against — asymmetric embedding. Calls are batched at 128 texts (Voyage's per-request ceiling), with exponential backoff on 429s, since the free tier caps out at three requests a minute.

04 · Store

Chunks and their vectors land in a vector store behind a small interface (VectorStore.php) with two swappable drivers, picked by an env var — no LangChain, no LlamaIndex, just a Qdrant or MongoDB Atlas client behind one contract.

$this->app->bind(VectorStore::class, fn () => match (config('chatbot.vector_store.driver')) {    'mongodb' => new MongoVectorStore,    'qdrant'  => new QdrantVectorStore,});

The default, Qdrant, runs self-hosted via the project's docker-compose.yml and is queried over its REST API with cosine distance and deterministic MD5-derived point IDs — so re-inserting a chunk during a re-crawl upserts instead of duplicating. The MongoDB path uses Atlas's $vectorSearch aggregation stage instead, also cosine, with numCandidates set to max(100, topK × 20) to give the ANN index a wide enough shortlist to work from.

05 · Retrieve

This is the first of the two stages that run inside a user's request, live. ChatController embeds the incoming question — with input_type: query — and asks the vector store for its five nearest chunks:

$queryEmbedding = $voyage->embedQuery($question);$matches = $store->similaritySearch($queryEmbedding, config('chatbot.top_k'));

top_k defaults to 5. No reranking step follows — whatever the ANN index returns is what the model sees. There's also no query rewriting: even mid-conversation, the vector search always runs against the visitor's latest raw message, not a history-aware reformulation of it. A follow-up like "what about pricing for that?" retrieves purely on those five words.

06 · Generate

The retrieved chunks don't get concatenated onto the question and thrown at the model raw — they're formatted into labeled, sourced excerpts first, inside ChatbotAgent (app/Ai/Agents/ChatbotAgent.php):

$context = collect($this->contextChunks)    ->map(fn ($chunk, $i) => sprintf(        "[%d] Source: %s (%s)\n%s",        $i + 1, $chunk['title'] ?: $chunk['url'], $chunk['url'], $chunk['text'],    ))    ->implode("\n\n---\n\n");

That block, plus the last few turns of conversation history, gets folded into a system prompt whose instructions matter as much as the retrieval itself: answer only from the context, don't invent anything, don't mention "the context" to the visitor, and if the answer isn't in there, say so plainly and suggest they reach out directly. The question goes in separately as the user turn, and the whole thing is sent to Claude:

$answer = empty($matches)    ? "I don't have any indexed content to answer that yet…"    : (new ChatbotAgent($matches, $history))->prompt($question)->text;

Notice what retrieval bought here: the model isn't recalling the site's pricing page from training data — it's reading the actual paragraph, indexed minutes or hours ago, sitting directly in its prompt.

Separately from the generated text, the controller also returns a structured sources field — url, title, similarity score, deduped by URL — pulled straight from the vector matches. The model is explicitly told not to cite [n] markers inline; citation is a UI-level field, not something baked into the prose:

Answer: "We don't have a fixed public price list — most plans are scoped tousage, so the fastest way to get an exact number is to book a quick callwith the team. Happy to point you to the right page if you want to seefeature tiers first."Sources:  /pricing      — 0.912  /contact      — 0.847  /faq#billing  — 0.803

What's missing — and why that's fine

Compared to a fully-loaded retrieval stack, a few pieces are deliberately absent here:

• No reranker. The top-k ANN result goes straight to the model, unfiltered by a second relevance pass.
• No caching. Every question re-embeds and re-searches, even a repeat one.
• No streaming. The API waits for the full Claude response and returns one JSON payload.
• No query rewriting. History informs the answer, not the search.

None of these are bugs — they're a specific system's answer to how much complexity the problem actually needs. A support site with modest traffic doesn't need a reranker or a cache; it needs a chunker that respects sentence meaning and a prompt that refuses to hallucinate. Bigger corpora or heavier query volumes are exactly when those missing pieces start paying for themselves.

Closing

Strip away the acronym and RAG is a search engine wired to a very good summarizer: index documents, find the closest ones to a question, hand them to the model, and get out of the way. Every "RAG" product is really just a set of decisions about those six stages — how to chunk, what to embed, what "closest" means, and how much to trust the model to stay inside the paragraph you gave it. ChatBotAi's answers to those questions — semantic chunking, asymmetric Voyage embeddings, cosine search over Qdrant, a prompt that separates citation from prose — are one reasonable set. Yours will look different, and that's the actual skill in building one of these: not knowing the acronym, but making six ordinary choices well.