How to Build a RAG Application: Architecture and Evaluation

Here’s what matters most before we get into the architecture, tuning decisions, and evaluation methods that make a RAG application production-ready:

Key Points

• LLMs can generate confident claims that aren’t supported by their training data or available evidence, so grounding answers in retrieved data can reduce unsupported claims that additional training alone can’t address.
• RAG is context engineering executed programmatically: a retrieval pipeline places curated grounding data into the model’s context.
• A retrieval agent can handle complex, multi-step queries that static RAG may miss, but it adds orchestration complexity and token spend.
• Groundedness is one of the most important evaluation checks because unsupported claims undermine the core purpose of a RAG system.

Retrieval-augmented generation (RAG) is a technique that, before answering, retrieves relevant passages from a trusted, curated knowledge source and inserts them into the model’s prompt. The model then answers using relevant information you control rather than relying solely on its pretrained knowledge, a distinction that matters once an AI system starts speaking for your company.

This guide gives a high-level overview of RAG architecture, compares RAG systems to the alternatives, and works through the hyperparameters that let you adapt a RAG pipeline to your own use case. From there, we get practical: how to select those hyperparameters, how to evaluate the pipeline once it’s running, and the failure modes worth watching for in production.

The Real-World Risks of LLM Hallucination

In March 2023, Steven Schwartz, a seasoned personal injury attorney, submitted a 10-page brief on behalf of his client, Roberto Mata, who was suing Avianca Airlines for injuries sustained during a 2019 flight. What happened next is now familiar: Avianca’s lawyers, after digging through the citations, found that Schwartz had cited cases that didn’t exist. The judge agreed with that assessment, and the lawyer was put under professional sanction.

The Avianca case was not an isolated incident. A public database maintained by HEC Paris research fellow Damien Charlotin had identified 1,598 court cases involving AI-fabricated citations or content as of June 9, 2026, up from roughly 200 cases a year earlier. The database counts cases in which a court or tribunal explicitly found or clearly implied that a party relied on hallucinated material. 

For the AI world, the “Avianca Incident” introduced a new fear. “Hallucination” entered the lexicon as an unrelenting threat for AI engineers. In some ways, hallucination is inherent to how an LLM works: next-token prediction is based on the likelihood of a particular token occurring, not on whether a claim is true. The training data also contains fiction, sarcasm, and hypothetical claims, which can be completely benign in one context and unsupported in another. More training alone doesn’t guarantee that an LLM will stop making factual errors.

The problem is particularly serious when an LLM is used with proprietary or business-critical information. RAG addresses part of the problem by allowing the model to retrieve external knowledge at inference time rather than relying entirely on information encoded in its model weights. It doesn’t eliminate hallucination, but it gives the application a mechanism for grounding responses in information the organization controls.

To address hallucination, engineers turned to retrieval-augmented generation, a technique theorized long before LLMs became commercially viable. Beyond reducing unsupported responses, RAG lets LLMs draw on proprietary data, making them far more useful inside the enterprise.

RAG Architecture

At the core, RAG combines information retrieval with generation. Here’s what that means in practice.

Prompt engineering is the art of constructing prompts that get predictable, high-quality responses from large language models. A more specialized approach, context engineering, involves assembling the information and instructions an LLM needs to carry out a task. RAG is context engineering executed programmatically: it uses a retrieval pipeline to find relevant information and add that information to the model’s context.

Selecting the Grounding Data

The first step in the pipeline is selecting the grounding data. The purpose of a RAG system is to give the model access to a managed, controllable knowledge source, so that knowledge source has to earn its role. The grounding data needs to be comprehensive, authoritative, and up to date. Build a process that checks the dataset for contradictions and maintains a consistent update schedule so the grounding data doesn’t go stale.

Most of that grounding data won’t arrive as clean rows and columns. A November 2025 Cloud Security Alliance survey of IT and security professionals found that unstructured data made up about 33% of enterprise data and semi-structured data another 21%, with 29% of respondents saying unstructured data accounted for more than half of their annual data growth. 

This matters because retrieval quality can’t compensate for poor source material. AWS recommends clear headings and subheadings, concise and focused documents, defined terminology, and breaking large documents into smaller, self-contained units where appropriate. These practices can make source material easier to index, retrieve, and interpret.

Selecting the Embedding Model

Next, select and test the embedding model. The embedding model translates documents in your grounding data into vectors: numerical representations that capture aspects of their semantic meaning. That representation supports mathematical comparison. For example, an embedding model may place “apple tree” closer to “banana tree” than to “nuclear power plant,” preserving conceptual similarity within the vector space.

That property drives the retrieval stage, where the system finds pieces of grounding data that are relevant to the user query.

Picking the right embedding model for your use case matters. Some embedding models are better suited to general-purpose search, while others may perform better with specialized terminology, languages, or domains. If your RAG system processes a large volume of industry-specific jargon, test different embedding models against representative queries rather than assuming one model will perform best.

Thorough testing is particularly important for sensitive workloads where accurate retrieval matters most.

Building the Vector Database

Once you’re happy with the embedding model, build the vector database or another vector store. A vector database specializes in retrieving information based on the similarity between vectors, making it efficient at finding the top N relevant pieces of grounding data for a given query.

Vector database providers range from self-hosted open source options to auto-scaling enterprise services, so there are options for nearly any market segment.

The vector database can also store metadata alongside each chunk. Publication date, document owner, business unit, version, document type, and access permissions can all become important retrieval signals. Semantic similarity alone isn’t always enough to identify the right context.

Building the Prompt Pipeline

Next comes the prompt pipeline. RAG prompts are more involved than typical prompt engineering because they’re constructed using programmatic retrieval tools and can look different depending on the state of the corpus and the wording of the user query.

Typically, the system intercepts the user query before it reaches the LLM and uses it to search the vector database or other search indexes for the grounding data most relevant to that query. It then constructs a new prompt instructing the LLM to answer within the context of the retrieved data.

This stage carries many of the hyperparameters that need to be tuned and evaluated. Modern RAG systems may combine vector search with full-text search, hybrid search, query rewriting, and re-ranking rather than relying on semantic search alone.

Deploying the RAG Application

Finally, deploy and test the RAG application. You’ll generally manage three major services or endpoints.

The first is user-facing: it takes the user’s input, orchestrates the RAG pipeline, constructs the RAG prompt, and returns the LLM’s answer.

The second is the retrieval layer, which may include a vector database, full-text search engine, or both. It retrieves the most relevant information for the query.

The third is the LLM endpoint that composes the final answer, usually through an API provided by an LLM provider or hyperscaler. Teams using open source models have other inference options.

RAG Pipeline

In an enterprise environment, this architecture also needs deliberate controls for authentication, authorization, data security, logging, monitoring, and auditability. The retrieval layer should respect the user’s permissions rather than exposing every document that happens to exist in the knowledge base.

Now that we understand the basic implementation architecture of RAG, let’s look at where RAG fits relative to other approaches.

RAG vs. Alternatives

RAG is a popular technique, but it’s one of several options for improving the reliability and usefulness of LLM applications. It’s worth understanding the strengths and weaknesses of each.

Fine-Tuning

Fine-tuning is the technique most often discussed alongside RAG. Instead of working primarily at the prompt level, it changes the model’s weights through additional training.

One advantage of fine-tuning is that the resulting behavior can be applied across many requests without repeatedly supplying the same instructions. This can make fine-tuning useful when the goal is to change behavior, style, classification, formatting, or task performance.

Fine-tuning is less attractive when the problem is frequently changing factual knowledge. Updating a knowledge base through RAG is generally more practical than retraining a model whenever a document changes.

Proper fine-tuning also demands specialized knowledge and thorough testing. It’s not a simple or reliable standalone solution for hallucination.

Long Context

A second approach is the large context window. As models have gained the ability to process increasingly large inputs, some teams have considered putting large volumes of grounding data directly into the prompt and skipping the retrieval stage.

It’s an appealingly simple approach: no embeddings, no vector database, straight to prompt engineering. But it has practical limitations.

The technique eventually hits a hard limit when the grounding corpus exceeds the model’s context window. Long before that, however, cost, latency, irrelevant information, and the model’s ability to identify the most useful information can become concerns.

The practical question isn’t simply whether the corpus fits inside the context window. It’s whether providing the entire corpus produces better results than retrieving a smaller amount of carefully selected context.

Retrieval Agents

The third approach is a retrieval agent. Semantic search has limitations, particularly when a query requires information from multiple sources or multiple retrieval steps.

Instead of a single static retrieval based on vector similarity, a multi-step architecture lets an agent use tools and iteratively query the knowledge base as it gathers more information. This can improve retrieval for complex questions, but it requires a substantially more complex architecture.

Tool definitions, retriever prompts, orchestration, state management, and additional model calls all require design and testing. Its multi-step nature can also add token and latency costs compared with a straightforward RAG pipeline.

Microsoft’s current Azure AI Search guidance similarly distinguishes classic RAG, where simplicity and control may be priorities, from agentic retrieval for complex conversational queries where higher relevance and accuracy are more important.

That complexity hasn’t slowed adoption. The US RAG market alone was valued at roughly $558.8 million in 2025 and is projected to reach $2.59 billion by 2030, a 35.9% CAGR, as enterprises move retrieval pipelines from pilot to production.

Now that we understand the alternatives and where RAG fits, let’s look at the architectural choices you’ll make when implementing a RAG pipeline and how they affect performance.

RAG Hyperparameters: Decisions and Tradeoffs

The first decision point is the embedding model. Most commercially available models handle general-purpose use cases well, but specialized datasets can expose meaningful differences.

For example, the phrase “we made an impression,” in a typical context, may refer to making an impact. In a medical or dental context, it may refer to creating a physical impression of someone’s teeth. If your RAG system processes a large volume of industry-specific jargon, test embedding models against representative data to determine which one best captures the semantic meaning that matters to your application.

The next decision point is the chunking strategy. RAG requires high-quality grounding data, and how you chunk or split the data makes a significant difference to that quality.

The best chunking strategy accounts for the structure of the underlying data. If your corpus consists of self-contained, list-like paragraphs, a smaller chunk size can keep each idea inside its own chunk. If the corpus conveys ideas across multiple, complex paragraphs, a larger chunk size may capture the full context of the input.

Chunking methodology matters as much as chunk size. If the input data has a natural structure, the chunking methodology should take advantage of it. HTML-marked pages, for instance, can use tags like <p>, <h1>, and <h2> to split the data into semantically consistent chunks. A book with well-defined chapters and sections offers the same kind of exploitable structure.

Dense tables and forms are a common trap. A chunking strategy that treats them as ordinary text can destroy the relationships between rows and columns, making dedicated table extraction necessary before the content can be retrieved reliably.

AWS likewise recommends structuring source documents with clear headings, defining terminology and context, and breaking large documents into smaller, self-contained units where appropriate.

Data Augmentation and Metadata

Data augmentation is the next consideration. Embedding-based semantic similarity gets you grounding data chunks that resemble each other, but semantic search alone often isn’t enough.

The data usually needs metadata to be genuinely useful. A RAG system grounded on newspaper articles, for example, will likely want to store publication date, title, author, and section alongside each chunk, both to support citations and to let the query filter on publication date.

In an enterprise knowledge base, metadata can also support access control, version selection, business-unit filtering, document ownership, and other retrieval constraints.

Retrieval Strategy

The last step before prompting is the retrieval strategy. The biggest variable is the number of chunks to retrieve and include in the prompt. Too few chunks and the model may lack necessary context; too many and the model may receive irrelevant information while consuming more tokens.

Chunk count and chunk size are deeply connected, since a larger chunk size creates a longer final context for the same chunk count. Run integration tests to confirm that the chosen context count supports the final answering LLM and that results hold up across representative queries.

It’s also worth testing more than pure vector search. Hybrid search combines keyword and vector retrieval, while re-ranking can score a smaller set of retrieved documents again before passing the highest-value results to the LLM. Query rewriting can also generate better retrieval queries when the original user query is poorly phrased or ambiguous. These techniques are now established components of modern RAG pipelines.

RAG retrieval strategy

Constructing the RAG Prompt

Now you’re ready to construct the RAG prompt itself. There are many ways to implement one, but a few best practices apply broadly.

First, clearly demarcate data from instruction. One of the thorniest LLM issues is that input data and instructions share the same context, and RAG makes this more complicated by inserting external text that can contain instruction-like language. Use consistent delimiters so the model knows which text represents instructions and which text is retrieved reference material.

Second, give the model a way to avoid hallucinating. Teams sometimes give models contradictory instructions by telling them to stay strictly grounded while also demanding an answer no matter what. When the grounding data doesn’t contain enough information but the model is still pushed to answer, it may generate unsupported content. Let the model say “I don’t know” when the grounding data falls short.

Third, consider structured output instead of free-form text. If you’re asking the model for citations, multiple output types, or other structured information, a format like JSON makes the output easier for downstream programs to parse.

Evaluating a RAG Pipeline

There are several ways to test a RAG pipeline’s effectiveness. LLM-as-a-judge is a practical evaluation technique, particularly when a team needs to evaluate many generated responses, but it shouldn’t be the only form of validation. For high-impact applications, human review and task-specific automated checks can complement LLM-based evaluation.

To evaluate a RAG pipeline with an LLM judge, start by preparing test data: a representative set of queries the pipeline is likely to encounter in production, spanning easy questions, moderately difficult ones, complex questions, and questions that fall outside the scope of the grounding data entirely.

Run the questions through the full pipeline, retrieval and final answering included, then evaluate the results against three attributes.

Groundedness

The first test is groundedness: whether the LLM’s answer is actually supported by the retrieved data.

This is one of the most important tests in the pipeline, since grounding is a core purpose of RAG. Judge this one strictly: unsupported claims are a meaningful failure signal.

When groundedness fails, the usual culprits can include poor retrieval, incomplete or contradictory source data, a prompt that doesn’t adequately constrain generation, or a model that isn’t well suited to the task.

Relevance

The second test is relevance: whether the LLM actually answered the user’s question rather than just summarizing the retrieved data.

This one calls for a more nuanced read, since relevance sits on a continuum rather than a simple pass/fail line, and the fix depends on how the pipeline failed.

High groundedness paired with very low relevance usually points to the retrieval pipeline itself: the retrieved data simply isn’t relevant enough to the query. High groundedness with middling relevance may mean the chunking strategy is the issue. The pipeline retrieved some relevant information, but not enough of it.

Possible fixes include changing chunk size or count, improving query rewriting, adding metadata filters, introducing hybrid search, or adding a re-ranking stage.

Low groundedness and low relevance together mean the fix starts with groundedness, not relevance.

Correctness

The third test is correctness, measured against a “ground truth” answer from an expert or another validated reference.

This test differs from the other two because it needs a golden answer added to the dataset, so correctness tests are usually reserved for a prioritized subset rather than the full test set, which typically includes out-of-scope or “unanswerable” questions.

It’s still a critical part of final testing because it confirms the model performs where the answers matter most. A low correctness score paired with high relevance or groundedness can point to a retrieval problem, insufficient source material, or an LLM that isn’t capable of synthesizing the retrieved information effectively.

If those factors check out and correctness is still low, RAG itself may be the wrong fit for the task, and a more advanced technique such as agentic retrieval may be worth pursuing.

RAG Evaluation Gate

From Prototype to Production

A simple RAG application can be relatively straightforward to demonstrate. Getting it into production is a different problem.

The application needs reliable source data, a retrieval strategy that performs against real queries, security controls that preserve document permissions, monitoring, evaluation, and an operating process for keeping the knowledge base current.

That also means treating RAG as a software system rather than a prompt wrapped around an LLM.

Source documents can change. User queries can change. Embedding models can change. Retrieval settings can drift as the corpus grows. An application that performs well against a small test dataset can therefore degrade after deployment.

For engineering leaders, the production question isn’t simply whether the team can build a RAG application. It’s whether the team can establish the engineering discipline to measure retrieval quality, maintain the knowledge source, control access, and detect failures as the system evolves.

What Makes RAG Work

This guide walked through RAG’s architecture, its hyperparameters, and how to evaluate a pipeline once it’s built. We covered the main components of a RAG system, the decisions you’ll face implementing one, and how it stacks up against fine-tuning, long context windows, and retrieval agents as approaches to improving LLM reliability.

RAG is a mature building block for applications that need access to private, specialized, or changing information. Its basic architecture is straightforward: retrieve relevant information, provide that context to an LLM, and generate a response.

The engineering challenge lies in everything around that loop. Grounding data quality affects retrieval. The embedding model affects semantic search. Chunking affects what can be retrieved. Metadata affects filtering. Retrieval strategy affects the context available to the model. Prompt construction affects how that context is used. Evaluation determines whether the resulting system is actually working.

For straightforward knowledge-base applications, a simple RAG system may be enough. More complex queries may justify hybrid search, query rewriting, re-ranking, or agentic retrieval. Fine-tuning and long-context approaches can also make sense when the underlying problem is different.

The right architecture depends on the workload. What matters is not whether a system uses RAG, but whether it reliably retrieves the right context, produces a useful response, and fails safely when the available knowledge isn’t sufficient.