How to Build a RAG Assistant on Your Company's Documents

What is a RAG assistant?

A retrieval-augmented generation (RAG) assistant is a chatbot that answers questions using your documents. Before calling the language model, it searches a knowledge base for the passages that are most relevant to the question and passes them to the model as context. The model is instructed to answer only from that context and to cite it.

This solves the two main problems of using an LLM alone in a company: it does not know your internal documents, and it invents answers when it does not know. I built this kind of assistant inside D&A Studio, the data platform I developed during my end-of-studies internship (see the project).

How does a RAG pipeline work?

There are two phases.

1. Indexing (done once, then on every document change)

1. Load documents (PDF, Word, web pages, database exports).
2. Clean the text and split it into chunks of a few hundred tokens.
3. Compute an embedding (a vector of numbers that represents meaning) for each chunk.
4. Store chunk text, metadata and vector in a vector index.

2. Answering (on every question)

1. Compute the embedding of the question.
2. Retrieve the k most similar chunks (cosine similarity).
3. Build a prompt with the instructions, the retrieved chunks and the question.
4. Call the LLM and return the answer with its sources.

Which tools should you use?

You do not need a new database. If you already run PostgreSQL, the pgvector extension adds a vector type and similarity search:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE doc_chunks (
  id        BIGSERIAL PRIMARY KEY,
  doc_id    TEXT NOT NULL,
  title     TEXT,
  content   TEXT NOT NULL,
  embedding VECTOR(768)
);

CREATE INDEX ON doc_chunks USING hnsw (embedding vector_cosine_ops);

Retrieval is then a single query (<=> is the cosine distance operator):

SELECT doc_id, title, content
FROM doc_chunks
ORDER BY embedding <=> $1   -- question embedding
LIMIT 5;

For the application layer, any stack works. In D&A Studio the API is in Spring Boot; for prototypes I use Python. A minimal answering function looks like this:

def answer(question: str) -> dict:
    q_vec = embed(question)                       # same model as indexing
    chunks = search_similar(q_vec, k=5)           # pgvector query above
    context = "\n\n".join(f"[{i+1}] {c.content}" for i, c in enumerate(chunks))
    prompt = (
        "Answer using ONLY the context below. "
        "If the answer is not in the context, say you don't know. "
        "Cite sources as [1], [2]...\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    return {"answer": llm(prompt), "sources": [c.doc_id for c in chunks]}

How do you chunk documents correctly?

Chunking has more impact on quality than the choice of model.

• Size: 300–500 tokens per chunk with ~50 tokens of overlap is a good default.
• Respect structure: split on headings and paragraphs, not in the middle of a table.
• Keep metadata: document title, section, date and URL go with each chunk; they help ranking and let you show sources.
• Clean first: remove headers, footers and page numbers that repeat on every page.

How do you stop the assistant from inventing answers?

• Tell the model explicitly to answer only from the context and to say "I don't know" otherwise.
• Return sources with every answer so users can check.
• Set a similarity threshold: if no chunk is close enough, do not call the LLM at all.
• Keep the temperature low (0–0.3) for factual answers.
• Restrict access: the assistant should only search documents the user is allowed to see (role-based access).

How do you evaluate a RAG assistant?

Write 20–50 real questions with the expected answer and the document that contains it, then measure:

• Retrieval hit rate: is the right chunk in the top 5?
• Answer faithfulness: is every statement supported by the retrieved chunks?
• Refusals: does it say "I don't know" for questions outside the documents?

Re-run this test set every time you change chunk size, embedding model or prompt.

Practical tips

On long technical documents, split by section headings rather than fixed lengths and prepend the section title to each chunk — retrieval gets noticeably more precise. And always show sources: users check answers and trust the assistant more when they can.

Need a RAG assistant for your business?

I build RAG assistants and AI automations for companies in Morocco and remotely. See my services or contact me.