# So, What Even Is RAG? AI-powered RAG agent. RAG this. RAG that. You: “What even is RAG?” Tbh, shit’s confusing as hell at first. But the deeper you go, the better it gets. ## **What LLMs OFTEN get wrong at** We all have had this moment before vivas. Cram all the information a few hours before your turn. Then yap at the external examiner and hope for the best. But then the examiner asks: “Okay, but why did you make that decision in your project?” Now all quiet at the viva hall. LLMs do just that. They are designed to generate plausible responses. Which means…they simply dunno how to say ‘I dunno’. And then there’s the problem of outdated information. Imagine your examiner asks: “Why did you use that approach when there’s a newer and better way of doing it?” You’re in trouble. LLMs can face the same problem when their knowledge doesn’t contain the latest information. So… **How do we give an LLM the information it needs when it needs it?** ## **Introducing Retrieval-Augmented Generation** Press enter or click to view image in full size Don’t get intimated by the name. Just a fancy name for a simple process: find relevant info., give it to the model and let it generate an answer. That’s it. Let’s break down what’s actually happening under the hood. ## **Understanding the problem** Picture this: you’re building an AI assistant for your company. Now, the company has several documents: employee handbook, internal FAQs, security policies, etc. Let’s say that you’re finished building it. People will want to test it for sure. So, someone asks the model: > *“How many days of parental leave do I get?”* Now the problem is, the model *doesn’t* know what your company policies are. Nor can you dump a 500 documents \[each of several hundred pages\] on a poor model and ask it to spit out an answer \[this leads to overfitting, context window overload, etc.\] ## **Process behind RAG** Instead of asking the LLM to answer the question from its own knowledge, we first retrieve relevant information from our own knowledge base. Then we give that information to the LLM. The LLM uses that information to generate the answer. This means the model doesn’t need to know everything. We just need to feed it the **right information** at the **right time**. But now we have another problem: **How do we figure out which information is relevant in the first place?** ## **Retrieval** We don’t want to send all of them to the LLM for every question. We first need to **find the relevant bits**, as finding every single one is resource-intensive, very time-consuming! So we split the documents into smaller pieces called **chunks**. One chunk might contain the parental leave policy, another the security policy, and another the expense policy. ### **How do we find the right chunk?** A user might ask: *“How long can I stay away from work after having a baby?”* The document says: *“Employees are entitled to 26 weeks of parental leave.”* Different words. Same meaning. For tackling this unique issue, we introduce **embeddings**. ### **Embeddings** An embedding converts text into numbers that capture its meaning. Similar text ends up close together in this mathematical space \[we use a terminology called vector distance: meaning, how far are 2 vectors away from each other?\]. We store these embeddings in a **vector database**. When a question comes in, we embed it too and search for the closest chunks. That gives us the relevant chunks, which are then passed to the LLM as context. ## **Augmentation** We’ve found the relevant chunks. Now we add them to the LLM’s **context** alongside the user’s question: ```plaintext Context: Employees are entitled to 26 weeks of parental leave. Question: How long can I stay away from work after having a baby? ``` The LLM now has the information it needs to answer. ## **Generation** Now the LLM does what it’s extremely good at: **generating the answer.** It takes the user’s question and the retrieved context, then produces a response: > *“You’re entitled to 26 weeks of parental leave.”* That’s the final step from the process of: **Retrieve → Augment → Generate** ### **But there’s a catch.** The LLM can only work with the **information we retrieve**. If we retrieve the wrong chunk, the answer can still be wrong. Here’s an example: **Question:** “How long is parental leave?” **Retrieved:** “Notify your manager 30 days before taking parental leave.” **Answer:** “You need to notify your manager 30 days before taking parental leave.” ### **How to make it better, then?** So, how do we make sure we retrieve the **right** information? A few things matter: **Chunking:** How we split documents can determine whether the useful context stays together. **Embeddings:** Better embeddings make it easier to match questions with relevant content. **Search:** We can combine semantic search with keyword search to catch both meaning and exact terms. **Reranking:** Retrieve a few candidates first, then rank them again to find the most relevant ones. The goal is simple: **Get the right context in front of the LLM.** ## **To conclude** RAG isn’t magic. Find the right information. Give it to the LLM. Let it answer. That’s it. The LLM doesn’t need to know everything. It just needs the **right context** at the **right time.** And in RAG, that’s the real game: Garbage in, garbage out and vice versa. Now you know what all the “RAG this, RAG that” is about.