What Is RAG and What Role Does Embedding Play?

1. Why Do LLMs Need a "Plug-in"? The Origin of RAG

Large language models like ChatGPT, Claude, and Gemini are powerful, but they have three inherent weaknesses:

1. Hallucination: Models can confidently fabricate facts—citing non-existent papers, inventing company financial data.
2. Knowledge cutoff: Models can't answer "what happened yesterday" or news after their training cutoff date.
3. Private data gap: General LLMs don't know your company's internal rules, product manuals, or patient records, and shouldn't touch confidential data.

Fine-tuning a model is extremely expensive (millions of dollars) and requires retraining every time data changes. Retrieval-Augmented Generation (RAG) solves all three problems with a low-cost "plug-in" approach: let the model "look up facts" before answering, then generate responses based on retrieved content.

2. The Standard RAG Pipeline: Two Phases, Three Steps

RAG is essentially a two-step "retrieve + generate" process, divided into an indexing phase (offline) and retrieval-generation phase (online).

Phase 1: Indexing (knowledge preprocessing)

1. Document loading: Read knowledge from PDFs, Word docs, web pages, databases, Notion, etc.
2. Chunking: Since LLMs have limited context windows (typically 4K-128K tokens), long documents must be split into chunks (typically 200-800 words each). Chunking strategy directly affects retrieval precision.
3. Embedding: Use an embedding model to convert each chunk into a high-dimensional vector (e.g., 1536 dimensions).
4. Storage in vector database: Store all vectors along with original text in specialized vector databases like Milvus, Pinecone, Chroma, or Weaviate.

Phase 2: Retrieval & Generation (when user asks a question)

1. Query embedding: Convert the user's question into a vector using the same embedding model.
2. Similarity search: Find the Top-K most similar chunks (K usually 3-10) in the vector database.
3. Prompt assembly: Combine "user question + retrieved chunks" into a prompt.
4. Generate answer: The LLM generates the final response based on the prompt and can cite source documents.

This "look up → answer" mechanism effectively suppresses hallucination and makes responses traceable and auditable.

3. Embedding: The Translator That Helps Machines Understand Semantics

3.1 What Is Embedding?

Embedding is a technique that converts discrete unstructured data (text, images, audio) into continuous high-dimensional numerical vectors.

The classic example is Word2Vec's "King − Man + Woman ≈ Queen"—through vector arithmetic, the machine discovers that "king" relates to "queen" the same way "man" relates to "woman." This proves that vector space can capture semantic relationships.

For a sentence like "Apple Inc. released the new iPhone," an embedding model outputs a vector like [0.012, -0.034, 0.087, ..., 0.056] (1536 dimensions). These numbers may seem meaningless, but they encode semantics:

• Sentences related to "iPhone release" cluster together
• Sentences about "apple fruit" are far from "iPhone" vectors

3.2 Evolution of Embedding Models

For Chinese scenarios, BGE series (by BAAI/Zhipu) and M3E are widely recognized as the best open-source choices. For multilingual or cross-border applications, OpenAI text-embedding-3-large remains the industry benchmark.

3.3 How to Measure Semantic Similarity Between Two Vectors?

The most common metric is Cosine Similarity:

• The closer the cosine value to 1, the more semantically similar
• 0 means orthogonal (unrelated)
• -1 means opposite (antonymic)

The formula is simple: cos(θ) = (A · B) / (‖A‖ × ‖B‖), the dot product divided by the product of vector magnitudes. Besides cosine, Euclidean distance and dot product are also common choices.

4. The Core Role of Embedding in RAG: The Bridge Between Semantics and Retrieval

Embedding is the soul component of RAG. Without embedding, RAG can only do keyword matching (like traditional search engines) and cannot understand that "I want to switch phones" and "how to buy the latest iPhone" are essentially the same intent.

Embedding takes on three critical responsibilities in RAG:

4.1 Semantic Encoding: Turning Text into "Meaning"

All text chunks to be retrieved must first be encoded into vectors via the embedding model. The quality of the embedding model directly determines the ceiling of the RAG system—a good embedding naturally brings "questions" and "answers" close in vector space, while a poor one will map obviously related sentences like "I want a refund" and "refund policy" to completely different positions.

4.2 Unified Language: Letting Questions and Documents Communicate

User queries must also be encoded using the same embedding model. Only with the same model does the "distance" in vector space have meaning. This is why enterprises that lock in an embedding model rarely switch—changing models means rebuilding the entire vector database.

4.3 Efficient Retrieval: Finding Answers Quickly in Millions of Texts

Vector databases pre-build indexes (e.g., HNSW, IVF) on embedding vectors. When a user asks a question, the system can find the most similar Top-K within milliseconds across millions of vectors. This Approximate Nearest Neighbor (ANN) search is the foundation that enables real-time RAG.

5. Advanced RAG: Pure Vector Search Isn't Enough, Hybrid Search Is the Standard

Although embedding vector search is powerful, it has a weakness: insensitive to exact matches of proper nouns, numbers, and model specifications. For example, when a user searches for "iPhone 17 Pro Max 256GB," pure vector search might return content about "iPhone 15" because they are "semantically similar."

Therefore, modern RAG systems universally adopt Hybrid Search:

1. Sparse Vector: Use BM25 or SPLADE traditional term-frequency algorithms specifically for matching keywords, model numbers, and digits.
2. Dense Vector: Use embedding models to capture semantic relations.
3. Rerank: Use specialized Cross-Encoder models (e.g., Cohere Rerank, BGE-Reranker) to precisely reorder recall results.

The combination of all three—keyword matching + semantic understanding + precise ranking—is the standard architecture for modern enterprise-grade RAG.

6. Practical Tips for Choosing Embedding Models

Choosing the right embedding model is half the battle for RAG success. Consider five dimensions:

1. Language coverage: For Chinese only, choose BGE; for Chinese-English mix, choose BGE-M3 or text-embedding-3; for multilingual, choose multilingual-e5.
2. Text length: For long documents (contracts, research reports), choose models supporting 8K+ tokens (BGE-M3 supports 8192).
3. Domain adaptation: For medical, legal, financial domains, fine-tune on domain corpora based on a general embedding.
4. Cost: OpenAI charges per token ($0.13/million tokens); open-source BGE-M3 self-hosted on GPU is more economical.
5. Benchmark scores: Reference the MTEB (Massive Text Embedding Benchmark) leaderboard, looking at combined scores across retrieval, classification, clustering, etc.

7. Future Trends in RAG and Embedding

Since 2026, RAG technology has been evolving rapidly:

• GraphRAG: Combines knowledge graphs, incorporating entity relationships into vector space to solve multi-hop reasoning problems.
• Agentic RAG: Agents autonomously decide "whether to search, what to search, when to stop," evolving from static retrieval to dynamic reasoning.
• Multimodal Embedding: Texts, images, audio, and video are unified into the same vector space, supporting cross-modal retrieval.
• Long-context Embedding: Supports million-token encoding, processing entire novels, legal case files, and research papers.
• On-device Embedding: Model miniaturization enables running directly on phones and edge devices, protecting data privacy.

8. Summary

RAG lets LLMs look up facts; Embedding lets LLMs understand facts.

• RAG solves "making models say the right thing"—through external retrieval, hallucination rates are suppressed to extremely low levels, and answers become traceable.
• Embedding solves "making retrieval find the right thing"—through semantic vector space, user intent is precisely aligned with relevant knowledge.

Together, they have become the standard tech stack for enterprise AI landing, knowledge base construction, intelligent customer service, legal assistants, medical consultation, and more scenarios. Mastering the principles and practice of RAG and Embedding is equivalent to holding the "application entry ticket" of the LLM era.

For developers, start building a minimal RAG system today using LangChain, LlamaIndex, or Spring AI frameworks. For enterprise decision-makers, prioritize RAG as a top pilot project for generative AI, starting with the highest-frequency knowledge retrieval scenarios (e.g., customer service, internal Q&A). Though small, embedding is the key to unlocking the practical door of large language models.