Retrieval-Augmented Generation (RAG) is the dominant architecture for building knowledge-aware AI applications. Instead of relying solely on what the model was trained on, RAG retrieves relevant documents from your data and feeds them into the generation context. This guide covers building RAG systems with Chinese embedding models and LLMs through TokenEase.
What is RAG?
RAG solves two critical LLM limitations:
- Knowledge cutoff: Models don't know about events after their training date
- Hallucination: Models invent facts when they lack information
By retrieving relevant documents before generating a response, RAG grounds the model in factual, up-to-date information from your knowledge base.
RAG Pipeline: User Query → Embed Query → Vector Search (Top-K docs) → Retrieve Chunks → Build Prompt (Query + Context) → Generate Answer
Chinese Embedding Landscape
Chinese text requires specialized embedding models. English-centric models like OpenAI's text-embedding-ada perform poorly on Chinese semantic tasks. Here are the top Chinese embedding models:
| Model | Dimensions | Context | Best For |
|---|---|---|---|
| BAAI/bge-large-zh-v1.5 | 1024 | 512 | General Chinese semantic search |
| BAAI/bge-m3 | 1024 | 8192 | Multilingual + long documents |
| maidalun1020/bce-embedding-base_v1 | 768 | 512 | High accuracy on C-MTEB |
| shibing624/text2vec-base-chinese | 768 | 256 | Lightweight, fast inference |
| ZhipuAI/embedding-3 | 2048 | 8192 | GLM ecosystem integration |
Building a Complete RAG System
Step 1: Document Chunking
Break documents into semantically meaningful chunks. For Chinese text, sentence and paragraph boundaries matter more than fixed token counts:
import re
def chunk_chinese_text(text: str, max_chars: int = 300, overlap: int = 50) -> list:
# Split on Chinese sentence boundaries
sentences = re.split(r'([。！？；\n])', text)
sentences = [s + p for s, p in zip(sentences[::2], sentences[1::2] + [''])]
chunks = []
current_chunk = ""
for sentence in sentences:
if len(current_chunk) + len(sentence) <= max_chars:
current_chunk += sentence
else:
if current_chunk:
chunks.append(current_chunk.strip())
current_chunk = sentence
if current_chunk:
chunks.append(current_chunk.strip())
return chunks
# Example
doc = "这是第一段。这是第二段，包含更多内容！这是第三段。"
chunks = chunk_chinese_text(doc)
print(f"Created {len(chunks)} chunks")
Step 2: Generate Embeddings
from sentence_transformers import SentenceTransformer
import numpy as np
# Load Chinese embedding model
model = SentenceTransformer('BAAI/bge-large-zh-v1.5')
# Generate embeddings for chunks
chunk_embeddings = model.encode(chunks, normalize_embeddings=True)
print(f"Embedding shape: {chunk_embeddings.shape}") # (num_chunks, 1024)
# Save for later
np.save("chunk_embeddings.npy", chunk_embeddings)
Step 3: Vector Database Setup
import chromadb
from chromadb.config import Settings
# Initialize ChromaDB (lightweight, no external dependencies)
client = chromadb.Client(Settings(anonymized_telemetry=False))
collection = client.create_collection(
name="chinese_docs",
metadata={"hnsw:space": "cosine"}
)
# Add documents
collection.add(
embeddings=chunk_embeddings.tolist(),
documents=chunks,
ids=[f"chunk_{i}" for i in range(len(chunks))],
metadatas=[{"source": "doc_1", "index": i} for i in range(len(chunks))]
)
print(f"Indexed {collection.count()} chunks")Step 4: Retrieval
def retrieve(query: str, top_k: int = 3) -> list:
# Embed the query
query_embedding = model.encode([query], normalize_embeddings=True)
# Search
results = collection.query(
query_embeddings=query_embedding.tolist(),
n_results=top_k,
include=["documents", "distances", "metadatas"]
)
return [
{
"text": doc,
"distance": dist,
"metadata": meta
}
for doc, dist, meta in zip(
results["documents"][0],
results["distances"][0],
results["metadatas"][0]
)
]
# Test
query = "什么是RAG系统？"
retrieved = retrieve(query)
for r in retrieved:
print(f"Score: {1-r['distance']:.3f} | {r['text'][:100]}...")Step 5: Generation with Retrieved Context
import requests
API_KEY = "your-tokenease-api-key"
BASE_URL = "https://tokenease.io/v1"
def rag_answer(query: str, top_k: int = 3) -> str:
# Retrieve relevant chunks
chunks = retrieve(query, top_k)
# Build context
context = "\n\n".join([f"[Document {i+1}]\n{c['text']}"
for i, c in enumerate(chunks)])
# Build prompt
prompt = f"""基于以下参考文档回答问题。如果文档中没有相关信息，请明确说明。
参考文档：
{context}
问题：{query}
请用中文回答："""
# Generate answer using Chinese LLM
response = requests.post(
f"{BASE_URL}/chat/completions",
headers={"Authorization": f"Bearer {API_KEY}"},
json={
"model": "deepseek", # or "zhipu", "qwen", "kimi"
"messages": [
{"role": "system", "content": "你是一个 helpful 的助手，基于提供的文档回答问题。"},
{"role": "user", "content": prompt}
],
"temperature": 0.3
}
)
return response.json()["choices"][0]["message"]["content"]
# Run RAG
answer = rag_answer("RAG系统有哪些优势？")
print(answer)Advanced RAG Techniques
1. Hybrid Search (Dense + Sparse)
Combine vector similarity with keyword matching for better recall:
from rank_bm25 import BM25Okapi
import jieba
# Build BM25 index
tokenized_chunks = [list(jieba.cut(chunk)) for chunk in chunks]
bm25 = BM25Okapi(tokenized_chunks)
def hybrid_search(query: str, top_k: int = 5, vector_weight: float = 0.7):
# Dense retrieval
vector_results = retrieve(query, top_k * 2)
# Sparse retrieval
query_tokens = list(jieba.cut(query))
bm25_scores = bm25.get_scores(query_tokens)
# Combine and rerank
combined = {}
for i, r in enumerate(vector_results):
combined[r["text"]] = vector_weight * (1 - r["distance"])
for i, score in enumerate(bm25_scores):
if chunks[i] in combined:
combined[chunks[i]] += (1 - vector_weight) * score / max(bm25_scores)
else:
combined[chunks[i]] = (1 - vector_weight) * score / max(bm25_scores)
# Return top_k
sorted_results = sorted(combined.items(), key=lambda x: x[1], reverse=True)
return [text for text, score in sorted_results[:top_k]]2. Re-ranking with Cross-Encoders
First retrieve 20-50 candidates with fast bi-encoder, then re-rank with a more accurate cross-encoder:
from sentence_transformers import CrossEncoder
# Load Chinese cross-encoder reranker
reranker = CrossEncoder('BAAI/bge-reranker-large')
def rerank_results(query: str, candidates: list, top_k: int = 3) -> list:
pairs = [[query, doc] for doc in candidates]
scores = reranker.predict(pairs)
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
return [doc for doc, score in ranked[:top_k]]3. Query Expansion
Use the LLM to generate multiple query variations, then aggregate results:
def expand_query(query: str) -> list:
expansion_prompt = f"生成3个与'{query}'语义相关的搜索查询变体，用中文回答，每行一个："
response = requests.post(
f"{BASE_URL}/chat/completions",
headers={"Authorization": f"Bearer {API_KEY}"},
json={
"model": "qwen",
"messages": [{"role": "user", "content": expansion_prompt}],
"temperature": 0.8
}
)
variations = response.json()["choices"][0]["message"]["content"].strip().split("\n")
return [query] + [v.strip("- ") for v in variations if v.strip()]
# Retrieve from multiple query variations
all_results = []
for q in expand_query("RAG系统"):
all_results.extend(retrieve(q, top_k=2))
# Deduplicate and rerank
# ... (deduplication logic)Evaluation Metrics
| Metric | What It Measures | Target |
|---|---|---|
| Hit Rate @K | Is the correct doc in top K? | > 90% |
| MRR (Mean Reciprocal Rank) | How high is the first relevant doc? | > 0.7 |
| Answer Relevance | Does the answer address the query? | > 4.0/5 |
| Faithfulness | Is the answer supported by retrieved docs? | > 90% |
Production Considerations
- Chunk size: 200-400 Chinese characters typically works best. Too small loses context; too large dilutes relevance.
- Embedding model: BGE-large-zh is the safest default. BGE-m3 for multilingual or long documents.
- Vector DB: ChromaDB for prototyping, Milvus or Pinecone for production scale.
- Update strategy: Incremental indexing for new documents. Full re-index only when embedding model changes.
- Caching: Cache frequent queries and their retrieved chunks to reduce latency and cost.
TokenEase Integration: All generation models (DeepSeek, GLM, Qwen, Kimi, Doubao) work identically in RAG pipelines. Switch models to optimize for cost, speed, or quality without changing your retrieval or chunking code.
Conclusion
RAG is the foundation of most production AI applications in 2026. With Chinese embedding models reaching parity with English counterparts and TokenEase providing unified access to all major Chinese LLMs, building knowledge-grounded AI systems has never been more accessible.
Start with a simple pipeline: chunk → embed → retrieve → generate. Add hybrid search and reranking as your accuracy requirements grow. The combination of BGE embeddings and DeepSeek/GLM generation delivers state-of-the-art results at a fraction of OpenAI's cost.