Tim kiem theo tu khoa truyen thong that bai khi nguoi dung dat cau hoi khac voi cach noi dung duoc viet. Semantic search giai quyet van de nay bang cach hieu y nghia thay vi chi so khop tu. Trong bai huong dan nay, ban se xay dung mot search engine hoan chinh co kha nang hieu truy van ngon ngu tu nhien su dung OpenAI embeddings va vector database.

Ket thuc bai nay, ban se co mot he thong hoat dong thuc su, co the tim kiem bat ky tap van ban nao voi kha nang hieu ngon ngu nhu con nguoi.

Ban Se Xay Duoc Gi

Kien truc kha don gian:

  1. Tai lieu duoc chuyen thanh vector embeddings qua OpenAI
  2. Embeddings duoc luu trong vector database (ChromaDB)
  3. Truy van cua nguoi dung cung duoc embed tuong tu
  4. He thong tim tai lieu co vector tuong dong nhat
  5. Ket qua duoc xep hang theo do lien quan ngu nghia

Day la nen tang cua cac he thong RAG (Retrieval-Augmented Generation) dang duoc su dung trong san pham cua cac cong ty nhu Notion, Stripe va Shopify.

Yeu Cau

Buoc 1: Thiet Lap Du An

Tao thu muc moi va cai dat cac thu vien can thiet:

mkdir ai-search-engine && cd ai-search-engine
python -m venv venv
source venv/bin/activate
pip install openai chromadb python-dotenv flask

Tao file .env de luu API key:

OPENAI_API_KEY=sk-your-key-here

Buoc 2: Tao Pipeline Embedding

Cot loi cua moi he thong semantic search la ham embedding. Ham nay chuyen van ban thanh vector nhieu chieu de nam bat y nghia.

# embeddings.py
import os
from openai import OpenAI
from dotenv import load_dotenv
 
load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
 
def get_embedding(text: str, model: str = "text-embedding-3-small") -> list[float]:
    """Chuyen van ban thanh vector embedding."""
    text = text.replace("\n", " ").strip()
    response = client.embeddings.create(input=[text], model=model)
    return response.data[0].embedding
 
def get_embeddings_batch(texts: list[str], model: str = "text-embedding-3-small") -> list[list[float]]:
    """Embed nhieu van ban cung luc de tiet kiem."""
    cleaned = [t.replace("\n", " ").strip() for t in texts]
    response = client.embeddings.create(input=cleaned, model=model)
    return [item.embedding for item in response.data]

Model text-embedding-3-small cho chat luong tot voi gia chi $0.02 cho moi trieu token. Neu can do chinh xac cao hon, dung text-embedding-3-large.

Buoc 3: Thiet Lap Vector Database

ChromaDB la vector database nhe, chay local ma khong can ha tang phuc tap.

# database.py
import chromadb
from embeddings import get_embedding, get_embeddings_batch
 
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_or_create_collection(
    name="documents",
    metadata={"hnsw:space": "cosine"}
)
 
def index_documents(documents: list[dict]):
    """Index mot lo tai lieu vao vector store."""
    texts = [doc["content"] for doc in documents]
    ids = [doc["id"] for doc in documents]
    metadatas = [{"title": doc.get("title", ""), "source": doc.get("source", "")} for doc in documents]
 
    embeddings = get_embeddings_batch(texts)
 
    collection.upsert(
        ids=ids,
        embeddings=embeddings,
        documents=texts,
        metadatas=metadatas
    )
    print(f"Da index {len(documents)} tai lieu.")
 
def search(query: str, n_results: int = 5) -> list[dict]:
    """Tim tai lieu tuong dong ngu nghia voi truy van."""
    query_embedding = get_embedding(query)
 
    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results,
        include=["documents", "metadatas", "distances"]
    )
 
    return [
        {
            "content": results["documents"][0][i],
            "metadata": results["metadatas"][0][i],
            "score": 1 - results["distances"][0][i]
        }
        for i in range(len(results["documents"][0]))
    ]

Buoc 4: Xay Script Nhap Du Lieu

Ban can cach de dua tai lieu vao search engine. Day la script xu ly cac file van ban tu mot thu muc:

# ingest.py
import os
import hashlib
from database import index_documents
 
def chunk_text(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
    """Chia van ban thanh cac doan chong lan de retrieval tot hon."""
    words = text.split()
    chunks = []
    for i in range(0, len(words), chunk_size - overlap):
        chunk = " ".join(words[i:i + chunk_size])
        if chunk.strip():
            chunks.append(chunk)
    return chunks
 
def ingest_directory(directory: str):
    """Xu ly tat ca file van ban trong thu muc."""
    documents = []
    for filename in os.listdir(directory):
        if not filename.endswith((".txt", ".md")):
            continue
        filepath = os.path.join(directory, filename)
        with open(filepath, "r") as f:
            content = f.read()
 
        chunks = chunk_text(content)
        for i, chunk in enumerate(chunks):
            doc_id = hashlib.md5(f"{filename}_{i}".encode()).hexdigest()
            documents.append({
                "id": doc_id,
                "content": chunk,
                "title": filename,
                "source": filepath
            })
 
    index_documents(documents)
    print(f"Da nhap {len(documents)} doan tu {directory}")
 
if __name__ == "__main__":
    ingest_directory("./data")

Buoc 5: Tao Search API

Goi tat ca trong Flask API don gian de cac ung dung khac co the truy van search engine cua ban:

# app.py
from flask import Flask, request, jsonify
from database import search
 
app = Flask(__name__)
 
@app.route("/search", methods=["POST"])
def search_endpoint():
    data = request.get_json()
    query = data.get("query", "")
    limit = data.get("limit", 5)
 
    if not query:
        return jsonify({"error": "Query is required"}), 400
 
    results = search(query, n_results=limit)
    return jsonify({"results": results, "query": query})
 
@app.route("/health", methods=["GET"])
def health():
    return jsonify({"status": "ok"})
 
if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8000, debug=True)

Buoc 6: Kiem Thu Search Engine

Truoc tien, tao thu muc data/ voi vai file van ban mau, sau do chay script nhap du lieu:

mkdir data
# Them cac file .txt hoac .md vao ./data/
python ingest.py

Khoi dong server va gui truy van thu:

python app.py &
curl -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -d '{"query": "xac thuc hoat dong nhu the nao", "limit": 3}'

Ban se thay ket qua lien quan ve ngu nghia ngay ca khi tu "xac thuc" khong xuat hien chinh xac trong tai lieu.

Meo Thuc Te

Chien luoc chia doan rat quan trong. Doan nho (200-500 tu) cho ket qua chinh xac hon nhung mat ngu canh. Doan lon (500-1000 tu) giu ngu canh nhung co the lam loang do lien quan. Hay thu nghiem voi du lieu cu the cua ban.

Su dung loc metadata. ChromaDB ho tro loc theo cac truong metadata. Them danh muc, ngay thang hoac tac gia metadata de nguoi dung co the thu hep pham vi tim kiem.

Cache embeddings triet de. Goi API embedding ton tien. Luu embeddings vao database va chi embed lai khi noi dung thay doi.

Can nhac hybrid search. Ket hop vector search voi keyword search (BM25) de co ket qua tot nhat. Thu vien rank-bm25 giup viec nay kha don gian.

Theo doi chi phi. Model text-embedding-3-small ton $0.02 cho moi trieu token. Mot tap 10,000 tai lieu (500 tu moi cai) ton khoang $0.05 de embed toan bo.

Buoc Tiep Theo

Khi search engine co ban da hoat dong, hay can nhac cac nang cap sau:

  • Them buoc reranking bang cross-encoder model de co ket qua top chinh xac hon
  • Ap dung query expansion de xu ly tim kiem mo ho
  • Them giao dien chat su dung ket qua tim kiem lam context (RAG day du)
  • Deploy len production voi managed vector database nhu Pinecone hoac Weaviate