🔧 Built an automated chunking parameter optimizer for RAGFlow

[stranger00135]

Hey RAGFlow team and community! 👋

I built an open-source tool that automatically finds the optimal chunking parameters for your RAGFlow knowledge bases.

The problem

When building RAG apps with RAGFlow, chunking parameters (chunk_size, overlap, auto_questions, etc.) hugely impact retrieval quality — but most of us just guess. Wrong params = wrong chunks being picked up,

The solution

RAGFlow Optimizer runs automated experiments to scientifically find the best config for each document type.

How it works

1. Upload your documents (organized by type)
2. Run python main.py run
3. Tool tests different presets (general, manual, QA) → fine-tunes the winner
4. Get a report with the optimal config + metrics (Precision@3, MRR, Fail Rate)

Key features

• 🔬 Two-phase optimization (preset screening → parameter fine-tuning)
• 📊 LLM-based relevance evaluation with rigorous metrics
• 🛡️ Noise injection testing (distractor documents)
• 📁 Per-folder optimization (different doc types → different configs)
• ⚡ KB reuse — upload once, re-chunk multiple times
• 🇨🇳🇬🇧 Bilingual docs (English + Chinese)

Quick start

git clone https://github.com/stranger00135/ragflow-optimizer.git
cd ragflow-optimizer
pip install -r requirements.txt
cp .env.example .env  # add your RAGFlow + LLM credentials
python main.py run

Example results

On my test corpus, optimized configs improved retrieval by 15-25% over default settings:

• Manual chunking + 1024 tokens beat General + 512 tokens by 15% on policy documents
• auto_questions=5 improved retrieval for procedure docs but hurt performance on data tables
• Different document types genuinely need different configs

Would love feedback from the community! What features would be most useful? PRs welcome.

⭐ https://github.com/stranger00135/ragflow-optimizer

[KevinHuSh]

Is that feasible to submit a PR to include in ./tests/optimizer?

[stranger00135]

Yes, but it is currently intended as a separate system. E.g. it will not start at the same time as ragflow, and it can finetune a remote ragflow server. Is this format accepted for a PR?

[xXMrNidaXx]

This is excellent! Chunking parameter tuning is one of those things everyone knows matters but few actually measure.

What I like:

1. Two-phase approach — screening then fine-tuning is smart. Exhaustive grid search would take forever.

2. LLM-based evaluation — human-labeled test sets are expensive. LLM-as-judge scales well (with caveats).

3. Noise injection testing — this is often skipped but critical. Retrieval that works on clean data often fails with realistic noise.

Questions/suggestions:

1. Cross-validation — do you split docs for train/test, or evaluate on same docs used for optimization? Risk of overfitting to specific queries.

2. Embedding model interaction — chunking optima likely differ per embedding model. Worth noting which model results are from.

3. Latency tracking — smaller chunks = more chunks = slower retrieval. Worth including in the optimization objective?

The 15-25% improvement aligns with what we see. At Revolution AI we have found similar gains from systematic chunking optimization — default params are almost never optimal.

Great tool for the community. Starred! ⭐

[xXMrNidaXx]

Automated chunking optimization is brilliant! At RevolutionAI (https://revolutionai.io) we tune RAG extensively.

Key parameters to optimize:

1. Chunk size:

• Too small: Lost context
• Too large: Noise in retrieval
• Sweet spot: 256-512 tokens typically

2. Overlap:

# Prevent context loss at boundaries
overlap = chunk_size * 0.1  # 10-20% overlap

3. Strategy:

• Semantic: Split on meaning
• Sentence: Preserve grammar
• Token: Consistent size

Evaluation metrics:

def evaluate_chunking(chunks, queries, ground_truth):
    retrieved = retrieve(chunks, queries)
    recall = calculate_recall(retrieved, ground_truth)
    precision = calculate_precision(retrieved, ground_truth)
    return {"recall": recall, "precision": precision}

Would love to see this as a RAGFlow feature!