[Bug]: document_metadata missing for most chunks in large dataset

[gileslloyd]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (Language Policy).
• Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) (Language Policy).
• Please do not modify this template :) and fill in all the required fields.

RAGFlow workspace code commit ID

cfdcceb

RAGFlow image version

v0.22.1

Other environment information

Actual behavior

When using the ragflow_retrieval MCP tool with a dataset containing more than 30 documents, chunks from documents 31+ do not contain document_metadata. This is because the metadata is fetched from the document metadata cache. That cache is built using the list documents endpoint with the default pagination parameters of page=1, page_size=30:

ragflow/mcp/server/server.py

Line 243 in 2fd5ac1

docs_res = self._get(f"/datasets/{dataset_id}/documents")

ragflow/api/apps/sdk/doc.py

Line 547 in 2fd5ac1

page = int(q.get("page", 1))

Ultimately that means we'll only ever be able to see the metadata for the first 30 docs. Even if the logic is altered to fetch all the pages when building the metadata cache there is also a hard limit of 128 documents.

ragflow/mcp/server/server.py

Line 60 in 2fd5ac1

_MAX_DOCUMENT_CACHE = 128

ragflow/mcp/server/server.py

Line 104 in 2fd5ac1

if len(self._dataset_metadata_cache) > self._MAX_DATASET_CACHE:

Expected behavior

When using the ragflow_retrieval tool, every chunk should include the relevant document_metadata

Steps to reproduce

1. Create a dataset,
2. Upload more than 30 documents,
3. Use the langflow_retrieval MCP tool, using a query which will return chunks from one of the later documents in the set,
4. Check the response to verify that those chunks do not include a document_metadata property

Additional information

No response

[dosubot]

Hi @gileslloyd! I'm Dosu and I’m helping the ragflow team.

The missing document_metadata for chunks beyond the 30th document is caused by two hard-coded limits: the metadata cache is built using the /datasets/{dataset_id}/documents endpoint with default pagination (page=1, page_size=30), and the cache itself is capped at 128 documents. These limits are set as constants in the code and cannot be changed via configuration or environment variables. As a result, only the first 30 documents' metadata are fetched and cached, leaving later documents without metadata in retrieval responses. There are no recent code changes or workarounds addressing this issue, and no comments in the code explain the rationale for these limits. Resolving this would require modifying the cache logic to fetch all pages and reconsidering or making the cache cap configurable. You can review the relevant code and limits here and here.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}