Missing custom metadata "file_url" when retrieving document in Dify

[Hories0106]

Body:

I'm using RAGFlow v0.24.0. In my knowledge base, I added a custom metadata field named file_url to a specific file (screenshot attached showing that I have added this metadata). However, when I retrieve this document through Dify, the returned metadata does not include the file_url field. Below is the metadata I received from Dify:

json
"metadata": {
"_source": "knowledge",
"data_source_type": "external",
"dataset_id": "9b1529ac-4beb-4067-8795-114fd7b03a3a",
"dataset_name": "人事总务（26/3/10）",
"doc_metadata": {
"dataset_id": "9b1529ac-4beb-4067-8795-114fd7b03a3a",
"dataset_name": "人事总务（26/3/10）",
"doc_id": "3a26cc001d1d11f1866c6d65ecb9f55a",
"score": 0.3046044653646284,
"title": "E-Mail管理规定（中）.pdf"
},
"document_id": "E-Mail管理规定（中）.pdf",
"document_name": "E-Mail管理规定（中）.pdf",
"position": 3,
"retriever_from": "workflow",
"score": 0.3046044653646284
},
"title": "E-Mail管理规定（中）.pdf"
As you can see, the custom field file_url is missing.

Steps to reproduce:

1. In RAGFlow v0.24.0, add a file to a knowledge base.

2. Add a custom metadata field file_url with some value (e.g., a local file path) to that file (see screenshot).

3. Use Dify to retrieve this document (e.g., via a retrieval workflow).

4. Observe the returned metadata – the file_url field is not present.

Expected behavior: The custom metadata file_url should be included in the metadata returned when the document is retrieved.

Actual behavior: The file_url field is absent.

Environment:

RAGFlow version: v0.24.0

Dify version: (unknown, but the issue seems to be on RAGFlow side)

Deployment: (e.g., local Docker, cloud – not specified)

Additional context: Screenshot attached showing the metadata added in RAGFlow. Please let me know if you need any further information.

[dosubot[bot]]

Hi @Hories0106! I'm Dosu and I’m helping the ragflow team.

This is a known limitation. RAGFlow's code does include custom metadata fields (like your file_url) in the Dify retrieval response, but there's a pagination bug that affects large knowledge bases.

The likely cause: When building the metadata cache for retrieval responses, RAGFlow uses a default pagination of page=1, page_size=30 with a hard cap of 128 documents. If your document is positioned beyond these limits in the knowledge base, its custom metadata won't be included (#11533).

To verify this is your issue, check how many documents are in your knowledge base and where your document falls in the list. If it's beyond position 30-128, this is likely the cause.

Possible workarounds:

1. If you have a smaller knowledge base (under 30 documents), the metadata should appear - double-check that the metadata was saved correctly after clicking "Save"
2. Consider splitting large knowledge bases into smaller ones if metadata is critical for your workflow

There are also other related Dify integration issues being tracked, including #13335 regarding missing document_id fields. You may want to check if an updated version of RAGFlow addresses some of these issues.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[Hories0106]

@dosu
I only have 27 files in this knowledge base, and this is also the latest version of Ragflow.

[dosubot[bot]]

Thanks for clarifying - with only 27 files, the pagination limit shouldn't be the issue.

Looking at the code, RAGFlow retrieves custom metadata via getattr(doc, 'meta_fields', {}), which returns an empty dict if the document doesn't have the meta_fields attribute populated. This suggests your custom metadata may not be getting stored/retrieved properly from the backend.

A few things to check:

1. Verify the metadata was saved: After adding file_url in the UI and clicking Save, try refreshing the page to confirm it persisted
2. Check RAGFlow directly: Use RAGFlow's own retrieval API (not via Dify) to see if file_url appears there - this would help isolate whether the issue is on the RAGFlow storage side or the Dify integration layer
3. Recent fixes: There have been metadata-related fixes in PR #13337 and PR #13509 - make sure these are included in your version

Could you test retrieving directly through RAGFlow's API to see if the custom metadata appears there?

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[Hories0106]

Hi @dosu,

Thanks for your detailed guidance. I followed your suggestion and tested the retrieval directly via RAGFlow's API using the Python script
The output shows:
文档名: E-Mail管理规定（中）.pdf 元数据: None meta_fields: None
I also checked the complete chunk data, but no file_url field appears anywhere. I have confirmed that the requests library is installed and that I'm using the correct Python interpreter (Python 3.13). The knowledge base contains only 27 files, so pagination should not be an issue.

It seems that the custom metadata file_url is not being stored or retrieved by RAGFlow internally. Could you advise what might be going wrong or what further steps I can take to debug this? Should I check something in the RAGFlow database or logs?

Thanks again for your help!

[dosubot[bot]]

Thanks for testing that - getting None from RAGFlow's API confirms the issue is on the storage side, not the Dify integration.

Here's what's happening: Custom metadata in RAGFlow is stored in Elasticsearch/Infinity (in an index named ragflow_doc_meta_{tenant_id}), not in MySQL. When you save metadata via the UI, it calls the /set_meta endpoint which writes to this ES index.

Debugging steps:

1. Check if the save actually succeeded - Look at RAGFlow logs when you click "Save" in the metadata editor. The service logs lines like:

   • Successfully inserted metadata for document {doc_id} (success)
   • Failed to insert metadata for document {doc_id} (failure)
   • Failed to create metadata table {index_name} (ES index creation issue)

2. Query the metadata directly - Use the /api/v1/document/infos endpoint with your doc_id to see what RAGFlow has stored:

   curl -X POST "http://your-ragflow-host/api/v1/document/infos" \
     -H "Authorization: Bearer YOUR_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"id": "3a26cc001d1d11f1866c6d65ecb9f55a"}'

3. Check Elasticsearch directly - If you have ES access, query the index:

   curl "http://your-es-host:9200/ragflow_doc_meta_YOUR_TENANT_ID/_search?q=id:3a26cc001d1d11f1866c6d65ecb9f55a"

The most likely cause is either the metadata save is failing silently (check logs), or there's an issue with the ES index not being created properly for your tenant.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}