[Bug]: Mismatch in Length of Values and Index in generate_text_embeddings

[zz44l43]

Do you need to file an issue?

• I have searched the existing issues and this bug is not already filed.
• My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here.
• I believe this is a legitimate bug, not just a question. If this is a question, please use the Discussions area.

Describe the bug

Encountered an error when running the generate_text_embeddings function in the pipeline. The error log indicates a mismatch between the length of values (338) and the index (366) when attempting to set values in a DataFrame.

Error Details: The error traceback provided below highlights the issue occurring within the generate_text_embeddings function in generate_text_embeddings.py:
ValueError: Length of values (338) does not match length of index (366)

pipeline with the following steps:

create_base_text_units
create_base_entity_graph
create_final_entities
create_final_nodes
create_final_communities
create_final_relationships
create_final_text_units
create_final_community_reports
create_final_documents
generate_text_embeddings

The pipeline fails at the generate_text_embeddings step with the error message.

LOG:
{
"type": "error",
"data": "Error executing verb "generate_text_embeddings" in generate_text_embeddings: Length of values (338) does not match length of index (366)",
"stack": "Traceback (most recent call last):\n File "C:\Users\DJ027103\AppData\Roaming\Python\Python311\site-packages\datashaper\workflow\workflow.py", line 415, in _execute_verb\n result = await result\n ^^^^^^^^^^^^\n File "D:\anaconda\Lib\site-packages\graphrag\index\workflows\v1\subflows\generate_text_embeddings.py", line 56, in generate_text_embeddings\n await generate_text_embeddings_flow(\n File "D:\anaconda\Lib\site-packages\graphrag\index\flows\generate_text_embeddings.py", line 106, in generate_text_embeddings\n await _run_and_snapshot_embeddings(\n File "D:\anaconda\Lib\site-packages\graphrag\index\flows\generate_text_embeddings.py", line 129, in _run_and_snapshot_embeddings\n data["embedding"] = await embed_text(\n ~~~~^^^^^^^^^^^^^\n File "C:\Users\DJ027103\AppData\Roaming\Python\Python311\site-packages\pandas\core\frame.py", line 4311, in setitem\n self._set_item(key, value)\n File "C:\Users\DJ027103\AppData\Roaming\Python\Python311\site-packages\pandas\core\frame.py", line 4524, in _set_item\n value, refs = self._sanitize_column(value)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "C:\Users\DJ027103\AppData\Roaming\Python\Python311\site-packages\pandas\core\frame.py", line 5266, in _sanitize_column\n com.require_length_match(value, self.index)\n File "C:\Users\DJ027103\AppData\Roaming\Python\Python311\site-packages\pandas\core\common.py", line 573, in require_length_match\n raise ValueError(\nValueError: Length of values (338) does not match length of index (366)\n",
"source": "Length of values (338) does not match length of index (366)",
"details": null
}
{
"type": "error",
"data": "Error running pipeline!",
"stack": "Traceback (most recent call last):\n File "D:\anaconda\Lib\site-packages\graphrag\index\run\run.py", line 269, in run_pipeline\n result = await _process_workflow(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File "D:\anaconda\Lib\site-packages\graphrag\index\run\workflow.py", line 105, in _process_workflow\n result = await workflow.run(context, callbacks)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "C:\Users\DJ027103\AppData\Roaming\Python\Python311\site-packages\datashaper\workflow\workflow.py", line 369, in run\n timing = await self._execute_verb(node, context, callbacks)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "C:\Users\DJ027103\AppData\Roaming\Python\Python311\site-packages\datashaper\workflow\workflow.py", line 415, in _execute_verb\n result = await result\n ^^^^^^^^^^^^\n File "D:\anaconda\Lib\site-packages\graphrag\index\workflows\v1\subflows\generate_text_embeddings.py", line 56, in generate_text_embeddings\n await generate_text_embeddings_flow(\n File "D:\anaconda\Lib\site-packages\graphrag\index\flows\generate_text_embeddings.py", line 106, in generate_text_embeddings\n await _run_and_snapshot_embeddings(\n File "D:\anaconda\Lib\site-packages\graphrag\index\flows\generate_text_embeddings.py", line 129, in _run_and_snapshot_embeddings\n data["embedding"] = await embed_text(\n ~~~~^^^^^^^^^^^^^\n File "C:\Users\DJ027103\AppData\Roaming\Python\Python311\site-packages\pandas\core\frame.py", line 4311, in setitem\n self._set_item(key, value)\n File "C:\Users\DJ027103\AppData\Roaming\Python\Python311\site-packages\pandas\core\frame.py", line 4524, in _set_item\n value, refs = self._sanitize_column(value)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "C:\Users\DJ027103\AppData\Roaming\Python\Python311\site-packages\pandas\core\frame.py", line 5266, in _sanitize_column\n com.require_length_match(value, self.index)\n File "C:\Users\DJ027103\AppData\Roaming\Python\Python311\site-packages\pandas\core\common.py", line 573, in require_length_match\n raise ValueError(\nValueError: Length of values (338) does not match length of index (366)\n",
"source": "Length of values (338) does not match length of index (366)",
"details": null
}

Using azure for both llm and embedding with text-embedding-ada-002

Steps to reproduce

running graphrag index with version of 0.4.0

Expected Behavior

No response

GraphRAG Config Used

# Paste your config here

Logs and screenshots

No response

Additional Information

• GraphRAG Version:
• Operating System:
• Python Version:
• Related Issues:

[davidloiret]

Hello,

I'm encountering the same error when following the "Getting Started" guide. Specifically, the issue arises when using the input text downloaded from:

curl https://www.gutenberg.org/cache/epub/24022/pg24022.txt

Any guidance on how to resolve this would be greatly appreciated. Thank you!

[Match-Yang]

Hello,

I'm encountering the same error when following the "Getting Started" guide. Specifically, the issue arises when using the input text downloaded from:

curl https://www.gutenberg.org/cache/epub/24022/pg24022.txt
Any guidance on how to resolve this would be greatly appreciated. Thank you!

Same!!! Need help

[skyqqcloud]

same problem with graphrag v0.5

[dipakmeher]

This issue got resolved for me when I changed chunk size from 1200 to 300 with Open AI embedding model and llama 3 for text generation.

[LiveWithTrance]

This bug might be the embedding model did not work. I solved this problem by editting the graphrag\index\operations\embed_text\embed_text.py. In line 187, there should not filter those embedding is None. Otherwise the later zip in this file in line 193 will throw error.!

[nijisakai]

This bug might be the embedding model did not work. I solved this problem by editting the graphrag\index\operations\embed_text\embed_text.py. In line 187, there should not filter those embedding is None. Otherwise the later zip in this file in line 193 will throw error.!

Your image is down. Can you fix it?

[lizhongv]

此错误可能是嵌入模型不起作用。我通过编辑 graphrag\index\operations\embed_text\embed_text.py 解决了这个问题。在第 187 行中，不应过滤那些嵌入为 None。否则此文件中第 193 行中后面的 zip 将引发错误。！

The issue is resolved!