Multimodal RAG with/without images issues

[jlfloresTeika]

Self Checks

• I have read the Contributing Guide and Language Policy.
• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report, otherwise it will be closed.
• 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
• Please do not modify this template :) and fill in all the required fields.

Dify version

latest docker release

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I did a Multimodal Knowledge with a docx containing images and text, I have created a tool workflow for an Agent to retrieve from the multimodal rag and I find this error consistently, error that in the non-multimodal rag workflows I don't find.

"tool_call_name": "multimodal_rag",
"tool_response": "tool invoke error: read tool response failed: request failed: Attachments variable is not array file or file type."

When I enable the parameter for the images (files array) and feed my chat workflow that is linked with an agent to the multimodal_rag tool workflow the error is:

Object of type File is not JSON serializable

Any help is appreciated, thanks DiFy team and contributors, this feature looks very promising!

For other details you may need to debug, i'm here. ty

✔️ Expected Behavior

The multimodal rag workflow should act like the non-multimodal regarding inputs and ouputs despite image input detail since is optional and it seems to not work as expected.

PD: When I test retrival directly from the retrieval tool it works good. That's why is strange that in an agent with the tool workflow attatched doesn't behave the same.

❌ Actual Behavior

The multimodal rag workflow tool I've configured doesn't behave as expected, I'm experiencing the avobe mentioned issues.

[dosubot]

The errors you’re seeing are due to how Dify handles file/image attachments in multimodal RAG agent workflows. The “Object of type File is not JSON serializable” error happens when a File object is passed directly instead of being converted to a JSON-serializable format (like a dict with metadata and a URL). The “Attachments variable is not array file or file type” error means the input validation expects an array of file references, but the input doesn’t match—often because the files weren’t properly transformed to URLs or plugin-compatible dicts before being sent to the tool workflow.

Direct retrieval works because it handles file/image serialization differently, but agent tool workflows require explicit conversion. Dify’s code expects files to be passed as references (URLs, file IDs, or plugin-compatible dicts), not as raw File objects. The File class provides methods like .to_dict() and .to_plugin_parameter() for this purpose, and the ToolFileMessageTransformer is responsible for converting File objects into URLs or plugin parameters for serialization in workflows. If this transformation is skipped, serialization errors occur1.

To resolve this:

• Ensure that any file/image attachments are passed as URLs or plugin-compatible dicts, not as raw File objects. Use .to_dict() or .to_plugin_parameter() if you’re working with custom plugins or integrations.
• Make sure your workflow nodes are configured to output file/image references, and that the LLM node’s image input is connected with the “vision” feature enabled.
• Set the environment variables MULTIMODAL_SEND_IMAGE_FORMAT (to url or base64, depending on your model’s requirements) and INTERNAL_FILES_URL (to a Docker-network-accessible URL) in your Docker deployment23.
• Avoid referencing image/file variables in the System Prompt; only use them in User Prompts, as only User Prompts are serialized properly in current Dify versions4.
• If you’re using official plugins, make sure you’re on the latest versions (OpenAI plugin >= 0.0.14, Azure OpenAI >= 0.0.11, OpenRouter >= 0.0.6) to avoid known serialization bugs5.

If you still see errors, double-check that all file/image data is correctly formatted and passed as references, and consider logging the serialized prompt messages before sending them to the plugin to debug further.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[jlfloresTeika]

But when I don't send images as input since in the multimodal rag images are optional how is that the error is: "tool_response": "tool invoke error: read tool response failed: request failed: Attachments variable is not array file or file type." ? also inside my agent workflow how can I convert the image / file inputs to serializable json like the tool expects? @dosu

[dosubot]

That error happens because Dify expects the attachments variable to always be an array—even if it's empty—not null or another type. If your agent workflow sends attachments: null or omits the field, or if it accidentally passes a raw File object instead of a serializable value, you'll get the "Attachments variable is not array file or file type" error. The validator only accepts arrays of file references (dicts/URLs), not File objects or other types (source).

To fix this, always send attachments: [] when there are no images/files, or attachments: [file1, file2, ...] where each file is a dict (not a File object). To convert a File object to a serializable dict in your agent workflow, use file.to_dict() or file.to_plugin_parameter() (source). For example:

# If you have a list of File objects
attachments = [file.to_dict() for file in files]  # or file.to_plugin_parameter()
# If no files/images:
attachments = []

Then, pass attachments as part of your tool input payload:

{
  "attachments": []
}

or, with files:

{
  "attachments": [
    {
      "filename": "example.png",
      "mime_type": "image/png",
      "url": "https://your-signed-url",
      ...
    }
  ]
}

Never pass the File object itself—always use the dict. This will keep your workflow compatible with Dify's multimodal RAG input validation and serialization logic (source).

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[jlfloresTeika]

Thanks @dosu but is still not clear how to configure my workflow to convert the attachments variable to always be an array, is there a step-by-step that you could explain to me? In short, how to configure my nodes to work like the multimodal RAG expects. Thanks.

[dosubot]

Here’s how to configure your workflow so the attachments variable is always an array, as required by multimodal RAG:

1. In the Dify workflow editor, make sure the node that provides files/images (like an upload or retrieval node) outputs a variable (e.g., attachments) and that this variable is set as a FILES type (not FILE). The FILES parameter type ensures that Dify always wraps the value as an array—even if it’s a single file or nothing at all. If no files are present, it will be an empty array, which is valid and prevents the “not array file or file type” error [source].

2. When connecting nodes, always pass the attachments variable as-is to the tool node. If you’re using custom code or plugins, convert each File object to a dict using .to_dict() or .to_plugin_parameter() before passing it along. For example:

   attachments = [file.to_dict() for file in files]  # or file.to_plugin_parameter()
   # If no files:
   attachments = []

   Never pass raw File objects—always use their dict representation [source].

3. If you’re not sending any files/images, make sure the attachments variable is set to an empty array ([]), not null or omitted. Dify’s backend will handle this correctly if the parameter is FILES type, but if you’re building payloads manually, always include "attachments": [].

4. Example payloads:

   • With files:

     {
       "attachments": [
         {
           "filename": "example.png",
           "mime_type": "image/png",
           "url": "https://your-signed-url"
         }
       ]
     }

   • With no files:

     {
       "attachments": []
     }

5. Common mistakes: Passing a File object directly (not serializable), omitting the attachments field, or using the wrong parameter type (FILE instead of FILES). Also, don’t reference image/file variables in the System Prompt—only in User Prompts.

6. Debug tip: If you’re still seeing errors, log the attachments variable right before the tool node is invoked to confirm it’s always an array of dicts or an empty array.

This setup ensures your workflow is always compatible with multimodal RAG’s input validation and serialization logic [source].

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[fatelei]

Self Checks

• I have read the Contributing Guide and Language Policy.[x] This is only for bug report, if you would like to ask a question, please head to Discussions.[x] I have searched for existing issues search for existing issues, including closed ones.[x] I confirm that I am using English to submit this report, otherwise it will be closed.[x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）[x] Please do not modify this template :) and fill in all the required fields.

Dify version

latest docker release

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I did a Multimodal Knowledge with a docx containing images and text, I have created a tool workflow for an Agent to retrieve from the multimodal rag and I find this error consistently, error that in the non-multimodal rag workflows I don't find.

"tool_call_name": "multimodal_rag", "tool_response": "tool invoke error: read tool response failed: request failed: Attachments variable is not array file or file type."

When I enable the parameter for the images (files array) and feed my chat workflow that is linked with an agent to the multimodal_rag tool workflow the error is:

Object of type File is not JSON serializable

Any help is appreciated, thanks DiFy team and contributors, this feature looks very promising!

For other details you may need to debug, i'm here. ty

✔️ Expected Behavior

The multimodal rag workflow should act like the non-multimodal regarding inputs and ouputs despite image input detail since is optional and it seems to not work as expected.

PD: When I test retrival directly from the retrieval tool it works good. That's why is strange that in an agent with the tool workflow attatched doesn't behave the same.

❌ Actual Behavior

The multimodal rag workflow tool I've configured doesn't behave as expected, I'm experiencing the avobe mentioned issues.

how about your dsl

[jlfloresTeika]

MultimodalRAG.zip

Here are the DSL's, thanks @fatelei

[fatelei]

MultimodalRAG.zip

Here are the DSL's, thanks @fatelei

ok

[jlfloresTeika]

I believe it may be related to this pull request: #30177 @fatelei