Skip to content

Read a message with a file attached as its words and the file, not as [object Object] - #506

Merged
davidmckayv merged 2 commits into
CopilotKit:mainfrom
kevin9327:fix/bot-reads-attachment-content
Sep 12, 2026
Merged

Read a message with a file attached as its words and the file, not as [object Object]#506
davidmckayv merged 2 commits into
CopilotKit:mainfrom
kevin9327:fix/bot-reads-attachment-content

Conversation

@kevin9327

Copy link
Copy Markdown
Contributor

What happens

Attach a file to a message in a channel whose coworker is the Bot in the box (agent-bot) or the LangGraph Bot (agent-langgraph), and ask about it:

How many rows say failed? (runs.csv attached)

The Bot answers as though it was asked nothing, because what its model was sent is:

[object Object],[object Object]

That string is in place of the question and the file. An image goes the same way.

Why

With no file attached, the composer sends a message as a string. With one attached, it sends a list of parts: what the person typed, then the file (channel-chat.tsx). The server resolves each file before the run leaves it. inlineAttachments in copilot.ts turns a text file into a text part carrying its contents, and an image into an image part with a data source. That comment says the reason it is done in the remote Bot's middleware is so a file does not reach the endpoint as a URL it cannot fetch.

Both Bots then read the message like this:

// agent-bot/src/history.ts
messages.push({ role: "user", content: String(message.content ?? "") });

// agent-langgraph/src/history.ts
messages.push(new HumanMessage(String(message.content ?? "")));

String() of an array of objects is [object Object],[object Object]. So the file arrives intact at the Bot, and the Bot's last step throws it away along with the words beside it.

Run against main, with the exact content shape inlineAttachments produces (typed text, a CSV's text, a PNG):

agent-bot user message content: "[object Object],[object Object],[object Object]"
agent-langgraph user message content: "[object Object],[object Object],[object Object]"

Built-in Bots are not affected. The runtime converts their content itself.

The change

One function, shared/user-content.ts, read by both Bots, for the reason NO_ANSWER_CAME lives in shared/: two copies of this would drift. It maps each part to the two content blocks both providers read:

  • a text part → { type: "text", text }, verbatim;
  • an image with a data source → { type: "image_url", image_url: { url: "data:<mime>;base64,<bytes>" } };
  • anything else → a text part naming it, e.g. [audio]. It is named rather than dropped, for the reason resultText in plugins/mcp.ts names a tool part it cannot read. A model told something was attached can say it cannot see it. A model handed nothing answers as though nothing was attached.

Those two block shapes are what OpenAI chat completions take. They are also what @langchain/openai, @langchain/anthropic and @langchain/google-genai each convert (each has an image_url branch that accepts a base64 data URL), so the LangGraph Bot works on all three of its providers.

A message that is a string, which is nearly every message, is returned exactly as before.

Where it runs

  • New state that outlives a request? None. A pure function of the message.
  • Second replica? No difference. Nothing is held.
  • Serialised / fanned out / new listener? None.

Boundary and audit

Untouched. No acting call, refusal or audit row is involved. This is the last step of turning a run's input into the model's prompt.

Changelog

A line under Unreleased, because a Bot that could not read an attached file now can.

Proof

bun test tests/history.test.ts in each Bot, on main with only this PR's tests applied (absolute paths shortened to the repository root, nothing else edited):

agent-bot: 10 pass, 3 fail
bun test v1.3.14 (0d9b296a)

tests\history.test.ts:
301 |     );
302 |     return user?.content;
303 |   }
304 | 
305 |   test("keeps what the person typed and the text of the file", () => {
306 |     expect(userContent([typed, { type: "text", text: csv }])).toEqual([
                                                                    ^
error: expect(received).toEqual(expected)

- [
-   {
-     "text": "How many rows say failed?",
-     "type": "text",
-   },
-   {
-     "text": 
- "Attached file "runs.csv":
- 
- id,status
- 1,failed
- 2,ok"
- ,
-     "type": "text",
-   },
- ]
+ "[object Object],[object Object]"

- Expected  - 16
+ Received  + 1

      at <anonymous> (agent-bot\tests\history.test.ts:306:63)
(fail) a message with a file attached > keeps what the person typed and the text of the file [1.38ms]
313 |     const image = {
314 |       type: "image",
315 |       source: { type: "data", value: "iVBORw0KGgo=", mimeType: "image/png" },
316 |       metadata: { attachmentId: "a1", filename: "chart.png" },
317 |     };
318 |     expect(userContent([typed, image])).toEqual([
                                              ^
error: expect(received).toEqual(expected)

- [
-   {
-     "text": "How many rows say failed?",
-     "type": "text",
-   },
-   {
-     "image_url": {
-       "url": "data:image/png;base64,iVBORw0KGgo=",
-     },
-     "type": "image_url",
-   },
- ]
+ "[object Object],[object Object]"

- Expected  - 12
+ Received  + 1

      at <anonymous> (agent-bot\tests\history.test.ts:318:41)
(fail) a message with a file attached > puts an attached image in front of the model [0.36ms]
329 |     // nothing answers as though nothing was attached.
330 |     const audio = {
331 |       type: "audio",
332 |       source: { type: "data", value: "UklGRg==", mimeType: "audio/wav" },
333 |     };
334 |     expect(userContent([typed, audio])).toEqual([
                                              ^
error: expect(received).toEqual(expected)

- [
-   {
-     "text": "How many rows say failed?",
-     "type": "text",
-   },
-   {
-     "text": "[audio]",
-     "type": "text",
-   },
- ]
+ "[object Object],[object Object]"

- Expected  - 10
+ Received  + 1

      at <anonymous> (agent-bot\tests\history.test.ts:334:41)
(fail) a message with a file attached > names a part it cannot read rather than dropping it [0.23ms]

 10 pass
 3 fail
 22 expect() calls
Ran 13 tests across 1 file. [33.00ms]
agent-langgraph: 6 pass, 3 fail (the same three, against LangChain's message)
(fail) a message with a file attached > keeps what the person typed and the text of the file [1.38ms]
(fail) a message with a file attached > puts an attached image in front of the model [0.50ms]
(fail) a message with a file attached > names a part it cannot read rather than dropping it [0.25ms]

 6 pass
 3 fail
 14 expect() calls
Ran 9 tests across 1 file.

Each failure's received value is "[object Object],[object Object]", identical to the agent-bot output above.

With the change: agent-bot 13 pass, 0 fail, agent-langgraph 9 pass, 0 fail.

Not a widening. In each file the fourth test, sends a message that is only text exactly as it was typed, passes both before and after. It pins that a string message is returned verbatim, so the only messages this changes are the ones carrying parts. Every test that was already in both files (tool-call pairing, out-of-order results, restored calls, A2UI context) passes before and after.

Whole packages (bun test in each Bot directory):

main this PR
agent-bot 12 pass, 0 fail 16 pass, 0 fail
agent-langgraph 29 pass, 0 fail 33 pass, 0 fail

The delta is exactly the four tests added to each.

bunx @biomejs/biome check and bunx prettier --check are clean on the five TypeScript files. prettier --check CHANGELOG.md reports style issues, but it does on main too: every suggestion is a missing blank line further down the file, none in the lines added here, so I have left that file's existing formatting alone.

Note on the CHANGELOG

The entry goes at the top of ## Unreleased, the one line every entry goes at, so it will conflict with any other PR open against that anchor. Happy to rebase whenever it suits you.

🤖 Generated with Claude Code

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

… [object Object]

With a file attached, the composer sends a message as a list of parts,
and the server resolves each file into a text or image part before the
run reaches a remote Bot. agent-bot and agent-langgraph both read the
message with String(), so the model was sent "[object Object],[object
Object]" in place of the question and the file.

Both Bots now read the parts through one shared function: text parts
verbatim, an image with a data source as an image_url data URL, and any
other part named rather than dropped. A string message is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kevin9327
kevin9327 force-pushed the fix/bot-reads-attachment-content branch from bd5b14a to 698dcab Compare September 12, 2026 22:08

@davidmckayv davidmckayv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep-reviewed clean (validation, no secret leak, fail-closed, agrees with existing layers). CI green.

@davidmckayv
davidmckayv merged commit e8f6d05 into CopilotKit:main Sep 12, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants