[WIP] [#521] Research Agent Tools

[sahilds1]

Description

Adds a second tool to the assistant and makes the eval able to show which tool the
model chose — the thing this branch exists to change.

• New ask_database tool — read-only SQL over the medication table, reusing the
  existing function in api/services/tools/database.py (SELECT-only + ALLOWED_TABLES
  guards not reimplemented). Schema built in the flattened Responses-API shape, which
  differs from Chat Completions' nested one.
• One Tool dataclass + one TOOLS list — replaces two parallel registries
  (get_tools_schema / make_tool_mapping) linked only by a duplicated, unenforced
  tool-name string, where a half-registered tool failed at runtime. user is now bound
  at dispatch time, which deletes the binder/closure machinery.
• No DB query at import time — the schema string is a hardcoded constant (4-column,
  effectively frozen table). Also drops the AppRegistryNotReady risk and lets us hide
  id from the model.
• run_assistant returns AssistantResult (output_text, response_id,
  tool_calls) instead of a tuple, carrying a ToolCall per call. status is a
  3-state enum, not a bool: a tool that raised and a tool name the model
  hallucinated are opposite diagnoses. Dataclass over widened tuple so later metrics
  are defaulted fields with zero call-site churn.
• Closes a swallowed-failure hole — tool exceptions are caught and fed back to the
  model as a string, so it writes fluent prose around a failed retrieval. Previously a
  run where ask_database threw on every question read as five clean rows; now it shows
  as tool_error_count > 0 while error is None.
• Eval CSV gains response_id, tools_called, tool_call_count,
  tool_error_count, tool_calls_json, duration_s.
• Housekeeping — 10 relative imports → absolute; search_tool.py split out (with
  the imports that split had left broken); one shared MODEL_NAME constant so the CSV
  can't mislabel the model.

API unchanged — views.py unpacks the dataclass; the JSON body is byte-identical.
tool_calls are intentionally not exposed to the frontend.

Related Issue

#521

Manual Tests

# unit suite (no API key, no fixtures)
docker compose exec backend python manage.py test api.views.assistant -v2

# eval CSV — needs OPENAI_API_KEY, a superuser, embedded docs
EVAL_BRANCH=521-research-agent-tools \
  docker compose exec backend python api/views/assistant/eval_assistant.py

# response shape unchanged (the refactor's regression check)
curl -X POST http://localhost:8000/v1/api/assistant \
  -H "Authorization: JWT <token>" -H "Content-Type: application/json" \
  -d '{"message": "What medications are recommended for bipolar depression?"}'
# expect: {"response_output_text": "...", "final_response_id": "resp_..."}

Automated Tests

19 unit tests, OpenAI + DB + tools all mocked — no live calls, no fixtures.

File	#	Covers
test_tool_services.py	13	Tool.run/schema()/TOOLS; a ToolCall recorded on all three dispatch branches (OK / FAILED / UNREGISTERED); loop termination, previous_response_id continuity, tool calls accumulating across iterations
test_assistant_services.py	4	Input shape, include/omit previous_response_id, and that TOOLS + user are forwarded to the loop
test_eval_assistant.py	2	A raising question becomes an error row, not an aborted batch; the swallowed-failure case (tool_error_count == 1, error is None)

Deliberately uncovered: search_documents' own paths (TODO in-file — mockable, no DB
needed) and main()'s CSV writing.

Documentation

No user-facing or CLAUDE.md changes — internal to the assistant module, HTTP contract
unchanged. Design rationale is documented inline where the code lives (why behavior is a
run field not a subclass method; why the schema is hardcoded; why tests patch use
sites, with a maintenance guard).

Reviewers

Notes

Targets develop.

Before merge

• ⚠️ Suite not run since the last commit; the eval has never run end-to-end, so a
  green suite is not evidence a CSV comes out. First run also tests the self-flagged
  sys.path caveat in eval_assistant.py ("../../../../" → /usr/src, not
  /usr/src/server).
• ⚠️ Dropping id from the schema string is a behavior change, not a pure refactor —
  worth confirming no prompt or eval expected it.

Known, documented rather than fixed

• If client.responses.create raises mid-loop, every ToolCall so far is lost with the
  exception (tool_call_count 0 despite real calls). Fixing it means deciding whether
  AssistantResult describes a successful run or whatever happened — a contract
  change.
• Tool overlap: semantic search and SQL can both answer the same question. The new
  columns exist to make that observable; sharper descriptions may be needed.
• Keep key order identical in both run_one row literals — as_completed is
  nondeterministic, so that's the only thing keeping CSV column order stable.

Deferred, design written down

• Token usage + turn count as defaulted fields (incl. cached_tokens — every turn
  resends context, so cost from input_tokens alone overstates spend).
• INSTRUCTIONS → {branch}-{timestamp}.prompt.txt sidecar beside the CSV.
• Scoring layer — the biggest gap. eval_assistant.py is still a generation
  harness: no ground truth, so the CSV shows what was said and which tools ran, but not
  whether the answer was right. Citation accuracy is the cheapest real signal (the
  mandated [Name, Page] format can be checked against what search_documents returned).