feat(rag): add multi-tenant hybrid search, LLM streaming, token metering, conversation memory & full-stack UI

[VimalN2005]

Summary of Changes

This PR introduces a comprehensive, production-grade Retrieval-Augmented Generation (RAG) & Conversational AI capability to the full-stack template, maintaining multi-tenant isolation, enterprise token governance, and modern UI integration.

---

Key Capabilities Added

1. Multi-Tenant Hybrid Search (pgvector + Full-Text Search)

   • Swapped PostgreSQL image to pgvector/pgvector:pg17 in compose.yml.
   • Added Document and DocumentChunk models with Vector(1536) embeddings and owner_id indexing for strict tenant isolation.
   • Combined dense semantic similarity (HNSW cosine index) and sparse keyword retrieval (PostgreSQL full-text search with tsvector) using Reciprocal Rank Fusion (RRF).
   • Zero-external-dependency fallback for deterministic offline vector generation during testing and local development.

2. Two-Stage Hybrid Retrieval & Cross-Encoder Re-Ranking

   • Stage 1: High-recall hybrid candidate retrieval.
   • Stage 2: Cross-attention relevance scoring with term coverage, phrase proximity bonus, and optional Cohere Rerank API integration (rerank-v3.5).

3. Abort-Aware LLM Token Streaming (SSE)

   • Implemented real-time token-by-token streaming over Server-Sent Events (SSE) following W3C standards (sources, token, done).
   • Proactive Disconnect Detection: Periodically polls await request.is_disconnected(). If a user closes their tab or cancels generation, upstream LLM connections and compute are aborted immediately to prevent token leakage.

4. AI Token Metering & Budget Quota Guardrails

   • Real-time tracking of prompt and completion tokens per request with cost estimation in USD.
   • Enforces per-user monthly token quotas with HTTP 429 Too Many Requests when limits are exceeded (Superusers are exempt).
   • Dedicated API endpoints for monthly usage inspection and administrative quota management.

5. Multi-Turn Conversation Memory & Chat Sessions

   • Persistent ChatSession and ChatMessage models with JSON source chunk metadata.
   • Sliding-window conversational context formatting and contextual query expansion for multi-turn RAG.

6. Async Document Ingestion via Background Tasks

   • POST /api/v1/rag/documents?background=true returns 202 Accepted and offloads chunking/embedding to FastAPI BackgroundTasks.
   • Added status (processing | ready | failed) and status polling endpoint GET /api/v1/rag/documents/{id}/status.

7. Full-Stack Frontend UI (React 19 + TanStack Router & Query + Tailwind CSS)

   • Knowledge Base Page (/documents):
     • Live document table with auto-polling for processing items.
     • Real-time monthly AI token progress bar, spend tracker, and quota limit badge.
     • Ingestion modal with background task toggle and 1-click sample document templates.
   • AI Chat & Streaming Page (/chat):
     • Multi-session sidebar (create, switch, delete conversations).
     • Live token-by-token streaming with typing animation.
     • Grounded citations accordion showing chunk indices, similarity match percentage, and retrieval strategy.
     • "Stop Generating" abort button linked to frontend AbortController.

---

Verification & Testing

• Backend: 90/90 tests passing (uv run pytest tests) covering hybrid search, reranking, SSE streaming, multi-turn memory, token metering quotas, and async ingestion.
• Frontend: bun run build and bun run lint pass with 0 errors (TypeScript compilation succeeded in 864ms).
• Backward Compatibility: Fully non-breaking change. Existing User and Items endpoints function identically without modification.

[github-actions]

This PR modifies dependency files (pyproject.toml or uv.lock), which is restricted to members of the fastapi organization on GitHub.

If you need a dependency change, please open a discussion describing what you need and why.

Closing this PR automatically.