From fa46d457faa11fde0c4b4904511e97167847296b Mon Sep 17 00:00:00 2001 From: "Hermes (agora)" Date: Tue, 15 Sep 2026 10:49:43 -0700 Subject: [PATCH 1/3] feat: extend GeminiLive for Gemini 3.8 models --- README.md | 16 +++ docs/concepts/vendors.md | 2 +- docs/guides/mllm-flow.md | 2 + docs/guides/preview-endpoint.md | 13 +- docs/index.md | 1 + docs/reference/vendors.md | 10 +- pyproject.toml | 1 + src/agora_agent/__init__.py | 2 + src/agora_agent/agentkit/__init__.py | 4 +- src/agora_agent/agentkit/agent.py | 4 +- src/agora_agent/agentkit/agent_session.py | 4 +- src/agora_agent/agentkit/preview/__init__.py | 12 ++ src/agora_agent/agentkit/preview/client.py | 129 ++++++++++++++++++- src/agora_agent/agentkit/preview/vendors.py | 89 +++++++++++++ src/agora_agent/agentkit/vendors/mllm.py | 27 +++- tests/custom/test_agentkit_agent.py | 14 ++ tests/custom/test_preview.py | 37 +++++- tests/custom/test_preview_gemini_merge.py | 65 ++++++++++ 18 files changed, 414 insertions(+), 18 deletions(-) create mode 100644 tests/custom/test_preview_gemini_merge.py diff --git a/README.md b/README.md index f250daf..2e0eaef 100644 --- a/README.md +++ b/README.md @@ -181,6 +181,7 @@ Use `with_mllm()` for OpenAI Realtime, Gemini Live, Vertex AI, or xAI Grok. No S ```python from agora_agent import Agent, Agora, Area, OpenAIRealtime +import os import time client = Agora( @@ -206,6 +207,21 @@ session = agent.create_session( session.start() ``` +For Gemini 3.8 Live Extended Thinking, use the same single `GeminiLive` class as the regular Live model: + +```python +from agora_agent import GeminiLive + +gemini_agent = Agent(client=client).with_mllm(GeminiLive( + api_key=os.environ["GOOGLE_API_KEY"], + model="models/gemini-3.8-live-extended-thinking", + thinking_level="medium", + greeting_message="Hello! Ready to chat.", +)) +``` + +Use `models/gemini-3.8-live` without `thinking_level` for the lower-latency model. Gemini sessions use the preview gateway and `agora-feature: gemini-live`; the Google key is sent as `mllm.api_key`. See the [Preview Endpoint guide](./docs/guides/preview-endpoint.md). + See the [MLLM Flow guide](./docs/guides/mllm-flow.md) for full examples with Gemini Live and Vertex AI. ## Documentation diff --git a/docs/concepts/vendors.md b/docs/concepts/vendors.md index c093e8d..3ba71f9 100644 --- a/docs/concepts/vendors.md +++ b/docs/concepts/vendors.md @@ -160,7 +160,7 @@ Used with `agent.with_mllm()` for the [MLLM flow](../guides/mllm-flow.md). These | `OpenAIRealtime` | OpenAI Realtime | Global | `api_key`; optional `turn_detection` | | `OpenAIGPTLive` (preview) | OpenAI GPT Live | Global | `api_key`; optional `greeting` | | `AzureOpenAIRealtime` | Azure OpenAI Realtime | Global | `api_key`, `url`, `turn_detection`; optional `max_history` | -| `GeminiLive` | Google Gemini Live API | Global | `api_key`, `model`; optional `turn_detection` | +| `GeminiLive` | Google Gemini Live API | Global | `api_key`; `model` defaults to `models/gemini-3.8-live`. The two 3.8 IDs use preview routing; older IDs use production. | | `VertexAI` | Vertex AI (Gemini Live) | Global | `model`, `project_id`, `location`, `adc_credentials_string`; optional `turn_detection` | | `XaiGrok` | xAI Grok (`mllm.vendor`: `xai`) | Global | `api_key`; optional `voice`, `language`, `sample_rate`, `turn_detection` | | `QwenOmni` | Alibaba Cloud Qwen Omni Realtime | CN | `api_key`, `url`; optional `turn_detection` | diff --git a/docs/guides/mllm-flow.md b/docs/guides/mllm-flow.md index ae1d036..4f4414a 100644 --- a/docs/guides/mllm-flow.md +++ b/docs/guides/mllm-flow.md @@ -95,6 +95,8 @@ asyncio.run(main()) ## Gemini Live +Use `GeminiLive` for the existing Gemini Live models and both Gemini 3.8 models. The 3.8 IDs select the preview route automatically; Extended Thinking also accepts `thinking_level`. See the [Preview Endpoint guide](./preview-endpoint.md). + Gemini Live uses a Google AI API key: ```python diff --git a/docs/guides/preview-endpoint.md b/docs/guides/preview-endpoint.md index 85bda0d..39148ea 100644 --- a/docs/guides/preview-endpoint.md +++ b/docs/guides/preview-endpoint.md @@ -10,7 +10,8 @@ Some providers may be released through a preview gateway before their production and `AsyncAgentSession` detect registered preview providers from the resolved start request and route the entire session automatically. -OpenAI GPT Live is registered for preview routing with the `live-models` feature. Gemini STT has graduated to +OpenAI GPT Live uses the `live-models` feature. Gemini 3.8 MLLMs use `gemini-live`. +Gemini STT has graduated to production and uses the normal regional endpoint. Existing imports of `GeminiSTT` and `GeminiSTTModels` from `agora_agent.agentkit.preview` remain supported as compatibility aliases. @@ -28,6 +29,14 @@ agent_id = session.start() This session uses the preview base URL and sends `agora-feature: live-models`. A session using `GeminiSTT` uses the client's normal GA regional endpoint without that header. +Use the single `GeminiLive(api_key=..., model=...)` class with `with_mllm`. +The model IDs are `models/gemini-3.8-live` and +`models/gemini-3.8-live-extended-thinking`; the low-latency ID is the default. +Set `thinking_level="medium"` for extended thinking. `GeminiLive` sends it +only for the extended-thinking ID. The Gemini +credential is sent once as `mllm.api_key`, never as `mllm.params.api_key`. +Gemini sessions send `agora-feature: gemini-live`, while GPT Live retains `live-models`. + ## Session-scoped routing Preview routing does not mutate the bound `Agora` or `AsyncAgora` client. A session that needs a preview feature @@ -37,7 +46,7 @@ receives private generated clients configured with: - `agora-feature` as the feature gate header. - All custom headers, authentication settings, timeouts, and the supplied `httpx` client from the original client. -The gate header is applied after caller-provided headers, so it cannot be accidentally blanked or replaced. It is +The gate header is applied after caller-provided and per-call headers, so it cannot be accidentally blanked or replaced. It is kept on every request made through that session. Production sessions created from the same client continue using the regional production endpoint. diff --git a/docs/index.md b/docs/index.md index 9178fcf..c1f8da3 100644 --- a/docs/index.md +++ b/docs/index.md @@ -54,6 +54,7 @@ The Agora Conversational AI Python SDK lets you build voice-powered AI agents on | [Pagination](./guides/pagination.md) | Iterate over paginated list endpoints | | [Advanced](./guides/advanced.md) | Raw response, retries, timeouts, custom httpx client | | [Low-Level API](./guides/low-level-api.md) | Generated REST APIs | +| [Preview Endpoint](./guides/preview-endpoint.md) | `AgoraPreview`, the `agora-feature` gate header, and preview routing | | [Client Reference](./reference/client.md) | Full `Agora` / `AsyncAgora` API | | [Agent Reference](./reference/agent.md) | Full `Agent` builder API | | [Session Reference](./reference/session.md) | Full `AgentSession` / `AsyncAgentSession` API | diff --git a/docs/reference/vendors.md b/docs/reference/vendors.md index 76b8328..d22d705 100644 --- a/docs/reference/vendors.md +++ b/docs/reference/vendors.md @@ -956,14 +956,18 @@ CN Alibaba Cloud Qwen Omni Realtime vendor (`mllm.vendor`: `"qwen_omni"`). Impor ### `GeminiLive` +`GeminiLive` supports existing Gemini Live models and both public Gemini 3.8 voice models. The 3.8 IDs select the preview gateway with `agora-feature: gemini-live`; older model IDs keep the production route. See [Preview Endpoint](../guides/preview-endpoint.md). + | Parameter | Type | Required | Default | Description | |---|---|---|---|---| | `api_key` | `str` | Yes | — | Google Gemini API key | -| `model` | `str` | Yes | — | Gemini Live model name | -| `url` | `str` | No | `None` | Custom WebSocket URL | +| `model` | `str` | No | `models/gemini-3.8-live` | Gemini Live model name | +| `thinking_level` | `str` | No | `None` | `low`, `medium`, or `high`; sent only for 3.8 Extended Thinking | +| `language_codes` | `List[str]` | No | `None` | 3.8 language codes in `mllm.params.language_codes` | +| `url` | `str` | No | `None` | Custom endpoint; 3.8 defaults to the Gemini Developer API host | | `instructions` | `str` | No | `None` | System instructions | | `voice` | `str` | No | `None` | Voice name | -| `greeting_message` | `str` | No | `None` | Greeting message | +| `greeting_message` | `str` | No | `None` | Greeting message; sent as `mllm.greeting` for 3.8 models | | `failure_message` | `str` | No | `None` | Message played when the model call fails | | `input_modalities` | `List[str]` | No | `None` | Input modalities | | `output_modalities` | `List[str]` | No | `None` | Output modalities | diff --git a/pyproject.toml b/pyproject.toml index 5b17970..bea7825 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,5 +1,6 @@ [project] name = "agora-agents" +dynamic = ["version"] [tool.poetry] name = "agora-agents" diff --git a/src/agora_agent/__init__.py b/src/agora_agent/__init__.py index 40fab84..5a1ad53 100644 --- a/src/agora_agent/__init__.py +++ b/src/agora_agent/__init__.py @@ -48,6 +48,8 @@ GeminiSTT, GeminiSTTModels, GeminiLive, + GeminiLiveModels, + GEMINI_MLLM_DEFAULT_MODEL, GenericAvatar, GenericTTS, GoogleSTT, diff --git a/src/agora_agent/agentkit/__init__.py b/src/agora_agent/agentkit/__init__.py index 3c07199..924431b 100644 --- a/src/agora_agent/agentkit/__init__.py +++ b/src/agora_agent/agentkit/__init__.py @@ -154,7 +154,7 @@ OpenAITtsPresetModels, normalize_preset_input, ) -from .preview import OpenAIGPTLive +from .preview import OpenAIGPTLive, GeminiLiveModels, GEMINI_MLLM_DEFAULT_MODEL from .vendors import ( AkoolAvatar, AmazonBedrock, @@ -391,6 +391,8 @@ "BaseSTT", "BaseMLLM", "OpenAIGPTLive", + "GeminiLiveModels", + "GEMINI_MLLM_DEFAULT_MODEL", "BaseAvatar", "SampleRate", "ElevenLabsSampleRate", diff --git a/src/agora_agent/agentkit/agent.py b/src/agora_agent/agentkit/agent.py index 4f25033..2ea93c7 100644 --- a/src/agora_agent/agentkit/agent.py +++ b/src/agora_agent/agentkit/agent.py @@ -890,7 +890,7 @@ def create_session( if resolved_client is None: raise ValueError("client is required. Pass client=... to Agent(...).") - session_name = name or f"agent-{int(time.time())}" + session_name = name or f"agent-{time.time_ns()}" return AgentSession( client=resolved_client, agent=self, @@ -937,7 +937,7 @@ def create_async_session( if resolved_client is None: raise ValueError("client is required. Pass client=... to Agent(...).") - session_name = name or f"agent-{int(time.time())}" + session_name = name or f"agent-{time.time_ns()}" return AsyncAgentSession( client=resolved_client, agent=self, diff --git a/src/agora_agent/agentkit/agent_session.py b/src/agora_agent/agentkit/agent_session.py index 18a2bac..e2b2604 100644 --- a/src/agora_agent/agentkit/agent_session.py +++ b/src/agora_agent/agentkit/agent_session.py @@ -43,7 +43,7 @@ normalize_preset_input, resolve_session_presets, ) -from .preview.client import create_preview_session_clients, required_preview_features +from .preview.client import apply_preview_shape, create_preview_session_clients, required_preview_features from .token import _parse_numeric_uid, generate_convo_ai_token @@ -632,6 +632,7 @@ def start(self) -> str: properties, ) + apply_preview_shape(resolved_properties) self._bind_session_clients(required_preview_features(resolved_properties)) if self._debug: @@ -999,6 +1000,7 @@ async def start(self) -> str: properties, ) + apply_preview_shape(resolved_properties) self._bind_session_clients(required_preview_features(resolved_properties)) if self._debug: diff --git a/src/agora_agent/agentkit/preview/__init__.py b/src/agora_agent/agentkit/preview/__init__.py index c640e01..70240ec 100644 --- a/src/agora_agent/agentkit/preview/__init__.py +++ b/src/agora_agent/agentkit/preview/__init__.py @@ -9,12 +9,18 @@ PREVIEW_FEATURE_HEADER, PreviewFeature, PreviewFeatures, + apply_preview_shape, create_preview_session_clients, required_preview_features, ) from .vendors import ( + GEMINI_MLLM_DEFAULT_MODEL, + GEMINI_PREVIEW_MLLM_URL, + GEMINI_THINKING_LEVELS, + GeminiLiveModels, GeminiSTT, GeminiSTTModels, + GeminiThinkingLevel, OpenAIGPTLive, ) @@ -24,6 +30,12 @@ "GeminiSTTModels", "GeminiSTT", "OpenAIGPTLive", + "GEMINI_MLLM_DEFAULT_MODEL", + "GEMINI_PREVIEW_MLLM_URL", + "GEMINI_THINKING_LEVELS", + "GeminiLiveModels", + "GeminiThinkingLevel", + "apply_preview_shape", "PreviewFeature", "PreviewFeatures", "create_preview_session_clients", diff --git a/src/agora_agent/agentkit/preview/client.py b/src/agora_agent/agentkit/preview/client.py index c9cb1cb..6f496eb 100644 --- a/src/agora_agent/agentkit/preview/client.py +++ b/src/agora_agent/agentkit/preview/client.py @@ -14,9 +14,11 @@ import typing +import httpx from ...agent_management.client import AgentManagementClient, AsyncAgentManagementClient from ...agents.client import AgentsClient, AsyncAgentsClient from ...core.client_wrapper import AsyncClientWrapper, SyncClientWrapper +from .vendors import GEMINI_PREVIEW_MLLM_URL #: Base URL that serves the preview providers. PREVIEW_API_BASE_URL = "https://partner.ai.agora.io/preview/api/conversational-ai-agent" @@ -36,7 +38,7 @@ class PreviewFeatures: vendors on the preview endpoint. """ - #: Deprecated compatibility value. Gemini ASR now uses the production endpoint. + #: Gemini preview MLLM gate. Gemini ASR uses the production endpoint. GEMINI_LIVE = "gemini-live" LIVE_MODELS = "live-models" @@ -44,6 +46,42 @@ class PreviewFeatures: PreviewFeature = str +def _pinned_headers(headers: typing.Any, feature_header: str) -> httpx.Headers: + merged = httpx.Headers(headers) + merged[PREVIEW_FEATURE_HEADER] = feature_header + return merged + + +class _GatedSyncHttpxClient: + """Pin the preview header after generated per-call header overrides.""" + + def __init__(self, inner: httpx.Client, features: typing.Sequence[str]): + self._inner = inner + self._feature_header = ",".join(features) + + def request(self, *args: typing.Any, **kwargs: typing.Any) -> httpx.Response: + kwargs["headers"] = _pinned_headers(kwargs.get("headers"), self._feature_header) + return self._inner.request(*args, **kwargs) + + def stream(self, *args: typing.Any, **kwargs: typing.Any) -> typing.Any: + kwargs["headers"] = _pinned_headers(kwargs.get("headers"), self._feature_header) + return self._inner.stream(*args, **kwargs) + + +class _GatedAsyncHttpxClient: + def __init__(self, inner: httpx.AsyncClient, features: typing.Sequence[str]): + self._inner = inner + self._feature_header = ",".join(features) + + async def request(self, *args: typing.Any, **kwargs: typing.Any) -> httpx.Response: + kwargs["headers"] = _pinned_headers(kwargs.get("headers"), self._feature_header) + return await self._inner.request(*args, **kwargs) + + def stream(self, *args: typing.Any, **kwargs: typing.Any) -> typing.Any: + kwargs["headers"] = _pinned_headers(kwargs.get("headers"), self._feature_header) + return self._inner.stream(*args, **kwargs) + + def _preview_headers( features: typing.Sequence[str], headers: typing.Optional[typing.Dict[str, str]], @@ -55,7 +93,10 @@ def _preview_headers( it routes to the production environment, where the preview providers do not exist. Use ``features`` to change the value. """ - merged: typing.Dict[str, str] = dict(headers or {}) + merged: typing.Dict[str, str] = { + key: value for key, value in (headers or {}).items() + if key.lower() != PREVIEW_FEATURE_HEADER + } merged[PREVIEW_FEATURE_HEADER] = ",".join(features) return merged @@ -73,15 +114,16 @@ def create_preview_session_clients( "headers": _preview_headers(features, source.get_custom_headers()), "base_url": PREVIEW_API_BASE_URL, "timeout": source.get_timeout(), - "httpx_client": source.httpx_client.httpx_client, } if isinstance(source, AsyncClientWrapper): + kwargs["httpx_client"] = typing.cast(httpx.AsyncClient, _GatedAsyncHttpxClient(source.httpx_client.httpx_client, features)) async_wrapper = AsyncClientWrapper(**kwargs) return ( AsyncAgentsClient(client_wrapper=async_wrapper), AsyncAgentManagementClient(client_wrapper=async_wrapper), ) if isinstance(source, SyncClientWrapper): + kwargs["httpx_client"] = typing.cast(httpx.Client, _GatedSyncHttpxClient(source.httpx_client.httpx_client, features)) sync_wrapper = SyncClientWrapper(**kwargs) return ( AgentsClient(client_wrapper=sync_wrapper), @@ -97,6 +139,84 @@ def create_preview_session_clients( } +_PREVIEW_MLLM_MODELS = frozenset( + { + "models/gemini-3.8-live", + "models/gemini-3.8-live-extended-thinking", + } +) + + +def _has_preview_mllm_envelope(mllm: typing.Mapping[str, typing.Any]) -> bool: + """Whether a config carries the envelope the preview MLLM classes emit. + + That envelope is a top-level ``mllm.api_key`` plus a ``url`` on the Gemini + Developer API host. ``GeminiLive`` configs for older model IDs use a + different URL (an empty string or WebSocket endpoint). + + This is the second recognition path, and it exists because keying only off + :data:`_PREVIEW_MLLM_MODELS` makes an unrecognised model name fail silently: + :func:`apply_preview_shape` would stop retargeting ``greeting_message``, the + greeting would land in a field these models ignore, and the agent would + simply never greet. A model name we have not listed yet is reachable by + following this SDK's own advice to override ``model`` when Google renames + one ahead of a release, so the failure has to not be silent. + """ + api_key = mllm.get("api_key") + url = mllm.get("url") + return isinstance(api_key, str) and isinstance(url, str) and url.startswith(GEMINI_PREVIEW_MLLM_URL) + + +def _is_preview_mllm(mllm: typing.Any) -> bool: + """Whether an MLLM config targets a preview model. + + Recognised by model name, or by the wire envelope only the preview vendor + classes produce. + """ + if not isinstance(mllm, dict) or mllm.get("vendor") != "gemini": + return False + params = mllm.get("params") + model = params.get("model") if isinstance(params, dict) else None + if isinstance(model, str) and model in _PREVIEW_MLLM_MODELS: + return True + return _has_preview_mllm_envelope(mllm) + + +#: MLLM wire keys the preview route spells differently from the Agora schema, +#: as production spelling -> preview spelling. +#: +#: ``failure_message`` is deliberately absent: it is an Agora engine feature +#: rather than a Gemini one, so it keeps its schema spelling. +_PREVIEW_MLLM_FIELD_RENAMES = {"greeting_message": "greeting"} + + +def apply_preview_shape(properties: typing.MutableMapping[str, typing.Any]) -> None: + """Retarget MLLM fields the shared builder wrote with production spellings. + + ``Agent`` fills ``mllm.greeting_message`` from an agent-level ``greeting`` + whenever the vendor has not set that key — correct for every GA vendor, but + the preview Gemini models read ``greeting``, so the value would land in a + field they ignore and the agent would silently never greet. + + Rather than teach the shared builder about preview providers, the + translation lives here and disappears with this package at GA. The vendor's + own value wins; the production-spelled one is the fallback, which also + migrates a hand-written ``greeting_message`` onto the preview key so an + existing config keeps working after only swapping the model. + + Mutates ``properties["mllm"]`` in place. Safe because the builder hands this + a fresh copy of the MLLM config rather than the Agent's stored one. + """ + mllm = properties.get("mllm") + if not isinstance(mllm, dict) or not _is_preview_mllm(mllm): + return + for production, preview in _PREVIEW_MLLM_FIELD_RENAMES.items(): + if production not in mllm: + continue + value = mllm.pop(production) + mllm.setdefault(preview, value) + + def required_preview_features(properties: typing.Mapping[str, typing.Any]) -> typing.List[str]: """Return the preview features a start request needs. @@ -114,6 +234,8 @@ def required_preview_features(properties: typing.Mapping[str, typing.Any]) -> ty feature = vendors.get(vendor) if feature is not None and feature not in features: features.append(feature) + if _is_preview_mllm(properties.get("mllm")) and PreviewFeatures.GEMINI_LIVE not in features: + features.append(PreviewFeatures.GEMINI_LIVE) return features @@ -122,6 +244,7 @@ def required_preview_features(properties: typing.Mapping[str, typing.Any]) -> ty "PREVIEW_FEATURE_HEADER", "PreviewFeature", "PreviewFeatures", + "apply_preview_shape", "create_preview_session_clients", "required_preview_features", ] diff --git a/src/agora_agent/agentkit/preview/vendors.py b/src/agora_agent/agentkit/preview/vendors.py index 5ee5b0d..1305007 100644 --- a/src/agora_agent/agentkit/preview/vendors.py +++ b/src/agora_agent/agentkit/preview/vendors.py @@ -12,6 +12,7 @@ from urllib.parse import urlsplit, urlunsplit from ..vendors.base import BaseMLLM, ensure_mcp_transport +from ..vendors.mllm import MllmTurnDetectionConfig from ..vendors.stt import GeminiSTT, GeminiSTTModels from pydantic import ConfigDict, Field from typing_extensions import Literal @@ -141,6 +142,94 @@ def to_config(self) -> Dict[str, Any]: __all__ = [ "OpenAIGPTLive", + "GEMINI_MLLM_DEFAULT_MODEL", + "GEMINI_PREVIEW_MLLM_URL", + "GEMINI_THINKING_LEVELS", + "GeminiLiveModels", + "build_gemini_preview_config", + "GeminiThinkingLevel", "GeminiSTTModels", "GeminiSTT", ] + + +class GeminiLiveModels: + """Preview MLLM model names. + + The ``models/`` prefix is part of each model ID. + """ + + LIVE_38 = "models/gemini-3.8-live" + LIVE_38_EXTENDED_THINKING = "models/gemini-3.8-live-extended-thinking" + + +#: The model name the Gemini MLLM sends by default. +#: +#: Low-latency Gemini voice is the default. +GEMINI_MLLM_DEFAULT_MODEL = GeminiLiveModels.LIVE_38 + + +GEMINI_THINKING_LEVELS = ("low", "medium", "high") + +GeminiThinkingLevel = Literal["low", "medium", "high"] + +#: The preview MLLM talks to the Gemini Developer API rather than a WebSocket host. +GEMINI_PREVIEW_MLLM_URL = "https://generativelanguage.googleapis.com" + + +def build_gemini_preview_config(self: Any) -> Dict[str, Any]: + """Serialize GeminiLive options for the Gemini 3.8 preview gateway.""" + model = (self.model or "").strip() or GEMINI_MLLM_DEFAULT_MODEL + voice = self.voice if self.voice is not None else "Puck" + url = self.url if self.url is not None else GEMINI_PREVIEW_MLLM_URL + + params: Dict[str, Any] = dict(self.additional_params or {}) + params.pop("api_key", None) + params["model"] = model + params["voice"] = voice + if model == GeminiLiveModels.LIVE_38_EXTENDED_THINKING: + if self.thinking_level is not None: + params["thinking_level"] = self.thinking_level + else: + params.pop("thinking_level", None) + # Plural array, and omitted when unset. The singular ``params.language`` + # belongs to xAI Grok in the Agora schema, and the production Gemini Live + # provider sends no language field at all. + if self.language_codes is not None: + params["language_codes"] = list(self.language_codes) + + if self.instructions is not None: + params["instructions"] = self.instructions + if self.transcribe_agent is not None: + params["transcribe_agent"] = self.transcribe_agent + if self.transcribe_user is not None: + params["transcribe_user"] = self.transcribe_user + if self.affective_dialog is not None: + params["affective_dialog"] = self.affective_dialog + if self.proactive_audio is not None: + params["proactive_audio"] = self.proactive_audio + if self.http_options is not None: + params["http_options"] = self.http_options + + config: Dict[str, Any] = { + "vendor": "gemini", + "api_key": self.api_key, + "url": url, + "params": params, + } + if self.messages is not None: + config["messages"] = self.messages + # ``greeting``, not ``greeting_message``: the preview Gemini models read + # this spelling. See the preview-endpoint guide. + if self.greeting_message is not None: + config["greeting"] = self.greeting_message + if self.failure_message is not None: + config["failure_message"] = self.failure_message + if self.input_modalities is not None: + config["input_modalities"] = self.input_modalities + if self.output_modalities is not None: + config["output_modalities"] = self.output_modalities + if self.turn_detection is not None: + config["turn_detection"] = self.turn_detection + + return config diff --git a/src/agora_agent/agentkit/vendors/mllm.py b/src/agora_agent/agentkit/vendors/mllm.py index bb207b7..e0d00a7 100644 --- a/src/agora_agent/agentkit/vendors/mllm.py +++ b/src/agora_agent/agentkit/vendors/mllm.py @@ -1,8 +1,9 @@ from typing import Any, Dict, List, Optional +from typing_extensions import Literal from ...types.mllm_turn_detection import MllmTurnDetection from .base import BaseMLLM -from pydantic import BaseModel, ConfigDict, Field +from pydantic import BaseModel, ConfigDict, Field, field_validator MllmTurnDetectionConfig = MllmTurnDetection @@ -261,8 +262,12 @@ class GeminiLiveOptions(BaseModel): model_config = ConfigDict(extra="forbid") api_key: str = Field(..., description="Google API key") - model: str = Field(..., description="Gemini Live model name") - url: Optional[str] = Field(default=None, description="WebSocket URL") + model: str = Field(default="models/gemini-3.8-live", description="Gemini Live model name") + thinking_level: Optional[Literal["low", "medium", "high"]] = Field( + default=None, description="Reasoning budget for the 3.8 extended-thinking model" + ) + language_codes: Optional[List[str]] = Field(default=None, description="Languages for Gemini 3.8") + url: Optional[str] = Field(default=None, description="Endpoint override; Gemini 3.8 defaults to the Developer API host") instructions: Optional[str] = Field(default=None, description="System instructions") voice: Optional[str] = Field(default=None, description="Voice name") affective_dialog: Optional[bool] = Field(default=None, description="Enable affective dialog") @@ -278,13 +283,27 @@ class GeminiLiveOptions(BaseModel): turn_detection: Optional[MllmTurnDetectionConfig] = Field(default=None, description="MLLM turn detection configuration") failure_message: Optional[str] = Field(default=None, description="Message played on failure") + @field_validator("api_key") + @classmethod + def _validate_api_key(cls, value: str) -> str: + value = value.strip() + if not value: + raise ValueError("GeminiLive requires api_key") + return value + class GeminiLive(GeminiLiveOptions, BaseMLLM): def to_config(self) -> Dict[str, Any]: + from ..preview.vendors import GeminiLiveModels, build_gemini_preview_config + + selected_model = self.model.strip() or GeminiLiveModels.LIVE_38 + if selected_model in (GeminiLiveModels.LIVE_38, GeminiLiveModels.LIVE_38_EXTENDED_THINKING): + return build_gemini_preview_config(self) + inner_params: Dict[str, Any] = {} if self.additional_params is not None: inner_params.update(self.additional_params) - inner_params["model"] = self.model + inner_params["model"] = selected_model if self.instructions is not None: inner_params["instructions"] = self.instructions if self.voice is not None: diff --git a/tests/custom/test_agentkit_agent.py b/tests/custom/test_agentkit_agent.py index 63b3c07..d7cf47e 100644 --- a/tests/custom/test_agentkit_agent.py +++ b/tests/custom/test_agentkit_agent.py @@ -50,6 +50,20 @@ def test_generated_core_aliases_are_public(): assert AvatarVendor is not None +def test_default_session_names_do_not_collide_within_one_second(monkeypatch): + import agora_agent.agentkit.agent as agent_module + + monkeypatch.setattr(agent_module.time, "time", lambda: 1_700_000_000.0) + timestamps = iter((1_700_000_000_000_000_001, 1_700_000_000_000_000_002)) + monkeypatch.setattr(agent_module.time, "time_ns", lambda: next(timestamps)) + agent = Agent(test_client()) + options = {"channel": "room", "agent_uid": "1", "remote_uids": ["100"]} + first = agent.create_session(**options) + second = agent.create_async_session(**options) + + assert first._name != second._name + + def test_model_copy_helper_supports_pydantic_v1_copy_api(): copied = Agent._copy_model_update(_CopyOnlyModel(enable_rtm=True), {"data_channel": "rtm"}) # noqa: SLF001 diff --git a/tests/custom/test_preview.py b/tests/custom/test_preview.py index 7f416e6..ca08b8b 100644 --- a/tests/custom/test_preview.py +++ b/tests/custom/test_preview.py @@ -139,4 +139,39 @@ def test_preview_client_factory_supports_sync_and_async_clients() -> None: for generated_client in (sync_agents, sync_management, async_agents, async_management): wrapper = generated_client._raw_client._client_wrapper assert wrapper.get_base_url() == PREVIEW_API_BASE_URL - assert wrapper.get_custom_headers()[PREVIEW_FEATURE_HEADER] == TEST_FEATURE + custom_headers = wrapper.get_custom_headers() + assert custom_headers is not None + assert custom_headers[PREVIEW_FEATURE_HEADER] == TEST_FEATURE + + +def test_sync_preview_gate_survives_case_variant_per_call_headers() -> None: + recorder = _Recorder() + client = Agora( + area=Area.US, app_id=APP_ID, app_certificate=APP_CERTIFICATE, + headers={"Agora-Feature": "caller-value"}, + httpx_client=httpx.Client(transport=recorder), + ) + agents, _ = create_preview_session_clients(client, [TEST_FEATURE]) + agents.stop(APP_ID, "agent-1", request_options={"additional_headers": {"AGORA-FEATURE": "wrong", "Authorization": "agora token=fake-token"}}) + assert recorder.requests[0].headers.get_list(PREVIEW_FEATURE_HEADER) == [TEST_FEATURE] + assert recorder.requests[0].headers["Authorization"] == "agora token=fake-token" + custom_headers = client._client_wrapper.get_custom_headers() + assert custom_headers is not None + assert custom_headers["Agora-Feature"] == "caller-value" + + +@pytest.mark.asyncio +async def test_async_preview_gate_survives_case_variant_per_call_headers() -> None: + recorder = _Recorder() + client = AsyncAgora( + area=Area.US, app_id=APP_ID, app_certificate=APP_CERTIFICATE, + headers={"Agora-Feature": "caller-value"}, + httpx_client=httpx.AsyncClient(transport=recorder), + ) + agents, _ = create_preview_session_clients(client, [TEST_FEATURE]) + await agents.stop(APP_ID, "agent-1", request_options={"additional_headers": {"AGORA-FEATURE": "wrong", "Authorization": "agora token=fake-token"}}) + assert recorder.requests[0].headers.get_list(PREVIEW_FEATURE_HEADER) == [TEST_FEATURE] + assert recorder.requests[0].headers["Authorization"] == "agora token=fake-token" + custom_headers = client._client_wrapper.get_custom_headers() + assert custom_headers is not None + assert custom_headers["Agora-Feature"] == "caller-value" diff --git a/tests/custom/test_preview_gemini_merge.py b/tests/custom/test_preview_gemini_merge.py new file mode 100644 index 0000000..6772d15 --- /dev/null +++ b/tests/custom/test_preview_gemini_merge.py @@ -0,0 +1,65 @@ +import pytest +from pydantic import ValidationError + +from agora_agent.agentkit.preview import ( + GeminiLiveModels, + OpenAIGPTLive, + PreviewFeatures, + apply_preview_shape, + required_preview_features, +) +from agora_agent.agentkit.vendors.mllm import GeminiLive + + +def test_gemini_and_gpt_live_use_their_own_feature_gates(): + for vendor in (GeminiLive(api_key="test-key", model=GeminiLiveModels.LIVE_38), GeminiLive(api_key="test-key", model=GeminiLiveModels.LIVE_38_EXTENDED_THINKING)): + properties = {"mllm": vendor.to_config()} + assert required_preview_features(properties) == [PreviewFeatures.GEMINI_LIVE] + assert required_preview_features({"mllm": OpenAIGPTLive(api_key="test-key").to_config()}) == [ + PreviewFeatures.LIVE_MODELS + ] + + +def test_gemini_greeting_uses_preview_wire_field(): + properties = {"mllm": GeminiLive(api_key="test-key", model=GeminiLiveModels.LIVE_38).to_config()} + properties["mllm"]["greeting_message"] = "Hello" + apply_preview_shape(properties) + assert properties["mllm"]["greeting"] == "Hello" + assert "greeting_message" not in properties["mllm"] + + +def test_one_gemini_preview_class_supports_public_38_ids(): + assert GeminiLive(api_key="test-key", model=" ").to_config()["params"]["model"] == GeminiLiveModels.LIVE_38 + models = ( + (GeminiLiveModels.LIVE_38, "medium"), + (GeminiLiveModels.LIVE_38_EXTENDED_THINKING, "medium"), + ) + for model, thinking in models: + config = GeminiLive(api_key="test-key", model=model, thinking_level=thinking, + additional_params={"thinking_level": "high", "api_key": "ignored-key"}).to_config() + assert config["api_key"] == "test-key" + assert "api_key" not in config["params"] + assert config["params"]["model"] == model + expected_thinking = thinking if model == GeminiLiveModels.LIVE_38_EXTENDED_THINKING else None + assert config["params"].get("thinking_level") == expected_thinking + assert required_preview_features({"mllm": config}) == [PreviewFeatures.GEMINI_LIVE] + # Explicit IDs must route even in a hand-written config without the + # SDK vendor's preview envelope. + assert required_preview_features({"mllm": {"vendor": "gemini", "params": {"model": model}}}) == [ + PreviewFeatures.GEMINI_LIVE + ] + + +def test_gemini_mllm_rejects_blank_api_key(): + with pytest.raises(ValidationError, match="GeminiLive requires api_key"): + GeminiLive(api_key=" ") + + +def test_unknown_gemini_model_keeps_preview_greeting_without_nested_api_key(): + properties = {"mllm": GeminiLive( + api_key="test-key", model="future-live-model", url="https://generativelanguage.googleapis.com" + ).to_config()} + properties["mllm"]["greeting_message"] = "Hello" + apply_preview_shape(properties) + assert properties["mllm"]["greeting"] == "Hello" + assert "greeting_message" not in properties["mllm"] From c204f469ce595082da1aee72044eee031a78aabb Mon Sep 17 00:00:00 2001 From: "Hermes (agora)" Date: Tue, 15 Sep 2026 14:04:16 -0700 Subject: [PATCH 2/3] chore: prepare v2.9.0 release --- changelog.md | 11 +++++++++++ compat/agora-agent-server-sdk/pyproject.toml | 4 ++-- pyproject.toml | 2 +- src/agora_agent/core/client_wrapper.py | 4 ++-- 4 files changed, 16 insertions(+), 5 deletions(-) diff --git a/changelog.md b/changelog.md index 995eec1..c16dca7 100644 --- a/changelog.md +++ b/changelog.md @@ -4,6 +4,17 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/). +## [v2.9.0] — 2026-09-15 + +### Added + +- **Gemini 3.8 Live MLLM** — `GeminiLive` now supports `models/gemini-3.8-live` and `models/gemini-3.8-live-extended-thinking`. The standard Live model is the default; Extended Thinking accepts `low`, `medium`, or `high` through `thinking_level`. + +### Changed + +- **Gemini Live routing and credentials** — Gemini 3.8 sessions use the preview gateway with `agora-feature: gemini-live`, send the Google credential as top-level `mllm.api_key`, and keep older Gemini Live model IDs on the production route. +- **Gemini Live documentation** — The README and vendor references document the existing `GeminiLive` API for both 3.8 models and the Extended Thinking level. + ## [v2.8.1] — 2026-09-11 ### Changed diff --git a/compat/agora-agent-server-sdk/pyproject.toml b/compat/agora-agent-server-sdk/pyproject.toml index 96aeb2d..a8eea54 100644 --- a/compat/agora-agent-server-sdk/pyproject.toml +++ b/compat/agora-agent-server-sdk/pyproject.toml @@ -3,7 +3,7 @@ name = "agora-agent-server-sdk" [tool.poetry] name = "agora-agent-server-sdk" -version = "v2.8.1" +version = "v2.9.0" description = "Compatibility shim for the renamed agora-agents package." readme = "README.md" authors = [] @@ -35,7 +35,7 @@ Repository = 'https://github.com/AgoraIO/agora-agents-python' [tool.poetry.dependencies] python = "^3.8" -agora-agents = ">=2.8.1,<3.0.0" +agora-agents = ">=2.9.0,<3.0.0" [build-system] requires = ["poetry-core"] diff --git a/pyproject.toml b/pyproject.toml index bea7825..3e39dc1 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ dynamic = ["version"] [tool.poetry] name = "agora-agents" -version = "v2.8.1" +version = "v2.9.0" description = "" readme = "README.md" authors = [] diff --git a/src/agora_agent/core/client_wrapper.py b/src/agora_agent/core/client_wrapper.py index 42c776b..728eb53 100644 --- a/src/agora_agent/core/client_wrapper.py +++ b/src/agora_agent/core/client_wrapper.py @@ -26,10 +26,10 @@ def __init__( def get_headers(self) -> typing.Dict[str, str]: headers: typing.Dict[str, str] = { - "User-Agent": "agora-agents/v2.8.1", + "User-Agent": "agora-agents/v2.9.0", "X-Fern-Language": "Python", "X-Fern-SDK-Name": "agora-agents", - "X-Fern-SDK-Version": "v2.8.1", + "X-Fern-SDK-Version": "v2.9.0", **(self.get_custom_headers() or {}), } headers["Authorization"] = httpx.BasicAuth(self._get_username(), self._get_password())._auth_header From c5bce3fd0414e68d97fee5e69f391b2397d55bb1 Mon Sep 17 00:00:00 2001 From: "Hermes (agora)" Date: Tue, 15 Sep 2026 14:19:20 -0700 Subject: [PATCH 3/3] docs: keep README MLLM guidance vendor agnostic --- README.md | 15 --------------- changelog.md | 2 +- 2 files changed, 1 insertion(+), 16 deletions(-) diff --git a/README.md b/README.md index 2e0eaef..67c1110 100644 --- a/README.md +++ b/README.md @@ -207,21 +207,6 @@ session = agent.create_session( session.start() ``` -For Gemini 3.8 Live Extended Thinking, use the same single `GeminiLive` class as the regular Live model: - -```python -from agora_agent import GeminiLive - -gemini_agent = Agent(client=client).with_mllm(GeminiLive( - api_key=os.environ["GOOGLE_API_KEY"], - model="models/gemini-3.8-live-extended-thinking", - thinking_level="medium", - greeting_message="Hello! Ready to chat.", -)) -``` - -Use `models/gemini-3.8-live` without `thinking_level` for the lower-latency model. Gemini sessions use the preview gateway and `agora-feature: gemini-live`; the Google key is sent as `mllm.api_key`. See the [Preview Endpoint guide](./docs/guides/preview-endpoint.md). - See the [MLLM Flow guide](./docs/guides/mllm-flow.md) for full examples with Gemini Live and Vertex AI. ## Documentation diff --git a/changelog.md b/changelog.md index c16dca7..c1c0996 100644 --- a/changelog.md +++ b/changelog.md @@ -13,7 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/). ### Changed - **Gemini Live routing and credentials** — Gemini 3.8 sessions use the preview gateway with `agora-feature: gemini-live`, send the Google credential as top-level `mllm.api_key`, and keep older Gemini Live model IDs on the production route. -- **Gemini Live documentation** — The README and vendor references document the existing `GeminiLive` API for both 3.8 models and the Extended Thinking level. +- **Gemini Live documentation** — The vendor reference and MLLM guide document the existing `GeminiLive` API for both 3.8 models and the Extended Thinking level. ## [v2.8.1] — 2026-09-11