agy-proxy-rs (Rust)

A high-performance self-hosted LLM gateway written in Rust, using the official Google Antigravity (agy) CLI as the upstream engine. It exposes OpenAI Chat Completions and Anthropic Messages compatible HTTP APIs, complete with real-time SSE streaming, token usage tracking, and structured logging.

Inspirations & Acknowledgements:

Sakiko15/agy-proxy

amlyczz/dsh-agy-link

Key Features

OpenAI Chat Completions (/v1/chat/completions):
- Non-streaming (chat.completion response)
- Real-time SSE streaming (chat.completion.chunk with delta.content, delta.reasoning_content, delta.reasoning, and [DONE])
- Multi-turn conversation digest preservation
- Flexible tool call & tool result continuation (supports arbitrary client IDs like call_xxx, toolu_xxx)
- Multimodal image support (data:image/...;base64,...)
- Stop sequences & max tokens control
Anthropic Messages (/v1/messages):
- Non-streaming (message response with stop_reason, usage.output_tokens_details.thinking_tokens)
- Real-time SSE streaming (message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop)
- Native thinking_delta streaming events
- Multimodal image support (base64 blocks)
Dynamic Workspace Resolution & Auto-Sniffing:
- Zero-Config Auto-Sniffing (自动嗅探): Automatically extracts referenced file paths from prompt context, walking up directory trees to detect project markers (.git, package.json, Cargo.toml, pyproject.toml, go.mod, etc.) and automatically lock the agy execution directory to the active project root.
- Magic Slash Commands (魔术指令): Temporarily switch working directories by prefixing prompts with /cwd <path> or /workspace <path> (supports ~ home directory expansion and quotes). The command line is cleanly stripped before sending the prompt to the upstream model.
- Header & Body Overrides: Explicitly specify execution root via X-Workspace-Root / X-Cwd HTTP headers or "workspace_root" / "cwd" request body fields.
- Multi-Directory Workspace: Support mounting additional directories via X-Add-Dir / X-Add-Dirs headers, "add_dirs" request field, or automatic multimodal media staging (mapped to agy --add-dir).
- Resolution Precedence: Magic Command (/cwd) > HTTP Header (X-Workspace-Root) > Request Body (cwd) > Context Auto-Sniffing > Default Global Fallback (AGY_PROXY_WORKSPACE_ROOT).
True Streaming Reasoning & Visual Tool Calling:
- StreamTagFilter state machine: Seamlessly detects and filters <think>...</think> and <thought>...</thought> tags across streaming chunk boundaries, routing them directly into reasoning channels.
- Native extraction of thought_delta, thinking, and agent_response step types from agy CLI stream-json.
- Real-time tool call visualization: Intercepts CLI tool execution events and renders formatted Markdown status cards in the reasoning stream:
  - Tool invocation: > ⚙️ [调用工具] **tool_name** (args...) ...
  - Tool completion: > ↳ 完成 (duration) with preview snippet.
  - Tool failure: > ↳ 失败 (duration) with diagnostic error snippet.
- Keeps connections alive during long bash commands and file operations, preventing timeouts and UI freezes in Trae, Cursor, Cline, etc.
UTF-8 Multi-byte Character Boundary Safety:
- Fully immune to UTF-8 character boundary slicing panics on multi-byte CJK text.
- truncate_safe function ensuring no multi-byte code point is split across byte boundaries.
- Byte-level UTF-8 buffering on child process stdout pipe, eliminating character corruption or replacement character (\u{FFFD}) artifacts across chunk boundaries.
Long-Running Task Reliability:
- Sliding activity watchdog: inactivity timer resets whenever stdout/stderr activity occurs, allowing complex tasks to run for hours without premature timeout.
- Extended CLI print timeout: automatically passes --print-timeout 240m (4 hours) to upstream agy.
- Continuous SSE heartbeat pings (every 15s) to prevent reverse proxy and client disconnects.
- High-capacity 4096-item MPSC channel buffers preventing backpressure deadlocks.
Dual-shape Model List:
- GET /v1/models (OpenAI format by default, Anthropic format if anthropic-version header is provided)
- GET /v1/anthropic/models (Anthropic format with after_id / before_id pagination)
- GET /v1/models/:id (OpenAI single model retrieval)
Token Estimation (/v1/messages/count_tokens):
- Deterministic CJK & Latin heuristic with x-agy-proxy-token-estimate: heuristic header.
Authentication & Concurrency Control:
- Constant-time timing-safe comparison.
- Supports Authorization: Bearer <key> and x-api-key: <key>.
- Semaphore-based concurrency limiter (AGY_PROXY_MAX_CONCURRENT).

Requirements

Rust: 1.75+
Google Antigravity (agy) CLI: Version 1.1.8+ installed and available on PATH (or specified via AGY_PROXY_BIN). Run agy login to authenticate.

Quick Start

1. Build and Run

cd agy-proxy-rs

# Build release binary
cargo build --release

# Run with defaults (Port 8082, API Key: change-me)
./target/release/agy-proxy-rs

Or run directly with cargo:

cargo run --release

Configuration

Configuration is managed via environment variables or a .env file in the project directory:

Environment Variable	Default Value	Description
`AGY_PROXY_PORT`	`8082`	HTTP listen port
`AGY_PROXY_HOST`	`0.0.0.0`	Bind host address
`AGY_PROXY_API_KEY`	`change-me`	API key for authentication (empty string disables auth)
`AGY_PROXY_BIN`	(auto-detected)	Path to `agy` CLI binary
`AGY_PROXY_ENABLED`	`true`	Master switch (`false` returns 503)
`AGY_PROXY_DEFAULT_MODEL`	`gemini-3.8-flash`	Default model when request omits `model`
`AGY_PROXY_DEFAULT_EFFORT`	`medium`	Default reasoning effort for Gemini models
`AGY_PROXY_MODE`	`plan`	agy permission mode (`plan`, `accept-edits`, `skip`)
`AGY_PROXY_MAX_CONCURRENT`	`3`	Maximum concurrent `agy` subprocesses
`AGY_PROXY_MAX_QUEUE_DEPTH`	`64`	Maximum queue depth before returning 429 BUSY
`AGY_PROXY_TIMEOUT_MS`	`600000`	Inactivity timeout watchdog in milliseconds (10 minutes)
`AGY_PROXY_MAX_TOKENS_DEFAULT`	`65536`	Default floor for max tokens
`AGY_PROXY_SSE_HEARTBEAT_MS`	`15000`	SSE heartbeat interval (ms; default 15s keeps connection alive)
`AGY_PROXY_DISABLE_TELEMETRY`	`true`	Set `DO_NOT_TRACK=1` and disable telemetry in agy
`AGY_PROXY_WORKSPACE_ROOT`	`~/.agy-proxy/workspace`	Default directory where `agy` executes if not specified
`AGY_PROXY_MEDIA_DIR`	`~/.agy-proxy/gateway/media`	Directory where inbound request images are staged
`AGY_PROXY_LOG_LEVEL`	`info`	Logging level (`trace`, `debug`, `info`, `warn`, `error`)

Working Directory & Workspace Management

When using agy-proxy-rs with IDE clients like Trae Work, Cursor, or Cline, you can control the project working directory dynamically:

1. Auto-Sniffing (Zero Configuration)

Simply ask questions or edit files normally in your IDE. When prompt text or system instructions reference file absolute paths (e.g. /Users/.../my-repo/src/index.ts), the gateway walks up directory levels searching for markers (.git, package.json, Cargo.toml, etc.) and automatically switches the active agy process to the project root.

2. Magic Slash Commands (`/cwd`, `/workspace`)

In any chat session, temporarily switch projects by typing a command on the first line:

/cwd /Users/tinytinycn/Desktop/my-project
Help me refactor the database module

Or with home directory abbreviation:

/workspace ~/Projects/frontend-app
Run unit tests and fix errors

The command line is automatically extracted, verified, and cleanly stripped before the prompt is dispatched to the model.

3. HTTP Header & Body Overrides

You can also explicitly pass the directory via standard API fields:

HTTP Header: X-Workspace-Root: /path/to/project or X-Cwd: /path/to/project
Request Body JSON: "cwd": "/path/to/project" or "workspace_root": "/path/to/project"
Additional Directories: X-Add-Dir: /path/to/extra/dir or "add_dirs": ["/path/to/extra/dir"]

API Endpoints & `curl` Examples

All examples assume the server is running on http://127.0.0.1:8082 with the default API key change-me.

Tip: If your terminal environment has http_proxy or all_proxy set, add --noproxy "*" or --noproxy 127.0.0.1 to curl so requests connect directly to localhost.

1. Health Check

curl --noproxy '*' -s http://127.0.0.1:8082/healthz

Response:

{"ok":true}

2. List Models (OpenAI Format)

curl --noproxy '*' -s http://127.0.0.1:8082/v1/models \
  -H "Authorization: Bearer change-me"

3. OpenAI Chat Completions (Real-time Streaming with Reasoning)

curl --noproxy '*' -N -s -X POST http://127.0.0.1:8082/v1/chat/completions \
  -H "Authorization: Bearer change-me" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3.8-flash",
    "stream": true,
    "messages": [
      {"role": "user", "content": "Explain quantum entanglement in one sentence."}
    ]
  }'

Output stream:

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"reasoning":"Thinking...","reasoning_content":"Thinking..."},"finish_reason":null}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Quantum entanglement is..."},"finish_reason":null}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]

4. Magic Command Working Directory Test

curl --noproxy '*' -s -X POST http://127.0.0.1:8082/v1/chat/completions \
  -H "Authorization: Bearer change-me" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3.8-flash",
    "messages": [
      {"role": "user", "content": "/cwd /tmp\nReply with exactly two words: Received."}
    ]
  }'

The gateway logs will confirm:

INFO resolved execution workspace workspace="/tmp" source=magic_command

5. Anthropic Messages (Real-time Streaming)

curl --noproxy '*' -N -s -X POST http://127.0.0.1:8082/v1/messages \
  -H "x-api-key: change-me" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6",
    "max_tokens": 1024,
    "stream": true,
    "messages": [
      {"role": "user", "content": "Count from 1 to 3."}
    ]
  }'

Output stream:

event: message_start
data: {"type":"message_start","message":{"id":"msg_...","type":"message","role":"assistant","model":"claude-sonnet-4-6","content":[],"stop_reason":null,"usage":{"input_tokens":13,"output_tokens":0}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"1, 2, 3\n"}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":16,"output_tokens_details":{"thinking_tokens":0}}}

event: message_stop
data: {"type":"message_stop"}

6. Multimodal Vision Example (OpenAI format)

curl --noproxy '*' -s -X POST http://127.0.0.1:8082/v1/chat/completions \
  -H "Authorization: Bearer change-me" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3.8-flash",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "What is in this image?"},
          {
            "type": "image_url",
            "image_url": {
              "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
            }
          }
        ]
      }
    ]
  }'

Structured Logging & Observability

agy-proxy-rs emits rich, structured telemetry via tracing:

2026-09-12T14:46:33.949299Z  INFO agy-proxy-rs listening for incoming requests addr=0.0.0.0:8082
2026-09-12T14:46:44.606992Z  INFO --> HTTP request received req_id=req_1fb7d7... method=POST uri=/v1/chat/completions
2026-09-12T14:46:44.607347Z  INFO resolved execution workspace req_id=req_1fb7d7... workspace="/tmp" source=magic_command
2026-09-12T14:46:44.607363Z  INFO processing chat completion req_id=req_1fb7d7... endpoint="chat/completions" model=gemini-3.8-flash stream=false prompt_len=30
2026-09-12T14:46:44.607651Z  INFO spawning agy engine process req_id=req_1fb7d7... run_id=run-f0038c... model=gemini-3.8-flash effort=Some("medium") bin="/Users/tinytinycn/.local/bin/agy" workspace="/tmp" prompt_len=30
2026-09-12T14:46:54.105836Z  INFO agy process finished req_id=req_1fb7d7... success=true exit_code=Some(0) total_chunks=7 duration_ms=9498
2026-09-12T14:46:54.106022Z  INFO <-- HTTP request completed req_id=req_1fb7d7... method=POST uri=/v1/chat/completions status=200 OK latency_ms=9499

Acknowledgements & Inspirations

Special thanks to the following upstream projects for their pioneering architecture and inspirations:

Sakiko15/agy-proxy: Foundational design for bridging Google Antigravity CLI to standard LLM protocols.
amlyczz/dsh-agy-link: Robust streaming watchdog, long-running task reliability patterns, and real-time keepalive architectures.

License

MIT

Name		Name	Last commit message	Last commit date
src		src
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
README.md		README.md
README_CN.md		README_CN.md

Navigation Menu

Repository files navigation

agy-proxy-rs (Rust)

Key Features

Requirements

Quick Start

1. Build and Run

Configuration

Working Directory & Workspace Management

1. Auto-Sniffing (Zero Configuration)

2. Magic Slash Commands (`/cwd`, `/workspace`)

3. HTTP Header & Body Overrides

API Endpoints & `curl` Examples

1. Health Check

2. List Models (OpenAI Format)

3. OpenAI Chat Completions (Real-time Streaming with Reasoning)

4. Magic Command Working Directory Test

5. Anthropic Messages (Real-time Streaming)

6. Multimodal Vision Example (OpenAI format)

Structured Logging & Observability

Acknowledgements & Inspirations

License

About

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

agy-proxy-rs (Rust)

Key Features

Requirements

Quick Start

1. Build and Run

Configuration

Working Directory & Workspace Management

1. Auto-Sniffing (Zero Configuration)

2. Magic Slash Commands (/cwd, /workspace)

3. HTTP Header & Body Overrides

API Endpoints & curl Examples

1. Health Check

2. List Models (OpenAI Format)

3. OpenAI Chat Completions (Real-time Streaming with Reasoning)

4. Magic Command Working Directory Test

5. Anthropic Messages (Real-time Streaming)

6. Multimodal Vision Example (OpenAI format)

Structured Logging & Observability

Acknowledgements & Inspirations

License

About

Resources

Stars

Watchers

Forks

Contributors

Languages

2. Magic Slash Commands (`/cwd`, `/workspace`)

API Endpoints & `curl` Examples