feat(providers): add Cloudflare Workers AI as an LLM and embedding provider#1111
Open
noises1990 wants to merge 2 commits into
Open
feat(providers): add Cloudflare Workers AI as an LLM and embedding provider#1111noises1990 wants to merge 2 commits into
noises1990 wants to merge 2 commits into
Conversation
…ovider Workers AI exposes OpenAI-compatible /ai/v1/chat/completions and /ai/v1/embeddings endpoints, so both providers reuse the existing fetchWithTimeout transport and the same request/response shapes as the OpenAI providers. CLOUDFLARE_AI_BASE_URL / CLOUDFLARE_EMBEDDING_BASE_URL also accept an AI Gateway URL for logging, caching and rate limiting. CLOUDFLARE_API_TOKEN alone enables Cloudflare for both roles. It sits after MINIMAX and before ANTHROPIC in the LLM detection order, and after OPENAI in the embedding order, so it never displaces an already-working setup. EMBEDDING_PROVIDER still overrides if you want to mix providers. Requests already flow through the account's default AI Gateway, so logging, caching, rate limiting and guardrails apply with no config. CLOUDFLARE_AI_GATEWAY_ID pins a named gateway instead — Cloudflare selects it via the cf-aig-gateway-id header, not a different base URL. Default chat model is @cf/meta/llama-3.1-8b-instruct-fp8: the non-fp8 @cf/meta/llama-3.1-8b-instruct was deprecated on 2026-05-30 and now returns HTTP 410, so it cannot be the default. Content extraction deliberately has no `reasoning` fallback. Reasoning models (@cf/zai-org/glm-*, qwq, deepseek-r1) return content:null plus a populated `reasoning` field once the token budget is spent thinking; treating that scratchpad as the answer would write chain-of-thought into memory and feed it to the XML parsers in summarize.ts. Truncation now raises an error naming MAX_TOKENS instead. Endpoint construction, auth headers and gateway selection live in _cloudflare-shared.ts so the chat and embedding providers do not mirror them, matching the existing _openai-shared.ts split. Embedding dimensions come from a known-models table (bge-small 384, bge-base 768, bge-large / bge-m3 / qwen3-embedding 1024, embeddinggemma-300m 768) because withDimensionGuard throws on every embed when the reported size is wrong. CLOUDFLARE_EMBEDDING_DIMENSIONS covers anything not in the table. Verified against live Workers AI: compress, summarize, the full loadConfig -> createProvider path, 384/768/1024-dim embeddings through withDimensionGuard, and gateway routing (unpinned, named, and a bad gateway id surfacing a 400). Verify: CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ACCOUNT_ID=... npx @agentmemory/agentmemory doctor npx vitest run test/cloudflare-provider.test.ts test/embedding-provider.test.ts Signed-off-by: Adrian Coroi <coroi_adrian@outlook.com>
|
@noises1990 is attempting to deploy a commit to the rohitg00's projects Team on Vercel. A member of the Team first needs to authorize it. |
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (8)
🚧 Files skipped from review as they are similar to previous changes (8)
📝 WalkthroughWalkthroughCloudflare Workers AI is added as an LLM and embedding provider. The change adds shared endpoint and gateway handling, provider detection and construction, embedding dimension validation, onboarding and diagnostics updates, documentation, and automated coverage. ChangesCloudflare provider integration
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant Config
participant CloudflareProvider
participant WorkersAI
CLI->>Config: Load provider configuration
Config->>CloudflareProvider: Construct cloudflare provider
CloudflareProvider->>WorkersAI: Send chat or embedding request
WorkersAI-->>CloudflareProvider: Return completion or embedding data
CloudflareProvider-->>CLI: Return provider result
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
README.md (1)
1379-1380: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDocument Cloudflare base URL overrides here.
The README lists token, account, model, and dimensions settings but omits
CLOUDFLARE_AI_BASE_URLandCLOUDFLARE_EMBEDDING_BASE_URL, despite both being supported configuration paths. Add those variables so users following the README can configure custom endpoints without relying on.env.example.Also applies to: 1416-1419
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@README.md` around lines 1379 - 1380, Update the Cloudflare configuration examples in the README to include CLOUDFLARE_AI_BASE_URL and CLOUDFLARE_EMBEDDING_BASE_URL alongside the existing token, account, model, and dimensions settings, documenting that they override the default endpoints.src/providers/embedding/cloudflare.ts (1)
28-39: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winDimensions override parsing is lenient, and the doc comment overstates enforcement.
parseInt(override, 10)accepts trailing garbage ("1024abc"→1024,"10.5"→10) silently, unlike the siblingparsePositiveIntincloudflare.tswhich strictly validates the whole string with/^\d+$/. Separately, the doc comment saysCLOUDFLARE_EMBEDDING_DIMENSIONSis "required for models not in the MODEL_DIMENSIONS table," but the code never throws for an unknown model without an override — it silently usesDEFAULT_DIMENSIONS.♻️ Align validation strictness with parsePositiveInt
- if (override !== undefined && override.trim().length > 0) { - const parsed = parseInt(override, 10); - if (!Number.isFinite(parsed) || parsed <= 0) { + if (override !== undefined && override.trim().length > 0) { + const trimmed = override.trim(); + const parsed = /^\d+$/.test(trimmed) ? Number(trimmed) : NaN; + if (!Number.isFinite(parsed) || parsed <= 0) { throw new Error( `CLOUDFLARE_EMBEDDING_DIMENSIONS must be a positive integer, got: ${override}`, ); } return parsed; }Also applies to: 56-57
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/providers/embedding/cloudflare.ts` around lines 28 - 39, Update resolveDimensions to validate the entire override string with the same strict positive-integer rules as the sibling parsePositiveInt, rejecting decimal values, trailing characters, and non-positive values before returning the parsed dimension. Correct the related documentation to state the actual fallback behavior for models absent from MODEL_DIMENSIONS, since unknown models without an override use DEFAULT_DIMENSIONS rather than throwing.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/cli.ts`:
- Around line 1733-1740: Update the setup hints to describe complete Cloudflare
configuration: in src/cli.ts lines 1733-1740, mention CLOUDFLARE_ACCOUNT_ID or
the relevant chat and embedding base-URL overrides in both Cloudflare hints; in
src/viewer/index.html line 3752, broaden the LLM banner beyond the token alone;
at line 3765 include CLOUDFLARE_ACCOUNT_ID or CLOUDFLARE_AI_BASE_URL in the LLM
snippet; and at line 3777 include CLOUDFLARE_ACCOUNT_ID or
CLOUDFLARE_EMBEDDING_BASE_URL in the embedding snippet.
---
Nitpick comments:
In `@README.md`:
- Around line 1379-1380: Update the Cloudflare configuration examples in the
README to include CLOUDFLARE_AI_BASE_URL and CLOUDFLARE_EMBEDDING_BASE_URL
alongside the existing token, account, model, and dimensions settings,
documenting that they override the default endpoints.
In `@src/providers/embedding/cloudflare.ts`:
- Around line 28-39: Update resolveDimensions to validate the entire override
string with the same strict positive-integer rules as the sibling
parsePositiveInt, rejecting decimal values, trailing characters, and
non-positive values before returning the parsed dimension. Correct the related
documentation to state the actual fallback behavior for models absent from
MODEL_DIMENSIONS, since unknown models without an override use
DEFAULT_DIMENSIONS rather than throwing.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 1f01f4b4-9dd5-446a-b72d-6cdcb2a27a78
📒 Files selected for processing (17)
.env.exampleREADME.mdsrc/cli.tssrc/cli/doctor-diagnostics.tssrc/cli/onboarding.tssrc/config.tssrc/functions/consolidation-pipeline.tssrc/functions/summarize.tssrc/providers/_cloudflare-shared.tssrc/providers/cloudflare.tssrc/providers/embedding/cloudflare.tssrc/providers/embedding/index.tssrc/providers/index.tssrc/types.tssrc/viewer/index.htmltest/cloudflare-provider.test.tstest/embedding-provider.test.ts
- Setup hints advertised CLOUDFLARE_API_TOKEN alone. Cloudflare is the only provider needing a second variable, so doctor, init, the diagnostics moreInfo and the three viewer banners now name CLOUDFLARE_ACCOUNT_ID (or the base-URL override) alongside the token. - CLOUDFLARE_EMBEDDING_DIMENSIONS used parseInt, which took "1024abc" as 1024 and "10.5" as 10 — silently producing vectors withDimensionGuard rejects on every embed. It now shares the strict parsePositiveInt used for CLOUDFLARE_TIMEOUT_MS, moved into _cloudflare-shared.ts so the two providers stop carrying separate copies. Covered by a table test. - MODEL_DIMENSIONS doc claimed the override was "required" for unknown models; the code falls back to the default size instead. Reworded to describe what actually happens. - README omitted CLOUDFLARE_AI_BASE_URL, CLOUDFLARE_EMBEDDING_BASE_URL and CLOUDFLARE_AI_GATEWAY_ID from the env blocks. Re-verified against live Workers AI: chat and 768-dim embeddings still resolve through the default path. Signed-off-by: Adrian Coroi <coroi_adrian@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Cloudflare Workers AI as a first-class LLM and embedding provider.
CLOUDFLARE_API_TOKENalone enables both roles. Workers AI exposes OpenAI-compatible/ai/v1/chat/completionsand/ai/v1/embeddingsendpoints, so both providers reuse the existingfetchWithTimeouttransport and the same wire shapes as the OpenAI providers — no new dependencies.Why
Workers AI is a cheap, no-egress option for the background compress/summarize workload, and it is the only major provider that covers both chat and embeddings from a single token. For anyone already on Cloudflare it removes a second vendor from the setup.
Concretely, on the compress-a-short-observation workload this PR's default model costs 0.8 neurons per call — roughly two orders of magnitude below a reasoning-class model (measured, table below).
Detection order
Cloudflare slots in so it never displaces a working setup:
OPENAI_API_KEY→MINIMAX_API_KEY→CLOUDFLARE_API_TOKEN→ANTHROPIC_API_KEY→GEMINI_API_KEY→OPENROUTER_API_KEY→ noopEMBEDDING_PROVIDER→GEMINI_API_KEY→OPENAI_API_KEY→CLOUDFLARE_API_TOKEN→VOYAGE_API_KEY→COHERE_API_KEY→OPENROUTER_API_KEY→ localEMBEDDING_PROVIDER=cloudflarestill works as an explicit override, andFALLBACK_PROVIDERS=cloudflareis accepted.Environment variables
CLOUDFLARE_API_TOKENCLOUDFLARE_ACCOUNT_ID*_BASE_URLis setCLOUDFLARE_MODEL@cf/meta/llama-3.1-8b-instruct-fp8)CLOUDFLARE_AI_BASE_URLCLOUDFLARE_AI_GATEWAY_IDCLOUDFLARE_TIMEOUT_MSAGENTMEMORY_LLM_TIMEOUT_MSCLOUDFLARE_EMBEDDING_MODEL@cf/baai/bge-base-en-v1.5)CLOUDFLARE_EMBEDDING_DIMENSIONSCLOUDFLARE_EMBEDDING_BASE_URLThree decisions worth reviewing
1. The default chat model is the
-fp8variant. The obvious pick,@cf/meta/llama-3.1-8b-instruct, was deprecated on 2026-05-30 and now returns HTTP 410 — it is gone from the model catalogue entirely.@cf/meta/llama-3.1-8b-instruct-fp8is the live successor and carries no deprecation flag.2. There is deliberately no
reasoningfallback when extracting content. Reasoning models (@cf/zai-org/glm-*,qwq,deepseek-r1) returncontent: nullalongside a populatedreasoningfield once the token budget is spent thinking. Treating that as the answer writes model scratchpad into memory and feeds it to the XML parsers insummarize.ts. Truncation now raises an error namingMAX_TOKENSinstead. Two tests cover this.3. Embedding dimensions come from a
MODEL_DIMENSIONStable, mirroringembedding/openai.ts.withDimensionGuardthrows on every embed when the reported size is wrong, so guessing one size for all models turns a config mistake into a total outage.CLOUDFLARE_EMBEDDING_DIMENSIONScovers anything not in the table.AI Gateway
The default endpoint already routes through the account's default gateway, so logging, caching, rate limiting and guardrails apply with no configuration.
CLOUDFLARE_AI_GATEWAY_IDpins a named gateway — Cloudflare selects it via thecf-aig-gateway-idheader, not a different base URL. Applies to both chat and embeddings.Structure
src/providers/_cloudflare-shared.tsholds endpoint construction, auth headers and gateway selection so the chat and embedding providers don't mirror them — the same split as the existing_openai-shared.ts.One knowing exception:
detectProvider()inconfig.tsrepeats the default model as a literal rather than importing the constant, becauseproviders/importsconfig.tsand importing back would cycle. Every other provider default in that function makes the same trade-off; there's a comment saying so.Testing
test/cloudflare-provider.test.ts— 20 tests covering endpoint resolution, the missing-account-id error, timeout precedence,detectProviderselection,FALLBACK_PROVIDERSacceptance, every dimension path, response parsing, and gateway header behaviour. Plus a detection case intest/embedding-provider.test.ts.Tests blank competing env keys to
""rather thandelete-ing them:getMergedEnv()layersprocess.envover the developer's real~/.agentmemory/.env, so deleting a key lets that file's value through and the suite fails on any machine that has anOPENAI_API_KEYconfigured.Also verified against live Workers AI (not just mocked):
compress,summarize, the fullloadConfig()→createProvider()→ResilientProviderpath, 384/768/1024-dim embeddings throughwithDimensionGuard, gateway routing (unpinned / named / bad-id → 400), and the bad-token error path.Measured cost, identical prompt,
neuronsas reported by the API:@cf/meta/llama-3.1-8b-instruct-fp8(default)@cf/meta/llama-3.2-3b-instruct@cf/zai-org/glm-5.2@cf/zai-org/glm-4.7-flashHow to verify
Checks
npm run build— cleannpm test— 1431 passing. The 5 failures intest/hook-project.test.tsare pre-existing and reproduce on an unmodifiedmain; they assertbasename(cwd) === "agentmemory"and fail whenever the clone directory has another name.npx tsc --noEmit— 25 errors, all pre-existing onmain, none in the files this PR touches.Not in scope
AGENTMEMORY_SUMMARIZE_MODELand friends). That needs task identity at theMemoryProviderboundary and overlaps feat(providers): route graph extraction via AGENTMEMORY_GRAPH_MODEL #954 — worth its own issue, and provider-agnostic rather than Cloudflare-specific.resolveDimensionsacrossembedding/openai.tsandembedding/cloudflare.ts. Real duplication, but collapsing it means editing the OpenAI provider, which doesn't belong in a PR that adds a provider. Happy to follow up.Summary by CodeRabbit