Skip to content

Connect a local model runtime from config, and ask it what it serves - #40

Open
its-janghoon wants to merge 3 commits into
developfrom
feature/local-provider-support
Open

its-janghoon wants to merge 3 commits into
developfrom
feature/local-provider-support

Conversation

@its-janghoon

Copy link
Copy Markdown
Contributor

Pointing Redrob Code at Ollama, LM Studio, llama.cpp or vLLM was not possible. ConfigProviderPlugin
refused to let a config file introduce any new AI-SDK provider, and its comment says exactly why: an
arbitrary npm package reaches DynamicProviderPlugin and gets installed and imported, which is arbitrary
code execution from a config file. That reasoning is correct and this change does not weaken it.

It blocked one case it did not need to. Every local runtime worth pointing at speaks the OpenAI-compatible
wire protocol, and @ai-sdk/openai-compatible is ALREADY the trusted, pinned package the console provider
itself uses -- so a local provider needs no new package at all. The hole opened here is exactly that
shape and no wider.

=== The gate ===

config/plugin/local-provider.ts decides it, and both conditions are necessary:

  • The package is EXACTLY @ai-sdk/openai-compatible. Anything else keeps the old refusal, so the
    install-an-arbitrary-package path stays shut. A local ADDRESS does not buy an arbitrary package.
  • The URL is loopback or private: 127/8, ::1, localhost, *.localhost (RFC 6761), 10/8, 172.16/12,
    192.168/16, 169.254/16, fc00::/7, fe80::/10. Without this, opening the door for local runtimes would
    also let a config file point a brand-new provider at any host on the internet -- quietly routing
    prompts, and whatever is in them, somewhere the user never chose.

A URL that does not parse is NOT local. That direction is deliberate: the cost of guessing wrong the other
way is a config file reaching the open internet, so anything unrecognised is refused.

=== Asking the runtime what it serves ===

config/plugin/local-models.ts fetches the OpenAI-standard GET <base>/models and adds those ids. A
hand-written model list goes stale the moment the user pulls a new model, which with Ollama is a one-line
command people run constantly.

Three properties, each chosen against a specific failure:

  • BEST-EFFORT. A local runtime that is not running is the normal state of a laptop, not an error. A
    failed fetch leaves the provider exactly as config declared it and logs at debug. Anything louder
    would make every start of the CLI complain about a model server the user is not using today.
  • RE-CHECKS THE ADDRESS RULE. This is an outbound request built from a config file, so it re-applies
    isLocalEndpoint rather than trusting the introduce gate. Two independent checks of one rule is the
    point: a later change that loosens one must not silently turn startup into a request to anywhere.
  • FAST TO FAIL. Two seconds, because the premise is a server on this machine or this LAN. The console's
    fetch can afford ten over the internet; waiting that long for loopback only delays the CLI when the
    runtime is down.

Model names are only DEFAULTED from the id, never overwritten -- a config file that gave a model a
readable name meant it, and the raw id is what that name replaced. Listing entries are decoded ONE AT A
TIME, because all-or-nothing decoding is what made the console catalog silently collapse to its six-id
fallback when a single entry was unexpected.

The fetch happens BEFORE the catalog transform, not inside it. ctx.catalog.transform takes
void | Effect<void, never, never> -- no requirements, no errors -- so HTTP cannot live there. Rather
than work around that, the providers are read from config directly, where they are declared anyway, and
the transform applies what is already in hand. That also drops an ordering dependency on whichever plugin
put the provider in the catalog.

=== Verification ===

The refusals carry the weight, because a predicate that is too permissive lets a config file reach the
internet while one that is too strict merely fails to find Ollama, which the user notices at once.

  • local-provider.test.ts (16): public addresses, the boundaries just outside each private range
    (172.32, 172.15, 11.0, 192.169), hostnames that merely CONTAIN a local name
    (localhost.evil.example), non-http schemes, and unparseable input failing closed.
  • local-models.test.ts (6): URL joining with and without trailing slashes, and the re-checked gate.
  • local-models-discovery.test.ts (3): the plugin against a REAL http server -- a stubbed client would
    only assert my own understanding of the wire format back at me. It pins that the request is actually
    made (GET /v1/models), that the served ids reach the catalog, that a public address produces ZERO
    requests, and that an unreachable runtime leaves the provider intact.
  • provider.test.ts (+3): a local provider is introduced, another package is refused, and a public
    address is refused even with the trusted package.

One of my own tests was wrong and the code was right: I asserted http://10.0.0/v1 should be refused, but
the URL parser normalises it to 10.0.0.0, which genuinely is private. Refusing it would refuse a real
private host, and this predicate is about where the request lands. The public counterpart (11.0.0) is
pinned as refused alongside it.

config group 79/79, core typecheck clean.

=== Known gap, not addressed here ===

I could not verify this through a running server: serve did not load a providers block from a project
redrob.json, a .redrob/redrob.json, or an isolated REDROB_CONFIG_DIR. The config itself is fine --
decoding that exact object against Config.Info succeeds and yields the provider -- and providers is
not a V1 key, so it takes the V2 path. So the loading problem is separate from this change, which is why the
verification above goes through the plugin directly. Worth its own look: config.ts's loadFile drops a
file silently on a decode failure (if (!info) return), which is the same no-error-no-warning failure a
neighbouring comment in that file already complains about.

… it serves

Pointing Redrob Code at Ollama, LM Studio, llama.cpp or vLLM was not possible. `ConfigProviderPlugin`
refused to let a config file introduce any new AI-SDK provider, and its comment says exactly why: an
arbitrary npm `package` reaches DynamicProviderPlugin and gets installed and imported, which is arbitrary
code execution from a config file. That reasoning is correct and this change does not weaken it.

It blocked one case it did not need to. Every local runtime worth pointing at speaks the OpenAI-compatible
wire protocol, and `@ai-sdk/openai-compatible` is ALREADY the trusted, pinned package the console provider
itself uses -- so a local provider needs no new package at all. The hole opened here is exactly that
shape and no wider.

=== The gate ===

`config/plugin/local-provider.ts` decides it, and both conditions are necessary:

  - The package is EXACTLY `@ai-sdk/openai-compatible`. Anything else keeps the old refusal, so the
    install-an-arbitrary-package path stays shut. A local ADDRESS does not buy an arbitrary package.
  - The URL is loopback or private: 127/8, ::1, localhost, *.localhost (RFC 6761), 10/8, 172.16/12,
    192.168/16, 169.254/16, fc00::/7, fe80::/10. Without this, opening the door for local runtimes would
    also let a config file point a brand-new provider at any host on the internet -- quietly routing
    prompts, and whatever is in them, somewhere the user never chose.

A URL that does not parse is NOT local. That direction is deliberate: the cost of guessing wrong the other
way is a config file reaching the open internet, so anything unrecognised is refused.

=== Asking the runtime what it serves ===

`config/plugin/local-models.ts` fetches the OpenAI-standard `GET <base>/models` and adds those ids. A
hand-written model list goes stale the moment the user pulls a new model, which with Ollama is a one-line
command people run constantly.

Three properties, each chosen against a specific failure:

  - BEST-EFFORT. A local runtime that is not running is the normal state of a laptop, not an error. A
    failed fetch leaves the provider exactly as config declared it and logs at debug. Anything louder
    would make every start of the CLI complain about a model server the user is not using today.
  - RE-CHECKS THE ADDRESS RULE. This is an outbound request built from a config file, so it re-applies
    `isLocalEndpoint` rather than trusting the introduce gate. Two independent checks of one rule is the
    point: a later change that loosens one must not silently turn startup into a request to anywhere.
  - FAST TO FAIL. Two seconds, because the premise is a server on this machine or this LAN. The console's
    fetch can afford ten over the internet; waiting that long for loopback only delays the CLI when the
    runtime is down.

Model names are only DEFAULTED from the id, never overwritten -- a config file that gave a model a
readable name meant it, and the raw id is what that name replaced. Listing entries are decoded ONE AT A
TIME, because all-or-nothing decoding is what made the console catalog silently collapse to its six-id
fallback when a single entry was unexpected.

The fetch happens BEFORE the catalog transform, not inside it. `ctx.catalog.transform` takes
`void | Effect<void, never, never>` -- no requirements, no errors -- so HTTP cannot live there. Rather
than work around that, the providers are read from config directly, where they are declared anyway, and
the transform applies what is already in hand. That also drops an ordering dependency on whichever plugin
put the provider in the catalog.

=== Verification ===

The refusals carry the weight, because a predicate that is too permissive lets a config file reach the
internet while one that is too strict merely fails to find Ollama, which the user notices at once.

  - `local-provider.test.ts` (16): public addresses, the boundaries just outside each private range
    (172.32, 172.15, 11.0, 192.169), hostnames that merely CONTAIN a local name
    (`localhost.evil.example`), non-http schemes, and unparseable input failing closed.
  - `local-models.test.ts` (6): URL joining with and without trailing slashes, and the re-checked gate.
  - `local-models-discovery.test.ts` (3): the plugin against a REAL http server -- a stubbed client would
    only assert my own understanding of the wire format back at me. It pins that the request is actually
    made (`GET /v1/models`), that the served ids reach the catalog, that a public address produces ZERO
    requests, and that an unreachable runtime leaves the provider intact.
  - `provider.test.ts` (+3): a local provider is introduced, another package is refused, and a public
    address is refused even with the trusted package.

One of my own tests was wrong and the code was right: I asserted `http://10.0.0/v1` should be refused, but
the URL parser normalises it to `10.0.0.0`, which genuinely is private. Refusing it would refuse a real
private host, and this predicate is about where the request lands. The public counterpart (`11.0.0`) is
pinned as refused alongside it.

config group 79/79, core typecheck clean.

=== Known gap, not addressed here ===

I could not verify this through a running server: `serve` did not load a `providers` block from a project
`redrob.json`, a `.redrob/redrob.json`, or an isolated `REDROB_CONFIG_DIR`. The config itself is fine --
decoding that exact object against `Config.Info` succeeds and yields the provider -- and `providers` is
not a V1 key, so it takes the V2 path. So the loading problem is separate from this change, which is why the
verification above goes through the plugin directly. Worth its own look: `config.ts`'s `loadFile` drops a
file silently on a decode failure (`if (!info) return`), which is the same no-error-no-warning failure a
neighbouring comment in that file already complains about.
…op apps read

The previous commit taught the V2 catalog to ask a local model runtime what it serves. The desktop apps do
not read that catalog. Tracing how cowork actually gets its provider list:

    apps/app/src/react-app/infra/provider-list-query.ts
      client.provider.list({ directory })        @opencode-ai/sdk/v2/client
      -> { all, connected, default }
      -> getConnectedProviderItems():
           connected.has(id) && (source !== "custom" || id === "opencode" || has models)

That `{ all, connected, default }` shape is built by `packages/redrob` -- the V1 shipping CLI -- not by
`packages/core`. So the V2 change reached the `lildax` preview CLI and nothing else.

Two things follow, and the second is the reason this is not cosmetic:

  - V1 already permits a config-declared provider at a local address, so connecting Ollama there worked.
  - But its models had to be WRITTEN OUT BY HAND, and cowork's filter above drops a `custom`-source
    provider unless it has at least one model. A local runtime declared without a model list is therefore
    not merely sparse in the model picker -- it is INVISIBLE there. A hand-written list also goes stale
    the moment the user pulls a new model, which with Ollama is a one-line command people run constantly.

So V1 now fetches the OpenAI-standard `GET <base>/models` for a config provider that sits at a local
address and declares no models of its own.

=== How ===

As a PRE-PASS that feeds the existing config merge, rather than by assembling model objects at a second
site. That merge already applies every default a model needs; a second construction path would drift from
it, and my first attempt at one was immediately caught by the type checker for missing `api.url`.

Hand-written entries win over discovered ones: a model the user described themselves keeps that
description. Listing entries are read ONE AT A TIME, because rejecting a whole listing over one
unexpected entry is what made the console catalog silently collapse to its fallback.

=== The gate is applied here too ===

`isLocalEndpoint` is re-checked on this path rather than assumed from elsewhere. This is an outbound
request assembled from a config file, and a project config travels with a cloned repository -- without the
check, opening someone's repo could make startup issue a request to any host it named. Two seconds of
timeout, because the premise is a server on this machine or this LAN.

Best-effort throughout: a local runtime that is not running is the normal state of a laptop, not an error,
and must not take the rest of the catalog down with it.

=== Verification ===

`local-discovery.test.ts` drives a real http server, because a stubbed fetch would only assert my own
understanding of the wire format back at me and that format is the entire interface to Ollama and
LM Studio. It pins that the request is actually made, that the served ids reach the provider's model map,
and that a provider pointed at a public address is NOT queried -- zero requests, no models.

The fixed port is deliberate and commented: `it.instance` takes its config at module-evaluation time, so
an OS-assigned port is not knowable in time to put in that config.

provider suite 103/103, new suite 2/2, typecheck clean.

=== Still open, deliberately not changed here ===

V1 places NO restriction on the npm package a config file names: `provider.npm` flows into
`Npm.add(model.api.npm)` and then `import`, and the module's first `create*` export is called. That is the
arbitrary-code-execution path the V2 comment exists to prevent, and it is open on the shipping path, with
a project config from a cloned repository as the realistic vector.

Closing it means refusing packages that work today, which breaks anyone who configured a custom provider
with another package. That is a product decision about an installed base, not a detail to slip into this
change, so it is reported rather than done.
CI caught this: `engine (linux 3/4)` went red with nine failures in `session.llm.stream`, all reporting
`GET /v1/models failed`.

The discovery pre-pass added in the previous commit probed any config provider that sat at a local address
and declared no models. That condition is too loose. **Pointing an ordinary provider at a local proxy is a
normal thing to do**, and pointing one at a mock server is what the test suite does — so the pre-pass was
issuing a request those servers never agreed to answer, during provider-list construction. A side effect
in that path is exactly the kind of thing a test is right to fail on.

Two more conditions are now required, chosen because they are true of a real local runtime and false of a
proxied provider:

  - It needs no credential: `env` empty and no `apiKey` option. A provider carrying a credential is
    somebody's hosted account reached through a local hop, not a model server on this machine.
  - It speaks the OpenAI-compatible protocol rather than naming a vendor package.

`session/llm` is back to 30/30, and all four redrob shards pass (677 / 1151 / 718 / 803). The discovery
tests still pass, including the public-address refusal.

=== How I nearly got this wrong ===

Worth recording, because the method failed before the code did. To check whether the failures were mine I
ran `git stash push -- packages/redrob/src/provider/provider.ts`, saw the suite still fail, and concluded
they were pre-existing. They were not: the file was already COMMITTED, so the stash saved nothing and both
measurements were taken with my change in place. The `&&` chain printed a reassuring "change removed"
message regardless.

`git checkout HEAD~1 -- <file>` gave the real answer immediately: 30/30 on the base, 9 failures with my
change. The lesson is not about git — it is that a control condition has to be VERIFIED to have taken
effect, not assumed from the command having exited zero.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant