Skip to content

generate-names: default to the frontier models, not the cheap tier - #16

Merged
ralyodio merged 1 commit into
masterfrom
better-default-models
Aug 19, 2026
Merged

generate-names: default to the frontier models, not the cheap tier#16
ralyodio merged 1 commit into
masterfrom
better-default-models

Conversation

@ralyodio

Copy link
Copy Markdown
Contributor

Defaults were gpt-4.1-mini and claude-haiku-4-5. Now gpt-5.6-sol and claude-fable-5.

Why

The names showed the cheap tier: generic startup vocabulary, the same few stems recycled, and a drift off-brief on any description longer than a sentence.

And the saving was never real. This tool makes exactly one call per run whatever --count says — that is the whole design, since asking a model for a thousand names directly repeats itself within a few hundred. The model is a rounding error against the value of a name you actually ship, so paying for the weaker one bought nothing.

What the Anthropic path needed first

  • max_tokens 4096 → 16000. Fable 5 always thinks, and thinking counts against max_tokens, so a budget that was ample for a non-thinking model can be spent before the JSON starts.
  • stop_reason: "refusal" is now reported as a refusal. It arrives as HTTP 200 with no text block, so unhandled it reads as "the model returned nothing" — which sends you to the parser rather than to the answer the API actually gave.

Reading the text block by type rather than by position already handled the leading thinking block; there is now a test pinning that, plus one pinning that this request never grows temperature/top_p/top_k — all three were removed on Fable 5 / Opus 5 / Sonnet 5 and are rejected with a 400.

Heads-up for anyone hitting this

Every Anthropic key in the team vaults is spend-capped until 2026-09-01 (400 invalid_request_error, "You will regain access on 2026-09-01"). So --provider anthropic fails until then and OpenAI is the working path. Verified against a live call today.

Checks

187 tests pass (was 180), typecheck clean.

Verified end to end: the full generate-names … | domainfree pipeline returned 41 available of 100 checked, and a bare run with no --model now reports openai/gpt-5.6-sol.

🤖 Generated with Claude Code

The defaults were gpt-4.1-mini and claude-haiku-4-5, and the names showed it:
generic startup vocabulary, the same few stems recycled, and a drift off-brief
on any description longer than a sentence. Now gpt-5.6-sol and claude-fable-5.

The cheap tier was never the saving it looked like. This tool makes exactly ONE
call per run whatever --count says — that is the whole design, since asking a
model for a thousand names directly repeats itself within a few hundred. So the
model is a rounding error against the value of a name you actually ship, and
paying for the weaker one bought nothing.

Two things the Anthropic path needed before claude-fable-5 could be the default:

- max_tokens 4096 -> 16000. Fable 5 always thinks and thinking counts against
  max_tokens, so a budget that was ample for a non-thinking model can be spent
  before the JSON starts.
- stop_reason "refusal" is now reported as a refusal. It arrives as HTTP 200
  with no text block, so unhandled it reads as "the model returned nothing" and
  sends you to the parser rather than to the answer the API actually gave.

Reading the text block by type rather than by position already handled the
leading thinking block; there is now a test pinning that, and one pinning that
this request never grows temperature/top_p/top_k — all three were removed on
Fable 5 / Opus 5 / Sonnet 5 and are rejected with a 400.

Note for anyone hitting this: every Anthropic key in the team vaults is
spend-capped until 2026-09-01, so --provider anthropic returns a 400 until then
and OpenAI is the working path.

187 tests pass (was 180), typecheck clean. Verified end to end against both the
piped domainfree run and a bare default-model run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

ThreatCrush Security Scan

6 finding(s)

MEDIUM: 1 | LOW: 5

Severity Rule Location
MEDIUM redos-nested-quantifier src/domain-free.ts:56
LOW insecure-temp-file test/blog.test.ts:73
LOW insecure-temp-file test/blog.test.ts:74
LOW insecure-temp-file test/credentials.test.ts:43
LOW insecure-temp-file test/credentials.test.ts:44
LOW secret-generic-api-key test/credentials.test.ts:208

Snippets are redacted; ThreatCrush never prints matched credential material.

@ralyodio
ralyodio merged commit cd8c650 into master Aug 19, 2026
5 checks passed
@ralyodio
ralyodio deleted the better-default-models branch August 19, 2026 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant