v0.7.52: span sanitization, highlight tables and files text to chat, settings UX improvements, security hardening

.agents/skills/add-block/SKILL.md

@@ -919,6 +919,12 @@ Derive templates from the service's real use cases. Each prompt should name a co
 - **Ground every skill in operations the block actually exposes** — cross-check each skill's steps against `tools.access`. Never describe an action the integration cannot perform.
 - **Derive skills from real, popular use cases found online — never invent them.** Web-search the service's documented use cases (vendor use-case/solutions pages, official docs describing the workflow, reputable "top automations for X" articles) and only add a skill you can source as something people genuinely do with the service. Do not hallucinate skills.
 
+## Generated tool metadata
+
+Adding a block on its own needs **no** regeneration — a block references existing tool IDs through `tools.access` and does not change any tool's shape.
+
+But if the same change also adds, edits **or removes** a tool, run `bun run tool-metadata:generate` and commit the result, or CI fails on stale artifacts. That matters here because a block's `outputs` are authored to match its tools' outputs, and the UI now reads those from the generated metadata rather than the executable registry — an unregenerated tool change makes the block's outputs disagree with what the panel renders. See `.agents/skills/tool-registry-boundary/SKILL.md`.
+
 ## Checklist Before Finishing
 
 - [ ] `integrationType` is set to the correct `IntegrationType` enum value
@@ -933,6 +939,7 @@ Derive templates from the service's real use cases. Each prompt should name a co
 - [ ] Tools.config.tool returns correct tool ID (snake_case)
 - [ ] Outputs match tool outputs
 - [ ] Block + meta registered in registry-maps.ts (`BLOCK_REGISTRY` / `BLOCK_META_REGISTRY`)
+- [ ] If any tool was added, changed or removed alongside the block: ran `bun run tool-metadata:generate` and committed the artifacts
 - [ ] If icon missing: asked user to provide SVG
 - [ ] If triggers exist: `triggers` config set, trigger subBlocks spread
 - [ ] Optional/rarely-used fields set to `mode: 'advanced'`

.agents/skills/add-integration/SKILL.md

@@ -415,6 +415,16 @@ export const tools: Record<string, ToolConfig> = {
 }
 ```
 
+Then regenerate the generated tool metadata and commit it:
+
+```bash
+bun run tool-metadata:generate
+```
+
+Client code reads `params`/`outputs` from these artifacts rather than importing
+the registry, so a tool you add, change or remove is invisible to the UI until they are regenerated,
+and CI fails on stale ones. See `.agents/skills/tool-registry-boundary/SKILL.md`.
+
 ### Block Registry (`apps/sim/blocks/registry-maps.ts`)
 
 The data maps (`BLOCK_REGISTRY` + `BLOCK_META_REGISTRY`) live in `registry-maps.ts`; `registry.ts` holds only the accessor functions. Add the import and an entry to each map alphabetically:
@@ -490,6 +500,7 @@ If creating V2 versions (API-aligned outputs):
 - [ ] All optional outputs have `optional: true`
 - [ ] Created `index.ts` barrel export
 - [ ] Registered all tools in `tools/registry.ts`
+- [ ] Ran `bun run tool-metadata:generate` and committed the regenerated artifacts
 
 ### Block
 - [ ] Created `blocks/blocks/{service}.ts`

.agents/skills/add-tools/SKILL.md

@@ -296,6 +296,17 @@ export const tools = {
 }
 ```
 
+3. Regenerate the tool metadata artifacts:
+
+```bash
+bun run tool-metadata:generate
+```
+
+Client code reads a tool's `params`/`outputs` from generated metadata rather than
+importing the registry, so a tool you add, change or remove is invisible to the UI until
+these are regenerated — and CI fails on stale artifacts. Commit the result. See
+`.agents/skills/tool-registry-boundary/SKILL.md`.
+
 ## Wiring Tools into the Block (Required)
 
 After registering in `tools/registry.ts`, you MUST also update the block definition at `apps/sim/blocks/blocks/{service}.ts`. This is not optional — tools are only usable from the UI if they are wired into the block.
@@ -443,6 +454,7 @@ All tool IDs MUST use `snake_case`: `{service}_{action}` (e.g., `x_create_tweet`
 - [ ] Types file has all interfaces
 - [ ] Index.ts exports all tools and re-exports types (`export * from './types'`)
 - [ ] Tools registered in `tools/registry.ts`
+- [ ] `bun run tool-metadata:generate` run and the regenerated artifacts committed
 - [ ] Block wired: `tools.access`, dropdown options, subBlocks, `tools.config`, outputs, inputs
 
 ## Final Validation (Required)

.agents/skills/tool-registry-boundary/SKILL.md

@@ -0,0 +1,94 @@
+---
+name: tool-registry-boundary
+description: Keep the executable tool registry out of client-reachable module graphs — when to read `@/tools/metadata` instead of `getTool`, how to measure whether an import edge pulls the registry, and how to regenerate the metadata artifacts. Use when touching `apps/sim/tools/registry.ts`, `tools/utils.ts`, `tools/params.ts`, or anything that calls `getTool`.
+---
+
+# Tool Registry Boundary Skill
+
+You keep the 4,300-tool executable registry out of module graphs that don't execute tools.
+
+## The rule
+
+> Client-reachable code reads tool **metadata**. Only code that actually executes a tool imports the **registry**.
+
+`@/tools/registry` is a ~9,000-line barrel importing every tool. Each `ToolConfig` mixes plain data (`params`, `outputs`, `name`) with closures — `request.url`, `request.headers`, `transformResponse`, `directExecution`, `postProcess`. Those closures reach the SDK clients, API helpers and parsers each integration needs, and that is what makes the barrel expensive: reaching it costs ~4,700 additional modules.
+
+`getTool()` returns the whole `ToolConfig`, so a single `getTool` import anywhere in a client-reachable file drags all of it in.
+
+## Which module to import
+
+| you need | import | notes |
+| --- | --- | --- |
+| whether a tool id exists | `hasToolId` from `@/tools/tool-ids` | ~110 KB — the cheapest module |
+| to resolve an unversioned name | `resolveToolId` from `@/tools/tool-ids` | |
+| every tool id | `getToolIds` from `@/tools/tool-ids` | |
+| a tool's params | `getToolParams` / `getToolMetadata` from `@/tools/metadata` | ~4 MB |
+| a tool's declared outputs | `getToolOutputsMetadata` from `@/tools/metadata-outputs` | ~4 MB, separate on purpose |
+| to **execute** a tool | `getTool` from `@/tools/utils`, or `@/tools/utils.server` | server paths only |
+
+Three modules, cheapest first. Ids are their own artifact because resolution and existence checks need only the key set; outputs are their own because they are the larger half of the data with a single consumer. `@/tools/metadata` and `@/tools/metadata-outputs` both resolve ids through `@/tools/tool-ids`, which is what keeps them independent of each other — do not "helpfully" re-export one from another, or every caller pays for all three.
+
+All lookups guard with `Object.hasOwn`. `JSON.parse` yields an object with the normal prototype, so a bare bracket lookup returns inherited members: `getToolMetadata('constructor')` returned a *function* typed as tool metadata before that was fixed.
+
+## The generated artifacts
+
+`apps/sim/tools/generated/tool-ids.ts`, `tool-metadata.ts` and `tool-outputs.ts` are produced by `scripts/sync-tool-metadata.ts`:
+
+```bash
+bun run tool-metadata:generate   # after adding/changing a tool
+bun run tool-metadata:check      # what CI runs; fails if stale
+```
+
+Never hand-edit them. If you add a tool or change a tool's `params`/`outputs`, regenerate and commit the result, or CI fails.
+
+Three non-obvious properties, each of which was measured and is easy to undo by accident:
+
+- **The data is a JSON string parsed at runtime, not an imported `.json` and not an object literal.** With `resolveJsonModule` (which this repo enables), a `.json` import makes TypeScript infer a literal type for all 4,300+ entries and takes `tsc --noEmit` from **12.6s to 8m07s** — a 38x regression. An ambient `declare module` does *not* short-circuit it, and an object literal costs the same. A single string literal is one cheap token for both the compiler and the bundler, and `JSON.parse` beats evaluating the equivalent literal at runtime. Do not "clean this up" into a `.json` import.
+- **The generator refuses to emit function values.** If you add a field to `METADATA_FIELDS` that contains a closure, generation fails loudly rather than shipping executable config to the client. `hosting` and `schemaEnrichment` are excluded for exactly this reason (`hosting.enabled`, `pricing`, and `enrichSchema` are functions) — they are server-only.
+- **Empty param entries are stripped.** The registry contains one (`stt_deepgram_v2`), which crashes callers that read `param.type` while iterating.
+- **Lookups resolve versions.** `getTool` maps an unversioned name onto the newest version, and 246 tools are versioned. A plain key lookup would silently report them missing — a quiet correctness bug, not a crash. `resolveToolId` reproduces that against the id set and is differentially tested against the original.
+
+## Testing code that reads tool metadata
+
+Mock the module the code under test actually reads. `vi.mock('@/tools/utils', () => toolsUtilsMock)` only controls `getTool`; code that reads `params`/`outputs`/`name` goes through `@/tools/metadata`, so mocking `tools/utils` there is a **no-op that still passes** — because the real generated artifacts happen to agree with the mock fixtures. The test looks green while controlling nothing.
+
+```ts
+import { blocksMock, toolsMetadataMock, toolsUtilsMock } from '@sim/testing/mocks'
+
+vi.mock('@/tools/utils', () => toolsUtilsMock)      // executable lookup
+vi.mock('@/tools/metadata', () => toolsMetadataMock) // params / outputs / name
+```
+
+Both are backed by the same `mockToolConfigs`, so mocking both gives one consistent tool universe. If you are unsure whether a mock is load-bearing, change a fixture value to a sentinel and confirm the test fails.
+## The guard
+
+`bun run check:tool-registry-boundary` (CI: "Tool registry client-boundary audit") walks the module graph from each workspace route and fails if `@/tools/registry` is reachable, printing the exact import chain that reintroduced it.
+
+If it fails, do not add the entry to an allowlist — there isn't one. Find the symbol the offending file actually needs and move it to a registry-free module, exactly as `mergeToolParameters` and `formatParameterLabel` were.
+
+Run it with `--verbose` to print per-route module counts, which is also the quickest way to see whether a change moved the graph.
+
+## How to verify an edge actually got cut
+
+Do not eyeball imports — the registry is reached through several redundant paths, so cutting one buys nothing while another survives. Walk the graph:
+
+1. From the entry you care about, follow `import` and `export … from` (skipping `import type`), resolving `@/` against `apps/sim`.
+2. Check whether `apps/sim/tools/registry.ts` is in the reachable set, and print the parent chain if it is.
+3. Compare the reachable module count before and after.
+
+Reference points measured on this repo:
+
+| entry | modules |
+| --- | --- |
+| `tools/registry.ts` reachable | ~4,900 |
+| `tools/merge-params.ts` (leaf) | 2 |
+| `providers/utils.ts` after cutting its `params` edge | 22 |
+| `app/workspace/[workspaceId]/w/page.tsx` (canvas) | 6,592 before, 1,908 after |
+
+The canvas route reached the registry through **four** redundant edges — `providers/utils` (via `tools/params`), `lib/workflows/blocks/block-outputs`, `lib/workflows/sanitization/validation`, and `serializer/index`. Cutting any one alone moved the module count by ~1. They all had to go before anything improved; measure the route, not the file you edited.
+
+## When adding a new caller
+
+Ask what the caller does with the config. If it reads `params`, `outputs`, `name`, `description` or just checks existence, it belongs on `@/tools/metadata` — no exceptions, even on a path you believe is server-only today, because a future client import will silently re-attach the registry to the graph.
+
+If it genuinely executes — builds a request, transforms a response, runs `directExecution` — use `getTool`, and keep that file off client-reachable paths.

.agents/skills/validate-integration/SKILL.md

@@ -295,13 +295,31 @@ Group findings by severity:
 
 After reporting, fix every **critical** and **warning** issue. Apply **suggestions** where they don't add unnecessary complexity.
 
+### Regenerate Derived Artifacts
+
+Several files are generated from tool and block definitions. Editing a tool or block WITHOUT regenerating them fails CI, so run these before pushing:
+
+```bash
+bun run tool-metadata:generate       # repo root — apps/sim/tools/generated/*
+cd apps/sim && bun run generate-docs # docs .mdx + lib/integrations/integrations.json + docs icons
+```
+
+- **`tool-metadata:generate`** — required whenever a tool's `outputs`, `params`, or descriptions change. CI enforces this with `bun run tool-metadata:check`, which fails with *"Generated tool metadata is stale"*. This is the easiest gate to miss, because nothing in the tool file hints that a generated artifact mirrors it.
+- **`generate-docs`** — required whenever block metadata changes (`bgColor`, `name`, `description`, operations, outputs). Regenerates the integration `.mdx`, `integrations.json`, and the docs copy of `components/icons.tsx`.
+
+**Always diff the regen output before committing.** These generators rewrite every file they own, so they will also sweep in unrelated drift that accumulated on the base branch — pages losing sections, unrelated icons appearing. Keep only the hunks belonging to the integration under validation and `git checkout --` the rest, otherwise an unrelated doc regression rides along in the PR. Verify no page was silently dropped by comparing the directory listing before and after.
+
+If an icon changed, `apps/sim/components/icons.tsx` is the source of truth and `apps/docs/components/icons.tsx` is its generated mirror — they must end up byte-identical for that component.
+
 ### Validation Output
 
 After fixing, confirm:
 1. `bun run lint` passes with no fixes needed
-2. TypeScript compiles clean (no type errors)
-3. Re-read all modified files to verify fixes are correct
-4. Any remaining unknown response schemas were explicitly reported to the user instead of guessed
+2. TypeScript compiles clean (no type errors) — check the error list is empty for the files you touched; pre-existing unrelated errors in a worktree usually mean workspace packages resolve to the main checkout
+3. The integration's tests pass, and any test you added actually fails without its fix (revert it once and watch it go red)
+4. Derived artifacts regenerated and their diffs reviewed (see above)
+5. Re-read all modified files to verify fixes are correct
+6. Any remaining unknown response schemas were explicitly reported to the user instead of guessed
 
 ## Checklist Summary
 
@@ -322,5 +340,8 @@ After fixing, confirm:
 - [ ] Validated `{Service}BlockMeta` exported with at least 7 templates
 - [ ] Reported all issues grouped by severity
 - [ ] Fixed all critical and warning issues
+- [ ] Ran `bun run tool-metadata:generate` if any tool outputs/params changed, and confirmed `bun run tool-metadata:check` passes
+- [ ] Ran `bun run generate-docs` if any block metadata changed, and reverted unrelated drift the generator swept in
 - [ ] Ran `bun run lint` after fixes
 - [ ] Verified TypeScript compiles clean
+- [ ] Verified added tests fail without their fix

.claude/commands/add-block.md

@@ -918,6 +918,12 @@ Derive templates from the service's real use cases. Each prompt should name a co
 - **Ground every skill in operations the block actually exposes** — cross-check each skill's steps against `tools.access`. Never describe an action the integration cannot perform.
 - **Derive skills from real, popular use cases found online — never invent them.** Web-search the service's documented use cases (vendor use-case/solutions pages, official docs describing the workflow, reputable "top automations for X" articles) and only add a skill you can source as something people genuinely do with the service. Do not hallucinate skills.
 
+## Generated tool metadata
+
+Adding a block on its own needs **no** regeneration — a block references existing tool IDs through `tools.access` and does not change any tool's shape.
+
+But if the same change also adds, edits **or removes** a tool, run `bun run tool-metadata:generate` and commit the result, or CI fails on stale artifacts. That matters here because a block's `outputs` are authored to match its tools' outputs, and the UI now reads those from the generated metadata rather than the executable registry — an unregenerated tool change makes the block's outputs disagree with what the panel renders. See `.agents/skills/tool-registry-boundary/SKILL.md`.
+
 ## Checklist Before Finishing
 
 - [ ] `integrationType` is set to the correct `IntegrationType` enum value
@@ -932,6 +938,7 @@ Derive templates from the service's real use cases. Each prompt should name a co
 - [ ] Tools.config.tool returns correct tool ID (snake_case)
 - [ ] Outputs match tool outputs
 - [ ] Block + meta registered in registry-maps.ts (`BLOCK_REGISTRY` / `BLOCK_META_REGISTRY`)
+- [ ] If any tool was added, changed or removed alongside the block: ran `bun run tool-metadata:generate` and committed the artifacts
 - [ ] If icon missing: asked user to provide SVG
 - [ ] If triggers exist: `triggers` config set, trigger subBlocks spread
 - [ ] Optional/rarely-used fields set to `mode: 'advanced'`

.claude/commands/add-integration.md

@@ -414,6 +414,16 @@ export const tools: Record<string, ToolConfig> = {
 }
 ```
 
+Then regenerate the generated tool metadata and commit it:
+
+```bash
+bun run tool-metadata:generate
+```
+
+Client code reads `params`/`outputs` from these artifacts rather than importing
+the registry, so a tool you add, change or remove is invisible to the UI until they are regenerated,
+and CI fails on stale ones. See `.agents/skills/tool-registry-boundary/SKILL.md`.
+
 ### Block Registry (`apps/sim/blocks/registry-maps.ts`)
 
 The data maps (`BLOCK_REGISTRY` + `BLOCK_META_REGISTRY`) live in `registry-maps.ts`; `registry.ts` holds only the accessor functions. Add the import and an entry to each map alphabetically:
@@ -489,6 +499,7 @@ If creating V2 versions (API-aligned outputs):
 - [ ] All optional outputs have `optional: true`
 - [ ] Created `index.ts` barrel export
 - [ ] Registered all tools in `tools/registry.ts`
+- [ ] Ran `bun run tool-metadata:generate` and committed the regenerated artifacts
 
 ### Block
 - [ ] Created `blocks/blocks/{service}.ts`

.claude/commands/add-tools.md

@@ -295,6 +295,17 @@ export const tools = {
 }
 ```
 
+3. Regenerate the tool metadata artifacts:
+
+```bash
+bun run tool-metadata:generate
+```
+
+Client code reads a tool's `params`/`outputs` from generated metadata rather than
+importing the registry, so a tool you add, change or remove is invisible to the UI until
+these are regenerated — and CI fails on stale artifacts. Commit the result. See
+`.agents/skills/tool-registry-boundary/SKILL.md`.
+
 ## Wiring Tools into the Block (Required)
 
 After registering in `tools/registry.ts`, you MUST also update the block definition at `apps/sim/blocks/blocks/{service}.ts`. This is not optional — tools are only usable from the UI if they are wired into the block.
@@ -442,6 +453,7 @@ All tool IDs MUST use `snake_case`: `{service}_{action}` (e.g., `x_create_tweet`
 - [ ] Types file has all interfaces
 - [ ] Index.ts exports all tools and re-exports types (`export * from './types'`)
 - [ ] Tools registered in `tools/registry.ts`
+- [ ] `bun run tool-metadata:generate` run and the regenerated artifacts committed
 - [ ] Block wired: `tools.access`, dropdown options, subBlocks, `tools.config`, outputs, inputs
 
 ## Final Validation (Required)