Add AI agent detection to CLI analytics

[amcaplan]

WHY are these changes introduced?

We can't currently distinguish CLI runs made by AI coding agents from runs made by humans or scripts. The Shopify AI toolkit has agents declare themselves through SHOPIFY_CLI_AGENT_INFO and SHOPIFY_CLI_AGENT_IDS, but agents outside that toolkit stay invisible in analytics.

https://github.com/shop/issues-develop/issues/23864

WHAT is this pull request doing?

Detect the agent with @vercel/detect-agent and report it through the existing sensitive env_shopify_variables field, so there is no Monorail schema change.

Detected names use the packed format producers already send — SHOPIFY_CLI_AGENT_INFO="n:<name>" — so the agent column resolves the same way whether a name was declared or detected. Only the n: tag is set: detection cannot resolve version, provider or model, so those tags are omitted rather than padded, leaving those columns null instead of a literal none.

Detection is a fallback. It is skipped when SHOPIFY_CLI_AGENT_INFO or SHOPIFY_CLI_AGENT_IDS is declared, because writing SHOPIFY_CLI_AGENT_INFO ourselves would clobber the producer's whole packed value. SHOPIFY_CLI_AGENT_DETECTED=true records that a name was derived rather than declared; nothing consumes it downstream yet.

Detected names have | stripped. It is the tag separator with no escape sequence, and the AI_AGENT convention passes arbitrary values through verbatim, so a name containing one could otherwise fabricate tags such as v:.

Detection runs only when the event will actually be sent, so an opted-out user pays nothing for it.

How to manually test your changes?

bin/dev.js runs the bundle rather than the source, so bundle first:

pnpm nx bundle --skip-nx-cache

A detected agent, with nothing declared:

env -u SHOPIFY_CLI_AGENT_INFO -u SHOPIFY_CLI_AGENT_IDS AI_AGENT=codex SHOPIFY_CLI_ALWAYS_LOG_ANALYTICS=1 node packages/cli/bin/dev.js version --verbose | grep env_shopify_variables

Reports {"SHOPIFY_CLI_AGENT_INFO":"n:codex","SHOPIFY_CLI_AGENT_DETECTED":"true"}.

A declared agent, which detection must leave alone:

env -u SHOPIFY_CLI_AGENT_IDS AI_AGENT=codex SHOPIFY_CLI_AGENT_INFO='n:shopify-ai-toolkit|v:1.0.0|p:anthropic|m:claude-opus-5' SHOPIFY_CLI_ALWAYS_LOG_ANALYTICS=1 node packages/cli/bin/dev.js version --verbose | grep env_shopify_variables

Reports the declared value unchanged, with no SHOPIFY_CLI_AGENT_DETECTED.

Checklist

• I've considered possible cross-platform impacts (Mac, Linux, Windows)
• I've considered possible documentation changes
• I've considered analytics changes to measure impact
• The change is user-facing — I've identified the correct bump type (patch for bug fixes · minor for new features · major for breaking changes) and added a changeset with pnpm changeset add

🤖 Generated with Claude Code

[github-actions]

Differences in type declarations

We detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:

• Some seemingly private modules might be re-exported through public modules.
• If the branch is behind main you might see odd diffs, rebase main into this branch.

New type declarations

packages/cli-kit/dist/private/node/context/agent.d.ts

export declare function detectedAgentEnvironmentVariables(env?: NodeJS.ProcessEnv): Promise<NodeJS.ProcessEnv>;

Existing type declarations

packages/cli-kit/dist/private/node/analytics.d.ts

@@ -31,4 +31,6 @@ export declare function getSensitiveEnvironmentData(config: Interfaces.Config):
     env_plugin_installed_all: string;
     env_shopify_variables: string;
 }>;
+export declare function monorailAnalyticsSkipped(): boolean;
+export declare function metricAnalyticsSkipped(): boolean;
 export {};
\ No newline at end of file