fix(seo): noindex non-production sim.ai hosts, redirect indexed 404s - #5988
Conversation
Non-production deployments (dev.sim.ai, staging.sim.ai) serve the same build as www.sim.ai and were fully crawlable. Send X-Robots-Tag noindex for those hosts, and 301 seven marketing paths that an external SEO audit found returning 404.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
PR SummaryLow Risk Overview Indexed marketing 404s get permanent redirects:
Reviewed by Cursor Bugbot for commit 4a304f1. Configure here. |
Greptile SummaryAdds SEO hardening for non-production sim.ai hosts and permanent redirects for audited marketing 404s.
Confidence Score: 5/5This PR appears safe to merge; prior review concerns around forwarded-host parsing, the /security redirect, and the urls mock are addressed in the current HEAD with no remaining blocking failures identified under follow-up scope. No blocking failure remains. Host-based noindex is limited to non-canonical *.sim.ai hosts, redirects cover the intended 404 paths without colliding with the security.txt Policy URI, and the testing mock exports the new symbols used by the proxy path.
|
| Filename | Overview |
|---|---|
| apps/sim/proxy.ts | Applies X-Robots-Tag noindex/nofollow on non-canonical sim.ai hosts inside track() using the first x-forwarded-host entry or Host. |
| apps/sim/lib/core/utils/urls.ts | Adds isNonCanonicalSimHost keyed off SITE_URL/CANONICAL_SITE_HOST with www/port stripping and first comma-joined host entry. |
| apps/sim/next.config.ts | Adds permanent redirects for six audited 404 marketing paths; omits /security to preserve the security.txt Policy URI. |
| packages/testing/src/mocks/urls.mock.ts | Mirrors CANONICAL_SITE_HOST and isNonCanonicalSimHost so global urlsMock stays complete for proxy tests. |
| apps/sim/lib/core/utils/urls.test.ts | Pins canonical vs non-canonical host classification including comma-joined forwarded hosts and lookalikes. |
Sequence Diagram
sequenceDiagram
participant Client
participant Proxy as apps/sim/proxy
participant Classify as isNonCanonicalSimHost
participant Next as Next.js response
Client->>Proxy: request (Host / x-forwarded-host)
Proxy->>Proxy: build NextResponse
Proxy->>Proxy: track(request, response)
Proxy->>Classify: host (first forwarded entry)
alt "non-canonical *.sim.ai (e.g. dev.sim.ai)"
Classify-->>Proxy: true
Proxy->>Next: set X-Robots-Tag noindex, nofollow
else www.sim.ai / sim.ai / other domains
Classify-->>Proxy: false
Proxy->>Next: no indexing header
end
Proxy-->>Client: response
Reviews (3): Last reviewed commit: "fix(seo): drop the /security redirect, c..." | Re-trigger Greptile
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
A multi-value x-forwarded-host survives the port split intact, so
endsWith('.sim.ai') matched on the trailing entry and could apply
noindex to the canonical site. Normalize with the same
split(',')[0].trim() the rest of the repo uses for forwarded headers.
|
@cursor review |
security.txt advertises /security as its RFC 9116 Policy URI, so a permanent redirect to marketing would mislead that link and shadow a real policy page added later. urlsMock is installed globally and documents itself as carrying every real export, but was missing CANONICAL_SITE_HOST and isNonCanonicalSimHost — any test loading proxy.ts would have called undefined in track().
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 4a304f1. Configure here.
1 participant
Summary
dev.sim.ai/staging.sim.aiserve the same build aswww.sim.aiand were fully crawlable (Allow: /) — they now returnX-Robots-Tag: noindex, nofollow/read,/research,/scrape→/integrations;/actions,/crawl,/fast,/security→/Notes
Emir's ask was a robots.txt
Disallowon dev.noindexis the correct mechanism instead: a disallowed URL can still be indexed when linked externally, and blocking the crawl stops Google from ever seeing the directive that removes pages already in the index.robots.txtis therefore left untouched and stays crawlable — it's already excluded from the proxy matcher.Host classification is a pure function keyed off
SITE_URL, sosim.aiandwww.sim.aiare both canonical, self-hosted domains and lookalikes (notsim.ai,sim.ai.evil.com) are unaffected, and the header is applied intrack()— the single chokepoint every proxy return path flows through.Not in scope, both outside this repo:
vou.sim.ai,dogutopia.sim.ai,oceanriser.sim.aietc. return Vercel's platform 404 — stale hostnames on the project with no deployment routed. Needs a Vercel/DNS cleanup.dev.sim.aiapex 301s tohttps://www.dev.sim.ai:443/— ALB leaking the port intoLocation.The ~112
/models/**/opengraph-imagerows in the tracker are already fixed (#5636); prod emits the hash-suffixed URL, which returns 200. The bare path 404s by design and should not be redirected.Type of Change
Testing
urls.test.ts,seo.test.ts,mailer.test.ts), including 12 new cases pinning host classificationredirects()tscandbiomecleanChecklist