Migrate registry-docgen to registry-dev

[thomashoneyman]

Closes #704. This moves @natefaubion's purescript-registry-docgen into this repository as the top-level docgen package and makes it the foundation for registry-owned package documentation, as discussed on Discord.

The goal is to build doc artifacts that we can store in a bucket, and those become the source of truth for Pursuit. HTML, search indexes, and other data can be rebuilt from the docs without needing to reach back out to GitHub or any dependency on purs publish directly.

What changed

The new docgen package includes lots of good things built by Nathan:

• A canonical docs schema, which is versioned and has the associated codecs
• Functions to convert docs from compiler/historical Pursuit documentation
• Deterministic package generation using the registry manifest and dependency resolutions
• Re-export resolution including selective imports, hidden imports, aliases, etc.
• Package and module HTML rendering, README rendering
• Exact packag eversion links for dependencies and re-exports
• Test covering codecs, generation, etc.

We use the registry manifest for almost everything, but since we have all the Pursuit backups we can use that for the documentation content itself. 107 package versions out of the ~6000 total have different dependency ranges in the registry and the docs used technically invalid versions, so we'd want to regenerate those.

Documentation storage

The main registry change in here, beyond the docgen target, is adding a separate DocsStorage effect with filesystem and S3 interpreters. Documentation is derived and replaceable, unlike immutable package tarballs, so it supports checking for existing artifacts, downloading, create-only uploading, explicitly replacing, and deleting docs.

The existing package storage effect has been renamed from Storage to PackageStorage to make the distinction clearer.

The production Nix configuration now supplies DOCS_BUCKET, defaulting to the purescript-registry-docs space that I created last week. DigitalOcean spaces cannot make the existence check and upload atomic, so production generation will still need to ensure there is only one writer for a given package version. That constraint is tracked in #525.

Historical documentation

This adds scripts for rendering documentation locally and validating historical Pursuit artifacts against registry manifests.

The migration audit found:

• 6,543 historical Pursuit artifacts;
• 4,240 artifacts corresponding to registry versions considered for migration;
• 4,133 which decode and agree with their authoritative registry manifests;
• 107 which should be regenerated from registry tarballs rather than imported;

All 107 regeneration candidates have stored tarballs and at least one recorded compatible compiler. This PR adds the conversion and validation machinery but does not perform the production migration.

Scope

This PR is for defining, validating, rendering, and storing documentation artifacts, but it doesn't do any of the following:

• generate or upload them during package publication;
• add retry or regeneration jobs;
• perform the production migration;
• replace the existing Pursuit application;
• add search or compiler-versioned Prim documentation;
• change package-source handling for monorepos.

Those are covered by #525, #789, #790, and #750.

The imported and adapted code is included under the repository's BSD-3-Clause license with permission from Nathan.

[thomashoneyman]

The Docgen tree is brought in basically as-is, which includes modules like this one — not sure if we need this separate JSON decoding module, but I didn't want to futz around with the implementation.

[natefaubion]

This was because:

• Legacy decoder doesn't need to round-trip. That would bloat the decoders significantly since there's a lot of sum types.
• We aren't depending on argonaut, AFAIK

There's not a great ecosystem for purescript-json, other than codec.

[greptile-apps]

Greptile Summary

This PR migrates purescript-registry-docgen into the monorepo as the top-level docgen package, introduces a separate DocsStorage effect for canonical documentation artifacts, and renames the existing Storage effect to PackageStorage to distinguish the two concerns.

• docgen package: canonical docs schema (schema version 1), codecs, legacy Pursuit artifact conversion, deterministic generatePackage from registry manifests, re-export resolution (selective/hidden/aliased/transitive), and package/module HTML rendering — all ported from Nathan's original with PureScript registry integration.
• DocsStorage effect: FS backend uses link/rename for atomic create-only/replace semantics; S3 backend explicitly documents the preflight-check TOCTOU limitation (tracked in Proposal: Reimplement 'purs publish' in the registry #525); handleReadOnly proxies reads to S3 while skipping writes. Full unit and E2E test coverage included.
• PackageStorage rename: purely mechanical rename of Storage → PackageStorage with consistent updates across app/, app-e2e/, tests, and the server environment wiring.

Confidence Score: 5/5

Safe to merge. The new DocsStorage effect is well-isolated, tested at both the unit and E2E levels, and the Storage→PackageStorage rename is mechanical and consistent throughout.

The DocsStorage FS backend achieves atomic create-only semantics with link and atomic replacement with rename, both verified by concurrent-upload unit tests. The S3 TOCTOU limitation is explicitly documented in code comments, AGENTS.md, the PR description, and tracked in a separate issue. The PackageStorage rename touches many files but is a pure identifier change with no behavioural difference. No incorrect logic, missing effects, or broken contracts were found.

Files Needing Attention: No files require special attention. The WireMock state-machine mappings in nix/test/config.nix are the most complex addition but are well-structured and consistent with the E2E test fixture values.

Important Files Changed

Filename	Overview
app/src/App/Effect/DocsStorage.purs	New DocsStorage effect with FS and S3 backends; uses hard-link (atomic create) and rename (atomic replace) for FS, explicit TOCTOU acknowledgement for S3. Node error-code decoding correctly updated from unsafeCoerce to Foreign-based decoding.
app/src/App/Effect/PackageStorage.purs	Renamed from Storage.purs to PackageStorage.purs; all identifiers updated consistently. No logic changes.
app/src/App/Server/Env.purs	Wires DocsStorage interpreter into ServerEffects and runEffects; adds docsBucket with a safe default; read-only mode correctly delegates to handleReadOnly.
docgen/src/Registry/Docgen/Generate.purs	Deterministic DocPackage generation from manifest + compiler output; validates module name consistency, uniqueness, package-relative source paths, and resolved dependencies before building the artifact.
docgen/src/Registry/Docgen/Reexports.purs	Complex re-export resolution using RefSet algebra for selective, hidden, aliased, and transitive re-exports; ReexportError reported, modules never silently dropped.
app/test/App/Effect/DocsStorage.purs	Unit tests cover the full FS lifecycle: upload, exists, download, identity validation, immutable duplicate rejection, replace, delete idempotency, and concurrent immutable upload atomicity via parSequence.
app-e2e/src/Test/E2E/DocsStorage.purs	E2E tests exercise the full S3 lifecycle against WireMock; verifies correct PUT/GET/DELETE request counts and that upload-immutability and replace semantics are observed end-to-end.
foreign/src/Foreign/S3.purs	Adds getObject binding using AWS SDK v3 GetObjectCommand; FFI converts Body readable stream to Node.js Buffer via transformToByteArray. Pattern consistent with existing putObject binding.
scripts/src/RenderDocs.purs	Script to render a canonical docs JSON artifact to Pursuit-style HTML; correctly reads REGISTRY_DOCGEN_ASSETS env var with a relative default, and is wrapped by Nix to set the absolute asset path.
nix/test/config.nix	Adds comprehensive WireMock state-machine mappings for docs storage lifecycle (list, get, put, delete), correctly modelling S3 object presence state transitions across all operations.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[PackageInput\nmanifest + compiler docs + source CST] --> B[generatePackage]
    B --> C{Validate modules\nnames · uniqueness\nsource paths · packages}
    C -->|error| D[GenerationError]
    C -->|ok| E[Convert from Legacy]
    E --> F[Reexports.modulesWithReexports\nselective · hidden · aliased · transitive]
    F -->|ReexportError| D
    F -->|ok| G[DocPackage\nschemaVersion=1]
    G --> H[Codec.docPackage\nJSON serialisation]
    H --> I{DocsStorage}
    I -->|upload create-only| J[FS: link tempfile to path\natomic EEXIST detection]
    I -->|upload create-only| K[S3: preflight exists check\nthen PutObject - TOCTOU noted]
    I -->|replace| L[FS: rename tempfile to path\natomic overwrite]
    I -->|replace| M[S3: PutObject unconditional]
    I -->|download| N[FS / S3: read + validateIdentity]
    I -->|exists| O[FS: FS.Sync.exists\nS3: listObjects prefix check]
    I -->|delete| P[FS: conditional remove\nS3: deleteObject idempotent]
    subgraph PackageStorage
      Q[Upload / Download / Delete / Query\npackage tarballs in purescript-registry bucket]
    end
    subgraph DocsStorageBucket
      J
      K
      L
      M
      N
      O
      P
    end

Loading

_{Reviews (4): Last reviewed commit: "Vendor documentation styles and harden t..." | Re-trigger Greptile}

[natefaubion]

I don't remember what this is about exactly, but it probably needs some consideration. Right now only absolute uri's are allowed but I don't know if we want to do any other kind of linking, or even munging of links.