📝 SPIKE Candidate Crossref SDK API by stevejpurves · Pull Request #27 · continuous-foundation/crossref-utils

stevejpurves · 2026-09-14T12:11:50Z

This PR includes a possible plan and example 'end state' documentation for an SDK layer on the crossref-utils. Motivation is to be able to re-use the core deposit building functionality from pure TS code in places where there is no filesystem, and everything is fetched / processed in memory.

fwkoch

Generally, I think this plan makes sense (though my gut feeling is it's easier than the 9-step(!) process implies). My comments provide more detailed thoughts, but fundamentally, I think it's just:

Tear the single package into two packages (-cli with filesystem and interactivity, "SDK" with everything else). Keep existing DTOs and function interfaces mostly unchanged.
Massage the details to make this work (e.g. decouple abstract extraction via MyST from light-weight abstract transformations; use logger interface rather than passing around MyST session; etc).
Decide if we need TS-native XML validation, and if so, replace xmllint with that.

Bonus: Remove Curvenote-specific stuff (though that is not actually an SDK feature, just something we should do...)

fwkoch · 2026-09-15T01:24:09Z

+
+## Goals
+
+- Expose an in-memory SDK at `crossref-utils/sdk` for serverless / headless callers.


Rather than an SDK subfolder, we could structure this like JATS and our other monorepos, where we have packages/crossref-cli and packages/crossref-utils (published individually, -cli depends on -utils, -utils is very lightweight).

I suspect an initial pass, pulling the existing codebase into these two packages, would get us 90% of the way there and make the remaining 10% easy.

fwkoch · 2026-09-15T01:32:56Z

+## Goals
+
+- Expose an in-memory SDK at `crossref-utils/sdk` for serverless / headless callers.
+- Primary input: Crossref-oriented DTO → deposit XML via a thin facade over existing builders.


We currently have 2 existing sets of DTOs: MyST frontmatter types (imported from myst-frontmatter) and crossref-specific types (already defined in this repo). We shouldn't need new DTOs for this scope of work.

I think the "sdk" should include both options as input - these already exist in our *FromMyst and *Xml functions. There's a tiny bit of refactoring the *FromMyst functions so they are not dependent on the Session (which I think is just used for logging - I suspect we could just swap in a logger interface).

fwkoch · 2026-09-15T04:07:20Z

+- Expose an in-memory SDK at `crossref-utils/sdk` for serverless / headless callers.
+- Primary input: Crossref-oriented DTO → deposit XML via a thin facade over existing builders.
+- Separately validate XML in-process (no `xmllint`, no filesystem).
+- Helpers: `abstractFromMdast`, `generateDoi`, `suggestDois`, optional `mystToDepositItem`.


We need to think a little about generateDoi - right now, we do 4 random letters and 4 random numbers. The interactive component ensures we don't accidentally make any DOIs with slurs or something (usually I try to avoid anything that even looks like it could be pronounced).

I think it would be fine to just ensure generateDoi is always used in a context where users can validate it before submitting to crossref, even if that workflow is moved out of the CLI.

Alternatively, we could see if there is a better way to generate our random string to avoid potential issues in the first place...

fwkoch · 2026-09-15T04:09:44Z

+
+- Implement `resolveDoiData(doi_data, resourceResolver, ctx)` used by `buildDeposit`.
+- Remove Curvenote URL defaults from the **SDK path** (legacy `*FromMyst` / CLI may keep current behavior until a later migration).
+- Prefer passing explicit `doi_data` into existing `*Xml` builders rather than going through `*FromMyst` where those hardcode Curvenote resources.


I think both should be available in the SDK - I mentioned this in my other comment - but yes, removing Curvenote hardcoding is good. 👍

fwkoch · 2026-09-15T04:25:57Z

+- Move CLI abstract pipeline pieces used after mdast exists into a shared module (e.g. `src/abstract.ts`):
+  - transforms from `src/cli/utils.ts` that are FS-free
+  - `JatsSerializer` + wrap / unwrap xref
+- Export `abstractFromMdast` from SDK.


We need to decide what format the SDK should take the abstract.

Currently, the CLI based workflow has a myst session loaded. If the abstract is already part of the processed MyST mdast, we pull the abstract "part" from the session. If the abstract is defined as text in the frontmatter, we use parseMyst to turn it into mdast. Then we run some lightweight transforms on it and convert it to JATS.

For the SDK, we have a few options:

We could include parseMyst and convert text -> JATS in the exact same way we currently do for frontmatter abstracts. The major downside is we have to ship MyST with the SDK - I don't think we want to do that.

We could just have the SDK take the abstract as mdast. The processing to get it there is outside the scope of the SDK, but the lightweight transforms and JATS conversion are in scope.

We could just take text and write that directly to the deposit as a simple jats paragraph. No mdast, no transforms, but also no ability to have citations, links, text formatting in the abstract.

I think (2) makes the most sense. (This also makes the CLI refactor pretty simple, we just split depositArticleFromSource in half...)

I think that the SDK will be appropriate for pulling built myst content from somewhere and not touching markdown, so that sounds like 2. 3 sounds too simple and lossy compared to what we do now?

fwkoch · 2026-09-15T04:26:55Z

+### 6. DOI helpers
+
+- Keep `generateDoi` in `src/utils.ts`.
+- Move `DOI_PREFIXES` (today’s CLI `PREFIX` map) to a shared module (e.g. `src/doi.ts`).


All the prefixes are Curvenote-specific. They should probably not be in this library at all... Or at least only exist as config, not code.

agreed this should be always passed in ?

fwkoch · 2026-09-15T04:30:21Z

+
+### 7. `validateDeposit`
+
+- Add in-memory validation module under `src/sdk/validate.ts` (or `src/validateMemory.ts`).


It would be great to be able to validate our deposits against the XSD - about half of our Crossref error responses would have been caught before submitting if we had done this.

I remember previously trying to do this in TS and having some difficulty (hence falling back to xmllint), but maybe there is something out there.

yes I think we should push for that, there are options but some seem to be rooted in python or C/C++ or Java, if we can find a node / pure JS fine, otherwise we may need to just assume and use an external service

alternatively we can convert the XSD schema to json, convert the xml ot json and validate there but that feels a bit removed and not ideal.
https://www.npmjs.com/package/xml-xsd-engine looks like a good candidate but the github org and repo has disappeared so not a good sign

agutierrezgit · 2026-09-15T16:01:59Z

From the platform side (curvenote CN-2509): we will consume SDK.md as written, preprint first. Two things we depend on: batch.id and batch.timestamp must be honoured as inputs when provided, since we store the batch id before posting, and buildDeposit must not default the resource URL, which the doc already states. Until the SDK entry lands we build against the root exports (preprintXml, contributorXml, DoiBatch).

stevejpurves · 2026-09-15T16:16:47Z

Plan updated (pushed)

Revised plan-sdk.md + SDK.md to align with @fwkoch's review:

Monorepo split: packages/crossref-utils + packages/crossref-cli
Keep existing Crossref *Xml + MyST *FromMyst — no new buildDeposit / DTO facade
Core vs adapters (MyST today; other X→Crossref later)
Abstracts: processed mdast in → abstractFromMdast in utils
In-process validateDeposit must land on utils
Drop Curvenote-hardcoded resource URLs / DOI prefix aliases from the library

Commit: 06fd88b

📝 Document headless Crossref SDK API and implementation plan

46298a8

stevejpurves requested a review from fwkoch September 14, 2026 12:11

fwkoch reviewed Sep 15, 2026

View reviewed changes

📝 Revise SDK plan for monorepo split per PR review

06fd88b


		## Goals

		- Expose an in-memory SDK at `crossref-utils/sdk` for serverless / headless callers.


		### 7. `validateDeposit`

		- Add in-memory validation module under `src/sdk/validate.ts` (or `src/validateMemory.ts`).

Uh oh!

Conversation

stevejpurves commented Sep 14, 2026

Uh oh!

fwkoch left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agutierrezgit commented Sep 15, 2026

Uh oh!

stevejpurves commented Sep 15, 2026

Plan updated (pushed)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fwkoch left a comment •

edited

Loading