docs(qwp): document canonical RFC 4122 UUID byte order - #516
Open
mtopolnik wants to merge 17 commits into
Open
Conversation
The C, C++, Rust, and Python clients take and return UUID values as 16 canonical RFC 4122 big-endian bytes, and byte-swap into the QWP wire encoding themselves. The docs still described the wire encoding's two little-endian 64-bit halves as what those APIs accept, which would lead a reader to swap the bytes a second time and corrupt every value. Along the way, binary Arrow columns now need an explicit claim to be anything other than opaque bytes: a 16-byte width on its own does not make a UUID. The claim comes from the `arrow.uuid` extension name, `questdb.column_type` field metadata, or one of the two new per-column overrides, which also work on variable-length binary columns and so give polars frames a route to UUID and LONG256. - rust.md: chunk `column_uuid` input, `bind_uuid` / `bind_long256` inputs, `Uuid` / `Long256` result access, and a note that the row buffer's `column_uuid(lo, hi)` is the one place that still takes wire halves. - c-and-cpp.md: the same setter, bind, and reader byte orders, plus a new section on how a binary column claims UUID or LONG256, and the note that the Arrow import entry points accept no overrides. - python.md: the two `schema_overrides` value lists, and a new paragraph covering binary columns, the claim rules, and byte order. - qwp-ingress-websocket.md: spell out that the wire order is the canonical order reversed, so the wire spec and the client APIs can no longer be read as contradicting each other. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
🚀 Build success! Latest successful preview: https://preview-516--questdb-documentation.netlify.app/docs/ Commit SHA: 30a2228
|
The Arrow binary columns section explained the three ways a column can claim UUID or LONG256 — the arrow.uuid extension label, questdb.column_type field metadata, and a per-flush override — but every runnable snippet on the page opted out of all of them, so a reader had no worked example of any route. The C and Rust pages passed empty override lists, and the Python page described two of the routes in prose only. C and C++: add an Arrow C++ snippet attaching the extension label and the questdb.column_type metadata to fields, and C/C++ tabs filling in a qwp_arrow_override array and passing it to the flush call where the earlier example passes NULL, 0. Cover the arg field, which carries geohash precision and nothing else, and the invalid_api_call failure modes. Rust: populate the Polars override list with Uuid and Long256 entries and note that &[] is the no-override form. Add a binary columns subsection mirroring the C and C++ one, calling out that Polars needs the override because it has no fixed-size binary dtype. Python: add snippets for the extension-type route and for the plain uuid.UUID object column that needs no claim at all. Retarget the schema_overrides example onto the trades model the rest of the page and the other client pages use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
py-questdb-client#140 adds UUID, IPV4, BINARY, CHAR, DATE, LONG256, and GEOHASH values to row(). The page still told the reader that these types have no row() value type and to reach for dataframe() or a SQL INSERT instead, which sends people the long way around for a value they can now pass directly. List the seven in the value-type table, and replace the redirection paragraph with what a caller needs: the QuestDB 10 and QWP-transport requirements, the four wrapper types for the values with no natural Python equivalent, a worked row() call, and the reserved NULL sentinels the client writes rather than rejects. The snippet is checked against the client on the paired branch: every value is accepted and reaches a QWP mock server as one binary frame. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two snippets did not compile. `Chunk::column_uuid` takes `&[[u8; 16]]`, one array per row, so `uuid.as_bytes()` is the element type rather than the slice; the text now wraps a single value with `slice::from_ref`. `bind_uuid` takes its 16 bytes by value, so it needs `*u.as_bytes()` or `u.into_bytes()`. Polars exports its `Binary` columns as Arrow `BinaryView`, not `Binary`, which is what the client asserts when it checks override applicability. Adds the new `Object` dtype rejection and a note that a bare `FixedSizeBinary(16)` or `(32)` no longer becomes UUID or LONG256 on width alone. That change and the byte-order change both alter an existing Arrow ingest without reporting an error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Arrow import path does take a type choice: `qwp_arrow_import_new` has a `symbol_mode` argument, so only the ipv4, char, geohash, uuid and long256 claims have to travel as field metadata. Adds the same behaviour-change warning as the Rust page: an unclaimed 16- or 32-byte column is now BINARY rather than UUID or LONG256, and code that passed wire-order UUID bytes now stores them reversed. Neither reports an error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The UUID section said clients reverse each value on the way to the wire, without qualification, while the same change documents the row-buffer call as the one that does not. Names it, and states the wire-to-canonical relation as a plain instruction. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A geohash precision mismatch does not fail the flush: `row()` raises straight away and rewinds the bad row, leaving the buffer usable. Fills three gaps. An unlabelled 16- or 32-byte column is refused, not written as BINARY, when the frame is not fully Arrow-backed. The `arrow.uuid` route needs pyarrow 18 and the extension type rather than the metadata key. `dataframe()` writes DATE from the column's Arrow type, and there is no `date` value for `schema_overrides`. Adds a section on writing a query result back. Five types cannot be told apart from their pandas dtype, so `to_pandas()` records what each column was in `df.attrs['questdb']` and `dataframe()` reads it back, including the `version` key a hand-written mapping needs. Turns the wrapper list and the NULL sentinel list into tables, and adds the five types the reader can now return to the result-type table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This reverts commit 29804fc.
Contributor
Author
Revised review: approve-with-nits, excluding the version blockerThe C/C++ and Rust version issue remains a required pre-merge fix, now recorded in the PR body. Setting that aside, the documentation covers the central workflows well. My earlier classification of the three Python caveats as Major was disproportionate to their practical impact; none should block merge. Non-blocking suggestions
What worksClear canonical UUID versus wire-order guidance, useful Arrow override examples, good coverage of immutable pandas type metadata, and an important correction of replay/deduplication guarantees. Navigation remains intact and changelog entries are included. Validation: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Updates the QWP documentation to match recent client behavior across C, C++, Rust, and Python:
row()and for preserving QuestDB types when writing query results back.Compatibility notes
Required before merge: C/C++ and Rust version boundary
The version issue must be resolved before merge. The published
c-questdb-client/ Rust7.0.0tag still uses wire-order UUID bytes and width-based binary type inference, and lacks the new UUID/LONG256 overrides. The documentation currently attributes the new behavior to 7.0.0. Update the migration warnings and dependency examples to the release containing c-questdb-client #186, or pin a containing commit until that release is available. Following the new byte-order guidance with 7.0.0 can silently reverse UUIDs.Related client changes
Validation
npx docusaurus build