CAGRA-ACE: Support Direct HNSW Index Creation During Build

[julianmi]

A hnsw::build using CAGRA-ACE on-disk builds a CAGRA index on disk and uses hnsw::from_cagra to build an HNSW index from the ACE intermediate files. This patch adds support to directly create an HNSW index during ACE builds. It streams partitioned graph data into HNSW format with transactional publication, KvikIO-based I/O, and GPU hierarchy support. This speeds up the end-to-end runtime by removing a disk read of the full CAGRA graph. An exemplary BIGANN-100M build on an L40S GPU shows 8.14% end-to-end speedup with direct POSIX I/O on a consumer NVMe.

The direct CAGRA-ACE disk path does not produce a byte-identical HNSW index. Mainline assigns hierarchy levels with hnswlib’s stateful RNG after ACE reordering; the fused path uses deterministic per-ID levels and partitioned hierarchy construction. This changes maximum levels, internal IDs, and neighbor lists while preserving a valid hnswlib layout. The maximum absolute recall difference was 0.0087 for three exemplary datasets:

ef	Wiki main	Wiki fused	OpenAI main	OpenAI fused	LAION main	LAION fused
24	0.81716	0.82577	0.86655	0.86689	0.91345	0.92028
48	0.89432	0.90063	0.92568	0.92503	0.94778	0.95093
96	0.93976	0.94562	0.96076	0.95935	0.96731	0.96825
192	0.96633	0.97075	0.97996	0.97929	0.97622	0.97677
384	0.98310	0.98494	0.99035	0.98989	0.98121	0.98088
768	0.99190	0.99257	0.99550	0.99518	0.98314	0.98317