Video search on top of the Review grid (rebase of #1743) - #1898
Merged
Conversation
New top-level Review tab (desktop: after Training; web: after Models) showing many annotations at once as cropped image chips, so a whole class (or everything carrying an attribute) can be checked across several datasets and wrong types corrected in place, without opening each sequence in the viewer. - Datasets view / Grid view toggle: pick datasets through the platform picker or listing (multicam parents expand into their cameras), then page through matching annotations in a rows x columns grid (5x4 by default, settable, with zoom in/out keeping the shape) and a context slider for the margin around each box (30% by default). - Query by type above a confidence threshold, or by attribute key/value on tracks and/or detections; sort by dataset, confidence or frame. The grid keeps its entries until the query is re-run so edits never reshuffle it. - Chips are cropped client-side from image sequences or by seeking a hidden video element (shared frame->time mapping extracted from VideoAnnotator into videoSeek.ts), through a concurrency-limited queue that renders the first box of every visible entry before any track's extra frames. Track entries then cycle through up to 8 boxes sampled along the track, with the object kept centred. - Types are edited per cell (assign, or mark correct) or for a whole page, batched and saved through saveDetections; unsaved edits are counted and guarded on navigation. - Clicking a chip opens the viewer on that dataset, seeking to the frame and selecting the track (new initialFrame / initialTrackId Viewer props read from the route query on both platforms). - Api gains an optional peekConfig for reading dataset configs without the viewer bookkeeping (desktop recents, web browse location); the web dataset store now shares its config merge with it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu
A square chip letterboxed into a wide cell wasted most of the cell, so the crop region now extends the padded square around the box to the cell's width/height ratio (coarsened to a tenth so window resizes rarely force a re-render), keeping the object centred whatever the grid shape. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu
Move the review page's paging, zoom, chip-resolution and keyboard handling into useReviewGrid, its grid-shape/context/pager row into ReviewGridControls, and the persisted grid settings into gridSettings.ts, so another view showing ReviewItems (e.g. video search results) gets the same behaviour without copying the page. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu
Per-dataset search index lifecycle (build via a process_video.py job, status from an index_meta.json sidecar + ITQ file check, delete) and a persistent QueryServiceManager wrapping viame.core.query_service over NDJSON stdio (open index, formulate from image chip + boxes, query with optional warm-start model, refine with +/- feedback, export the SVM as a runnable trained pipeline in DIVE_Pipelines). One index open at a time; the embedded postgres is stopped gracefully on app cleanup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New in-viewer context panel (desktop only) for index-backed video search and rapid model generation: build/rebuild/delete the dataset's search index (via the GPU job queue), query from the selected annotation, an external image file, or a saved .svm model warm-start, adjudicate ranked results +/- with cropped chip thumbnails and click-to-seek, refine iteratively, and save good models as runnable trained pipelines. Shared request/response types live in apispec so web can implement later. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace per-dataset index directories with a single shared search database (DIVE_SearchIndex): every table keys rows on a per-video stream identifier, so datasets are added, updated, and removed independently while one IQR session searches everything at once. - Add/Update ingests a dataset into the shared database as a job (initializing it on first use), re-using stream identifiers derived from the dataset id (image sequences) or video filename stem. - Remove deletes the dataset's rows through the query service and drops it from the membership metadata; deleting a dataset from DIVE also removes it from the index automatically. - Query results attribute back to their source dataset via stream_id; the panel labels cross-dataset results, seeks within the current dataset, crops thumbnails from each result's own media, and can filter the display to the current dataset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New fullscreen 5x4 grid page, opened from the Video Search panel, showing the ranked results across every indexed video as cropped chips with accept/reject buttons; marks share the panel's session state and feed the same IQR refine loop. Chip rendering moves to a shared, concurrency-limited loader (useResultChips) with context padding and the result box outlined, and the cell (AdjudicationChip) is presentation-only so future annotation cluster rows can reuse it. Also: extracted video frames are now cached per source video (hash-keyed) instead of colliding across videos on frame number, and cross-dataset chip media resolves the file the media server actually serves, so transcoded copies work when originals have moved.
Track results load their first-state chip immediately, then the grid queues up to 8 frames evenly sampled along the track (visible page only, at lower priority than primary chips) and each cell cycles through whichever frames have loaded, with a filmstrip badge marking animated cells. Static detection results are unchanged.
Follow renames that landed on main since the branch was cut: the desktop project store type is JsonConfig and is read with loadJsonConfig from the project's datasetFileAbsPath, the frontend metadata loader is loadConfig, the typed IPC invoke is used in the GPU job queue, and a spawn without a pid now fails the index build instead of storing an undefined pid. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu
The full-window results grid is now the review chip grid: results map to ReviewItems (searchResultItems.ts), chips are cropped client-side through the review frame sources (a lazy per-dataset registry, so cross-dataset results need no backend frame extraction), and the grid shape, zoom, context and paging come from useReviewGrid / ReviewGridControls with the same persisted settings as the Review page. Cells keep the accept/reject actions through ReviewCell's slots and a highlight border; results without a box show the whole frame. useResultChips.ts and AdjudicationChip.vue are replaced by useSearchChips.ts, which also feeds the side panel's row thumbnails. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu
The shared search index no longer needs an embedded PostgreSQL server: run_bulk.py --index-backend files writes one set of files per stream (manifest, descriptor CSV, float32 descriptor array, uids, ITQ hash codes, tracks) that the query service reads directly. index_meta.json records the backend the index was built with, every later build joins it, and removing a dataset deletes its stream files. The PostgreSQL backend is still supported for indexes recorded (or predating) that backend, through the same code paths as before. Also follow the VIAME tool renames (run_bulk.py, database.py, index_generic.pipe) and drop the initdb requirement from the install check unless the postgres backend is in use. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu
Member
Author
|
Update: the index storage moved from embedded PostgreSQL to per-stream file bundles, following the storage comparison.
Verified against a VIAME main build: index build, open, query, refine and model export on both backends through |
The desktop backend now invokes the viame index applet (index.py add / remove) instead of assembling run_bulk.py arguments and issuing psql scripts itself: the applet initialises and starts PostgreSQL when that backend is in use, runs the ingest pipeline, and refreshes the hashes. The install check looks for index.py. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu
Member
Author
|
Follow-up: DIVE now drives the new |
Member
Author
|
This branch has been merged into the VIAME fork's |
…ng, overlays and sharper chips The Results panel is now the first and default panel, with the Datasets panel second; with no dataset added it says so and links across. Cell type fields and captions scale up as the grid shows fewer entries, and the threshold and context sliders and fields are larger. Right clicking a cell (or its edit action) opens the frame it is showing for editing in place: the box gains drag handles, and polygon vertices and head/tail points can be dragged too; Enter/Apply keeps the edit, Esc/Cancel drops it, and the chip is re-cropped around the new box. Polygons and head/tail points are drawn over every chip. Opening the viewer from a cell seeks to the frame the cell is showing and selects the track. Paging quickly only loads the page landed on: queued work for pages passed over is dropped at once and new loads wait for paging to settle. Chips are rendered at the cell's resolution with high-quality resampling and stored losslessly when upscaled, so small objects are as sharp as the source allows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu
…ame stepping and auto-save Polygon vertices now move while dragged (the draft arrays are replaced instead of index-assigned, which Vue 2 cannot see). Editing handles are the circles the annotator draws, in the type's colour and red while dragged, and a right click while editing locks the change in. A single click on a chip no longer opens the viewer (double click or the action does), so a slip after an edit does not raise the unsaved-changes prompt. Track cells gain arrows to step through their sampled frames, which pauses the cycling until resumed. When auto-save is enabled in the settings, review edits are saved after the same delay the annotator uses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu
…zoom, live queries and toolbar polish Starting an edit pauses a track's cycling and it stays paused until resumed. Cycling runs at the dataset's real-time rate: consecutive frames advance every 1/fps seconds and sparser samples wait proportionally longer. The box is drawn over the chip instead of into it, so an edit no longer re-crops the chip and the view keeps its zoom. While editing, the mouse wheel zooms the chip about the cursor (up to 16x) and dragging empty space pans it. The type field's arrow closes its list on a second press, the hover actions grow under the mouse, entries sort by confidence (highest first) by default, and the tooltip over each chip is gone. Query changes apply as soon as they settle, so the Show button is gone; datasets loading or leaving refresh the grid on their own. A settings gear next to Save opens the annotator's settings dialog for auto-save. All toolbar fields read at the same size as the Results and Datasets buttons. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu
… with add-box and delete actions The cameras of a multicamera dataset now share a grid entry per track, shown as one chip per camera side by side and labelled with the camera. The chips show the same sampled frames on every side; where a camera has no detection on a frame the track has elsewhere, that side is cropped at a position interpolated from its own neighbouring boxes, shows no box, and offers an add-box action that creates the detection there ready to be adjusted. Type edits apply to the track in every camera and opening the viewer opens the rig. The chip (image, overlay, editing) is split out of the cell into ReviewChip so a cell can hold several. A red X next to the accept action deletes the track, written on the next save. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu
…ept clear of the camera label Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu
…dexes Run the query when formulation returns no results (whole-image exemplars). Hide indexed datasets from the picker, show each entry's index type, add the whole-frames index method, and rename the Indexes toggle to Index.
mattdawkins
marked this pull request as ready for review
September 13, 2026 02:17
Typing a type or editing a box adopts a result into its dataset through the review service, linking to an overlapping annotation or inserting a new track. Save/Discard on the results toolbar; Hide reviewed drops adjudicated entries.
Park the search session, review edits, chips, view and grid position when leaving the page and take them back on return, so adjudication continues after opening a result in the viewer.
Accepted results and typed ones are written (rejected only when typed); each gets its own track instead of editing existing annotations in place. Results overlapping originals prompt: keep originals, replace overlapping, replace all, or discard.
Member
Author
|
Detection and tracking indexes might be broken on the latest viame release, but detection are fine (both are fixed in the source) |
Pass page.results into createSearchChips so the toggle actually limits which hits appear for adjudication. Co-authored-by: Cursor <cursoragent@cursor.com>
…ch index. Match parent/camera dataset ids as well as exact ids so index cleanup does not leave orphaned streams. Co-authored-by: Cursor <cursoragent@cursor.com>
Frame extraction needs fps from disk; wire ffprobe through the desktop backend route the frontend already calls. Co-authored-by: Cursor <cursoragent@cursor.com>
Avoid searching with a box drawn on one frame against an image from another. Co-authored-by: Cursor <cursoragent@cursor.com>
Adopt boxless hits as frameless keyframes so marking them correct can persist on save. Co-authored-by: Cursor <cursoragent@cursor.com>
Use video-info duration and dataset fps so text search stays within the clip instead of aborting past EOF. Co-authored-by: Cursor <cursoragent@cursor.com>
Failed loads no longer stick in error until the Query session is restarted. Co-authored-by: Cursor <cursoragent@cursor.com>
Index builds close the backend session, so invalidate the in-memory query session (results, adjudications, generation) when jobs finish or membership changes. Pass the filtered results list into search review so change counts and saves only touch hits the grid is showing. Co-authored-by: Cursor <cursoragent@cursor.com>
process.kill(pid, 0) can fail with permission errors for a still-running job (other uid or a restricted environment). Only ESRCH means the process is gone, so resumable-training discovery no longer misclassifies active runs as interrupted. Co-authored-by: Cursor <cursoragent@cursor.com>
BryonLewis
approved these changes
Sep 14, 2026
BryonLewis
left a comment
BryonLewis
left a comment
Collaborator
There was a problem hiding this comment.
- Review page: chip-grid annotation audit across datasets (web + desktop), with in-place type/geometry edits, accept/delete, zoom/pan, stereo/multi-cam sync, and session resume
- Query page (Desktop): shared search index over selected datasets; image, video-frame, and text queries; IQR refine loop; results in the same grid as Review
- Video Search: desktop/IQR backend plus viewer sidebar that launches searches on Query; file-backed indexes via VIAME
- Search adjudication: edit hits as annotations, hide reviewed, save accepted results (incl. whole-frame), and resolve overlaps with existing annotations
- DatasetPicker: shared picker used on Review, Query, Training, and Pipelines (select all / remove all / stable selection list)
- Jobs & indexing UX: index prep/logs in Jobs, pending vs ready index rows, delete index entries, clearer indexing failure dialogs
- Docs: Query.md and Review updates
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1897 (base branch
dev/review-grid). Rebases the video search / IQR work from #1743 onto it and shows search results in the new Review chip grid.Rebase
The five commits of #1743 apply on top of the review branch with these adaptations to changes that landed on main in the meantime (one follow-up commit):
JsonConfig, read withloadJsonConfigfromdatasetFileAbsPathloadConfiginvoke<DesktopJob>undefinedResults in the Review grid
VideoSearchResultsGridis rebuilt on the shared review components: results map toReviewItems (dive-common/review/searchResultItems.ts), and the grid usesReviewGrid/ReviewCell/ReviewGridControlsanduseReviewGrid, so grid shape, zoom, context margin, paging and keyboard behaviour are the same as the Review tab and share its persisted settings.createFrameSourceRegistry), so cross-dataset results load their own media (image sequence or hidden<video>seek) with no backend frame extraction for display. Results without a box show the whole frame.ReviewCell's actions slot, get a green / red highlight border, show the relevancy score as the badge, and clicking a chip from the open dataset seeks the viewer to it.useSearchChips.tsreplacesuseResultChips.tsandAdjudicationChip.vue; it also feeds the side panel's row thumbnails from the same chip store.docs/Review.mdgains a short "Search results" section.Testing
npm run lint,npm run typecheckandnpm testpass.🤖 Generated with Claude Code
https://claude.ai/code/session_01W1C4QY6hxjHaUPJfWPfQuu