Add a foundation-stereo option for warping detections between stereo cameras - #1875
Add a foundation-stereo option for warping detections between stereo cameras#1875mattdawkins wants to merge 2 commits into
Conversation
…cameras The existing client-side transfer matches each point by NCC template matching along its epipolar curve, which needs the source patch to be photometrically matchable in the other view. Where that fails -- obstructed viewpoints, repetitive substrate, low contrast -- it mismatches or declines. Adds a second correspondence method that runs a Fast-FoundationStereo ONNX export instead: rectify the pair once per frame, run a dense disparity network, and read each point's shift out of the disparity map. Both methods satisfy a shared StereoMatcher interface, so box/line/polygon warping, measurement and bulk transfer are unchanged either way, and a Correspondence method dropdown under Track Settings -> Stereo Settings picks between them. The foundation model is not bundled -- the exports run ~100 MB -- so it is supplied by URL and the method no-ops with a warning when none is served. NCC remains the default.
|
A 100MB stereo file shouldn't be bundled in the app or in the repo. I think for the Web version it should be in a viame.kitware.com location and there should be a configuration option for the server that points at either the direct URL or girder Item. Maybe it is a place where we can configure multiple models in the future if we start using moe onnx models. For the desktop version I would probably have that onnx model embedded inside of the VIAME install and the express server in electron would serve that to the client when requested. It could fall back to using the web version if it has an internet connection Then the client when requested the first time would download and cache it for use. I think this makes it a bit easier in the future for deployment of other models as well if we are going to rely on onnx web runtimes for other functionality. |
…pute its disparity per frame Girder resolves FAST-FDN-STEREO from download_viame_addons.csv, caches it, and serves it; the client caches by md5. Fixes ImageNet/RGB preprocessing and rectify-with-resize; LRU disparity cache filled on frame change. Settings labels: "Lower Accuracy, Higher Speed" / "Higher Accuracy, Lower Speed" (web only). Claude-Session: https://claude.ai/code/session_013oFD6dP2EQY3R9UdXzx7dL
2 participants
What
Adds a second stereo correspondence method next to the NCC template matcher: the Fast-FoundationStereo ONNX export VIAME ships in its
FAST-FDN-STEREOadd-on. Track Settings → Stereo Settings → Point matching picks between Lower Accuracy, Higher Speed (NCC, default) and Higher Accuracy, Lower Speed (foundation). Web only; desktop's method comes from the VIAME interactive stereo config.How
GET dive_configuration/stereo_foundation_model[/spec]. Girder reads the add-on's URL + md5 from VIAME'sdownload_viame_addons.csv, downloads the zip once intoDIVE_MODEL_CACHE_DIR(compose volumegirder_model_cache), keeps the.onnx+ sidecar yaml, and streams it. The client caches the bytes in the browser Cache API keyed by md5, so a re-published add-on is picked up with no DIVE change.plugins/onnx/fast_foundation_stereo.py. Both matchers now take RGBA frames.useStereoOnnxWebwatches the viewer frame and prepares both warp directions whenever the foundation method and auto-compute are on. The matcher keeps an LRU of 8 disparity maps, runs one inference at a time, dedupes concurrent requests, and drops stale prefetches unless a warp is waiting on them.Status
Convasymmetric pads, then 3DConvTranspose; native EP: 48-inputConcat), and it carries 1.5 GB fp32 intermediates. A web-viable export needs to come from VIAME — details inclient/dive-common/use/stereo/README.md. Until then the method reports a clear error once and the NCC method is unaffected.🤖 Generated with Claude Code
https://claude.ai/code/session_013oFD6dP2EQY3R9UdXzx7dL