Case study · CUDA · Computer Vision
CudaSift: GPU keypoints that keep the best, not the first
I forked CudaSift to select keypoints by score instead of truncating by detection order, with an optional per-octave cap and a pybind11 Python wrapper.
Top-k
kept by score, not the first found
Per-octave cap
across every octave, not just the first to fill
3–5 ms
of filtering overhead
SIFT is still one of the most used feature detectors in computer vision, and CudaSift is a CUDA implementation that runs it on the GPU at high speed. While integrating it I hit a detail that hurts keypoint quality: when an image produces more responses than the buffer holds, CudaSift keeps the first ones it finds and discards the rest.
I forked CudaSift to fix that: it now keeps the strongest keypoints by score, optionally splits the budget across octaves, and can be called from Python through a pybind11 wrapper. The original detector is by Mårten Björkman; my changes sit on top of that work.
The problem: the buffer truncates by order, not by quality#
CudaSift allocates a fixed buffer of maxPts keypoints. Detection appends candidates with atomicInc and, once the buffer is full, the kernel clamps the write index to the last slot: every extra candidate from then on overwrites that same final slot. So the returned set is whichever keypoints were written first (detection order), and a strong feature can be lost just because it was detected late.
For a matching or SLAM front end this matters: you want the most repeatable, highest response keypoints, not an arbitrary subset biased by the kernel scan order.
My contribution: score selection, a per-octave cap, and a wrapper#
I decoupled the working capacity from the final cap. The device now stores up to maxWorkPts = 4 × maxPts candidates without overwriting (anything beyond is dropped, not written onto the last slot). After detection, each octave is filtered on the host to the top-k candidates by abs(sharpness), the extraction time DoG response, with nth_element and sort, and only those are returned. The original legacy mode is still available behind a flag.
I added an optional per-octave cap: the maxPts budget is split evenly across octaves (the remainder goes to the finest ones), so every scale is represented instead of a single octave filling the buffer. And I wrote a pybind11 wrapper (cudasift_py) that takes a NumPy image and returns keypoints, scales, orientations, scores, and descriptors as arrays, without going through the C++ executable.
The idea is simple: let detection over-collect, then keep the best by score instead of the first found. Three modes: legacy, global score, and per-octave score.
How to use it and where I tested it#
From Python it is one call: cudasift_py.extract(img, ...) with use_score_filter and use_per_octave_cap flags. It returns keypoints (N,2), scales, orientations, scores (the extraction sharpness), and descriptors (N,128). The CUDA architecture is parametrizable from CMake, so it builds for whatever GPU you have.
I tested it inside EndoCartoScope, the colonoscopy SLAM system I research on, as the front end extractor. Keeping the strongest keypoints and spreading them across scales gives the matcher a more stable base than truncating by order.
Top-k selection by sharpness instead of detection order truncation.
Optional per-octave cap to cover every scale.
pybind11 wrapper: a NumPy image goes in, keypoints and descriptors come out.
Stack#
FAQ#
What is CudaSift?
A CUDA implementation of the SIFT detector and descriptor, originally by Mårten Björkman, that runs on the GPU at high speed. I forked that repo to improve keypoint selection and add a Python wrapper.
What exactly did you change?
Three things: keypoint selection by score (top-k by sharpness) instead of truncating by detection order, an optional per-octave cap to cover every scale, and a pybind11 wrapper to call it from Python with NumPy.
Why does keeping keypoints by score matter?
Because the original buffer, once full, overwrites the last slot with every extra candidate: you keep the first detected, not the best. For matching or SLAM you want the strongest, most repeatable ones.
Can I still use the original behavior?
Yes. The legacy mode is still available behind a flag (use_score_filter=False), and there is also a global score mode with no per-octave cap.
Want to talk computer vision?
If you work on CUDA, matching, or any hard vision problem, get in touch.