iRM-Vision-2026 - Camera Pipeline Bring-up (MindVision)

Scope of this file. For now this README documents exactly one path: bringing up the MindVision camera pipeline and proving that the imaging pipeline is pinned (every ISP parameter explicitly written, then read back). The complete project documentation - architecture, auto-aim theory, WSL/x86_64 replay setup, inference backends (TensorRT / ONNX Runtime / OpenCV DNN), every other test program - lives in readme_zh.md (Chinese), which is the current full Chinese README. More sections will be ported here over time.

Engineering rules for this port live in AGENTS.md: section 8.7 (detection-frequency test tool + MindVision keys) and section 8.8 (headless streaming + pinned pipeline).

MindVision USB camera (MV-SUA133GC-T1V-C, f622:0001, 1280x1024 Bayer8)
  -> io::MindVision::open()          io/mindvision/mindvision.cpp   (MindVision SDK)
  -> pinned ISP pipeline             explicit write + readback of 18 parameters
  -> io::Camera::read() -> cv::Mat   consumers: detect_freq_visual_test, camera_test, uav, ...

configs/mv_sua133gc.yaml is a standalone config for this USB2.0 test camera: camera section + detector section only. The end-to-end programs (uav, minimum_vision_system) additionally need calibration / planner / cboard keys, so use a full config (configs/demo.yaml etc.) and swap in the camera section from configs/mv_sua133gc.yaml.

• Linux x86_64 or aarch64 (verified on a Jetson Orin Nano, JetPack 5.1.3).

• MindVision SDK: nothing to install, it is vendored in this repo - headers in io/mindvision/include/ and io/mindvision/lib/arm64/libMVSDK.so (or lib/amd64/ on x86_64), selected by io/CMakeLists.txt from CMAKE_SYSTEM_PROCESSOR. Only libusb has to be installed.

• Base libraries:

  sudo apt install -y libopencv-dev libfmt-dev libeigen3-dev libspdlog-dev libyaml-cpp-dev \
      libusb-1.0-0-dev

• Camera plugged in. A USB2.0 port is enough: at 1280x1024 Bayer8 the stream is roughly 1.25 MB per frame (~45 MB/s), which already saturates a 480 Mbps link and still yields ~35.7 fps. Exactly one process may hold the camera at a time.

cmake -B build                                   # only needed once; build/ is usually configured
make -C build/ detect_freq_visual_test -j$(nproc)
make -C build/ camera_test -j$(nproc)            # optional: capture-only minimal program

# with a display (X / WSLg):
./build/detect_freq_visual_test -c=configs/mv_sua133gc.yaml -m=live -d

# on a headless machine (Jetson over SSH) - stream the image, see section 5:
./build/detect_freq_visual_test -c=configs/mv_sua133gc.yaml -m=live -stream=8080 -no-yolo

Flags used above and nearby (full table in AGENTS.md 8.7):

Warning: every program in this repo parses arguments with cv::CommandLineParser, so a short option must be written as -c=<path>. A space (-c <path>) parses as an empty value and aborts with [YAML] Failed to load file: bad file.

./build/detect_freq_visual_test -c=configs/mv_sua133gc.yaml -m=live -stream=8080
curl -s -o /tmp/snap.jpg http://127.0.0.1:8080/snapshot.jpg     # script self-check
# browser / VLC / ffplay on any machine that can reach the board:
#   http://<board-ip>:8080/

The startup log lists every local IPv4 address together with its URL. The server lives in tools/mjpeg_server.{hpp,cpp} (POSIX sockets + cv::imencode, no third-party dependency):

• JPEG encoding happens only while at least one client is connected, and each client has its own thread sharing the newest frame, so a slow client drops frames instead of stalling the pipeline.
• -stream without an explicit -d disables cv::imshow automatically. If imshow throws anyway (no DISPLAY), it is caught, logged as a warning, and the program keeps running.
• Measured on a 1280x1194 canvas: 10.2 ms/frame with no client, 16.1 ms/frame with one client (JPEG quality 80 costs about 6 ms). Both stay below the 28 ms camera frame interval, so throughput remains camera-bound: 35.7 fps captured, 34.8 fps end to end, no dropped frames.

io::MindVision::open() writes every mv_* parameter present in the yaml, in a fixed order, and then reads them all back into the startup log (search the log for the readback block; its labels are Chinese on the device). With configs/mv_sua133gc.yaml the readback must show:

Consistency check - run twice and compare the startup block. The readback block sits inside the first 40 lines, before the first per-frame line containing fps, so a fixed-width head is enough:

./build/detect_freq_visual_test -c=configs/mv_sua133gc.yaml -m=bench -n=60 -no-yolo > /tmp/rb1.log 2>&1
./build/detect_freq_visual_test -c=configs/mv_sua133gc.yaml -m=bench -n=60 -no-yolo > /tmp/rb2.log 2>&1
for f in 1 2; do sed -E 's/^\[[^]]*\] \[[^]]*\] //' /tmp/rb$f.log | head -40 > /tmp/rb$f.txt; done
diff /tmp/rb1.txt /tmp/rb2.txt && echo IDENTICAL

On the Jetson this prints IDENTICAL (empty diff). If your camera prints more startup lines, raise head -40 until it covers the whole readback block, but keep it before the first fps line. Any difference between two runs means something else is still changing the imaging.

Rules that came out of measurement - do not undo them:

• Ordering: CameraSetParameterMode makes the SDK reload its parameter sheet, which silently resets the exposure side to the defaults (auto exposure on, about 10 ms, gamma 100). Therefore setup_parameter_source() runs before the exposure settings in io::MindVision::open(), while auto-exposure / exposure / gamma run after the structural settings (output format, trigger, frame speed, gains, resolution). The readback block exposes any regression immediately.
• Unsupported on this camera model: CameraSetWbMode and CameraSetFrameRate return -4 on the MV-SUA133GC, so mv_wb_mode and mv_frame_rate must stay commented out (every start would log a warning). Both still appear in the readback (manual white balance, 0 Hz = unlimited), because their values are read, not written.
• Golden sheet (copy the imaging to another machine): set mv_parameter_save_file: "assets/mv_sua133gc_pipeline.config" to dump the effective parameters on exit (about 74 KB) and mv_parameter_file to the same path to restore them at startup. Both go through the same reload path as above, so the ordering rule applies. That .config file is not committed to the repository.
• To test whether a write actually took effect, use a temporary yaml with deliberately different values (mv_analog_gain: 96, mv_sharpness: 5, mv_light_frequency: 1, ...) and check the readback. Keeping a single key in the file is enough to tell "write rejected" apart from "overwritten by a later call".

Every mv_* key has a default, so configs that do not mention them behave exactly as before.

Changing the resolution invalidates the camera calibration, so keep 1280x1024 unless you redo it.

On the Jetson, install TensorRT before enabling detection:

sudo apt install -y tensorrt libnvinfer-dev libnvinfer-plugin-dev
cd /home/irm/iRM-Vision-2026
/usr/src/tensorrt/bin/trtexec --onnx=/home/irm/iRM-Vision-2026/assets/yolov5_0526.onnx \
  --saveEngine=/home/irm/iRM-Vision-2026/assets/yolov5_0526_fp16.engine \
  --fp16 --memPoolSize=workspace:1024 --buildOnly
cmake -S /home/irm/iRM-Vision-2026 -B /home/irm/iRM-Vision-2026/build
make -C /home/irm/iRM-Vision-2026/build detect_freq_visual_test -j2
/home/irm/iRM-Vision-2026/build/detect_freq_visual_test \
  -c=/home/irm/iRM-Vision-2026/configs/mv_sua133gc.yaml -m=live -stream=8080

Keep the repository as the working directory because model paths in YAML are relative. If an existing build cache uses the old /home/irm/sp_vision_25 symlink, use that path for cmake -S instead. Stop any capture-only session with Ctrl+C before starting detection. Do not pass -no-yolo. The startup log must show backend=tensorrt and yolo=on; NO YOLO means detection is unavailable, even if the stream still works. The overlay displays armor corners, confidence, colour and number. This program does not control the gimbal or firing hardware.

Measured on October 3, 2026 with TensorRT 8.5.2: engine generation took 638.5 seconds. The first 60 demo frames matched a Python OpenCV 5.0 ONNX reference: 55 detected frames on both sides, identical counts and labels, maximum confidence difference 0.00321 and maximum corner-coordinate difference 2.47 pixels (existing limits: 0.01 and 3 pixels). A 120-frame camera run with snapshot saving measured 23.63 ms average detection time and 31.24 fps end to end; capture reported 165 valid frames and zero drops. A snapshot request returned HTTP 200. This initial benchmark viewed a desk without a visible armor target.

Real-target confirmation (October 3, 2026): after testing armor plates with the live camera detection stream, the user reported that recognition worked very well. This confirms successful real-target use of the MindVision -> TensorRT -> annotated MJPEG stream. It is qualitative user feedback, not a measured precision/recall result or a validation across all distances and lighting conditions. The benchmark numbers above remain from the initial run. Engine files are local artifacts and must not be committed.

• documentation/intrinsic_calibration.md: camera-only browser calibration, sample management, error reports, YAML export, and the future hand-eye extension contract. No CBoard is required.
• AGENTS.md section 8.7: detect_freq_visual_test and the MindVision keys; section 8.8: headless MJPEG streaming and the pinned pipeline (engineering rules, Chinese).
• readme_zh.md: the current full Chinese README of this project (a superset of this file - everything not ported here yet, including the inference backends and the local regression runner, lives there).
• configs/mv_sua133gc.yaml: the config file itself, with the measurement notes inline.