SmartCam 多传感器机器人SmartCam · RGB-D Robot Perception
Rice ELEC 594 团队机器人项目：实现 RealSense RGB-D 目标检测与相机坐标估计，并开展 Gazebo 规则式跟踪、数据采集和 SmolVLA 离线实验；机器人通信、建图与 AEB 按团队阶段成果展示。A Rice ELEC 594 team robotics project combining RealSense RGB-D detection and camera-frame estimation with Gazebo rule-based tracking, data collection and SmolVLA offline experiments. Robot communication, mapping and AEB are presented as staged team results.
- 01输入InputRealSense D415RealSense D415
- 彩色帧RGB Frame
- 对齐后的深度帧Aligned Depth Frame
- 02处理ProcessingYOLOv8n 目标检测YOLOv8n Detection
- 检测框深度选择Bounding-Box Depth Selection
- 有效深度过滤Valid-Depth Filtering
- 03输出Output目标距离 / XYZDistance / XYZ
- 目标类别与距离Object Class & Distance
- 相机坐标系 XYZCamera-Frame XYZ
项目背景与工程目标Project Overview & Engineering Objective
Rice ELEC 594 · 团队项目Rice ELEC 594 · Team project
我的工作围绕两条主线展开：RealSense RGB-D 与 YOLO 目标感知，以及 Gazebo 中从规则示教、轨迹整理到 SmolVLA 离线预测的数据与模型流程。My work follows two tracks: RealSense RGB-D and YOLO perception, and a Gazebo workflow connecting rule-based demonstrations, trajectory preparation and SmolVLA offline prediction.
感知端输出目标类别、距离与相机坐标 XYZ；仿真端把图像、里程计和速度动作按时间戳对齐，转换为 LeRobot 数据，并完成训练流程与 Checkpoint 重载测试。The perception pipeline outputs object class, distance and camera-frame XYZ. The simulation pipeline aligns images, odometry and velocity actions by timestamp, converts them to LeRobot data, and exercises training and checkpoint reload.
资料来源Sources
来源：2026 春季 / 秋季汇报；本地感知实现Sources: spring/fall 2026 presentations; local perception implementation
机器人平台与系统架构Robot Platform & System Architecture
硬件版本分开记录Hardware revisions tracked separately
团队以 MentorPi 为移动平台，Raspberry Pi 连接机器人侧接口，Jetson 承担视觉计算与跨设备图像发布。传感器包括 RealSense RGB-D、LiDAR 和轮编码器；系统按采集、感知、ROS 2 接口与运动执行分层。The team uses MentorPi as its mobile platform, Raspberry Pi for robot-side interfaces, and Jetson for vision compute and cross-device image publication. RealSense RGB-D, LiDAR and wheel encoders feed acquisition, perception, ROS 2 interfaces and motion layers.
春季架构使用 Jetson Nano；本地相机脚本加载 yolov8n.pt。机器人平台、地图与 AEB 模块的演示材料保留在下方，便于对照我的感知与仿真工作和团队系统集成。The spring architecture uses Jetson Nano, while the local camera script loads yolov8n.pt. Platform, mapping and AEB demonstrations below provide context for my perception and simulation work within the team integration.
资料来源Sources
来源：Spring Final p.6、10、11；Spring Midterm p.18；smartcam_pipeline.pySources: Spring Final pp.6,10,11; Spring Midterm p.18; smartcam_pipeline.py
机器人硬件Robot Hardware Gallery
系统架构图System Architecture Figures
RGB-D 感知与 YOLOv8RGB-D Perception & YOLOv8
本地源码实现Local source implementation
相机脚本配置 640×480 的 bgr8 彩色流与 z16 深度流，请求 15 FPS，并通过 rs.align 将深度对齐到彩色图像。The camera script configures 640×480 bgr8 color and z16 depth streams at a requested 15 FPS, aligning depth to color through rs.align.
对置信度大于 0.5 的 YOLO 检测框裁剪边界，剔除零值深度后取中值，乘 depth_scale 得到米制距离；再结合框中心与对齐后的内参反投影为相机坐标 XYZ，通过 OpenCV 显示类别和空间信息。YOLO boxes above 0.5 confidence are clipped to image bounds. Nonzero depth values are reduced to a median and scaled to meters; box centers and aligned intrinsics are then deprojected to camera-frame XYZ. OpenCV displays class and spatial information.
这一实现将检测、深度关联和可视化拆成可检查的步骤。中值估计对少量极端深度值更稳健；背景混入、外参和坐标变换是接入机器人控制时的关键设计问题。This implementation separates detection, depth association and visualization into inspectable steps. Median depth is robust to a few extreme values; background contamination, extrinsics and coordinate transforms remain key integration considerations.
资料来源Sources
来源：camera_test/smartcam_pipeline.py，main() 的采集、检测和反投影流程Source: camera_test/smartcam_pipeline.py, acquisition, detection and deprojection in main()
RGB-D 与 YOLO 结果RGB-D & YOLO Results
训练版本与验证范围Training Revisions & Evaluation Scope
训练配置与结果文件Training configurations & result files
训练记录按 run 管理：4 月 12 日包含 YOLOv8n、416 px 和 batch 32 的配置；4 月 23 日 exp_0423_1742 使用 YOLOv8s、640 px、batch 16、60 epochs、fraction 0.5、AMP 与 mosaic 0.5，results.csv 保存了 60 个 epoch 行。Training records are tracked by run. The April 12 configuration uses YOLOv8n, 416 px and batch 32. The April 23 exp_0423_1742 run uses YOLOv8s, 640 px, batch 16, 60 epochs, fraction 0.5, AMP and mosaic 0.5, with 60 epoch rows in results.csv.
train_point.py 优先恢复最新 last.pt，因此复现时以保存的 args.yaml、权重和数据划分为依据。检测样例、混淆矩阵与 PR 曲线与各自训练记录一起展示。Because train_point.py preferentially resumes the latest last.pt, reproduction uses the saved args.yaml, weights and data split. Detection examples, confusion matrices and PR curves are presented with their corresponding run records.
资料来源Sources
来源：train_point.py；2026-04-12 训练日志；exp_0423_1742/args.yaml 与 results.csvSources: train_point.py; April 12, 2026 training log; exp_0423_1742/args.yaml and results.csv
ROS 2 跨设备通信ROS 2 Communication
团队集成记录Team integration records
团队通信实验使用 Jetson 发布视觉图像、Raspberry Pi 5 订阅。我的独立 Python 感知脚本负责 RGB-D 处理与可视化，机器人侧 ROS 2 通信作为单独模块集成。Team communication experiments use a Jetson image publisher and a Raspberry Pi 5 subscriber. My standalone Python perception script handles RGB-D processing and visualization; robot-side ROS 2 communication is integrated as a separate module.
接口调试围绕话题类型、时间戳、坐标 frame、网络发现和 QoS 展开，按发布端与接收端记录检查数据新鲜度，为后续控制接口对接提供依据。Interface debugging focuses on topic types, timestamps, coordinate frames, discovery and QoS, using publisher and subscriber observations to inspect freshness before control integration.
资料来源Sources
来源：素材包机器人侧通信摘要；本地 Python 源码交叉核对Sources: package robot-side communication summary; cross-check against local Python source
SLAM 与 AEB 团队实验SLAM & AEB Team Results
团队阶段性实验Team module experiments
团队春季实验比较 LiDAR 与 LiDAR + 编码器建图，秋季材料记录路径估计与参考距离；AEB 台架录像展示超声波感知和模块响应。相关图表与录像按实验阶段展示。Team spring experiments compare LiDAR mapping with LiDAR plus encoders; fall material records estimated paths against reference distance. AEB bench recordings show ultrasonic sensing and module response, with figures and videos grouped by experimental stage.
这些材料提供整机协作背景。我的主要工作集中在 RGB-D 感知和秋季仿真数据 / VLA 流程，SLAM、AEB 与 RL 消融按团队成果归属。These records provide system context. My main work is RGB-D perception and fall simulation-data/VLA workflows; SLAM, AEB and RL ablations are attributed to the team.
资料来源Sources
来源：Spring Final p.10、11、13；Fall Midterm p.7、14、15；团队录像Sources: Spring Final pp.10,11,13; Fall Midterm pp.7,14,15; team recordings
SLAM 与 AEB 团队实验SLAM & AEB Experimental Evidence
Gazebo 仿真与 VLA 数据链Gazebo Simulation & VLA
规则控制 / 离线可行性Rule-based control / offline feasibility
在 Gazebo Classic 与 ROS 2 Humble 中开展 MentorPi 仿真，使用 RGB、LiDAR 和里程计观测，完成红蓝目标规则式视觉跟踪与限幅速度指令流程。I worked on MentorPi simulation in Gazebo Classic and ROS 2 Humble, using RGB, LiDAR and odometry observations for rule-based red/blue target tracking with bounded velocity commands.
AutoTeacher 记录相机、odom、LiDAR、cmd_vel 与任务标签，经 ROS bag 审核和时间戳对齐后，整理为 LeRobot 图像—动作样本。期中示例包含约 10 Hz 的 340 帧部分轨迹。AutoTeacher records camera, odometry, LiDAR, cmd_vel and task labels. ROS bag auditing and timestamp alignment prepare LeRobot image–action samples. The midterm example contains a partial trajectory of 340 frames at roughly 10 Hz.
SmolVLA 已完成 100 步 BF16 训练流程测试、Checkpoint 重载及 9 个训练轨迹帧的离线动作预测。误差分析发现初始转向偏小、停止阶段仍有非零角速度，为后续动作标定与停止行为验证提供了具体问题。SmolVLA was exercised through a 100-step BF16 training test, checkpoint reload and offline prediction on nine training-trajectory frames. Error analysis found underestimated initial turns and nonzero angular velocity at stops, identifying concrete targets for action calibration and stopping tests.
资料来源Sources
来源：Fall Midterm p.12、16–19；2026-10-05 周会 p.8Sources: Fall Midterm pp.12,16–19; October 5, 2026 weekly meeting p.8
Gazebo 与 VLA 研究Gazebo & VLA Research Gallery
工程成果与后续验证Engineering Results & Validation
按实验阶段评价Results by experimental stage
已形成从 RGB-D 感知实现到仿真数据与离线模型实验的可追溯工作链，覆盖源码、训练配置、数据转换、预测结果及阶段演示。The work connects RGB-D implementation with simulation data and offline model experiments through source code, training configurations, data conversion, prediction results and staged demonstrations.
下一阶段优先验证独立场景的检测与深度误差、仿真停止行为、任务完成率与失效恢复，再推进带安全限幅和急停机制的机器人接口集成。Next priorities are detection and depth errors on independent scenes, simulation stopping behavior, task completion and failure recovery, followed by robot-interface integration with bounded commands and emergency stopping.
资料来源Sources
来源：本地代码核查与春秋季材料；后续验证项为工程建议Sources: local source review and spring/fall material; future checks are engineering proposals
我的实际工作与团队分工My Contributions & Team Scope
感知 / 仿真数据 / 离线评估Perception / simulation data / offline evaluation
个人工作：RealSense RGB-D / YOLO 感知流程、室内障碍物训练实验，以及秋季 Gazebo 视觉跟踪、AutoTeacher 数据整理与 SmolVLA 离线可行性验证。My work covers the RealSense RGB-D/YOLO pipeline, indoor-obstacle training experiments, fall Gazebo visual tracking, AutoTeacher data preparation and SmolVLA offline feasibility.
团队协作：机器人平台、ROS 2 系统集成、SLAM、AEB 与 RL 实验。下表集中说明成果归属和验证范围。Team work covers the robot platform, ROS 2 system integration, SLAM, AEB and RL experiments. The table below brings contribution and evaluation scope together.
资料来源Sources
来源：私人交接包贡献摘要；Fall 2026 个人周会；本地感知代码Sources: contribution summary in the private handoff; fall 2026 individual update; local perception code
完成状态与验证范围Completion status & evaluation scope
| 模块Module | 完成阶段与归属Stage & contribution | 验证范围Evaluation scope |
|---|---|---|
| RGB-D 感知RGB-D perception | 个人实现 · 本地运行演示Personal implementation · local demo | 类别、距离与相机坐标 XYZ；机器人 / 地图坐标变换为后续接口工作。请求 15 FPS 为采集配置，非模型吞吐量。Class, distance and camera-frame XYZ. Robot/map transforms are further interface work. Requested 15 FPS is capture configuration, not model throughput. |
| YOLO 训练YOLO training | 个人训练实验 · 按 run 归档Personal training experiments · per-run records | 保留实际配置、权重与训练结果；指标只对应各自数据划分和平台，独立场景评估待开展。Saved configurations, weights and training results. Metrics retain their own data split and platform; independent-scene evaluation remains future work. |
| AutoTeacher / LeRobotAutoTeacher / LeRobot | 个人秋季工作 · 仿真数据链Personal fall work · simulation data pipeline | 规则教师、采集审核与数据转换；340 帧约 10 Hz 为部分轨迹，任务完成率与独立测试集待验证。Rule-based teacher, collection auditing and conversion. The 340 frames at about 10 Hz are a partial trajectory; task completion and held-out testing remain to be evaluated. |
| SmolVLASmolVLA | 个人秋季工作 · 离线流程测试Personal fall work · offline pipeline test | 100 步 BF16、Checkpoint 重载、9 帧训练轨迹预测；当前阶段为流程可行性，尚无模型收敛、泛化或实机闭环结果。100 BF16 steps, checkpoint reload and prediction on nine training-trajectory frames. This establishes pipeline feasibility; convergence, generalization and physical closed-loop results remain unverified. |
| 通信 / SLAM / AEB / RLCommunication / SLAM / AEB / RL | 团队系统与阶段实验Team system and module experiments | 按录像与报告记录模块阶段；AEB 采集、传输和停车时间分别评价，多机器人地图融合及量化安全结论待验证。Module stages follow recordings and reports. AEB acquisition, transfer and stopping times are evaluated separately; multi-robot map fusion and quantitative safety conclusions remain unverified. |
| 平台与版本Platforms and revisions | 按实验环境区分Tracked by experiment environment | Jetson Nano 架构、本地相机演示与 Gazebo 实验分别记录；Orin Nano / TensorRT 仍为部署方向。Jetson Nano architecture, local camera demos and Gazebo experiments are recorded separately. Orin Nano / TensorRT remains a deployment direction. |
技术栈Tech Stack
工程技术深入Technical Deep Dive
从一段可读的相机实现追踪数据：检测框怎样选取深度、距离怎样变成 XYZ，以及哪些日志能帮助区分采集故障与深度估计偏差。Follow the data through the camera implementation: how boxes select depth, how distance becomes XYZ, and which records help separate capture failures from estimation errors.
工程问题与模块职责Engineering question and responsibilities
如何在无效深度、目标框背景混入和取帧超时下，仍保留可检查的空间输出？How can spatial output remain inspectable when depth is invalid, boxes include background, or frame capture times out?
本地脚本负责采集、检测、深度关联与可视化；ROS 2 Topic、LiDAR/IMU 及控制接口属于独立的机器人系统集成环节。Local scripts handle acquisition, detection, depth association and visualization. ROS 2 topics, LiDAR/IMU and control interfaces belong to separate robot-system integration work.
检测框到相机坐标的真实数据路径From detection boxes to camera coordinates
本地相机实现Local camera implementation
- 01彩色 / 深度对齐Align color / depth
- 02框边界裁剪Clip box bounds
- 03剔除零值深度Reject zero depth
- 04中值 × 深度比例尺Median × depth scale
- 05内参反投影 XYZDeproject to XYZ
01RGB-D 接口与采集配置RGB-D interfaces and capture configuration
实现解析Implementation Reading
RealSense 彩色与深度流分别配置为 bgr8 和 z16；所读感知脚本使用 640×480、15 FPS，请求帧率与实际处理吞吐量分开理解。另一份双流预览脚本使用 30 FPS，不能把两个配置混作同一次性能测试。Color and depth streams use bgr8 and z16. The reviewed perception script requests 640×480 at 15 FPS; requested frame rate is distinct from measured processing throughput. A separate dual-stream preview requests 30 FPS, so the two settings do not represent one benchmark.
深度通过 rs.align 对齐至彩色流，再读取对齐后帧的内参及设备深度比例尺。空间输出属于相机坐标系；它并不是已经变换到地图坐标的障碍物位置。Depth is aligned to color with rs.align, followed by reading the aligned-frame intrinsics and device depth scale. Spatial output is in the camera frame, not automatically a map-frame obstacle position.
来源：相机配置与帧对齐实现Source: camera configuration and frame-alignment implementation
02框、深度与三维输出的实现Boxes, depth and spatial output
实现解析Implementation Reading
YOLO 输出提供 xyxy 框、类别和置信度；该脚本只处理 conf > 0.5 的框，先裁剪到图像边界，再读取同一区域深度。阈值是代码条件，不是“50%准确率”。YOLO provides xyxy boxes, classes and confidence. This script processes boxes with conf > 0.5, clips them to image bounds and selects the corresponding depth region. The threshold is a code condition, not a 50% accuracy result.
非零深度集合非空时计算中值，乘以 depth_scale 得到米；用框中心像素和内参调用反投影函数。它把一个代表距离赋给框中心，不估计目标的完整三维形状。For a nonempty set of nonzero depths, the median is multiplied by depth_scale to obtain meters. The box-center pixel and intrinsics are passed to deprojection. This assigns a representative distance to the box center; it does not reconstruct the object’s complete 3D shape.
depth_region = depth_image[y1:y2, x1:x2]
valid_depths = depth_region[depth_region > 0]
if len(valid_depths) > 0:
median_depth_raw = np.median(valid_depths)
median_distance_m = median_depth_raw * depth_scale来源：检测结果解析与深度关联代码Source: detection parsing and depth-association code
03中值估计的取舍Trade-offs of median depth
工程分析Engineering Analysis
剔除零值可以避免无效像素拉低估计；中值比直接均值更不易被少数异常深度影响。但当检测框内背景占多数时，中值仍可能落在背景上，遮挡和框抖动也会改变采样集合。Removing zeros prevents invalid pixels from depressing the estimate, and the median is less sensitive than the mean to a few extreme values. However, a background-dominated box can still yield background depth; occlusion and box jitter also change the sample set.
可比较中心区域取样、深度聚类或实例掩膜，但这些是后续分析方向，当前实现没有据此宣称更高准确率。相机到机器人坐标的外参标定和时间同步应另行验证。Center-region sampling, depth clustering and instance masks are alternatives to evaluate, not implemented accuracy improvements claimed here. Camera-to-robot extrinsics and time synchronization require separate validation.
工程分析：依据当前 ROI 中值实现Engineering analysis: based on the current ROI-median implementation
04采集异常与数据可诊断性Capture faults and diagnostic records
实现解析Implementation Reading
代码先查询设备与 USB 类型，包含硬件复位、启动错误处理、预热帧和 2000 ms 取帧超时分支；无效帧跳过。超时参数不是处理延迟测量，复位流程也不等于故障已被永久解决。The code queries devices and USB type, includes reset and startup-error handling, discards warm-up frames and handles a 2000 ms frame-wait timeout. Invalid frames are skipped. The timeout is not a latency measurement, and reset handling does not establish permanent fault resolution.
映射脚本记录 Timestamp、Class、Confidence 和 X/Y/Z（米）。排查时可沿设备枚举→有效帧→检测框→有效深度→XYZ逐段检查，并将采集错误与空间估计偏差分开。The mapping script logs Timestamp, Class, Confidence and X/Y/Z in meters. A diagnostic sequence follows device enumeration, valid frames, boxes, valid depth and XYZ, separating capture faults from estimation errors.
来源：异常分支与 CSV 写入实现Source: exception branches and CSV-writing implementation
05已有记录与验证范围Available records and validation scope
项目资料Project Records
归档中存在与映射字段一致的障碍物 CSV 记录，可核对输出结构与时间顺序。它不是带真值标注的测距精度测试，也不能单凭记录中的时间间隔推导端到端制动时延。An archived obstacle CSV contains the mapping fields and recorded rows, supporting inspection of output structure and ordering. It is not a ground-truth ranging benchmark, and its timestamp intervals do not establish end-to-end braking latency.
本人已有工作范围仍为相机处理脚本、边缘设备及传感接口联调；这里的代码阅读不增加完整 ROS 2 发布节点、闭环制动或 VLA 部署的完成声明。My established scope remains camera-processing scripts, edge devices and sensor-interface integration. This code reading does not add claims of a complete ROS 2 publisher, closed-loop braking or deployed VLA control.
来源：归档日志及既有项目职责Source: archived log and existing project contribution scope
面试深入讨论Interview Deep Dive
从设计问题与已有实现出发，进一步讨论边界、观察量和验证方法。Follow the design questions and implementation details into scope, observables and validation methods.
中值深度为什么还可能选错目标？Why can median depth still select the wrong surface?
中值只描述 ROI 内占优势的深度分布；背景占比、遮挡及无效像素位置都影响结果，应同时检查框与深度 ROI。The median reflects the dominant ROI depth distribution. Background proportion, occlusion and the positions of invalid pixels matter, so boxes and depth ROIs should be inspected together.
XYZ 能否直接给导航使用？Can navigation consume XYZ directly?
当前 XYZ 属于相机坐标；机器人/地图坐标还需要外参、坐标变换和时间对应，不能省略。Current XYZ is camera-frame output. Robot/map coordinates additionally require extrinsics, transforms and time alignment.
15 FPS 与 2000 ms 分别说明什么？What do 15 FPS and 2000 ms describe?
前者是请求的采集配置，后者是等待帧的超时；两者均不是已测吞吐量或端到端系统延迟。The first is a requested capture setting; the second is a frame-wait timeout. Neither is measured throughput or end-to-end latency.
工程备注 · 阶段与版本Engineering Notes · stage & revisions
- 本地相机脚本处理 RGB-D 并输出可视化，本身不发布 ROS 2 消息；机器人侧 ROS 2 节点运行和多设备闭环按具体实验区分。The local camera script processes RGB-D data and visualizes results; it does not itself publish ROS 2 messages. Robot-side ROS 2 node execution and multi-device closed loops are experiment-specific.
- Gazebo 与 AEB 按场景实验处理；Zenoh、VLA 属于扩展工作。当前不发布端到端制动延迟或带宽改善指标。Gazebo and AEB are scenario experiments; Zenoh and VLA are extension work. No end-to-end braking-latency or bandwidth-improvement metric is claimed.