TinyML 惯性手势识别与相机控制TinyML · Inertial Gesture Camera Control
使用 ESP32 与 MPU6050 采集惯性手势，完成轻量模型训练与端侧推理，将识别结果映射为相机指令；研究成果发表于 IJET（2025）。ESP32/MPU6050 gesture acquisition, lightweight model training and on-device inference, mapped to camera commands. Published in IJET in 2025.
- 01采集AcquisitionMPU6050MPU6050
- 加速度 / 角速度 · I²CAcceleration / Angular velocity · I²C
- 02预处理Preprocess信号处理Signal Processing
- 平滑、归一化与窗口Smoothing, normalization and windows
- 03推理InferenceESP32 / TensorFlow LiteESP32 / TensorFlow Lite
- 轻量模型 · 手势类别Lightweight model · Gesture classes
- 04执行Execution相机指令Camera Command
- UART · MaixCamUART · MaixCam
项目背景Project Overview
将惯性数据采集、模型训练、嵌入式推理与相机执行组织为一个系统。训练模式把手势数据传到主机；识别模式在 ESP32 上运行轻量模型，由相机执行端解析控制指令。The system connects inertial-data acquisition, model training, embedded inference and camera execution. Training mode transfers gesture data to a host; recognition mode runs a lightweight model on the ESP32 and sends commands to the camera.
技术目标Engineering Objective
在嵌入式计算与内存约束下，识别动态手势并触发明确的相机动作。关键工作是保持训练和推理的输入处理一致，并把分类结果连接到可检查的指令协议。Recognize dynamic gestures under embedded compute and memory constraints and map them to explicit camera actions. The engineering task is to keep training and inference preprocessing consistent and connect classification to an inspectable command protocol.
系统架构System Architecture
- 01采集：ESP32 经 I²C 读取 MPU6050 加速度和角速度。Acquisition: ESP32 reads MPU6050 acceleration and angular velocity over I²C.
- 02训练：Wi-Fi UDP 传输数据到主机，整理时序输入并训练 TensorFlow/Keras 模型。Training: transfer data to a host over Wi-Fi UDP, prepare time-series inputs and train a TensorFlow/Keras model.
- 03推理：转换为 TensorFlow Lite，在 ESP32 上输出手势类别概率。Inference: convert to TensorFlow Lite and produce gesture-class probabilities on ESP32.
- 04执行：串口指令映射到 MaixCam 拍照、录像或回放操作。Execution: map serial commands to MaixCam capture, recording or playback.
关键技术实现Technical Implementation
IMU 数据与双模式流程IMU data and operating modes
论文描述了加速度/角速度的平滑、归一化与固定窗口采集。训练模式保存按手势分类的数据，识别模式重复同类预处理，再将窗口送入轻量模型。The paper describes smoothing, normalization and fixed-window acquisition of accelerometer and gyroscope signals. Training mode stores labeled gestures; recognition mode applies corresponding preprocessing before inference.
模型训练与部署Model training and deployment
在主机端使用 TensorFlow/Keras 训练轻量全连接分类模型，经模型转换后由 TensorFlow Lite 在 ESP32 上推理。论文中的三类手势为 cross、circle 和 w。A lightweight fully connected classifier is trained with TensorFlow/Keras on the host, converted and run with TensorFlow Lite on ESP32. The paper studies three gesture classes: cross, circle and w.
相机控制协议Camera-control protocol
识别结果映射为 m_a、m_b、m_c 一类串口命令。相机侧按起始字符、分隔符和动作字符逐字节解析，连接分类输出与拍照、录像和回放操作。Recognized gestures map to serial commands such as m_a, m_b and m_c. The camera parser processes a start character, separator and action character to connect classification with capture, recording and playback.
实验评估Experimental evaluation
论文分别讨论分类、实时交互和通信实验。项目讲解将离线分类、实时识别与相机响应分开，具体数值应连同数据划分、测试条件和测量对象理解。The paper discusses classification, live interaction and communication experiments. Offline classification, live recognition and camera response are distinct evaluations whose numerical results depend on their data splits, conditions and measurement definitions.
硬件与两条通信链路Hardware and Two Communication Paths
ESP32 与 MPU6050 构成惯性采集节点，I²C 连接用于读取三轴加速度和三轴角速度。最终报告图5保存了采集板与 MaixCam 装置的实物记录；相机是动作执行端，不是惯性手势分类的图像输入。ESP32 and MPU6050 form the inertial acquisition node, with I²C carrying three-axis acceleration and angular velocity. Figure 5 of the final report records the acquisition board and MaixCam hardware. The camera executes actions; camera images are not the input to inertial gesture classification.
训练时，经 Wi-Fi/UDP 向主机收集样本；识别到手势后，经相机 UART 接口传递动作指令。采集传输与相机执行是不同链路，需要分别核对数据完整性和命令执行状态。Training samples travel to the host over Wi-Fi/UDP; gesture-triggered actions use the camera UART interface. Acquisition transport and camera execution are separate paths, requiring separate checks of data completeness and command execution.
最终报告 PDF 第14、17–19页（正文第10、13–15页）。Final-report PDF pp. 14, 17–19 (printed pp. 10, 13–15).采样、窗口与数据完整性Sampling, Windows and Data Integrity
报告描述100 Hz采集和五次采样累加，并明确累加输出仍需在外部除以采样次数。180帧六轴窗口对应1080个输入值；轴顺序、采样间隔和归一化必须在训练与推理两侧保持一致。The report describes 100 Hz acquisition and accumulation over five readings, explicitly noting that division is still required to obtain an average. A 180-frame, six-axis window produces 1,080 input values; axis order, timing and normalization must match between training and inference.
通信设计中的序号可用于检测丢包，但报告同时出现20字节和40字节的数据包描述。现有资料不足以确定最终线格式；页面不将任一长度或日志中的传输成功率当作已复核参数。Sequence numbers can support loss detection, but the report describes both 20-byte and 40-byte packets. The final wire format is not established by the available material; neither length nor logbook transmission-success claims are treated as verified parameters.
最终报告第14、17–19页；本轮未找到独立 ESP32 采集固件。Final-report pp. 14, 17–19; no standalone ESP32 acquisition firmware was found in this review.模型训练与端侧导出Model Training and Embedded Export
附录 ReadDataFile 将数值除以32768，并按180×6组织窗口；CreateModel 使用 Dense(32) 与 Dense(16) 的 ReLU 隐藏层。TrainModel 按顺序切分80%/20%，以800 epochs、batch size 16训练；这种切分不能自行证明跨用户独立评估。The appendix ReadDataFile divides values by 32,768 and forms 180×6 windows. CreateModel uses Dense(32) and Dense(16) ReLU hidden layers. TrainModel makes a sequential 80/20 split and specifies 800 epochs with batch size 16; this split alone does not establish subject-independent evaluation.
ConvertModel 调用 TensorFlow Lite 转换器，SaveModel 写入 model.tflite，Hex2H 提供 C 数组导出路径。这是可核查的转换代码描述；本轮未找到可与最终实验绑定的模型二进制，也未重新训练或刷写设备。ConvertModel invokes the TensorFlow Lite converter, SaveModel writes model.tflite, and Hex2H provides a C-array export path. These are inspectable conversion-code descriptions; no model binary tied to the final experiment was found, and no retraining or flashing was performed.
附录实际为 Dense(2) 与双列标签，仅拼接 circle、cross；虽然读取了 w.csv，却没有加入返回数据。因此这份代码应作为双分类修订阅读，不能与正文三类结果直接合并。The appendix actually uses Dense(2), two-column labels and only concatenates circle and cross. It reads w.csv but does not include it in the returned data. This code is a two-class revision and must not be merged directly with the three-class results in the main text.
最终报告附录 PDF 第40–46页。Final-report appendix, PDF pp. 40–46.相机命令与实现阶段Camera Commands and Implementation Stage
本地 MaixCam model.py 配置115200波特率，定义拍照、显示照片和录像指令。拍照回调读取并保存图像；串口解析草稿具有帧边界、长度和加和校验逻辑。The local MaixCam model.py configures 115,200 baud and defines photo, display-photo and record-video commands. The photo callback reads and saves an image; the serial-parser draft includes framing, length and additive-checksum logic.
该修订从缓冲区提取两字节命令，却与三字节常量比较；录像分支仍为 pass，部分状态和显示对象未在文件内初始化。因此它支持接口设计分析，不支持宣称完整 UART 相机控制已在此版本跑通。This revision extracts two command bytes but compares them with three-byte constants. Its recording branch remains pass, and some state and display objects are not initialized in the file. It supports interface-design analysis, not a claim that complete UART camera control runs in this revision.
本地 MaixCam 串口接收拍照任务 model.py；静态阅读，未进行硬件复测。Local MaixCam serial-photo-task model.py; static inspection without a hardware rerun.实验版本与结果解读Experiment Versions and Results
最终报告评估 circle、cross、W 三类手势，各表以一名训练者及两名测试者分别记录20次尝试。Log Book 记载的是左右、上下、顺时针圆、逆时针圆和 Z 五类；两套标签与数据集范围分别保留。The final report evaluates circle, cross and W, recording 20 attempts for one trainer and two testers in each table. The logbook instead lists horizontal swipe, vertical swipe, clockwise circle, counter-clockwise circle and Z. These label sets and dataset scopes are kept separate.
Cross Tester 2 在最终报告表5写为12/20及80%，算术上12/20为60%；Poster表2则写16/20及80%。这能确认文档冲突，但不足以判断哪次实验记录正确，因此不据此发布总体准确率。Final-report Table 5 lists Cross Tester 2 as 12/20 and 80%, although 12/20 equals 60%; Poster Table 2 lists 16/20 and 80%. This establishes a document conflict, not which experiment record is correct, so no aggregate accuracy is published from these entries.
Poster正文的500样本、1000次迭代和85%与表格的200样本、200次迭代不一致；最终报告表格又标为100样本。分析重点是训练者与新测试者之间的差异，以及动作速度、幅度和数据划分对评估的影响。The poster narrative describes 500 samples, 1,000 iterations and 85%, while its tables specify 200 samples and 200 iterations; the final-report tables specify 100 samples. Analysis focuses on differences between the trainer and new testers, and how motion speed, amplitude and data splitting affect evaluation.
最终报告 PDF 第33–34页；Poster第1页；Log Book 2025-03-24记录。Final-report PDF pp. 33–34; poster slide 1; logbook entry dated 2025-03-24.实物装置与工程图Hardware and Engineering Figures
我的实际工作My Contributions
- 完成手势数据采集与预处理、模型训练和嵌入式部署的研究实现。Implemented the research pipeline for gesture acquisition, preprocessing, model training and embedded deployment.
- 连接 ESP32 识别输出与相机控制接口，组织交互实验。Connected ESP32 recognition output to the camera-control interface and conducted interaction experiments.
- 撰写 Camera Control System Based on TinyML Gesture Recognition，以单篇论文整理方法与实验。Authored Camera Control System Based on TinyML Gesture Recognition, documenting the methods and experiments as one publication.
项目成果与验证Results & Validation
技术栈Tech Stack
工程技术深入Technical Deep Dive
依据 IJET 论文的方法部分梳理六轴窗口、归一化、端侧推理和相机命令；将论文实验描述与本地相机脚本草稿分别标注，不把未完整的分支当作运行证明。Trace six-axis windows, normalization, on-device inference and camera commands through the IJET methods, separating paper-reported experiments from a local camera-script draft rather than treating incomplete branches as runtime proof.
工程问题与模块职责Engineering question and responsibilities
怎样使训练与端侧推理使用同一种惯性输入，并把分类输出连接到可检查的相机动作接口？How can training and embedded inference share one inertial input representation and connect classification to an inspectable camera-action interface?
论文的训练模式与识别模式属于同一工作链：ESP32/MPU6050 采集、主机建模、TensorFlow Lite 推理、相机执行。关联 MaixCam 实验不另算一个独立大型项目。Training and recognition are one work chain: ESP32/MPU6050 acquisition, host modeling, TensorFlow Lite inference and camera execution. Related MaixCam experiments are not counted as another large independent project.
论文方法的数据表示与动作映射Paper-method representation and action mapping
论文描述的方法Method described in the paper
- 01六轴惯性窗口Six-axis inertial window
- 02180×6 → 1080 特征180×6 → 1080 features
- 03归一化 / 模型推理Normalize / infer
- 04类别 → 命令 → 相机Class → command → camera
01传感、训练通道与执行接口Sensor, training transport and action interface
论文方法Paper Method
论文将 MPU6050 经 I²C 连接 ESP32，以 Wi-Fi UDP 将采集数据送主机用于训练；相机执行侧由 MaixCam 与串口链路接收命令。训练数据通路与动作命令通路应分开检查。The paper connects MPU6050 to ESP32 over I²C and sends acquisition data to a training host over Wi-Fi UDP. MaixCam receives action commands through a serial link. Training transport and action-command transport are distinct paths.
来源：IJET 论文硬件与通信方法，期刊 pp183–184Source: IJET hardware/communication methods, journal pp183–184
02窗口、平滑与输入维度Windows, smoothing and input dimensions
论文方法Paper Method
方法部分描述 MPU6050 三轴加速度与三轴角速度，采用五次采样平滑，并指出累计值在外部除法后才是平均值。数据准备使用 1/32768 的缩放；180 帧×6 轴展平为 1080 维输入。The methods describe three acceleration and three angular-velocity axes, five-sample smoothing, and the need to divide accumulated values externally to obtain an average. Data preparation uses a 1/32768 scale, flattening 180 frames × 6 axes into 1080 inputs.
这是一种保留固定时间窗的表示，要求训练与识别端在轴顺序、窗口长度、缩放和动作边界上相同。论文写到的采样设定属于论文条件，不在这里扩展为新的设备计时结果。This fixed-window representation requires matching axis order, window length, scaling and action boundaries between training and recognition. Sampling settings stated by the paper are treated as paper conditions, not new device-timing measurements.
来源：论文采集、数据准备与训练方法Source: paper acquisition, data-preparation and training methods
03训练、转换与端侧推理Training, conversion and embedded inference
论文方法Paper Method
训练模式收集 cross、circle、w 三类手势，主机完成 TensorFlow/Keras 建模并转换为 TensorFlow Lite；识别模式在 ESP32 上对相同输入表示产生类别输出，再映射为动作。Training mode collects cross, circle and w gestures; the host performs TensorFlow/Keras modeling and TensorFlow Lite conversion. Recognition mode produces class output on ESP32 from the same representation and maps it to actions.
论文不同段落对部分数据包长度、网络示意和时间指标的描述不完全一致，因此这里不固化一个未经最终模型文件确认的层数/量化配置，也不新增准确率或延迟图。Some paper sections differ in packet-length, network-diagram and timing descriptions. No layer count or quantization configuration is fixed here without the final model artifact, and no new accuracy or latency plot is introduced.
来源：论文模式流程与模型部署描述Source: paper mode flow and model-deployment description
04相机实验脚本与协议检查Camera experiment and protocol checks
归档代码Archived Code
本地一份相机实验草稿声明 m_a 拍照、m_b 显示照片、m_c 录像，并配置 115200 串口；该版本的录像分支仍是 pass。命令常量、帧解析和可执行动作必须按版本匹配，不能凭命令名称宣称功能已完成。One local camera draft declares m_a for capture, m_b for photo display and m_c for recording, with a 115200 UART setting; its recording branch is still pass. Command constants, framing and implemented actions must match by version; names alone do not establish a completed function.
这份草稿的解析长度、未初始化状态及显示对象仍需在可运行版本中核对；它不是论文实验的复测记录。格式、简单和校验与 CRC 是不同机制，本文不将它们混称。Parser length, initialization state and display objects in this draft still need checking against a runnable version. It is not a repeat of the paper’s experiments. Framing, additive checksums and CRC are distinct mechanisms.
CMD_TAKE_PHOTO = b'm_a'
CMD_SHOW_PHOTO = b'm_b'
CMD_RECORD_VIDEO = b'm_c'来源：本地相机控制实验草稿；非最终协议Source: local camera-control draft, not a final protocol
05固定窗口与交互稳定性Fixed windows and interaction stability
工程分析Engineering Analysis
固定窗口简化输入维度与端侧内存规划，但动作起止、跨用户姿态和连续动作触发会改变分布。分析应分别比较离线分类、实时触发和命令执行，不把一个分类结果等同于完整相机响应。A fixed window simplifies input shape and embedded-memory planning, but action boundaries, user/orientation variation and continuous triggering can shift the distribution. Offline classification, live triggering and command execution should be evaluated separately.
UDP 采集通道还需检查序号、丢包和窗口完整性；串口执行通道则需检查帧边界、非法命令和动作状态。两条链路的可靠性不能用同一个未定义指标代替。The UDP path needs sequence, packet-loss and window-integrity checks; serial execution needs framing, invalid-command and action-state checks. One undefined reliability metric cannot stand in for both.
工程分析：论文条件与实验接口的局限Engineering analysis: limits of the paper conditions and experimental interface
06论文实验与本轮验证的区别Paper experiments versus this review
项目资料Project Records
论文发表于 IJET 2025，17(3)，183–188，DOI 10.7763/IJET.2025.V17.1324。页面保留论文作为方法与实验成果，不把本轮源码阅读写成重新训练、跨用户评估或相机功能复测。The paper appeared in IJET 2025, 17(3), pp183–188, DOI 10.7763/IJET.2025.V17.1324. It remains the method/experiment publication; this code review is not retraining, cross-user evaluation or a repeated camera-functional test.
本人研究实现与论文写作范围保留；模型工具和设备框架属于使用的基础设施。原始论文图、测试图片和含个人信息的素材未经许可不加入公开媒体。My established research implementation and authorship scope is retained. Modeling tools and device frameworks are infrastructure used by the work. Original paper figures, test images and material containing personal information are not added without approval.
来源：期刊论文及既有贡献范围Source: journal paper and existing contribution scope
07附录模型与串口修订分析Appendix Model and Serial Revision Analysis
归档代码Archived Code
附录实际为 Dense(2) 与双列标签，仅拼接 circle、cross；虽然读取了 w.csv，却没有加入返回数据。因此这份代码应作为双分类修订阅读，不能与正文三类结果直接合并。The appendix actually uses Dense(2), two-column labels and only concatenates circle and cross. It reads w.csv but does not include it in the returned data. This code is a two-class revision and must not be merged directly with the three-class results in the main text.
该修订从缓冲区提取两字节命令，却与三字节常量比较；录像分支仍为 pass，部分状态和显示对象未在文件内初始化。因此它支持接口设计分析，不支持宣称完整 UART 相机控制已在此版本跑通。This revision extracts two command bytes but compares them with three-byte constants. Its recording branch remains pass, and some state and display objects are not initialized in the file. It supports interface-design analysis, not a claim that complete UART camera control runs in this revision.
最终报告附录第40–46页；本地 MaixCam model.py。Final-report appendix pp. 40–46; local MaixCam model.py.
面试深入讨论Interview Deep Dive
从设计问题与已有实现出发，进一步讨论边界、观察量和验证方法。Follow the design questions and implementation details into scope, observables and validation methods.
为什么强调 180×6 的一致性？Why emphasize consistency of 180×6?
它决定展平输入长度和时序排列；端侧窗口或轴顺序变化会使模型收到不同语义的数据。It determines flattened length and temporal ordering. A changed window or axis order changes the meaning of model input.
识别准确是否等于相机控制成功？Does recognition accuracy mean camera-control success?
分类、传输、解析和执行是不同环节，必须分别定义成功条件与测试对象。Classification, transport, parsing and execution are separate stages with separate success definitions and test targets.
如何解释相机草稿中的未完成分支？How should incomplete camera-draft branches be described?
它们是待核对的实验实现，不是已跑通证明；应绑定可运行版本和动作测试记录后再描述完成范围。They are experimental implementation requiring verification, not proof of successful operation. A runnable revision and action-test record are needed to establish scope.
工程备注 · 阶段与版本Engineering Notes · stage & revisions
- 论文、毕业设计与相关 MaixCam 实验作为同一研究工作的关联材料。TensorFlow、MaixCDK 和设备框架属于使用的工具与基础设施。The paper, thesis and related MaixCam experiments are related parts of this work. TensorFlow, MaixCDK and device frameworks are tools and infrastructure used by the project.
- 实验覆盖有限手势类别与研究条件；跨用户、姿态变化、连续动作误触发和无线丢包仍是系统局限。协议格式解析与 CRC 是不同机制。The experiments cover a limited gesture set and research conditions. Cross-user variation, orientation, unintended triggers during continuous motion and wireless loss remain limitations. Format parsing and CRC are different mechanisms.