OpenMV Self-Training Neural Network for Target Point Detection – Traffic Sign Recognition in Real-Road Environments

Artificial intelligence has two very common tasks in the field of computer vision: classification and object detection.

Category:Determine which category the image belongs to—for example, whether it is a cat or a dog.

Object Detection:Used to locate the positions, quantities, and dimensions of multiple distinct objects within an image.

In previous video tutorials, we explained how to use the OpenMV4 Plus to train neural networks online for classifying different objects. This enables real-time determination of whether an image within the OpenMV’s field of view corresponds to a specific object—for example, whether a face is wearing a mask—but it cannot output the positional coordinates of faces or masks, nor the number of faces.

Target detection is significantly more complex and computationally intensive than classification, making it rare in the microcontroller domain. The OpenMV main control MCU is the STM32H7; we have now added a new capability to OpenMV that enables target detection–like tasks to run on microcontrollers, delivering outstanding performance.

Today we will explainTraining demonstrates traffic sign detection in real-road environments, detecting traffic signs on our actual roadssuch as no-horn zones, no-parking zones, and speed limits of 80 km/h, rather than a laboratory environment free from interference.

You can view the entire project content, including all code and models, at the following address: https://book.openmv.cc/project/traffic-sign.html

It is widely recognised that real-world environments—particularly road environments—are highly complex and subject to numerous sources of interference. Using smartphones and other devices, we captured images of traffic signs in actual environments and trained a model on the EdgeImpulse online platform, a partner of OpenMV. Approximately 270 images were annotated, and model training required only ten minutes. The resulting model achieved an F1 score accuracy of 92% and delivered a frame rate of ten frames per second on OpenMV, ensuring both smooth operation and high accuracy.

Users can train the system to detect any target of interest—such as different numerals, various fruits, distinct markers, diverse components, or even any specific irregular object—by following our video tutorials, thereby enabling detection of the target’s quantity, coordinates, and object class name.

Note:This video tutorial covers the neural network keypoint detection feature, which is compatible with both the OpenMV4 H7 Plus and the OpenMV4 H7. The trained FOMO keypoint detection model is compact—only tens of kilobytes in size—yet delivers excellent performance. Even though the OpenMV4 H7 has less RAM than the Plus variant, a slightly smaller model can be trained for successful deployment on the OpenMV4.

Below is a brief introduction to the principle:

The primary design decision behind FOMO object detection on OpenMV is based on the idea that many object detection tasks do not actually require knowledge of an object’s size, but only its location within the image. Once the object’s location is known, subsequent actions can be performed—such as using OpenMV to control a vehicle to move toward the detected target, enabling a drone to land at a precise location, or controlling a robotic arm to grasp a specific target object.

The FOMO model is a model specially designed by EdgeImpulse for microcontroller environments; unlike conventional object detection, it can detect only the positions and number of multiple objects, not their specific dimensions.

The underlying principle of the FOMO model is extremely simple and flexible. It first divides an image into patches, each measuring 8×8 pixels. For a 96×96 resolution image, this results in a 12×12 grid of cells; for a 360×360 image, it results in a 40×40 grid of cells. Image classification is then performed independently on each cell.

FOMO performs significantly better than YOLO V5 or MobileNet SSD on large numbers of small objects. FOMO object point detection is 30 times faster than MobileNet SSD.

FOMO performs better when the objects to be detected are of similar size—for example, when the markers to be identified are broadly comparable in size, rather than varying significantly from very large to very small.

OpenMV uses EdgeImpulse to train neural network object detection models online, which mainly involves the following steps: collecting and uploading the image dataset, annotating objects, training, testing the model, and deployment.

* Collecting an image dataset: Capturing images of specific objects in the actual environment where recognition will be performed. Images are collected in the same environment where recognition will occur, ensuring that the actual training environment closely matches the actual detection environment, thereby achieving better performance.

* Annotation Target: Draw bounding boxes around each target to be detected in our dataset and label them with the corresponding target names to facilitate subsequent training for detecting target locations and names.

* Training the model: After annotation is complete, train the model using the convolutional neural network parameters designed by us.

* Test model: We conduct testing using the trained model; if the results are unsatisfactory, we accordingly expand the training dataset or adjust the training parameters and continue training until a satisfactory model is achieved.

*Deployment: We can run the trained model file directly from the built-in USB drive of the OpenMV device.

We now demonstrate the specific process below:

01. Collect and upload the image dataset

1. First, log in to the website edgeimpulse.com, select “Log In”, enter the name of the new project, and select “Images” → “Classify multiple objects”

II. Select an image to upload. There are several ways to obtain the image dataset:

1. Capture images in real time using the OpenMV IDE. Use the “Dataset Editor” tool within the OpenMV IDE to create a new dataset and capture images in real time; for detailed instructions, refer to our previous video tutorial on object classification for mask detection.

2. Images downloaded from the internet or captured using a mobile phone. Ideally, these should match your actual detection environment. We have prepared a dataset comprising approximately four to five hundred images of traffic signs, including “No Parking”, “No Honking”, and “Speed Limit 80 km/h”. All images were captured in real-world road environments. This dataset is available for download from our GitHub repository and tutorial website.

02. Annotation Target

Upload approximately 100 images for each category; once the images have been uploaded, proceed to annotate them by marking the positions and categories of the objects to be identified.

03. Train the model

Configure the training parameters, change the resolution to 128 × 128, and select all default parameter settings; then click Save.

Generate features, with three colours representing three distinct markers.

Configure the target detection training parameters. The default number of training epochs is 60, the learning rate is 0.001, and the validation set proportion is 20%. Enable data augmentation. For transfer learning, select either the default MobileNetV2 0.35 or MobileNetV2 0.1. (Note: SSD cannot be used, as it is not supported on OpenMV.)

Select Start Training. The trained neural network model achieves an F1 Score of 91.2%, and we can save the current version.

Select Storage, enter your description, and then select Save.

04. Deploy the model

Export the trained model file. Only the library needs to be exported; select OpenMV and click Build to begin deploying and exporting the model.

It will automatically download the exported model, which in this case is the model trained on our 300 images, thereby completing the entire training process.

05. Run the model

Run the model on OpenMV:

Connect the OpenMV Plus, save the three trained files to the OpenMV’s built-in USB drive, and open the ei_object_detection.py file in the OpenMV IDE. Click Run, and the results will appear in the serial terminal.

This feature is also compatible with the OpenMV4. As the OpenMV4 does not have external SDRAM and has less memory, we can reduce the model size to enable it to run on the OpenMV4.

There are two methods to reduce the model’s size:

1. Reduce the training resolution from 128×128 to 96×96.

II. Modify the transfer learning model used, changing MobileNetV2 0.35 to MobileNetV2 0.1.

Finally, load the trained new file onto the OpenMV4 and run it.

The above is our target point detection tutorial. We look forward to your results!