Skip to content

Implementing a handwritten digits recognition application on the ESP32-CAM dev board using Tensorflow Lite for Microcontrollers & PlatformIO

License

Notifications You must be signed in to change notification settings

JarmouniA/TFLM_Inference_ESP32_PlatformIO

Repository files navigation

Implementing a handwritten digits recognition application on an ESP32-CAM board using Tensorflow Lite for Microcontrollers.

The goal here is to perform inference on the 28000 test images, from this Kaggle competition, on an ESP32-CAM board and to submit the results in a csv file automatically to Kaggle using the Kaggle API.

The 28000 images will be served to an http Client on ESP32 by a Python http Server (over a wifi connection), and in the same way each inference result will be sent back to the server. When the processing of all the 28000 images is done, a POST request will be sent by the ESP32 to the server so that the inference results can be submitted to Kaggle.


First of all, we need to build a machine learning model to perform the digits' recognition task. For that, I used TensorFlow and Keras in a Kaggle notebook to build, train and evaluate a simple Convolutional Neural Network model using the MNIST dataset provided by the Digit Recognizer kaggle competition .

After making sure that the model is working, we need to compress it so that it can be used for inference on memory-constrained devices like the ESP32. For that, we have to generate a TensorFlow Lite model, using one of the 2 techniques described in the notebook :

Now that we have our TFLite model, we can generate a C file containing the model's weights and characteristics, which will be used by TensorFlow Lite for Microcontrollers, using the Xxd tool as described in the last step in the notebook.


Our TFLite Micro model is now ready to be deployed on the edge, we just need the TensorFlow Lite For Microcontrollers' library compiled for ESP32. To get the library, the best option is to use the Docker image provided by the following Github repository (Thank you @atomic14) to compile the library with the TensorFlow & ESP-IDF versions of your choice (I recommend using the latest stable versions).

The resulting TFLM library will be used with the Visual Studio Code extension PlatformIO which really facilities the building process for ESP32 applications. Here is an official PlatformIO guide that explains the different configuration options for ESP32 projects. Also, this project is build with the official Espressif IoT Development Framework (ESP-IDF) not the ESP32 Arduino Framework. The library must be placed in both the /components & /lib directories.

The platformio.ini file already contains the necessary configuration options to build & deploy this project on the ESP32-CAM, and it will be detected automatically once the project in opened in VSCode with the PlatformIO extension installed.