BrainMove: a Web App for Interactive Cognitive Training Games - (MCT Howest)

BrainMove is an interactive physical and cognitive training system that consists of a central hub and four wireless, proximity-sensitive pods. This project was originally designed for teachers organizing children's camps to make standard exercises more interactive.

The system works by linking physical interaction to browser-based games such as Color Sprint, Memory, and Falling Colors. The setup comprises a Raspberry Pi acting as the central server and four hardware modules (red, blue, yellow, and green) that are operated by hand. When a game indicates a certain color on the screen, the user must move withing the proximity the corresponding physical pod as quickly as possible. In this guide, we explain how you can build this system yourself.

System Architecture:

BrainMove is a locally running IoT application built with the following components. The full open-source code can be found on the GitHub Repository.

1. Hardware Communication: Four ESP32 microcontrollers send sensor data wirelessly via the locally hosted MQTT protocol (Mosquitto) to the central server.
2. Backend: An asynchronous FastAPI (Python) server processes the incoming hardware actions, manages the game logic, and stores scores in a SQLite database.
3. Frontend: The user interface is a Vue 3 (Vite) Single Page Application that receives and sends real-time updates via Socket.IO (WebSockets).

Make sure you have the following ready before you begin. Because this project requires 4 independent pods, the supplies for the pods are described per set of 4.

Central Hub (The Server):

1. 1x Raspberry Pi 5 with a microSD card (32GB or larger recommended)
2. 1x USB-C power supply for the Raspberry Pi (original or good quality, e.g., 27W)

Hardware per set of 4 Pods (Electronics):

1. 4x ESP32-C3 microcontrollers (the reliable brain of each pod)
2. 4x VL53L0X Time-of-Flight (ToF) sensors (laser distance sensor for interaction/touch detection)
3. 4x Passive buzzers (to give a sound signal on correct/incorrect)
4. 4x Push buttons (for powering on/off or resetting)
5. 4x 18650 Lithium-ion batteries (for a fully wireless experience)
6. 4x 18650 Battery charger modules (e.g., TP4056) (to safely charge and protect the battery)
7. Resistors (8 pieces in total: e.g., 2x 10kΩ or 100kΩ per pod for the voltage divider to measure the battery percentage via the ESP32, since the battery is max 4.2V and the ESP ADC measures up to 3.3V)
8. Sufficient jumper wires and heat shrink tubing
9. Soldering iron and solder

Enclosure (Optional but recommended):

1. 4x 3D-printed pods in the recognizable colors (Red, Blue, Yellow, Green)

Software & Libraries:

1. GitHub Repo: This includes all the code, installation manual, user manual, hardware spec sheets, ...
2. Backend (Data & Game Logic): Python 3.11+ with asynchronous FastAPI (hosted via Uvicorn). Caching and scores are saved locally to an SQLite3 database. The communication bridge with the hardware runs via a Mosquitto MQTT broker.
3. Frontend (Visuals & Real-time): Vue 3 (Composition API) built with Vite for maximum performance. Socket.IO is used for real-time, low-latency updates between the screen and the players.
4. Microcontrollers (Edge): C++ in the Arduino IDE. Use of PubSubClient for MQTT payloads and VL53L0X libraries for I2C sensor processing.

Before we bring the sensors to life, we need a "post office" that handles all the messages between the pods and the games. For this, we use an MQTT broker on our Raspberry Pi.

1. Install the operating system on your Raspberry Pi.
2. Open the terminal and install the Mosquitto broker by running the following:

1. Configure anonymous access so that our ESP32s can connect directly locally without complex passwords.

Although you can build this project loosely on a table, it is much more fun and safer in an enclosure.

1. Go to the Github Repository (see the link at the top of this project) and download the .STL files in the /hardware/3d-models folder.
2. Print a total of 4 enclosures. Pay attention to the color! You need red, blue, yellow, and green filament for the complete experience (PLA or PETG is recommended). Alternatively use grey filament and spray paint the pods each in their color.
3. We recommend an infill of at least 15% so they can withstand a bump (or an enthusiastic smack).

Before we upload the software, we need to solder and assemble all physical components into the pod (or use a test setup first using breadboards and solder later). Follow this schematic for each of the 4 pods:

1. VL53L0X (Laser Sensor):
2. Connect VCC to the 3.3V pin of the ESP32.
3. Connect GND to the GND pin of the ESP32.
4. Connect SDA to Pin 6.
5. Connect SCL to Pin 7.
6. Connect the XSHUT pin to Pin 4.
7. The Buzzer (Sound):
8. Connect the negative pin to GND and the positive pin to Pin 5.
9. The Push Button:
10. Connect one side to GND and the other side to Pin 3 (we use the internal pull-up of the ESP).
11. The Battery & Charger:
12. Connect the 18650 battery to the B+ and B- ports of your tp4056 charger module.
13. The OUT+ of the charger goes (via the on/off switch) to the 5V/VIN of the ESP, and OUT- to GND.
14. Battery Measurement (Important!): To read the battery, we create a voltage divider with two resistors between the OUT+ and GND. You tap the middle to Pin 2 (ADC). This allows the ESP to measure if it is almost empty and safely put itself into sleep mode to prevent damage.
15. Neatly mount everything into your (optional) 3D-printed enclosure.

Now that the physical wires are correct, we open the software for the ESP32s.

Take a look at your code at this essential block in the ESP32 firmware (brainmove_mqtt.ino). This piece of logic ensures that the pod knows what to do when the game sends a command:

What happens here in simple terms?

When the Raspberry Pi shouts "start" via WiFi, the parameter isPolling is turned on. The sensor in the pod wakes up, so to speak, and searches extremely fast for an object (for example your hand or the kids trying to hit the right pod!). If the game says "stop", the sensor goes back into sleep mode to save the 18650 battery. There are also commands here to have the speaker in the pod play a selected sound depending on whether you hit the correctly colored pod.

1. Open brainmove_mqtt.ino in the Arduino IDE.
2. At the top, adjust your WiFi name (WIFI_SSID), WiFi password (WIFI_PASSWORD), and the IP address of your Raspberry Pi (MQTT_BROKER) in the constants.
3. Flash this script via the Arduino IDE to all four of your ESP32 boards and systematically change the DEVICE_COLOR at the top of the script explicitly to "red", "blue", "yellow" and "green" before you click "upload"!

The backend checks all results, saves the scores, and sends the games via APIs to the frontend.

1. Clone the code and navigate in your terminal (on your Raspberry Pi) to the BrainMoveG1 folder.
2. Create a virtual Python environment (python3 -m venv venv) and install the requirements via pip install -r requirements.txt.
3. Then start the two scripts in backend/scripts (namely init_db.py and seed_games.py) to create your SQLite database and fill it with games like Color Sprint and Memory.

The frontend runs on Vue 3 and wraps everything up for that beautiful end result.

1. Enter the frontend folder in your terminal.
2. Run npm install to fetch all Vue packages.
3. In your files, open the .env.development file (or .env) and set VITE_API_BASE_URL to the IP of the Raspberry Pi.

To run the system locally, you test if the hardware and software are talking to each other.

1. Start the Backend:
2. Activate your virtual environment and run the backend with the fastapi server via:

1. Start the Frontend:
2. Open a new terminal, navigate to the frontend folder and run:

1. Turn on your 4 pods (hardware), they will automatically search for your WiFi network and the Mosquitto broker on your server.
2. Using a browser on the same network (e.g., a tablet or PC), navigate to http://localhost:3000.
3. Start the Color Sprint game and move your hand over the specified colored pod that lights up on your screen. Does the game react immediately? Congratulations, it works!