Co-Founder · Algorithm Engineer
MotionMentor
AI dance feedback tool
The user uploads a video of their dance, and our algorithm compares it to the reference video, producing a series of analytics.
Features
Visual Comparison
On the user's video, we overlay a colored "skeleton" of the reference video's pose at the corresponding moment.
By viewing the discrepancy between the skeleton and their own poses, the user can visually identify where their mistakes are.
AI Teacher Feedback
Based on the differences between the user's video and the reference video, MotionMentor identifies the biggest mistakes and suggests actionable improvement plans.
Accuracy Timelines
To help the user identify when and where they made mistakes in the dance, we designed a plot showing how accurate each of their body parts is at each point in the dance. The user can click on the dips in the graph to view the video comparison at that moment, and identify what improvements to make.
Tech Stack
The feedback pipeline works roughly as follows:
A Human Pose Estimation model extracts the 2D coordinates of joints from both videos, frame-by-frame.
Another model raises the 2D coordinates to 3D, still for individual frames, without temporal information. This step and the one above both run in the MMPose framework.
We treat the final 3D coordinates with various signal processing algorithms, such as the One-Euro Filter, to integrate temporal information for more accurate joint coordinate data.
An algorithm extracts the dance-relevant information from the mess of coordinates and formats it in a way understandable to an LLM.
The LLM returns feedback for the user based on the extracted dance information.
This whole pipeline is mostly implemented in Python and runs every time a user uploads a video for feedback.
Video Human Pose Estimation Demo
The following video demonstrates the level of stability and accuracy that this pipeline achieves. Note that the pipeline actually produces 3D coordinates; the video shows 2D coordinates for a clearer demonstration.