← Back to Projects
Spell It Out
Nintendo Switch RPG where you cast spells by swinging the Joy-Con like a real wand.
A gesture-driven action RPG built for Nintendo Switch. Players cast spells by physically performing directional arm movements with the Joy-Con — no button presses required. (Nintendo-specific SDK code removed for NDA compliance.)
- Utilized the Nintendo Switch SDK to build a real-time motion-to-2D-stroke pipeline by processing raw Gyroscope sensor data from Joy-Con controllers.
- Engineered a gesture recognition system using Weighted Edit Distance on 8-direction tokens to handle shaky/complex inputs.
- Increased the success rate of the most complex gesture (Heal) from 6% to 86% and Magic from 57% to 95% by significantly improving matching tolerance.
- Established a GitHub Actions CI/CD pipeline to automate WebGL builds and itch.io deployment, facilitating rapid external testing.
Technical Deep Dive
Real-Time Motion-to-Stroke Pipeline
Processed raw gyroscope and accelerometer data from Joy-Con controllers via the Nintendo Switch SDK. Designed a signal pipeline that discretizes continuous IMU data into a sequence of 8-direction stroke tokens, converting physical arm gestures into a comparable digital format in real-time.
Weighted Edit Distance Recognition Engine
Strict token matching failed on real hardware — 6% accuracy for complex gestures due to natural hand shake. Replaced it with a Weighted Edit Distance algorithm that penalizes substitution/insertion/deletion based on directional proximity. Result: ‘Heal’ gesture from 6% → 86%, ‘Magic’ from 57% → 95%, validated over 100 trials each.