Exploratory Design of a Hands-Free Video Game Controller for a Quadriplegic Individual
In short
Video games expect two hands on a controller. I designed a system that reads six facial expressions through a webcam and turns them into game controls, with speech as a backup. I was its co-designer and its first player, across three games we built for it.
Why this work
Game input needs fast, two-handed control. Switches, mouth and tongue controllers, brain-computer interfaces and eye gaze each supply far fewer inputs than most games demand, and adaptive controllers still assume some use of the hands.
I used to love racing games. Once I could no longer use standard controllers, I stopped playing and took to watching other people play.
What we built
The system uses the OpenFace toolkit to detect facial action units, the small muscle movements that make up an expression, from live webcam video. Six expressions, among them a smile, wide eyes and a jaw drop, each press a key.
An expression has to hold for five video frames before it counts, which stopped accidental triggers. Spoken keywords are a backup, and a settings window adjusts how strongly each expression must be made. We built three games in Unreal Engine: a walking adventure, a first-person shooter and a car racing game.
How we worked
This was design with one person, not a user study. Over six months of weekly remote sessions I tested each version at home, worked out which combinations of facial movements I could repeat comfortably, and helped set the thresholds.
What we learned
Expressions built from two or three facial movements worked best: more are tiring, and fewer cause false triggers. Speech suited the slow walking game but lagged too much for the shooter.
The shooter and the racing game were too fast at first, so we slowed them down, added auto-aim, and replaced held keys with on-and-off toggles. With macros, the same system let me play four commercial puzzle games without changing them.
The paper ends with four lessons: fit the input to the player’s ability, weigh how often each game action occurs, offer more than one kind of input and feedback, and build games so they can be personalized.
Cite this paper
Atieh Taheri, Ziv Weissman, and Misha Sra. 2021. Exploratory Design of a Hands-Free Video Game Controller for a Quadriplegic Individual. In Proceedings of the Augmented Humans International Conference 2021 (AHs ’21). Association for Computing Machinery, New York, NY, USA, 131–140. https://doi.org/10.1145/3458709.3458946
BibTeX
@inproceedings{10.1145/3458709.3458946,
author = {Taheri, Atieh and Weissman, Ziv and Sra, Misha},
title = {Exploratory Design of a Hands-Free Video Game Controller for a
Quadriplegic Individual},
booktitle = {Proceedings of the Augmented Humans International Conference
2021},
series = {AHs '21},
year = {2021},
pages = {131--140},
numpages = {10},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
doi = {10.1145/3458709.3458946}
}