Breaking Barriers to Creative Expression: Co-Designing and Implementing an Accessible Text-to-Image Interface
In short
PromptAssist helps people who find typing hard to write prompts for text-to-image models. A language model suggests and extends ideas step by step, so a detailed prompt takes a few clicks or keystrokes. It was designed and tested by a team in which most of us have motor disabilities.
Why this work
Making pictures from text removes the need for fine motor control with a brush or a mouse. But a good prompt can be long, and typing it is its own barrier.
When we started, the interface to a leading image model was a single text box and a button, with no keyboard shortcuts and no way to cancel a slow generation.
What we built
PromptAssist is a web tool that walks through a prompt in steps: environment, subjects, actions, scene and style. At each step you can type, ask the model for suggestions, or skip.
Type one word, such as beach, and it can grow into a full scene. Click any word in a suggested scene to swap it. The finished prompt can be pasted into any text-to-image model, and the tool works with a keyboard alone, a pointer alone, or both.
How we designed it
Four of the five authors identify as having a motor disability, and we were the tool’s first users. Over four sessions we tried existing tools, then a first version of PromptAssist, then a revision, thinking aloud as we went.
The first version forced a fixed order of steps, and we rejected it. The revision let people start from their own prompt and call on suggestions only when they wanted them.
What we learned
Everyone on the team found the revised version easier to navigate and better for following their own ideas.
The larger lesson is a tension between ease and authorship. Suggestions save effort, but relying on them too much can make it seem that the system, not the person, made the picture. So every suggestion can be accepted, changed or rejected.
Cite this paper
Atieh Taheri, Mohammad Izadi, Gururaj Shriram, Negar Rostamzadeh, and Shaun Kane. 2023. Breaking Barriers to Creative Expression: Co-Designing and Implementing an Accessible Text-to-Image Interface. In NeurIPS 2023 Workshop on Machine Learning for Creativity and Design. arXiv:2309.02402. https://arxiv.org/abs/2309.02402
BibTeX
@misc{taheri2023breakingbarrierscreativeexpression,
title = {Breaking Barriers to Creative Expression: Co-Designing and
Implementing an Accessible Text-to-Image Interface},
author = {Atieh Taheri and Mohammad Izadi and Gururaj Shriram and Negar
Rostamzadeh and Shaun Kane},
year = {2023},
eprint = {2309.02402},
archivePrefix = {arXiv},
primaryClass = {cs.HC},
url = {https://arxiv.org/abs/2309.02402}
}