Child interacting with ShadowPlay projection, making a llama shadow puppet

Interactive Installation

ShadowPlay

My Role Product Designer & Developer
Timeline Spring 2025
Tools TouchDesigner, Python, MediaPipe
Location The Center for Puppetry Arts, Atlanta

An interactive projection installation that uses hand-tracking and gesture recognition to bring shadow puppetry to life. Visitors form hand shadow shapes that are recognized in real-time, triggering animated storylines projected on the wall.

  • how can technology enhance a traditional art form like shadow puppetry?
  • how do you design an intuitive, gesture-based interaction for children and families?
  • balancing structured storytelling with free play and exploration

Process

1. Developed initial concept storyboard exploring how a user progresses through the interaction: a written prompt presents a shadow shape to match, the user forms the shape with their hands, and the shadow animates to life.

Initial concept storyboard showing three-step shadow puppet interaction flow

concept storyboard — prompt, match, animate

2. Iterated on concepts through sketching and user feedback sessions, refining the interaction flow and visual language.

Low-fidelity concept sketches exploring interaction variants User feedback session discussing concept sketches

left: lo-fi concept explorations, right: user feedback session

3. Designed high-fidelity screen flows for the projected interface — home screen with animal selection ordered by difficulty, guided shadow-making with size guides, and a growing storyline that responds to the user's shadow size.

High-fidelity screen designs showing Home, Making a small shadow, and Growing screens with light and dark variants

UI screens — home, shadow-making, and growing storyline

4. Built gesture recognition pipeline in TouchDesigner using Python and MediaPipe for real-time hand landmark detection and shadow shape classification.

TouchDesigner development environment showing gesture recognition pipeline with MediaPipe hand tracking

TouchDesigner development — gesture recognition with MediaPipe

5. Prototyped and tested the projection setup, calibrating for the physical space and optimizing gesture recognition accuracy.

Projection setup testing with projector and wall Early projection prototype testing User testing the projection installation

projection setup and testing

6. Deployed at the Center for Puppetry Arts. Visitors interact with the projection by forming shadow puppets — the system recognizes shapes like cats, llamas, wolves, and birds, each triggering unique animated storylines.

Live installation — cat shadow recognized on projection Children playing with the ShadowPlay installation, making bird shadows

demo video

Key Achievements

  • Real-time gesture recognition using MediaPipe hand tracking
  • Interactive storytelling that responds to shadow puppet shapes and sizes
  • Designed for children and families with intuitive, guided interactions
  • Successfully deployed at the Center for Puppetry Arts
Child interacting with ShadowPlay installation, making a llama shadow puppet with projected storyline