Visual Motion to MIDI
- Year: 2025
- Tech: MaxMSP, VCV Rack, loopMIDI
- Focus: Video Processing, Motion Detection, Modular Synthesis
This project is designed to transform visual movement on a screen into sound. Users can interact with various visual media—such as films like Safety Last, browser games like the Chrome T-Rex game, or live webcam feeds—and the system will detect motion and translate it into MIDI data.
The technical workflow consists of three main stages:
Screen Capture & Analysis: A MaxMSP patch captures activity from a secondary monitor. It analyzes the video feed for motion by detecting pixel differences and calculates average screen brightness, converting it into RGB color values.
Data Routing: The system locates the area of motion and converts both movement and brightness fluctuations into MIDI signals. These messages are sent through virtual ports created by loopMIDI.
Modular Synthesis: In VCV Rack, MIDI-CV modules receive the incoming data. Brightness data is routed from Ports 1-3 to oscillators to create ambient textures, while motion data from Port 4 generates melodic elements.
By mapping these visual elements to audio parameters, the system acts as a creative tool for live performance, generative music, and interactive audiovisual installations.