Recording Sample-Free

(Image from Evan Ziporyn's excellent solo clarinet album)

In a recent post I wrote on putting electrodes in our brains and using representative computer code to guide machines in playing real instruments; earlier, I wrote on Melodyne software that can pick out individual notes from individual instruments on a recording for pitch shifting or time stretching.

Last month at the International Conference on Acoustics, Speech and Signal Processing, Mark Bocko introduced a new project he's been working on at the University of Rochester that seeks to eliminate sampling from digital recording. Instead of recording live instruments and sampling at the highest rate possible for CD-quality sound, researchers developed two software models: one that replicates every parameter of a clarinet's sound, and another that codes a clarinet player's physical actions in performance. Once they make a real audio recording of a clarinetist, they can use the second model to determine the various performance actions and run this info into the clarinet sound module to produce a virtual rendition of the original performance.

One of the interesting developments here is the tiny amount of information needed to store such a virtual recording; because the audio is not sampled hundreds of times per second but reproduced based on actual physical gestures and frequencies of a player and nothing more, these recordings are quite small. The twenty-second clarinet solo filled under one kilobyte.

Bocko says, "Maybe the future of music recording lies in reproducing performers and not recording them." This to me sounds like a kind of compromise: let's still use real people initially but then convert their sound into something synthetic, better, and with a minimum of information. I have this image of Charlie Chaplin from Modern Times in my head, but now he's miming all the motions while a robot mimics him, actually screwing on the bolts.


So why not ditch real players and instead program virtuosic performances into the physical model, much like MIDI? This could be a whole new, more flexible version of MIDI, in that MIDI has only 128 (0-127) possibilities for each musical parameter, while this new software may have infinite expressive potential. (From the short article, it sounds like these computer models have continuous, non-incremental parameters, like we do.)

A very relevant comparison to make here is with the highly advanced, Hi-Res MIDI-driven live piano recreations of recordings by Zenph Studios - essentially, high tech player piano. (Hi-Res MIDI uses 1024 possibilities, versus 128.) I'm gonna write up a post on this soon, but Zenph can take an old recording of, say, Glenn Gould playing Bach, code it into Hi-Res MIDI, and play it back in a "re-performance" on an actual Yamaha Disklavier Pro concert grand piano. A convergence of Zenph and Rochester could be very interesting: a Disklavier performance of Art Tatum recorded and re-virtualized with Rochester's modeling technology.

The Rochester folks still have a long way to go in terms of sound quality; the demo recording sounds like low quality MIDI; Vienna Symphonic Library, for example, sounds infinitely better. Still, the potential for vast improvement is there, and also for manufacturing technology-esque 'musical redundancies.'