Following yesterday’s interview with Kyle McDonald on FaceOSC, his custom webcam + tracking application that can make music with your face, here’s another face-controlled music demo. This one uses Ableton Live for jamming. I should add, since I somewhat obscured the fact, that this isn’t Kinect: it works entirely with a built-in webcam, which means it’s completely free to try and you don’t have to tote any extra hardware, so long as you have a laptop with a built-in cam. More on this technology as we watch it evolve…
Two spacey ways of finding media: music collections, heirarchy, and images of planets in Planetary for iPad, top. Sound and loop collections, “magnetic” relations, algorithmic categorization, and rapid torchlight auditioning in Soundtorch 2.0 for Windows, bottom.
If your music and sound collections seem like outwardly-expanding universes, two new tools promise to bring order by representing media as virtual planets and stars. One works on albums and tracks on the iPad; the other uses computer-aided analysis of loops and samples (not just music) on Windows. One will make your eyeballs pop; one might help you manage gigs of samples for a game design project.
Built in the open-source framework Cinder by an all-star team of media artist-designers (Ben Cerveny, Tom Carden, Jesper Sparre Andersen and Robert Hodgin), Planetary should satisfy space nuts and eye candy lovers. The metaphor is pretty direct: artists are stars, albums are planets around the artists, tracks are moons around the planets, and you can filter “constellations” by letter. That means the actual structure is heavily hierarchical, actually, in the tradition of iTunes (and, before it, its predecessor SoundJam). I’m not sure what happens with, say, compilations. But let’s face it: the real draw is that it’s incredibly beautiful to look at. I’d be just as entertained looking at a visualization of my system folder if it looked this pretty.
For now, Planetary is some fascinating eye candy with at least basic playback capabilities, iPad-only. That brings some good news – Airplay wireless works, and since it makes use of standard media code, even features like Last.fm scrobbles function. It also brings some bad — while Apple added support for libraries to third-party apps, Home Sharing isn’t included, so you’re limited to what’s on your iPad. Playlists aren’t supported, either. But hook this up to a projector or large screen TV with some of your favorite music, and I don’t think you’ll be complaining. And as a free tool, it’s incredible.
Less pretty, but with greater facilities on the utility side, is the Windows-only Soundtorch. (Thanks to Kristian Gohlke for the tip!) Visually, it offers a similar metaphor: media assets live on a continuous plane. Functionally, though, it’s more algorithmic than hierarchic, using something called the Computer Aided Sound Exploration engine (C.A.S.E.). The set of algorithms, which the creators say were based on evaluation of human listening, performs a sophisticated set of extractions of some 600 features from each sound file.
Rather than limit itself to albums and tracks, C.A.S.E. is tuned for audio files and loops. It’s fast enough that it can plow quickly through gigs of material. So, if you’re on Windows and have amassed an enormous collection of loops, samples, field recordings, sound effects, and the like, Soundtorch will use C.A.S.E. to first map all those relationship, then visualize them. You can use the mouse to produce new collections of assets, map relationships visually, export those relationship to XML, copy sounds to the clipboard, export to WAV, or open them in Windows Explorer. That is, all that eye candy is a genuine interface, not a barrier between you and what you might do (as so often happens with these sorts of experimental interfaces).
In fact, you might argue that, despite outward appearances, Soundtorch is entirely different from Planetary, but they share one common conceptual assumption. Related media “orbit” or attract to common materials. The difference is that Soundtorch is relational. In Soundtorch, if you “magnetize” a file, it – and any similar files – become attracted to attractors called “magnets.”
As is appropriate searching for media, the “torchlight” metaphor shines a light through files. Everything under the light plays back simultaneously, so you don’t have to audition sounds one at a time. (That sounds slightly terrifying to me, but I have to spend more time with it in an actual library.)
The creators describe the magic thusly:
Have you ever listened to a sound and felt that there was a similar one somewhere on your hard disk? And the sound you can’t find would just work so much better right now? Well, Soundtorch also remembers all sounds that you ever listened to. Just select any sound on Soundtorch, and let the system suggest the most similar ones from your whole collection.
In other words, SoundTorch is as much about what you can’t see as what you can – the intelligence to determine similarity behind the scenes. Check out the tech talk in the video above for more information on how “aurally and visually-enhanced audio search” could also apply this technology. More research at: http://www.accessive-tools.com/
Finally, if you want to hear the “Optimist” track by Zoe Keating without that voiceover and just enjoy Planetary’s gorgeous visuals, here you go:
From innovation in the visual interface to the intelligence underneath that changes how the computer interprets relationships between files, finally, there’s hope. Music and sound might not forever be trapped in views borrowed from spreadsheets, tables modeled on the needs of accountants 30 years ago.
You’ve seen the demos. You like the idea of tracking tags in the real world to create visuals. And now you want to try augmented reality for yourself - and, incidentally, you’re a Flash developer.
Reader Eric Socolofsky writes to share a framework he’s created that makes it much easier to work with the Flash-based, open source FLARToolkit, called FLARManager. Version 0.4 is just released:
FLARManager has a number of features that improve upon the existing work done by FLARToolkit:
Building the apps themselves is easier. Fire up the framework with Flex Builder (or Flash, or Eclipse, or FlashDevelop), and you have access to all the libraries you need, so you can start playing more or less out of the box. Hello, world, indeed.
You don’t have to rely on Papervision if you don’t want to. Papervision, the faux-3D library for Flash, is included with the distribution. But marker tracking is decoupled from Papervision, so you don’t have to use it if you don’t need it.
Better event management. Marker adding, updating, and removal, multiple pattern detection and management, and the like are all extended in FLARManager.
Great documentation. Eric has taken the time to read some fantastic getting started tutorials, all accessible from the site above so you can go play.
Now, you wouldn’t pick Flash for speed - that’s not the idea.
This is about the slowest implementation of ARToolkit you’ll find. But you’d use it for compatibility, because of easy deployment to the browser. Speaking of speed, the NyARToolkit Java implementation actually outperforms the original C version. I’m the last person you should talk to about writing efficient, optimized code, but I can tell you that the notion that Java is “slower” than C is simply wrong. There are a great many other, more important variables, and in some cases Java can in fact outperform C. That doesn’t mean that Java is always the right tool for the job any more than C is, though, and in fact because Java’s Garbage Collector and event scheduling aren’t really built for real-time performance, and because “native” code is suited to certain situations, there are in fact times when you wouldn’t want to use Java. Understanding the application is what really matters - and that’s why it’s nice that NyARToolkit and siblings are available for AS3, C#, the mobile Android platform, iPhone, and others.
Eric doesn’t just do this to fool around, either; he works in interactive design for museums, and has what has to be one of the world’s sweetest “day jobs” - working for the legendary Exploratorium. He’s also working with Processing and the wonderful reacTIVision library on a separate project that’s working with tangible table-based interfaces; more on that separately. Thanks, Eric!
If you get interesting work in augmented reality going, let us know. And if you need some inspiration, my current favorite is from our friend Marco Tempest, working with Zach Lieberman in OpenFrameworks on augmented magic.
