Today, we note the availability on Android of Control, a WebKit-based touch interface also on iOS.
For visualists and interactive designers, it’s worth paying attention to one feature in particular: dynamic interface creation. Perhaps biased by the musicians who have tended to embrace them, touch interfaces have tended to rely on the static layouts favored by physical knobs and faders. That’s arguably the worst of both worlds: you lose the tactile feedback of physical controls, but you don’t add any of the flexibility of a display.
Control is an open-source application rendered in HTML5, powered by JavaScript and JSON, so it’s capable of anything you can imagine. But Charlie Roberts has already demonstrated how a dynamic interface could work. Using OSC, you can make control layouts on the fly. That could lead to more sophisticated software integration for visual and musical performance, new chances for collaboration and live rigs, and the ability to make an interface on someone’s device in an interactive situation.
We saw the last of these scenarios in the case of the iOS app mrmr, developed by Eric Redlinger. As proof of concept, I and others put together a gallery show using mrmr, at which interactive pieces were able to build interfaces on-the-fly on user’s iPhones and iPads. With Control, those horizons expand, no longer constrained to individual proprietary UI widgets on one platform (like iOS), but cross-platform, Web-based, and dynamic.
But dynamic layouts could go in many, many directions. Since this is especially relevant to visual performance, perhaps in modes of interaction not really possible in music, I’d love to hear what readers imagine. And do try Charlie’s app, whether on iOS, Android, or both: Control
– and if you’re really ambitious, have a look at the source!
Two spacey ways of finding media: music collections, heirarchy, and images of planets in Planetary for iPad, top. Sound and loop collections, “magnetic” relations, algorithmic categorization, and rapid torchlight auditioning in Soundtorch 2.0 for Windows, bottom.
If your music and sound collections seem like outwardly-expanding universes, two new tools promise to bring order by representing media as virtual planets and stars. One works on albums and tracks on the iPad; the other uses computer-aided analysis of loops and samples (not just music) on Windows. One will make your eyeballs pop; one might help you manage gigs of samples for a game design project.
Built in the open-source framework Cinder by an all-star team of media artist-designers (Ben Cerveny, Tom Carden, Jesper Sparre Andersen and Robert Hodgin), Planetary should satisfy space nuts and eye candy lovers. The metaphor is pretty direct: artists are stars, albums are planets around the artists, tracks are moons around the planets, and you can filter “constellations” by letter. That means the actual structure is heavily hierarchical, actually, in the tradition of iTunes (and, before it, its predecessor SoundJam). I’m not sure what happens with, say, compilations. But let’s face it: the real draw is that it’s incredibly beautiful to look at. I’d be just as entertained looking at a visualization of my system folder if it looked this pretty.
For now, Planetary is some fascinating eye candy with at least basic playback capabilities, iPad-only. That brings some good news – Airplay wireless works, and since it makes use of standard media code, even features like Last.fm scrobbles function. It also brings some bad — while Apple added support for libraries to third-party apps, Home Sharing isn’t included, so you’re limited to what’s on your iPad. Playlists aren’t supported, either. But hook this up to a projector or large screen TV with some of your favorite music, and I don’t think you’ll be complaining. And as a free tool, it’s incredible.
Less pretty, but with greater facilities on the utility side, is the Windows-only Soundtorch. (Thanks to Kristian Gohlke for the tip!) Visually, it offers a similar metaphor: media assets live on a continuous plane. Functionally, though, it’s more algorithmic than hierarchic, using something called the Computer Aided Sound Exploration engine (C.A.S.E.). The set of algorithms, which the creators say were based on evaluation of human listening, performs a sophisticated set of extractions of some 600 features from each sound file.
Rather than limit itself to albums and tracks, C.A.S.E. is tuned for audio files and loops. It’s fast enough that it can plow quickly through gigs of material. So, if you’re on Windows and have amassed an enormous collection of loops, samples, field recordings, sound effects, and the like, Soundtorch will use C.A.S.E. to first map all those relationship, then visualize them. You can use the mouse to produce new collections of assets, map relationships visually, export those relationship to XML, copy sounds to the clipboard, export to WAV, or open them in Windows Explorer. That is, all that eye candy is a genuine interface, not a barrier between you and what you might do (as so often happens with these sorts of experimental interfaces).
In fact, you might argue that, despite outward appearances, Soundtorch is entirely different from Planetary, but they share one common conceptual assumption. Related media “orbit” or attract to common materials. The difference is that Soundtorch is relational. In Soundtorch, if you “magnetize” a file, it – and any similar files – become attracted to attractors called “magnets.”
As is appropriate searching for media, the “torchlight” metaphor shines a light through files. Everything under the light plays back simultaneously, so you don’t have to audition sounds one at a time. (That sounds slightly terrifying to me, but I have to spend more time with it in an actual library.)
The creators describe the magic thusly:
Have you ever listened to a sound and felt that there was a similar one somewhere on your hard disk? And the sound you can’t find would just work so much better right now? Well, Soundtorch also remembers all sounds that you ever listened to. Just select any sound on Soundtorch, and let the system suggest the most similar ones from your whole collection.
In other words, SoundTorch is as much about what you can’t see as what you can – the intelligence to determine similarity behind the scenes. Check out the tech talk in the video above for more information on how “aurally and visually-enhanced audio search” could also apply this technology. More research at: http://www.accessive-tools.com/
Finally, if you want to hear the “Optimist” track by Zoe Keating without that voiceover and just enjoy Planetary’s gorgeous visuals, here you go:
From innovation in the visual interface to the intelligence underneath that changes how the computer interprets relationships between files, finally, there’s hope. Music and sound might not forever be trapped in views borrowed from spreadsheets, tables modeled on the needs of accountants 30 years ago.
The remarkable thing about Processing on Android is that you can use your (desktop) sketchbook development environment as always, then run on the Google emulator or your device.
Google’s open-source Android mobile platform runs on Java. Processing, the elegant coding language for visualization, art, and media, is built in Java. The marriage of the two, therefore, is one we’ve long been anticipating. Processing’s ability to focus on lightweight, portable implementation is a perfect match for the demands of mobile development. For artists and visualists wanting to make handheld devices and tablets more expressive, I have good news: it’s coming. While not fully implemented or ready for widespread testing yet, Processing is in active development on the platform.
Ben Fry, co-creator of Processing, has been hard at work making Processing run on Android. Interestingly, in some ways Android itself vindicates the direction Processing charted years ago. Google made their platform work based on open-source development tools. They removed parts of the Java platform that would have overwhelmed the limited processing power of handhelds – even as a handful of purist Java developers cried foul. Graphics focuses on lightweight 2D and OpenGL drawing output, much in the same way that Processing strips Java to its bare essentials.
Best of all, doing Processing development for Android promises the same streamlined, sketchbook-like production flow that Processing does on desktop. To create Android apps, you simply set Processing to “Android mode.” (Right now, you have to point to Google’s SDK, but that happens only once and may be removed in future versions.) Hit run, and your sketch launches in the Emulator. Plug in an Android device via USB, and it’ll run on your device. Soon, you should have one-click-export of apps in the way that you do Processing desktop apps.
This also suggests some great possibilities for cheap, handheld devices, installations, and reliable portable devices for visual performance. Sure, right now a lot of Android-powered devices require a phone contract and don’t output video, but a lot of devices slated for later this year are media devices with video output and tablets, too.
Processing for Android is an early work-in-progress. If you’re thinking about playing with it and you don’t know what you’re doing, or you have little patience for bugs and wrinkles, you should wait until the waters grow safer. Right now, the team working on testing the port is intended to be limited, although as someone testing it myself, I can say it’s already a lot of fun and holds a lot of promise for the future. OpenGL support is currently not available, but it’s coming – and I’ll say again, if you want to know what Processing’s 3D future looks like, check out the superb GLGraphics library. Anyone who thinks Processing can’t be fast, or do intensive GPU work, or mix HD video (okay, not on Android, but on desktop at least), check out this library. OpenCL should also be possible soon.
BIG, BIG disclaimers! Please don’t go testing Processing on Android assuming it’ll work out of the box – for the brave ONLY, at least for now. (That should change very, very soon, but I couldn’t keep the news under my hat any longer.) As the disclaimer says (hilariously):
Do not use this code while operating heavy equipment. Do not rely on this code for thesis or diploma work, as you will not graduate. Do not use this code if you’re prone to whining about incomplete software that you download for free.
A number of us have already begun talking about the possibility of adding libraries to connect Processing’s capabilities to Google’s own APIs for the phone, SMS, sensors, and so on. Sound and synthesis via external libraries should also be possible.
I think Processing for Android is the perfect complement to openFrameworks for native code on mobile platforms. Right now, OF has already been used for terrific work on the iPhone. It’s not quite as user-friendly as Processing for Android for a number of reasons, but it has Processing-derived syntax, aesthetics, and philosophy, and it’s also free and open. For a superb guide to developing on the iPhone with OF, plus some links:
OF does not appear to be a practical solution for Android development, not least because Android requires apps to be distributed as Java, and uses Java to talk to all the native APIs. But it’s a good option for some of these other devices, and you can run the two alongside one another in Eclipse if you like, more advanced users.
With OF on the iPhone (and presumably other native platforms in future – Nokia N900, anyone?), and Processing for Android, plus tools like Pd and SuperCollider for synthesis (more on that soon), there’s no reason not to go completely open with portable interactive art on mobile devices.
