something in the way

If your music and sound collections seem like outwardly-expanding universes, two new tools promise to bring order by representing media as virtual planets and stars. One works on albums and tracks on the iPad; the other uses computer-aided analysis of loops and samples (not just music) on Windows. One will make your eyeballs pop; one might help you manage gigs of samples for a game design project.

Built in the open-source framework Cinder by an all-star team of media artist-designers (Ben Cerveny, Tom Carden, Jesper Sparre Andersen and Robert Hodgin), Planetary should satisfy space nuts and eye candy lovers. The metaphor is pretty direct: artists are stars, albums are planets around the artists, tracks are moons around the planets, and you can filter “constellations” by letter. That means the actual structure is heavily hierarchical, actually, in the tradition of iTunes (and, before it, its predecessor SoundJam). I’m not sure what happens with, say, compilations. But let’s face it: the real draw is that it’s incredibly beautiful to look at. I’d be just as entertained looking at a visualization of my system folder if it looked this pretty.

For now, Planetary is some fascinating eye candy with at least basic playback capabilities, iPad-only. That brings some good news – Airplay wireless works, and since it makes use of standard media code, even features like Last.fm scrobbles function. It also brings some bad — while Apple added support for libraries to third-party apps, Home Sharing isn’t included, so you’re limited to what’s on your iPad. Playlists aren’t supported, either. But hook this up to a projector or large screen TV with some of your favorite music, and I don’t think you’ll be complaining. And as a free tool, it’s incredible.

Planetary is available now; free for the iPad. As seen on creativeapplications.
http://planetary.bloom.io/
iTunes link

Less pretty, but with greater facilities on the utility side, is the Windows-only Soundtorch. (Thanks to Kristian Gohlke for the tip!) Visually, it offers a similar metaphor: media assets live on a continuous plane. Functionally, though, it’s more algorithmic than hierarchic, using something called the Computer Aided Sound Exploration engine (C.A.S.E.). The set of algorithms, which the creators say were based on evaluation of human listening, performs a sophisticated set of extractions of some 600 features from each sound file.

Rather than limit itself to albums and tracks, C.A.S.E. is tuned for audio files and loops. It’s fast enough that it can plow quickly through gigs of material. So, if you’re on Windows and have amassed an enormous collection of loops, samples, field recordings, sound effects, and the like, Soundtorch will use C.A.S.E. to first map all those relationship, then visualize them. You can use the mouse to produce new collections of assets, map relationships visually, export those relationship to XML, copy sounds to the clipboard, export to WAV, or open them in Windows Explorer. That is, all that eye candy is a genuine interface, not a barrier between you and what you might do (as so often happens with these sorts of experimental interfaces).

In fact, you might argue that, despite outward appearances, Soundtorch is entirely different from Planetary, but they share one common conceptual assumption. Related media “orbit” or attract to common materials. The difference is that Soundtorch is relational. In Soundtorch, if you “magnetize” a file, it – and any similar files – become attracted to attractors called “magnets.”

As is appropriate searching for media, the “torchlight” metaphor shines a light through files. Everything under the light plays back simultaneously, so you don’t have to audition sounds one at a time. (That sounds slightly terrifying to me, but I have to spend more time with it in an actual library.)

The creators describe the magic thusly:

Have you ever listened to a sound and felt that there was a similar one somewhere on your hard disk? And the sound you can’t find would just work so much better right now? Well, Soundtorch also remembers all sounds that you ever listened to. Just select any sound on Soundtorch, and let the system suggest the most similar ones from your whole collection.

In other words, SoundTorch is as much about what you can’t see as what you can – the intelligence to determine similarity behind the scenes. Check out the tech talk in the video above for more information on how “aurally and visually-enhanced audio search” could also apply this technology. More research at:
http://www.accessive-tools.com/

Soundtorch 2.0 entered a free public beta last week. It was developed in Microsoft’s C#-based XNA framework.

Grab the download:
http://soundtorch.com

Finally, if you want to hear the “Optimist” track by Zoe Keating without that voiceover and just enjoy Planetary’s gorgeous visuals, here you go:

From innovation in the visual interface to the intelligence underneath that changes how the computer interprets relationships between files, finally, there’s hope. Music and sound might not forever be trapped in views borrowed from spreadsheets, tables modeled on the needs of accountants 30 years ago.

Weekend fun, part 1: as Reactable makes the leap from custom, tangible interface to go-anywhere iPhone/iPad app, here’s a look at what the fingertip-controlled music creation experience is like. The first of a series of tutorials, the video above walks you through some basic music production. It should lay to rest any question about whether this kind of interface can work in musical performance or live arrangement. I still think some artists will want to bang on something rather than just gently finger-paint their way through music, but as arrangement tool, it’s intriguing – and this video makes clearer what the thing is about.

(Just a couple of quick posts by way of apology for breaking one of our two servers, causing an outage. My fault.)

Pick up a pen and draw a sketch. There, that was easy – however crude, you can get out an idea. Sketching with paper is still the fastest way for most of us to imagine something. But between that immediacy and the end result, you need prototypes.

The Processing language has long been one of the easiest ways to sketch an idea in code – best after you’ve first put pen to paper, but as an immediate next step (and for ideas you just can’t draw). Built in Java, the creation of Ben Fry and Casey Reas and a broad community of free software makers, it runs on Mac, Windows, and Linux. Via Processing.js, the same API works in a browser via JavaScript. It has inspired the OpenFrameworks project, which uses nearly the same API. Those tools let you intermingle Java, JavaScript, and C++, as they’re written natively in those languages.

Processing now runs just as easily on a mobile platform with Android. You can try it with the free SDK and emulator, but it’s most fun with a device. Using all free software, you can sketch easily on mobile and desktop from one environment, and with only minor modifications, run the same code on a desktop, a browser, and a mobile device.

Translation: with one, elegant API, you can “sketch” visual ideas on screens from an Android phone to a browser to a projection or installation. As a prototyping tool, or for finished projects, that makes it an exceptional, expressive tool for making interactive visuals for screens.

This is a first-draft tutorial, as I make the same presentation in Stockholm at the info-rich Android Only conference. I’m eager to put it out there and find out that people have problems, as I can then improve the documentation. So do give it a try – especially if you’ve got (or can borrow) an Android phone.

Processing on Android performs all of the core functions of Processing on desktop – 2D and 3D visuals, data manipulation, images, and type – and you can mix in Android code using the standard Android APIs, right in the same project. Processing controls the screen on which it’s drawing, for now, but you can mix and match everything else, to support multi-touch, sensors, and even NDK code. (I’m using it with Pd for Android, with Pd doing sound and Processing doing visuals.)

I’m assuming basic familiarity with Processing, so if you haven’t tried it out yet, check out the excellent tutorials available online to get rolling.

Install Processing and the Android SDK

Definitely read the latest official instructions:
http://wiki.processing.org/w/Android

1. Download the latest pre-release version of Processing. Eventually, the standard Processing download will support Android automatically out of the box – very cool. The current stable version doesn’t yet have Android support, however, so for now, you’ll need the latest pre-release build, which includes all the code for the testing release of Processing for desktop and Android. I tested with 0190.

2. Download the Android SDK. For now, you do need to download the SDK for Android separately, which is an extra step – but at least it’s completely free, and runs on any OS. See:
developer.android.com (This step will eventually go away.)

You can put the SDK anywhere you want; just make a note of where you’ve installed it, as you’ll need to point Processing to that location later. Follow the instructions carefully on Google’s site. On Windows, you’ll need to download the USB driver. On Linux, you’ll want to read the complete Developing on a Device instructions, as there are a few short commands you have to enter at the command line because of the way Linux talks to USB devices.

Lastly, even though that SDK is a big download, you’re not done until you download additional components from Google. This allows a single recent version of the SDK – like the 2.2 SDK I tested – to support a variety of older versions for backwards compatibility.

I have Eclipse installed, because it’s a handy tool for developing Processing for desktop and Processing for Android on any OS. Once you’ve installed the Eclipse plug-in for the Android SDK, you can add components from inside Eclipse. Inside that IDE, choose Window > Android SDK and AVD Manager Manager and you’ll get a graphical interface for adding these components (pictured).

You can now also access the same interface from within Processing. Choose Android > Android SDK and AVD Manager.

For Processing 0190, you need the SDK Platform 2.1, API 7 and Google APIs by Google, Android API 7. (If you’re an Android developer, you’ll probably also select some other components here, as well.) Components are available under the Available Packages.

If for some reason you prefer to use the command line, head to the Windows, Mac, or Linux command line, and follow the commands in the Quick Start under the Android SDK download instructions.

3. Run Processing, and switch to Android Mode. Load the Processing IDE you just installed, and switch to Android Mode.

The first time you do this, you’ll need to direct Processing to your Android SDK. Choose the root directory of the SDK – the directory that contains “add-ons,” “docs,” “platforms,” “tools,” and the like.

4. Try running a sketch on the emulator. I’m compiling a set of simple sketches you can use with Android, but even many of the basic examples will work out of the box. For fun, you can try sketches like those found in the Examples > Basics folder and some of the other included Processing Examples. They don’t take into account varying screen resolutions, which is a cool feature of Processing for Android, but it’s still fun to watch them run.

It’s simple to run a sketch. Open the sketch you want, go to Android > Android Mode to enable Android Mode for the sketch, and then hit Run. Instead of opening in a window as desktop Processing sketches do, you’ll see an Android emulator appear.

Be prepared for the emulator to take … some … time. You’ll spend a long time looking at the load screen. When it does finally load and install, there may still be a delay as you stare at the main screen. You may have to hit run again. You may see error messages saying an application has become unresponsive – don’t worry about that; Android has a low threshold for applications timing out, so while the emulator is loading, just choose “wait.” And because the emulator defaults to a large screen size, it’ll also take up some room on-screen.

It’s just slow, painful, slow, and unreliable. Fortunately, running on the device is near-instantaneous and speedy, so beg or borrow access to one and try the next step.

5. Try running a sketch on a device (if you’ve got one). Running on the emulator is a pain, but running on a device is very quick – even quicker than you might imagine if you’ve come from a background on iOS. (Remember, you’re working with Java, not native code.)

In a clever design feature, to run on the device instead of the emulator, simply choose Present instead of Run. (The quickest way is simply to shift-click the play button.)

So long as you have a device connected with USB, and USB debugging is switched on, and you’ve done the setup correctly above, you’ll see your sketch appear on the device.

Now, that’s more like it.

“But, wait,” you say. “Android devices have all different screen sizes. Augh, fragmentation, fragmentation!”

Fret not: computers have different screen sizes. Projectors have different screen sizes. And Processing on Android can easily adapt to the size of the screen. We won’t need much Android-specific code, but let’s get started with our own Processing sketch and see how it works when you’re preparing for mobile.

Try Writing Some Code

Okay, so what does a Processing sketch for Android look like? Here’s a really simple example, just to show off basic drawing, managing different screen sizes, and simple (single) touch. (I’ll leave density for another day; multitouch and other touch events are also possible.)

float sw, sh, touchX, touchY;

void setup() {
  size(screenWidth, screenHeight, A2D);
  println(screenWidth);
  println(screenHeight);
  sw = screenWidth;
  sh = screenHeight;
}

void draw() {
  background(0);
  smooth();
  fill(255);
  noStroke();
  ellipse(sw/2, sh/2, sw/4, sw/4);
  noFill();
  stroke(255,0,0);
  strokeWeight(2);
  ellipse(touchX, touchY, sw/4, sw/4);
}

void mouseDragged() {
  touchX = mouseX;
  touchY = mouseY;
}

Two things to note, in particular:

size(screenWidth, screenHeight, A2D);

(Warning: don’t put variables in that size() command.)

Processing for Android will map the desktop renderer names to either A2D (for 2D graphics) or A3D (for 3D). (I’m using the Android names here, but the other names will work just fine.)

Also, since static pixel sizes don’t make sense on mobile platforms with different screens, you can substitute “screenWidth” or “screenHeight” and get dimensions from the device on which you’re running. (I just used “sw” and “sh” because they’e shorter, and then I can make everything else relative.)

void mouseDragged() {
  touchX = mouseX;
  touchY = mouseY;
}

Here’s the really cool part — touch input works exactly the same way as it does on Processing on a desktop, for relevant mouse commands (nothing involving these strange “buttons” of which you speak). mousePressed, mouseDragged, and mouseX and mouseY all work as expected, so long as you’re satisfied with simple, single touch and don’t need gestures, multitouch, and the like. Don’t be confused by my touchX and touchY variables — those are just to demonstrate that my circle only moves when it’s “dragged” — or your finger slides across the screen. What we have is the beginnings of a drawing app, which I’ll be expanding over coming tutorials.

Here’s the other cool thing: modifying only the references to screenWidth and screenHeight, and the renderer, I can run exactly the same code in Processing on the desktop, or even in Processing.js on a browser.

Tune in Next Time…

This is not a replacement for the official wiki documentation, so do read that, please.

In coming installments, I’ll share:
1. How to get Processing for Android working inside Eclipse. In short – it works. That makes adding Processing to an existing Android app easy. (Hoping to follow that up with gedit and Ant)
2. More advanced tricks for dealing with screen dimensions and touch.
3. Android-native snippets for connecting sensors.
4. How you can combine sound and visuals with Pd and Processing, both running on Pd.

Other things that are possible to do with Processing for Android (which I’ll cover):

• Use install custom or installed fonts
• Data visualization (just as on the desktop version – making this much easier on mobile than in the past)
• Touch events, relative motion, and key events
• Force a specific orientation
• Add security requirements to the manifest (for things like sensors), right from the PDE app – no need to edit a file or go to Eclipse

I also hope to take some notes on what happens here at Android Only.

Processing for Android is still in development, and this is pre-release software. So be prepared to encounter issues. That said, I want to refine this documentation, so please let me know if you encounter issues with my instructions.

Presentation slides

Here’s my presentation from Stockholm:

Share what you’re doing…

Noisepages registration is open again as we finish the site. Share what you’re doing, devices you’re testing, or other thoughts:
http://noisepages.com/groups/processing/forum/

SenseWall (preliminary) from Tiago Serra on Vimeo.

Multi-touch walls have been a closely-guarded novelty, but they’re evolving into something else: a real, usable platform that focuses on content and not just gimmicks. In the process, a hard-working community is building richer, standards-based, cross-platform, free and open source tools. The result: faster iteration, broader access of artists to the technology, and soon, hopefully, better and better work.

Tiago Serra writes with his latest project, based on the Community Core Vision project. He includes a call for works that I think a number of CDM readers may want to check out.

I recently developed a multitouch wall called SenseWall, located at a Computer Science and Design university in Portugal.

In terms of hardware, the display has an area of 2.8m x 1.05m and it consists of 2 XGA ultra short-throw projectors amounting to a total resolution of 2048×768. For the multitouch sensing, this is an LLP setup using 8 infrared lasers, 2 PS3 Eye cameras and a custom compiled version of the excellent CCV tracker (ccv.nuigroup.com), giving us a touch resolution of 1280×480.

This installation is an open platform for anyone interested in doing something with it. Although it is a multitouch enabled surface it also has a pair of cameras at the top for computer vision applications, a microphone for sound input, speakers for sound output and an RFID reader (http://www.touchatag.com). So there’s lots one can do with it.

By open platform I mean that our intent is to have college students use it at will, being in class courses or just for fun.
A webapp is being developed by a few students so that anyone can upload their apps through a web interface making them available instantly on the SenseWall. This will be also open sourced once finished on GoogleCode.

At the SenseWall there’s an “app launcher” showing the available applications name, author and date.

My main purpose with this installation is to let pupils learn new HCI concepts and hilight their creativity by giving them the tool to do so. Only if I had something like it when in college

I really enjoy you’re work so I’d like to invite you to submit any TUIO (http://tuio.org) based app, or any other app that uses micro/vision/rfid, you’d like to see there, and I’ll be more than happy to send you a video of it.
I do feel that it is also a great opportunity to showcase any artist/designer/coder.

Everyone who actually programs in Flash knows that there is now full multi-touch and gesture support available. Unfortunately, some people still like to make claims to the contrary in an effort to make Flash appear behind the times. It is fortunately very easy to prove them wrong. Below is a great example built by StruckAxiom that showcases the new APIs that we have made available in Flash Player 10.1.

You can also read this great blog post that talks about some of the techniques used to create this touch experience.

Lee

Puddle of Life - Darwin Exhibit from Tiago Serra on Vimeo.

Here’s a really elegantly-designed multi-touch table. It uses two really key pieces of open source technology: the Community Core Vision (“CCV”) tracker, formerly known as tbeta, and a lovely framework for coding visuals called PyMT.

PyMT, as the name implies, is a Python-based framework. I’ve gotten to know Nathanaël Lécaudé, a talented artist and coder who was nice enough to put me up a couple of nights while I was in Montreal; he’s one of several core coders. They’re doing a lot to really encapsulate functionality in widgets in a nice way. Features of PyMT include an event framework, specialized widgets for gesture, touch, and layout, and connections to OpenGL, OpenGL shaders, and sound. You can even work with the enduring, evergreen synthesis language Csound using its Python bridge, the oddly-named but powerful Ounk.

Of course, that’s all plumbing. It’s nice to see this applied to something powerful and educational. From the Vimeo description:

"Puddle of Life" is an educational multi-touch installation that was designed for Coimbra’s Science Museum (Darwin exhibit), demonstrating the theory of natural selection, as part of Darwin’s 200th anniversary.

The installation is composed of a round multi-touch table whose surface represents a virtual environment where 4 species of creatures live in. Each of these little creatures have different physical characteristics visually expressed by it’s quantity and/or size: Vision, Locomotion and Fur.

The player interacts with the game by touching the surface. He has to maintain full awareness of his creatures emotions (visually represented by a cartoon like ballon) and using this information to properly choose the mating partners on the control console. This console also allows the user to select the most appropriate descendent from 4 possible mutations, resulting from the reproduction.

The player’s objective is to help the species he controls achieve the highest number of creatures of its kind. Since this world suffers from climate change the player must assure that his creatures are well adapted to this ever-changing environment by choosing the best balance between mating partners and descendants.

The species move in a swarm, but they split when the predator is near. When a creature sees food it warns the nearby siblings and they all run towards it. Of course only the ones who have best locomotion reach it. The vision gives them the ability to see the predator sooner and the ability to see the food further ahead too. The fur is useful to them when the temperature is low but harmful when it’s hot, leading the creature to a shorter lifespan.

The player wins when they reach about 18 living creatures and loses when all of it’s creatures die from cold/hot, predation, famine or old age.

Photos: flickr.com/photos/tserra/sets/72157619458503007/with/3619335559/

Technical Specs:
LLP multi-touch table
CCV tracker - nuicode.com/projects/tbeta
pyMT framework - code.google.com/p/pymt/
Cython - cython.org/
Rabbyt sprite library for pyglet - matthewmarshall.org/projects/rabbyt/
Animation and artwork: Adobe’s AE, Ai and PS with Fasticon.com icons

The project is the work of Tiago Serra and his collaborators, who have also been PyMT contributors (see comments). Great work, gang. We’ll be watching for more.