something in the way

Today, we note the availability on Android of Control, a WebKit-based touch interface also on iOS.

For visualists and interactive designers, it’s worth paying attention to one feature in particular: dynamic interface creation. Perhaps biased by the musicians who have tended to embrace them, touch interfaces have tended to rely on the static layouts favored by physical knobs and faders. That’s arguably the worst of both worlds: you lose the tactile feedback of physical controls, but you don’t add any of the flexibility of a display.

Control is an open-source application rendered in HTML5, powered by JavaScript and JSON, so it’s capable of anything you can imagine. But Charlie Roberts has already demonstrated how a dynamic interface could work. Using OSC, you can make control layouts on the fly. That could lead to more sophisticated software integration for visual and musical performance, new chances for collaboration and live rigs, and the ability to make an interface on someone’s device in an interactive situation.

We saw the last of these scenarios in the case of the iOS app mrmr, developed by Eric Redlinger. As proof of concept, I and others put together a gallery show using mrmr, at which interactive pieces were able to build interfaces on-the-fly on user’s iPhones and iPads. With Control, those horizons expand, no longer constrained to individual proprietary UI widgets on one platform (like iOS), but cross-platform, Web-based, and dynamic.

The video above I think does a good job of scratching the surface of what’s possible. More on that here:
Control 1.3: Dynamic Interfaces, jQuery integration & more

But dynamic layouts could go in many, many directions. Since this is especially relevant to visual performance, perhaps in modes of interaction not really possible in music, I’d love to hear what readers imagine. And do try Charlie’s app, whether on iOS, Android, or both:
Control

– and if you’re really ambitious, have a look at the source!

Following yesterday’s interview with Kyle McDonald on FaceOSC, his custom webcam + tracking application that can make music with your face, here’s another face-controlled music demo. This one uses Ableton Live for jamming. I should add, since I somewhat obscured the fact, that this isn’t Kinect: it works entirely with a built-in webcam, which means it’s completely free to try and you don’t have to tote any extra hardware, so long as you have a laptop with a built-in cam. More on this technology as we watch it evolve…

For people coding for visuals, Processing just keeps getting better. And for people who aren’t … well, you might just want to give it a second look, as a growing global army of people who never fancied themselves coders suddenly start typing up new creations. A new release makes mobile development easier and corrects lots of bugs. But specifically of interest to readers here, powerful libraries for 3D help make Processing an intensive tool for creating visuals. With the aid of running of your GPU, they can also deliver eye-popping real-time performance not normally associated with Java.

Processing 1.5

In a surprise release, much of what’s coming in Processing 2.0 this summer is now available in a stable, general release of Processing 1.5. It lacks the new built-in renderer – OPENGL2 – but does incorporate new features for developing for Android and, via the third-party library GLGraphics, you get all sorts of new OpenGL goodies. (Note: Processing 1.5 = 0196. The previous pre-release version, 0195, is worth downloading if you want to play with the new OPENGL2 renderer and its examples. Both are on the download page, and they can be installed side-by-side.)

Most importantly, for anyone publishing to the Web, you need to download this version now and re-export sketches. Applets were seeming to run very slow in Chrome and Firefox 4. (While I vastly prefer the Java version for things like performance tools or installations, I also appreciate the Processing.js JavaScript fork for Web delivery – but this fix does make applets work pretty well.)

New to Processing 1.5:

• Tons of editor fixes in the PDE development environment
• An essential bug fix that corrects slow performance when exporting for the Web.
• Android development support – in preview builds, but now in a stable build. (I think this is still considered a pre-release feature, but it means you can run the stable build to try it out.) Now go make stuff for phones and tablets easily.

Processing 1.5 is a good start, but for me, the live visual workflow isn’t complete without two additional libraries, toxiclibs and GLGraphics. A new GLGraphics update improves integration with both toxiclibs and the new Processing release.

toxiclibs

toxiclibs represents over 270 classes in 7 libraries, some 25,000 lines of code, written by digital artist Karsten “toxi” Schmidt. There’s gobs of stuff in there. The most useful is a basic set of classes for things like geometries, meshes, and 3D vectors which you’d otherwise have to build from scratch, and which are generally built in fairly standard ways. With elegant math, Verlet physics, and color libraries, and wonderfully-usable API design, Karsten gives you all the essentials in a way that will inspire you to use them and make something truly original. (A full tour is probably a good subject for another post. Just ignore the toxi.audio packages; with the libpd crew, I’m working on something with Pd I think you’ll find more useful and stable.)

I had a discussion with one colleague who felt that, indeed, toxi’s libraries are so powerful that people are simply using it as a crutch. That may be true to an extent: people should prominently credit toxi’s work, and to do otherwise is plagiarism. But with proper credit, I feel that standing on the shoulders of someone else’s work can be a good thing. Digging through toxi’s libraries is like going to school for the sorts of math and geometry that you need to learn to understand 3D generation. For many of these classes, involving essential mathematics operations and 3D modeling, I’d have no idea where to start, would spend weeks or months writing something inefficient, and would come out with something that reinvented the wheel. A lot of the techniques themselves in those 27,000 lines of code weren’t developed by Karsten, either: it’s more like seeing the wisdom of a master teacher, assembled from a wide variety of sources and passed on.

In fact, this is a rant that I should probably invoke elsewhere, but to me received knowledge is the essence of any craft. Composers don’t invent new rules of harmony (well, at least, not tonal harmony). Engineers don’t work out the laws of physics from scratch each time they build something.

And most importantly, because all of this exists in code, you can read and modify anything you find. It’s a black box if you want it to be, but I very often dig directly into the code to understand how things work.

There’s a bunch of documentation and a showcase for great work:
http://toxiclibs.org/

GLGraphics

All of this, though, brings me to GLGraphics. As I’ve been saying for – gee, years now – GLGraphics is the future of Processing. Now, that’s come to pass: gifted developer Andres Colubri authored the new OPENGL2 renderer that is similarly based on native OpenGL calls, and now runs the Processing Android port and upcoming Processing 2.0.

GLGraphics is separate from OPENGL2, but it’s your best bet for work on the stable Processing 1.5 build. Version 0.95 adds compatibility for that latest release, and adds two essential other features:

• An example of how to integrate with toxiclibs
• GLSL shader support, which can in turn be used for complex mesh generation.

See yesterday’s blog post announcing the update:
Processing 1.5 / GLGraphics 0.95

And there should be still more coolness to talk about soon, at least for Mac users, with the availability of Syphon for Processing. Stay tuned on that.

I think I have to hide away in a hole and do nothing but code this weekend. Anyone want to hop on IRC or PiratePad and pass code snippets back and forth?

SenseWall (preliminary) from Tiago Serra on Vimeo.

Multi-touch walls have been a closely-guarded novelty, but they’re evolving into something else: a real, usable platform that focuses on content and not just gimmicks. In the process, a hard-working community is building richer, standards-based, cross-platform, free and open source tools. The result: faster iteration, broader access of artists to the technology, and soon, hopefully, better and better work.

Tiago Serra writes with his latest project, based on the Community Core Vision project. He includes a call for works that I think a number of CDM readers may want to check out.

I recently developed a multitouch wall called SenseWall, located at a Computer Science and Design university in Portugal.

In terms of hardware, the display has an area of 2.8m x 1.05m and it consists of 2 XGA ultra short-throw projectors amounting to a total resolution of 2048×768. For the multitouch sensing, this is an LLP setup using 8 infrared lasers, 2 PS3 Eye cameras and a custom compiled version of the excellent CCV tracker (ccv.nuigroup.com), giving us a touch resolution of 1280×480.

This installation is an open platform for anyone interested in doing something with it. Although it is a multitouch enabled surface it also has a pair of cameras at the top for computer vision applications, a microphone for sound input, speakers for sound output and an RFID reader (http://www.touchatag.com). So there’s lots one can do with it.

By open platform I mean that our intent is to have college students use it at will, being in class courses or just for fun.
A webapp is being developed by a few students so that anyone can upload their apps through a web interface making them available instantly on the SenseWall. This will be also open sourced once finished on GoogleCode.

At the SenseWall there’s an “app launcher” showing the available applications name, author and date.

My main purpose with this installation is to let pupils learn new HCI concepts and hilight their creativity by giving them the tool to do so. Only if I had something like it when in college

I really enjoy you’re work so I’d like to invite you to submit any TUIO (http://tuio.org) based app, or any other app that uses micro/vision/rfid, you’d like to see there, and I’ll be more than happy to send you a video of it.
I do feel that it is also a great opportunity to showcase any artist/designer/coder.

You’ve seen the demos. You like the idea of tracking tags in the real world to create visuals. And now you want to try augmented reality for yourself - and, incidentally, you’re a Flash developer.

Reader Eric Socolofsky writes to share a framework he’s created that makes it much easier to work with the Flash-based, open source FLARToolkit, called FLARManager. Version 0.4 is just released:

http://words.transmote.com/wp/20090618/flarmanager-v04/

FLARManager has a number of features that improve upon the existing work done by FLARToolkit:

• Building the apps themselves is easier. Fire up the framework with Flex Builder (or Flash, or Eclipse, or FlashDevelop), and you have access to all the libraries you need, so you can start playing more or less out of the box. Hello, world, indeed.
• You don’t have to rely on Papervision if you don’t want to. Papervision, the faux-3D library for Flash, is included with the distribution. But marker tracking is decoupled from Papervision, so you don’t have to use it if you don’t need it.
• Better event management. Marker adding, updating, and removal, multiple pattern detection and management, and the like are all extended in FLARManager.
• Great documentation. Eric has taken the time to read some fantastic getting started tutorials, all accessible from the site above so you can go play.

Now, you wouldn’t pick Flash for speed - that’s not the idea.

This is about the slowest implementation of ARToolkit you’ll find. But you’d use it for compatibility, because of easy deployment to the browser. Speaking of speed, the NyARToolkit Java implementation actually outperforms the original C version. I’m the last person you should talk to about writing efficient, optimized code, but I can tell you that the notion that Java is “slower” than C is simply wrong. There are a great many other, more important variables, and in some cases Java can in fact outperform C. That doesn’t mean that Java is always the right tool for the job any more than C is, though, and in fact because Java’s Garbage Collector and event scheduling aren’t really built for real-time performance, and because “native” code is suited to certain situations, there are in fact times when you wouldn’t want to use Java. Understanding the application is what really matters - and that’s why it’s nice that NyARToolkit and siblings are available for AS3, C#, the mobile Android platform, iPhone, and others.

Eric doesn’t just do this to fool around, either; he works in interactive design for museums, and has what has to be one of the world’s sweetest “day jobs” - working for the legendary Exploratorium. He’s also working with Processing and the wonderful reacTIVision library on a separate project that’s working with tangible table-based interfaces; more on that separately. Thanks, Eric!

If you get interesting work in augmented reality going, let us know. And if you need some inspiration, my current favorite is from our friend Marco Tempest, working with Zach Lieberman in OpenFrameworks on augmented magic.

Interface 27 from CyberPatrolUnit on Vimeo.

There has been a long tradition in live visuals and motion graphics, inherited from many other media, of maintaining a “secret sauce,” or the guarded formula of eleven herbs and spices. Ironically, for all you hear today “DIY” and “open source” in the same sentence, a lot of the motivation for doing something yourself has historically been doing something no one else can. Keep your secrets, and raise your value.

As our friend Bryant Place / CyberPatrolUnit sends over this latest set of live clips from a recent gig, and I browse through the comments, and reflect on the conversations I had last week at OFFF and during and following my own talk there, though, I’m struck.

The world has changed. First off, the Internet isn’t really about secrets. Your value is almost in direct proportion to how much you can share. Connections are forged through links of mutual exchange and good will. It’s not just about sharing your output or getting fans (the MySpace model), but sharing with a network of enthusiasts, and fellow artists. Those are the people from whom you often get real support (artistic, technical, and personal), gigs – and inspiration. (Even if you hate 8-bit music, that community is a really amazing model: their work to support each other and advocate for the whole subgenre has been I think the single biggest ingredient in their viral success.)

The visualist community increasingly itches not only to improve the quality of their own individual work, but everyone around them. A lot of us are in a battle for the future of this whole medium. Some parts of the world are devoid of live visuals, while others have mass-produced club visuals filling the nightlife.

Before I get carried away, the video itself is just the latest from the ongoing Interface 27 series. It employs a touch interface to control abstract visual pictures formed from streams of particles.

The reason I’m pulling back into the larger question is that these visuals are enabled by a library for Processing, a library we’ve seen here previously, developed by Memo Atken:

MSAFluid for processing (and Java)

If you’d rather use openFrameworks, there’s that version, too, as pictured below running blazingly fast:

ofxMSAFluid for openFrameworks

There’s even an ActionScript 3 port, in case you want to code Flash on the beach.

ofxMSAFluid for openFrameworks from Memo Akten on Vimeo.

So, why do I bring this up? Well, the work done on Processing (Ben Fry, Casey Reas, contributors like Karsten Schmidt, and others), on openFrameworks (Zachary Lieberman, Theo Watson, and their own team), and Memo’s own library, based in turn on many other libraries and implementations, was all a big risk.

It’s not an easy thing to put blood, sweat, and tears into open source. None of those people has exactly gotten rich in the process – not even via the ways you’re supposed to profit from open source, doing the lecture circuit and such. But on the other hand, we’re seeing things that would have been otherwise impossible.

And there’s artistic merit, too. Bryant’s work looks different than Memo’s. The library actually takes on a new life as it gets in someone else’s hands. Bryant actually just wrote me:

As for the Interface video - mention how cool it is that people like Memo post code for other VJ’s to tweak and use.  Mention "FaderTouch" - a 100buk touchscreen off ebay that "vjFader" programmed - using a rear projection onto a translucent screen/ touch sensor we were able to use processing in a very intuitive way.

I got the “mention” part down, I guess. ;)

The responsibility is partly ours to make all of this work: file bug reports, fix bugs if you can, document your work, properly credit the people making it, write documentation for projects, and so on. But it’s not hard to see an ideal start to happen:

1. Person x makes a library / framework.

2. Person y build on that library to make their own tool – and contributes it.

3. Artist uses the tool, gives back to the project, goes in a new direction.

4. More and better work spreads, the project grows, the medium grows, and the audience grows.

None of this happens automatically. We all have a lot more work to do. But having stood onstage in front of a few thousand people calling for just this, it’s nice to keep opening my inbox and seeing it happening. We’re seeing the first seeds planted for what could ultimately be a larger ecosystem. Now, I know there’s also a big gap left – Processing doesn’t have nearly enough contributors, bug squashers, or documenters, and it’s one of the biggest projects, so you can imagine what happens when you get upstream to libraries and the like.

Over the coming months, I think we’ll continue to look for opportunities to help structure some of that involvement and to explaining how you can contribute, too. Stay tuned.

In the meantime, go play with some particles.

For more on Bryant, here he is on his current activities:

- I just did Coachella with [Friend of CDM and contributor] Momo, and in the near future, will be heading to Detroit for http://www.myspace.com/detroitmusicfest

I’m not on the website, however, Kero.fm and Derek Michael - two people who essentially helped build the festival from the ground up 10 years ago - are booking me to play with various acts including CLP, Richard Devine, Drumcell, Busy P (which I did a solo VJ set with at Coachella) so I am super excited to be a part for the first time this year.

Here is a cool video from previous Interface 26:

After Detroit - Mutek.

http://www.mutek.org/

There are also some killer podcasts from past Mutek - http://www.mutek.org/podcast

I am going to meet artists, see the latest AV performances, attend workshops.

I’ll be at Mutek, too, so see you there.