something in the way

Today, we note the availability on Android of Control, a WebKit-based touch interface also on iOS.

For visualists and interactive designers, it’s worth paying attention to one feature in particular: dynamic interface creation. Perhaps biased by the musicians who have tended to embrace them, touch interfaces have tended to rely on the static layouts favored by physical knobs and faders. That’s arguably the worst of both worlds: you lose the tactile feedback of physical controls, but you don’t add any of the flexibility of a display.

Control is an open-source application rendered in HTML5, powered by JavaScript and JSON, so it’s capable of anything you can imagine. But Charlie Roberts has already demonstrated how a dynamic interface could work. Using OSC, you can make control layouts on the fly. That could lead to more sophisticated software integration for visual and musical performance, new chances for collaboration and live rigs, and the ability to make an interface on someone’s device in an interactive situation.

We saw the last of these scenarios in the case of the iOS app mrmr, developed by Eric Redlinger. As proof of concept, I and others put together a gallery show using mrmr, at which interactive pieces were able to build interfaces on-the-fly on user’s iPhones and iPads. With Control, those horizons expand, no longer constrained to individual proprietary UI widgets on one platform (like iOS), but cross-platform, Web-based, and dynamic.

The video above I think does a good job of scratching the surface of what’s possible. More on that here:
Control 1.3: Dynamic Interfaces, jQuery integration & more

But dynamic layouts could go in many, many directions. Since this is especially relevant to visual performance, perhaps in modes of interaction not really possible in music, I’d love to hear what readers imagine. And do try Charlie’s app, whether on iOS, Android, or both:
Control

– and if you’re really ambitious, have a look at the source!

For people coding for visuals, Processing just keeps getting better. And for people who aren’t … well, you might just want to give it a second look, as a growing global army of people who never fancied themselves coders suddenly start typing up new creations. A new release makes mobile development easier and corrects lots of bugs. But specifically of interest to readers here, powerful libraries for 3D help make Processing an intensive tool for creating visuals. With the aid of running of your GPU, they can also deliver eye-popping real-time performance not normally associated with Java.

Processing 1.5

In a surprise release, much of what’s coming in Processing 2.0 this summer is now available in a stable, general release of Processing 1.5. It lacks the new built-in renderer – OPENGL2 – but does incorporate new features for developing for Android and, via the third-party library GLGraphics, you get all sorts of new OpenGL goodies. (Note: Processing 1.5 = 0196. The previous pre-release version, 0195, is worth downloading if you want to play with the new OPENGL2 renderer and its examples. Both are on the download page, and they can be installed side-by-side.)

Most importantly, for anyone publishing to the Web, you need to download this version now and re-export sketches. Applets were seeming to run very slow in Chrome and Firefox 4. (While I vastly prefer the Java version for things like performance tools or installations, I also appreciate the Processing.js JavaScript fork for Web delivery – but this fix does make applets work pretty well.)

New to Processing 1.5:

• Tons of editor fixes in the PDE development environment
• An essential bug fix that corrects slow performance when exporting for the Web.
• Android development support – in preview builds, but now in a stable build. (I think this is still considered a pre-release feature, but it means you can run the stable build to try it out.) Now go make stuff for phones and tablets easily.

Processing 1.5 is a good start, but for me, the live visual workflow isn’t complete without two additional libraries, toxiclibs and GLGraphics. A new GLGraphics update improves integration with both toxiclibs and the new Processing release.

toxiclibs

toxiclibs represents over 270 classes in 7 libraries, some 25,000 lines of code, written by digital artist Karsten “toxi” Schmidt. There’s gobs of stuff in there. The most useful is a basic set of classes for things like geometries, meshes, and 3D vectors which you’d otherwise have to build from scratch, and which are generally built in fairly standard ways. With elegant math, Verlet physics, and color libraries, and wonderfully-usable API design, Karsten gives you all the essentials in a way that will inspire you to use them and make something truly original. (A full tour is probably a good subject for another post. Just ignore the toxi.audio packages; with the libpd crew, I’m working on something with Pd I think you’ll find more useful and stable.)

I had a discussion with one colleague who felt that, indeed, toxi’s libraries are so powerful that people are simply using it as a crutch. That may be true to an extent: people should prominently credit toxi’s work, and to do otherwise is plagiarism. But with proper credit, I feel that standing on the shoulders of someone else’s work can be a good thing. Digging through toxi’s libraries is like going to school for the sorts of math and geometry that you need to learn to understand 3D generation. For many of these classes, involving essential mathematics operations and 3D modeling, I’d have no idea where to start, would spend weeks or months writing something inefficient, and would come out with something that reinvented the wheel. A lot of the techniques themselves in those 27,000 lines of code weren’t developed by Karsten, either: it’s more like seeing the wisdom of a master teacher, assembled from a wide variety of sources and passed on.

In fact, this is a rant that I should probably invoke elsewhere, but to me received knowledge is the essence of any craft. Composers don’t invent new rules of harmony (well, at least, not tonal harmony). Engineers don’t work out the laws of physics from scratch each time they build something.

And most importantly, because all of this exists in code, you can read and modify anything you find. It’s a black box if you want it to be, but I very often dig directly into the code to understand how things work.

There’s a bunch of documentation and a showcase for great work:
http://toxiclibs.org/

GLGraphics

All of this, though, brings me to GLGraphics. As I’ve been saying for – gee, years now – GLGraphics is the future of Processing. Now, that’s come to pass: gifted developer Andres Colubri authored the new OPENGL2 renderer that is similarly based on native OpenGL calls, and now runs the Processing Android port and upcoming Processing 2.0.

GLGraphics is separate from OPENGL2, but it’s your best bet for work on the stable Processing 1.5 build. Version 0.95 adds compatibility for that latest release, and adds two essential other features:

• An example of how to integrate with toxiclibs
• GLSL shader support, which can in turn be used for complex mesh generation.

See yesterday’s blog post announcing the update:
Processing 1.5 / GLGraphics 0.95

And there should be still more coolness to talk about soon, at least for Mac users, with the availability of Syphon for Processing. Stay tuned on that.

I think I have to hide away in a hole and do nothing but code this weekend. Anyone want to hop on IRC or PiratePad and pass code snippets back and forth?

Mobile games are on slower hardware, similar to later 90′s computers so native is a great way to go for 3d and 2d game development because of this limitation at the current time and well into the next few years.  Take this time to learn you some native gamedev. andengine isn’t native directly as it is Java based but compiled with the Dalvik JIT virtual machine. Another way to go native on Android is the Android NDK which allows C and C++.

• andengine download at google code
• andengine examples at google code
• andengine blog
• Mercurial repository at google code
• cocos2d-android port of cocos2d-iphone with BSD license
• rokon android is another BSD licensed android game engine

The engine also has extensions that can be easily added and some great ones exist already.

SenseWall (preliminary) from Tiago Serra on Vimeo.

Multi-touch walls have been a closely-guarded novelty, but they’re evolving into something else: a real, usable platform that focuses on content and not just gimmicks. In the process, a hard-working community is building richer, standards-based, cross-platform, free and open source tools. The result: faster iteration, broader access of artists to the technology, and soon, hopefully, better and better work.

Tiago Serra writes with his latest project, based on the Community Core Vision project. He includes a call for works that I think a number of CDM readers may want to check out.

I recently developed a multitouch wall called SenseWall, located at a Computer Science and Design university in Portugal.

In terms of hardware, the display has an area of 2.8m x 1.05m and it consists of 2 XGA ultra short-throw projectors amounting to a total resolution of 2048×768. For the multitouch sensing, this is an LLP setup using 8 infrared lasers, 2 PS3 Eye cameras and a custom compiled version of the excellent CCV tracker (ccv.nuigroup.com), giving us a touch resolution of 1280×480.

This installation is an open platform for anyone interested in doing something with it. Although it is a multitouch enabled surface it also has a pair of cameras at the top for computer vision applications, a microphone for sound input, speakers for sound output and an RFID reader (http://www.touchatag.com). So there’s lots one can do with it.

By open platform I mean that our intent is to have college students use it at will, being in class courses or just for fun.
A webapp is being developed by a few students so that anyone can upload their apps through a web interface making them available instantly on the SenseWall. This will be also open sourced once finished on GoogleCode.

At the SenseWall there’s an “app launcher” showing the available applications name, author and date.

My main purpose with this installation is to let pupils learn new HCI concepts and hilight their creativity by giving them the tool to do so. Only if I had something like it when in college

I really enjoy you’re work so I’d like to invite you to submit any TUIO (http://tuio.org) based app, or any other app that uses micro/vision/rfid, you’d like to see there, and I’ll be more than happy to send you a video of it.
I do feel that it is also a great opportunity to showcase any artist/designer/coder.

You’ll hear odd cynicism about people working on free software / open source projects. Something like, “well, harumph, it’s not as though a bunch of people will make this stuff in their free time.”

Not only are these folks wrong, but you can actually visualize the contributions to source trees – and the results look spectacularly hypnotic. It’s free software – the music video.

Okay, now, granted, I may get so mesmerized by the results that I’ll just spend time staring at that instead of getting actual work done, but – working too hard isn’t good for you, anyway. It’s an organic high, audiovisuals.

At top, Ryadh Amar sends in a visualization of the excellent, lightweight LXDE windowing environment for Linux. (Actually, I’m inspired to give LXDE a fresh install.) At bottom, a collage of various projects showing that these data visualizations can take on various identities. Gource can support just about any project repository, too: Git, Bazaar (popular on Ubuntu), and Mercurial (recently added to Google Code, incidentally) are available native, and CVS and SVN are available as third-party extensions for those of you kickin’ it oldskool and non-distributed. (Though, really, come join the 21st Century – it’s awesome.)

And Gource can visualize itself. Freaky. It’s all thanks to the ongoing awesomeness of OpenGL.

http://code.google.com/p/gource/

I’d love to see this added to project management so you’d have a sort of live, superb visual to inspire you to keep the code moving forward.

Who knew source code would turn out to be so visually inspiring? (Now I just need a new way of visualizing me writing bad code and then correcting and cleaning it up. I think it could be best represented as a set of stick figures getting stuck in quicksand and hitting each other over the head. Then there could be a big Smoke Monster that represented the Evil Force of Procrastination.)

But wait! There’s more! You can visualize web logs, too. (It works with Apache; I have to see if I can make it work with our nginx logs, as visualizers could actually be very beneficial with the kind of complex data you get in something like a web log):

Boards Interactive Magazine – Walkthrough from Theo Watson on Vimeo.

Augmented reality has inspired plenty of experiments, magazines included – some successful, some failing to get far beyond the gimmick. What’s nice about this work is that it introduces the concept of motion to the typically-static pages of print, and inspires readers to imagine a world beyond the bounds of the page. There’s also an elegant expression of the theme.

It’s all made with OpenFrameworks (again – yes, OF is one of the easiest ways to hook into augmented reality). And none other than Theo Watson was involved.

For the March 2010 issue of Boards Magazine, Emily Gobeille and I worked with Nexus Productions to develop an interactive cover experience called Rise and Fall. Here is a little preview of the experience.

You can download the software and the cover from: boardsmag.com/RiseAndFall

Update: Found out you can buy a copy of the magzine for $7 by emailing – BoardsCustomerCare@boardsmag.com . You can also download the cover as a pdf from the link above.

The project uses the Ferns library for tracking ( cvlab.epfl.ch/software/ferns/index.php ) and the whole project is open source released under the GPL v2.0 . Grab the source code here: boardsmag.com/RiseAndFall

Credits:

Digital Directors:
Emily Gobeille – zanyparade.com
Theo Watson – theowatson.com

Produced by:
Nexus Productions – nexusproductions.com

Sound Design:
MOST Original Soundtracks – m-ost.nl

Software:
Made with openFrameworks – openframeworks.cc
Using the Ferns library for tracking – cvlab.epfl.ch/software/ferns/index.php

Thanks to @wetterberg via Twitter for sending this our way.

Google’s open-source Android mobile platform runs on Java. Processing, the elegant coding language for visualization, art, and media, is built in Java. The marriage of the two, therefore, is one we’ve long been anticipating. Processing’s ability to focus on lightweight, portable implementation is a perfect match for the demands of mobile development. For artists and visualists wanting to make handheld devices and tablets more expressive, I have good news: it’s coming. While not fully implemented or ready for widespread testing yet, Processing is in active development on the platform.

Ben Fry, co-creator of Processing, has been hard at work making Processing run on Android. Interestingly, in some ways Android itself vindicates the direction Processing charted years ago. Google made their platform work based on open-source development tools. They removed parts of the Java platform that would have overwhelmed the limited processing power of handhelds – even as a handful of purist Java developers cried foul. Graphics focuses on lightweight 2D and OpenGL drawing output, much in the same way that Processing strips Java to its bare essentials.

Best of all, doing Processing development for Android promises the same streamlined, sketchbook-like production flow that Processing does on desktop. To create Android apps, you simply set Processing to “Android mode.” (Right now, you have to point to Google’s SDK, but that happens only once and may be removed in future versions.) Hit run, and your sketch launches in the Emulator. Plug in an Android device via USB, and it’ll run on your device. Soon, you should have one-click-export of apps in the way that you do Processing desktop apps.

This also suggests some great possibilities for cheap, handheld devices, installations, and reliable portable devices for visual performance. Sure, right now a lot of Android-powered devices require a phone contract and don’t output video, but a lot of devices slated for later this year are media devices with video output and tablets, too.

Processing for Android is an early work-in-progress. If you’re thinking about playing with it and you don’t know what you’re doing, or you have little patience for bugs and wrinkles, you should wait until the waters grow safer. Right now, the team working on testing the port is intended to be limited, although as someone testing it myself, I can say it’s already a lot of fun and holds a lot of promise for the future. OpenGL support is currently not available, but it’s coming – and I’ll say again, if you want to know what Processing’s 3D future looks like, check out the superb GLGraphics library. Anyone who thinks Processing can’t be fast, or do intensive GPU work, or mix HD video (okay, not on Android, but on desktop at least), check out this library. OpenCL should also be possible soon.

http://android.processing.org/

BIG, BIG disclaimers! Please don’t go testing Processing on Android assuming it’ll work out of the box – for the brave ONLY, at least for now. (That should change very, very soon, but I couldn’t keep the news under my hat any longer.) As the disclaimer says (hilariously):

Do not use this code while operating heavy equipment. Do not rely on this code for thesis or diploma work, as you will not graduate. Do not use this code if you’re prone to whining about incomplete software that you download for free.

A number of us have already begun talking about the possibility of adding libraries to connect Processing’s capabilities to Google’s own APIs for the phone, SMS, sensors, and so on. Sound and synthesis via external libraries should also be possible.

I think Processing for Android is the perfect complement to openFrameworks for native code on mobile platforms. Right now, OF has already been used for terrific work on the iPhone. It’s not quite as user-friendly as Processing for Android for a number of reasons, but it has Processing-derived syntax, aesthetics, and philosophy, and it’s also free and open. For a superb guide to developing on the iPhone with OF, plus some links:

http://www.openframeworks.cc/setup/iphone

OF does not appear to be a practical solution for Android development, not least because Android requires apps to be distributed as Java, and uses Java to talk to all the native APIs. But it’s a good option for some of these other devices, and you can run the two alongside one another in Eclipse if you like, more advanced users.

With OF on the iPhone (and presumably other native platforms in future – Nokia N900, anyone?), and Processing for Android, plus tools like Pd and SuperCollider for synthesis (more on that soon), there’s no reason not to go completely open with portable interactive art on mobile devices.

Recently I've been playing a lot with OpenCL, the new API / framework designed to handle cross-platform parallel computing (i.e. a simple way of running code simultaneously on all cores of your CPU, GPU or other processors). Implementations have been cropping up this year in NVidia drivers or ATI drivers, but most famously it's included with Mac OSX 10.6 Snow Leopard.

To cut a long story short I've been working on a simple-to-use C++ wrapper for some of the most common functions, imaginatively called ofxOpenCL and here is a little demo of 1 million particles running at 100-200fps.

NOTE: The Vimeo compression destroys most of the particles, so I suggest downloading the quicktime directly from the vimeo page at

Frame-rate is around 100-200fps running on a macbook pro with GF 9600GT. That's 100-200fps on a laptop! (albeit a pretty decent one), but I'm dying to try this on a GF 285 GTX - which has 7.5x the number of cores, 2.5x the fillrate and 3.5x the memory bandwidth - for only £250!!

The kernel for this is surprisingly simple:

__kernel void updateParticleWithoutCollision(__global Particle* pIn, __global float2* pOut, const float2 mousePos, const float2 dimensions){
        int id = get_global_id(0);
        __global Particle *p = &pIn[id];
 
        float2 diff = mousePos - pOut[id];
        float invDistSQ = 1.0f / dot(diff, diff);
        diff *= 300.0f * invDistSQ;
 
        p->vel += (dimensions*0.5 - pOut[id]) * CENTER_FORCE2 - diff* p->mass;
        pOut[id] += p->vel;
        p->vel *= DAMP2;
 
        float speed2 = dot(p->vel, p->vel);
        if(speed2<MIN_SPEED2) pOut[id] = mousePos + diff * (1 + p->mass);
}

This example is based on Rui's opencl example at .

Discussion on the matter at http://www.openframeworks.cc/forum/viewtopic.php?f=10&t=2728&p=15107#p15...

source code for ofxOpenCL and the above example at
http://code.google.com/p/ofxmsaof/downloads/list
(the SVN is likely to be more recent).

AIR is very popular for creating twitter clients, Sönke Rohde just made it much easier to make AIR apps for twitter with an AS3 library for Twitter. This library is built on top of core oauth as3 library by iotashan.  Core OAuth as3 library is a standard OAuth library this can be used for your own OAuth backends or connecting to other OAuth services as well.

A very nice feature of this library, in addition to being coded cleanly and as3 style, is the ability to have the Twitter OAuth page render inside of Flash.

Instead of opening the Twitter authorization page in the browser the library also contains OAuthLoader which is a wrapper around HTMLLoader which enables to directly show the authorization page within an AIR window:

// use this in the requestTokenHandler instead of navigateToURL
var loader:OAuthLoader = new OAuthLoader();
loader.load(request);
loader.percentWidth = 100;
loader.percentHeight = 100;
var w:Window = new Window();
w.width = 800;
w.height = 400;
w.title = req.url;
w.addChild(loader);
w.open();

• Shannon Hicks core oauth as3 library
• High level AS3/Flex library for OAuth
• oauth-as3 Google Code page
• OAuth
• Twitter OAuth FAQ
• Twitter API Documentation
• Getting Started with OAuth

You’ve seen the demos. You like the idea of tracking tags in the real world to create visuals. And now you want to try augmented reality for yourself - and, incidentally, you’re a Flash developer.

Reader Eric Socolofsky writes to share a framework he’s created that makes it much easier to work with the Flash-based, open source FLARToolkit, called FLARManager. Version 0.4 is just released:

http://words.transmote.com/wp/20090618/flarmanager-v04/

FLARManager has a number of features that improve upon the existing work done by FLARToolkit:

• Building the apps themselves is easier. Fire up the framework with Flex Builder (or Flash, or Eclipse, or FlashDevelop), and you have access to all the libraries you need, so you can start playing more or less out of the box. Hello, world, indeed.
• You don’t have to rely on Papervision if you don’t want to. Papervision, the faux-3D library for Flash, is included with the distribution. But marker tracking is decoupled from Papervision, so you don’t have to use it if you don’t need it.
• Better event management. Marker adding, updating, and removal, multiple pattern detection and management, and the like are all extended in FLARManager.
• Great documentation. Eric has taken the time to read some fantastic getting started tutorials, all accessible from the site above so you can go play.

Now, you wouldn’t pick Flash for speed - that’s not the idea.

This is about the slowest implementation of ARToolkit you’ll find. But you’d use it for compatibility, because of easy deployment to the browser. Speaking of speed, the NyARToolkit Java implementation actually outperforms the original C version. I’m the last person you should talk to about writing efficient, optimized code, but I can tell you that the notion that Java is “slower” than C is simply wrong. There are a great many other, more important variables, and in some cases Java can in fact outperform C. That doesn’t mean that Java is always the right tool for the job any more than C is, though, and in fact because Java’s Garbage Collector and event scheduling aren’t really built for real-time performance, and because “native” code is suited to certain situations, there are in fact times when you wouldn’t want to use Java. Understanding the application is what really matters - and that’s why it’s nice that NyARToolkit and siblings are available for AS3, C#, the mobile Android platform, iPhone, and others.

Eric doesn’t just do this to fool around, either; he works in interactive design for museums, and has what has to be one of the world’s sweetest “day jobs” - working for the legendary Exploratorium. He’s also working with Processing and the wonderful reacTIVision library on a separate project that’s working with tangible table-based interfaces; more on that separately. Thanks, Eric!

If you get interesting work in augmented reality going, let us know. And if you need some inspiration, my current favorite is from our friend Marco Tempest, working with Zach Lieberman in OpenFrameworks on augmented magic.