something in the way

Whew!

It’s been a massively intense few weeks getting my PixelPhones project ready in time for my presentation at FOTB. I’ve been coding day and night for months, and I was up until 4am the night before making sure that I’d thought of everything.

The worst part was that I had no idea if it would work – it’s kinda difficult to get hundreds of phones together to test it. But it did work – and the results were better than I could have hoped for!

thanks to Momo for the amazing video

PixelPhones is this crazy project where I turn each phone in the audience into pixels in a large display. It runs in your phone browser so you don’t need to install an app.

Built with openFrameworks in C++, the app sets up its own WebSocket server and phones connect straight into it over the local wifi network (although in future I hope to get this working over 3G).

Of course socket connections are two way so I couldn’t help but take this chance to involve the audience too – I made a game where Nyan Cat runs from screen to screen, and whoever catches him fastest wins! (partly inspired by Rob Davis’ Cat on Yer Head game)

The tricky bit was figuring out where the phones are – GPS just isn’t accurate enough. I was inspired by the Junkyard Jumbotron – it puts markers on each phone, then you take a picture of them all. But in my app a marker would be way too small so instead I use a unique sequence of flashes.

It uses OpenCV to find the flashing objects, read the pixels, and decode the sequence.

Latency was also an issue, even on the fastest networks the packets arrive at slightly different times. And when you want super fast smooth animation, unpredictable timing is unacceptable. I solved it thanks to Jobe Makar’s excellent method of syncronisation outlined in his ActionScript multi-player gaming book.

I’m so happy and relieved that it worked so well and the timing was split-second perfect. The next day I put myself through the whole ordeal again, this time with a larger audience and only 10 minutes to demo (at the FOTB Jam session). Over 220 devices connected and I could have handled more but we ran out of time.

This is just the start, and I have many many ideas that I have yet to try. I’ll also be open sourcing this code when it’s finished – I can’t wait to see what the rest of you do with it! Both sessions were recorded in full, I’ll let you know when they’re uploaded.

No related posts.

Here is a little game that Nick and I built using the iPhone and Android as the game controller. The characters are controlled using your finger gestures. You need to eat as many berries as possible while avoiding the blocks.

Boards Interactive Magazine – Walkthrough from Theo Watson on Vimeo.

Augmented reality has inspired plenty of experiments, magazines included – some successful, some failing to get far beyond the gimmick. What’s nice about this work is that it introduces the concept of motion to the typically-static pages of print, and inspires readers to imagine a world beyond the bounds of the page. There’s also an elegant expression of the theme.

It’s all made with OpenFrameworks (again – yes, OF is one of the easiest ways to hook into augmented reality). And none other than Theo Watson was involved.

For the March 2010 issue of Boards Magazine, Emily Gobeille and I worked with Nexus Productions to develop an interactive cover experience called Rise and Fall. Here is a little preview of the experience.

You can download the software and the cover from: boardsmag.com/RiseAndFall

Update: Found out you can buy a copy of the magzine for $7 by emailing – BoardsCustomerCare@boardsmag.com . You can also download the cover as a pdf from the link above.

The project uses the Ferns library for tracking ( cvlab.epfl.ch/software/ferns/index.php ) and the whole project is open source released under the GPL v2.0 . Grab the source code here: boardsmag.com/RiseAndFall

Credits:

Digital Directors:
Emily Gobeille – zanyparade.com
Theo Watson – theowatson.com

Produced by:
Nexus Productions – nexusproductions.com

Sound Design:
MOST Original Soundtracks – m-ost.nl

Software:
Made with openFrameworks – openframeworks.cc
Using the Ferns library for tracking – cvlab.epfl.ch/software/ferns/index.php

Thanks to @wetterberg via Twitter for sending this our way.

Recently I've been playing a lot with OpenCL, the new API / framework designed to handle cross-platform parallel computing (i.e. a simple way of running code simultaneously on all cores of your CPU, GPU or other processors). Implementations have been cropping up this year in NVidia drivers or ATI drivers, but most famously it's included with Mac OSX 10.6 Snow Leopard.

To cut a long story short I've been working on a simple-to-use C++ wrapper for some of the most common functions, imaginatively called ofxOpenCL and here is a little demo of 1 million particles running at 100-200fps.

NOTE: The Vimeo compression destroys most of the particles, so I suggest downloading the quicktime directly from the vimeo page at

Frame-rate is around 100-200fps running on a macbook pro with GF 9600GT. That's 100-200fps on a laptop! (albeit a pretty decent one), but I'm dying to try this on a GF 285 GTX - which has 7.5x the number of cores, 2.5x the fillrate and 3.5x the memory bandwidth - for only £250!!

The kernel for this is surprisingly simple:

__kernel void updateParticleWithoutCollision(__global Particle* pIn, __global float2* pOut, const float2 mousePos, const float2 dimensions){
        int id = get_global_id(0);
        __global Particle *p = &pIn[id];
 
        float2 diff = mousePos - pOut[id];
        float invDistSQ = 1.0f / dot(diff, diff);
        diff *= 300.0f * invDistSQ;
 
        p->vel += (dimensions*0.5 - pOut[id]) * CENTER_FORCE2 - diff* p->mass;
        pOut[id] += p->vel;
        p->vel *= DAMP2;
 
        float speed2 = dot(p->vel, p->vel);
        if(speed2<MIN_SPEED2) pOut[id] = mousePos + diff * (1 + p->mass);
}

This example is based on Rui's opencl example at .

Discussion on the matter at http://www.openframeworks.cc/forum/viewtopic.php?f=10&t=2728&p=15107#p15...

source code for ofxOpenCL and the above example at
http://code.google.com/p/ofxmsaof/downloads/list
(the SVN is likely to be more recent).

Eloi of playmodes sends in this beautiful, extended (20 minutes) projection mapping performance, for the Ingravid Festival, in Figueres, Spain.

Telenoika Audiovisual Mapping @ Ingravid Festival, Figueres 9/2009 [FULL] from Telenoika on Vimeo.

Created by Telenoika, this performance is a tour de force of established projection mapping techniques and styles: Virtual lighting, 3D augmentation and distortion, edge highlighting and surface painting.

Mixed with lovely sound design, they’ve also added some nice new touches: The “light bulb generation” at 4:12 is lovely, as is the subsequent thunderstorm and electrocution of the building.

What this piece really does, that I’m excited to see more of, is establish a narrative. We’ve got our basic projection mapping building blocks now. Clever people will think of new, cool stuff to do, but while that’s going on, it’s time to start using these techniques to really tell some stories. Not just about things happening to the building or space, but inside it.

Another exciting development is that the “warping and video player software” was developed in OpenFrameworks [on CDMo] by Eloi, and he will be sharing it soon. In the meantime, you can check out some of his other development on the Playmodes Blog.

Elsewhere:
Playmodes/Eloi on Vimeo.
Telenoika on Vimeo.
Ingravid Festival

Interface 27 from CyberPatrolUnit on Vimeo.

There has been a long tradition in live visuals and motion graphics, inherited from many other media, of maintaining a “secret sauce,” or the guarded formula of eleven herbs and spices. Ironically, for all you hear today “DIY” and “open source” in the same sentence, a lot of the motivation for doing something yourself has historically been doing something no one else can. Keep your secrets, and raise your value.

As our friend Bryant Place / CyberPatrolUnit sends over this latest set of live clips from a recent gig, and I browse through the comments, and reflect on the conversations I had last week at OFFF and during and following my own talk there, though, I’m struck.

The world has changed. First off, the Internet isn’t really about secrets. Your value is almost in direct proportion to how much you can share. Connections are forged through links of mutual exchange and good will. It’s not just about sharing your output or getting fans (the MySpace model), but sharing with a network of enthusiasts, and fellow artists. Those are the people from whom you often get real support (artistic, technical, and personal), gigs – and inspiration. (Even if you hate 8-bit music, that community is a really amazing model: their work to support each other and advocate for the whole subgenre has been I think the single biggest ingredient in their viral success.)

The visualist community increasingly itches not only to improve the quality of their own individual work, but everyone around them. A lot of us are in a battle for the future of this whole medium. Some parts of the world are devoid of live visuals, while others have mass-produced club visuals filling the nightlife.

Before I get carried away, the video itself is just the latest from the ongoing Interface 27 series. It employs a touch interface to control abstract visual pictures formed from streams of particles.

The reason I’m pulling back into the larger question is that these visuals are enabled by a library for Processing, a library we’ve seen here previously, developed by Memo Atken:

MSAFluid for processing (and Java)

If you’d rather use openFrameworks, there’s that version, too, as pictured below running blazingly fast:

ofxMSAFluid for openFrameworks

There’s even an ActionScript 3 port, in case you want to code Flash on the beach.

ofxMSAFluid for openFrameworks from Memo Akten on Vimeo.

So, why do I bring this up? Well, the work done on Processing (Ben Fry, Casey Reas, contributors like Karsten Schmidt, and others), on openFrameworks (Zachary Lieberman, Theo Watson, and their own team), and Memo’s own library, based in turn on many other libraries and implementations, was all a big risk.

It’s not an easy thing to put blood, sweat, and tears into open source. None of those people has exactly gotten rich in the process – not even via the ways you’re supposed to profit from open source, doing the lecture circuit and such. But on the other hand, we’re seeing things that would have been otherwise impossible.

And there’s artistic merit, too. Bryant’s work looks different than Memo’s. The library actually takes on a new life as it gets in someone else’s hands. Bryant actually just wrote me:

As for the Interface video - mention how cool it is that people like Memo post code for other VJ’s to tweak and use.  Mention "FaderTouch" - a 100buk touchscreen off ebay that "vjFader" programmed - using a rear projection onto a translucent screen/ touch sensor we were able to use processing in a very intuitive way.

I got the “mention” part down, I guess. ;)

The responsibility is partly ours to make all of this work: file bug reports, fix bugs if you can, document your work, properly credit the people making it, write documentation for projects, and so on. But it’s not hard to see an ideal start to happen:

1. Person x makes a library / framework.

2. Person y build on that library to make their own tool – and contributes it.

3. Artist uses the tool, gives back to the project, goes in a new direction.

4. More and better work spreads, the project grows, the medium grows, and the audience grows.

None of this happens automatically. We all have a lot more work to do. But having stood onstage in front of a few thousand people calling for just this, it’s nice to keep opening my inbox and seeing it happening. We’re seeing the first seeds planted for what could ultimately be a larger ecosystem. Now, I know there’s also a big gap left – Processing doesn’t have nearly enough contributors, bug squashers, or documenters, and it’s one of the biggest projects, so you can imagine what happens when you get upstream to libraries and the like.

Over the coming months, I think we’ll continue to look for opportunities to help structure some of that involvement and to explaining how you can contribute, too. Stay tuned.

In the meantime, go play with some particles.

For more on Bryant, here he is on his current activities:

- I just did Coachella with [Friend of CDM and contributor] Momo, and in the near future, will be heading to Detroit for http://www.myspace.com/detroitmusicfest

I’m not on the website, however, Kero.fm and Derek Michael - two people who essentially helped build the festival from the ground up 10 years ago - are booking me to play with various acts including CLP, Richard Devine, Drumcell, Busy P (which I did a solo VJ set with at Coachella) so I am super excited to be a part for the first time this year.

Here is a cool video from previous Interface 26:

After Detroit - Mutek.

http://www.mutek.org/

There are also some killer podcasts from past Mutek - http://www.mutek.org/podcast

I am going to meet artists, see the latest AV performances, attend workshops.

I’ll be at Mutek, too, so see you there.