something in the way

For people coding for visuals, Processing just keeps getting better. And for people who aren’t … well, you might just want to give it a second look, as a growing global army of people who never fancied themselves coders suddenly start typing up new creations. A new release makes mobile development easier and corrects lots of bugs. But specifically of interest to readers here, powerful libraries for 3D help make Processing an intensive tool for creating visuals. With the aid of running of your GPU, they can also deliver eye-popping real-time performance not normally associated with Java.

Processing 1.5

In a surprise release, much of what’s coming in Processing 2.0 this summer is now available in a stable, general release of Processing 1.5. It lacks the new built-in renderer – OPENGL2 – but does incorporate new features for developing for Android and, via the third-party library GLGraphics, you get all sorts of new OpenGL goodies. (Note: Processing 1.5 = 0196. The previous pre-release version, 0195, is worth downloading if you want to play with the new OPENGL2 renderer and its examples. Both are on the download page, and they can be installed side-by-side.)

Most importantly, for anyone publishing to the Web, you need to download this version now and re-export sketches. Applets were seeming to run very slow in Chrome and Firefox 4. (While I vastly prefer the Java version for things like performance tools or installations, I also appreciate the Processing.js JavaScript fork for Web delivery – but this fix does make applets work pretty well.)

New to Processing 1.5:

• Tons of editor fixes in the PDE development environment
• An essential bug fix that corrects slow performance when exporting for the Web.
• Android development support – in preview builds, but now in a stable build. (I think this is still considered a pre-release feature, but it means you can run the stable build to try it out.) Now go make stuff for phones and tablets easily.

Processing 1.5 is a good start, but for me, the live visual workflow isn’t complete without two additional libraries, toxiclibs and GLGraphics. A new GLGraphics update improves integration with both toxiclibs and the new Processing release.

toxiclibs

toxiclibs represents over 270 classes in 7 libraries, some 25,000 lines of code, written by digital artist Karsten “toxi” Schmidt. There’s gobs of stuff in there. The most useful is a basic set of classes for things like geometries, meshes, and 3D vectors which you’d otherwise have to build from scratch, and which are generally built in fairly standard ways. With elegant math, Verlet physics, and color libraries, and wonderfully-usable API design, Karsten gives you all the essentials in a way that will inspire you to use them and make something truly original. (A full tour is probably a good subject for another post. Just ignore the toxi.audio packages; with the libpd crew, I’m working on something with Pd I think you’ll find more useful and stable.)

I had a discussion with one colleague who felt that, indeed, toxi’s libraries are so powerful that people are simply using it as a crutch. That may be true to an extent: people should prominently credit toxi’s work, and to do otherwise is plagiarism. But with proper credit, I feel that standing on the shoulders of someone else’s work can be a good thing. Digging through toxi’s libraries is like going to school for the sorts of math and geometry that you need to learn to understand 3D generation. For many of these classes, involving essential mathematics operations and 3D modeling, I’d have no idea where to start, would spend weeks or months writing something inefficient, and would come out with something that reinvented the wheel. A lot of the techniques themselves in those 27,000 lines of code weren’t developed by Karsten, either: it’s more like seeing the wisdom of a master teacher, assembled from a wide variety of sources and passed on.

In fact, this is a rant that I should probably invoke elsewhere, but to me received knowledge is the essence of any craft. Composers don’t invent new rules of harmony (well, at least, not tonal harmony). Engineers don’t work out the laws of physics from scratch each time they build something.

And most importantly, because all of this exists in code, you can read and modify anything you find. It’s a black box if you want it to be, but I very often dig directly into the code to understand how things work.

There’s a bunch of documentation and a showcase for great work:
http://toxiclibs.org/

GLGraphics

All of this, though, brings me to GLGraphics. As I’ve been saying for – gee, years now – GLGraphics is the future of Processing. Now, that’s come to pass: gifted developer Andres Colubri authored the new OPENGL2 renderer that is similarly based on native OpenGL calls, and now runs the Processing Android port and upcoming Processing 2.0.

GLGraphics is separate from OPENGL2, but it’s your best bet for work on the stable Processing 1.5 build. Version 0.95 adds compatibility for that latest release, and adds two essential other features:

• An example of how to integrate with toxiclibs
• GLSL shader support, which can in turn be used for complex mesh generation.

See yesterday’s blog post announcing the update:
Processing 1.5 / GLGraphics 0.95

And there should be still more coolness to talk about soon, at least for Mac users, with the availability of Syphon for Processing. Stay tuned on that.

I think I have to hide away in a hole and do nothing but code this weekend. Anyone want to hop on IRC or PiratePad and pass code snippets back and forth?

Attention, 3D fans: OpenGL in the browser has gradually gotten real. WebGL is a browser-friendly API for OpenGL graphics, and it’s pretty darned close to OpenGL ES 2.0, which in turn will be familiar to anyone doing modern mobile 3D development. WebGL isn’t part of HTML5, but HTML5 makes it possible: the Canvas element is what allows WebGL to work its magic. And WebGL goes nicely with technologies that are part of HTML5 or modern browser experiments, including the web audio API and browser video support. (The superb 20 Things I Learned About Browsers & The Web has a 3D in the browser section, well worth reading.) And you can use JavaScript (among other modern languages) to code 3D creations.

If you love the idea of sharing 3D as easily as a webpage, this is big news. It’s a huge step forward from the clunky, unpredictable, confusing use of Java for browser OpenGL, and unlike that solution, it’s part of the page on which it’s delivered, not part of a plug-in or launched app.

In recent days, we’ve seen the first stable browser with WebGL enabled by default, Google Chrome. Right now, Chrome or Firefox 4 beta are likely the easiest and most stable way to test WebGL graphics. I’ve been testing Firefox 4 beta on Linux and more recently the stable Chrome on Mac, Windows, and Linux, and it’s pretty fantastic.

Read Google’s announcement from Thursday, along with the other enhancements to Chrome:
A dash of speed, 3D and apps

Perhaps more exciting than the Chrome update is the superb Chrome Experiments, which recently added 3D goodness, from creative tools to eye candy to useful tools like an exploration of human anatomy:
WebGL Experiments

Grabbing the latest Chrome or installing Firefox beta will let you see them, but here are a couple of picks are a cool place to start, and have videos attached if you aren’t near a bleeding-edge browser:

http://www.chromeexperiments.com/detail/webgl-aquarium/?f=webgl”>WebGL Aquarium, Human Engines and Gregg Tavares

Field, Gregg Tavares

I’m pretty impressed with performance of experiments like the aquarium. I’m on a fairly low-end, last generation laptop with an NVIDIA 9500M, and they run easily.

WebGL is still early in development – Chrome is the first and only stable browser with support – but we’re getting to the phase when you could actually distribute stuff with it, and it could hit prime time very soon.

Which browsers support WebGL?

Chrome’s release is a very big deal. As I write this, WebGL is available in:

• Chrome stable, from now on
• Firefox 4.0b8 and later, meaning once Firefox 4 stable is shipped, stable Firefox, too
• Nightly builds of Safari/WebKit (which I believe includes both Mac and Windows support, though I haven’t tested on Windows)

Opera plans support, but no public build is available yet.

Microsoft appears not to be planning support for IE9, meaning it’s most likely to be odd man out … again. But you can get support for WebGL inside IE using the free Chrome Frame plug-in.

Really, if you want to try this out, installing Chrome is a good idea. It’s also no accident that Google’s Chrome Web App store means people with interesting creations have an avenue for distribution, which should soon also be true with an open Mozilla-based store for Firefox et al.

http://www.khronos.org/webgl/

Can you use Processing.js with WebGL?

Yes! Processing.js is actually a pretty decent way to fiddle around with it. The caveat is that WebGL support in Processing.js is a work in progress; if you want to get deeper, you’ll probably want to get into direct JavaScript control of WebGL. But that hasn’t stopped people from making some interesting hacks and work, and it’s a great place to start. Some demos –

A Processing.js Web IDE that uses WebGL:

– and a stunning music visualizer we’ve seen here before:

What about Google’s O3D?

O3D is some impressive technology and for many of us was the first truly compelling vision of 3D in a browser. The downside – it’s currently a plug-in. But Google does sometimes live up to their “open Web” hype. They’ve said they’re focused on improving WebGL, and that they’ll take the ideas from O3D (like its scene graph) and port to JavaScript and WebGL. There’s even an early version of the work.

It’s worth reading the (official Google) Chromium blog on the matter, partly to see how they’ve come around on JavaScript performance.

The future of O3D

Why wouldn’t you use WebGL?

This is all compelling stuff, so let’s all abandon everything we’re doing and switch to WebGL — right? Well, not necessarily:

• It’s not done yet. WebGL the spec appears pretty stable, but browser support is still forthcoming. When we (presumably) see stable Safari and Firefox builds in the near term, though, I think the whole thing will get a lot bigger.
• Universal browser support is a long ways off. Microsoft’s lack of support in IE could be a side effect of the lack of universal OpenGL drivers on the Windows platform. Whatever the reason, count out IE. And likewise, count out anyone with a capable GPU card. Even compared to the mess of video support, 3D is likely to be a “nice-to-have” feature on the Web, not the universal feature the traditional 2D page is.
• Mobile WebGL isn’t even on the table yet. So, JavaScript – yep, it’s faster on desktop computers. But mobile implementations are still evolving, mobile browsers still lag their desktop counterparts (even sometimes when they’re both based on the same Web engine, like WebKit), and performance on much less-capable mobile chips isn’t there just yet. That said, see the last bullet in this list…
• Live visuals, art will still often need “native” tools. Want to output to a second monitor, or monitors, plural? Doing something crazy like routing textures between apps? Live visualists are pushing the kinds of features that won’t be accessible immediately on the browser.
• Full-blown OpenGL isn’t available. OpenGL ES 2 is pretty great, but if you need the full OpenGL API, this isn’t designed to be that. And…
• Performance is still better with C/C++. Don’t get me wrong – performance with JavaScript is stunningly good, good enough that those Google engineers changed their assessment. But it seems to me this depends on your goals. If you’re really concerned with squeezing every last ounce of performance out of your graphics, necessary if your work is about visuals with greater complexity, this still really isn’t for you.
• This isn’t an either/or choice – OpenGL wins! Here’s the major point for me. You don’t have to choose WebGL as an exclusive solution, partly because you don’t have to choose WebGL. Invest your time in OpenGL, and learn the basic nuances when comparing, say, OpenGL 3.2 on the desktop to OpenGL ES 2 on Android and iOS to WebGL in the browser, and you can be everywhere with relatively minor adjustments.

What’s surprising to me just writing that list is, while it appears long, the advantages of WebGL are still clear, and it makes sense that some of these differences will disappear. I imagine we will still need desktop software. Google, while characterized as some sort of browser-only religion, themselves continue adding native support in their Android platform, so presumably they understand game developers and other parties want that native OpenGL access. The question may not be whether WebGL “replaces” those tools, but whether people find smarter workflows and integrated higher-level APIs to work across the platforms.

Let’s sum it up this way: if you love 3D, and if you’re an OpenGL nerd, you’re in very happy times, indeed.

And regardless, you get to watch a cool jellyfish in Chrome any time you need to unwind.

http://www.khronos.org/webgl/
http://planet-webgl.org/

Pretty promising project over at PhobosLab of the Impact Game Engine running on the iOS platform using the JavaScriptCore Framework. The best part is it has all Canvas calls passed into run on OpenGL ES thus the speedy demo.

Biolab Disaster on the iPhone 3GS from Dominic Szablewski on Vimeo.

The game is running in its own process and is not using the iPhone’s browser at all. Instead, it’s just using the JavaScriptCore Framework to run the game. All the necessary calls to the Canvas API have been reimplemented with OpenGL-ES and the touch input is passed over to JavaScript to be evaluated by the engine. I of course had to make some changes to the engine, but the game source code is exactly the same as for the web version.

This would probably never fly on the App Store because it executes code or interprets it but is a very nice experiment. More discussion at Hacker News..

Mobile games are on slower hardware, similar to later 90′s computers so native is a great way to go for 3d and 2d game development because of this limitation at the current time and well into the next few years.  Take this time to learn you some native gamedev. andengine isn’t native directly as it is Java based but compiled with the Dalvik JIT virtual machine. Another way to go native on Android is the Android NDK which allows C and C++.

• andengine download at google code
• andengine examples at google code
• andengine blog
• Mercurial repository at google code
• cocos2d-android port of cocos2d-iphone with BSD license
• rokon android is another BSD licensed android game engine

The engine also has extensions that can be easily added and some great ones exist already.

You’ll hear odd cynicism about people working on free software / open source projects. Something like, “well, harumph, it’s not as though a bunch of people will make this stuff in their free time.”

Not only are these folks wrong, but you can actually visualize the contributions to source trees – and the results look spectacularly hypnotic. It’s free software – the music video.

Okay, now, granted, I may get so mesmerized by the results that I’ll just spend time staring at that instead of getting actual work done, but – working too hard isn’t good for you, anyway. It’s an organic high, audiovisuals.

At top, Ryadh Amar sends in a visualization of the excellent, lightweight LXDE windowing environment for Linux. (Actually, I’m inspired to give LXDE a fresh install.) At bottom, a collage of various projects showing that these data visualizations can take on various identities. Gource can support just about any project repository, too: Git, Bazaar (popular on Ubuntu), and Mercurial (recently added to Google Code, incidentally) are available native, and CVS and SVN are available as third-party extensions for those of you kickin’ it oldskool and non-distributed. (Though, really, come join the 21st Century – it’s awesome.)

And Gource can visualize itself. Freaky. It’s all thanks to the ongoing awesomeness of OpenGL.

http://code.google.com/p/gource/

I’d love to see this added to project management so you’d have a sort of live, superb visual to inspire you to keep the code moving forward.

Who knew source code would turn out to be so visually inspiring? (Now I just need a new way of visualizing me writing bad code and then correcting and cleaning it up. I think it could be best represented as a set of stick figures getting stuck in quicksand and hitting each other over the head. Then there could be a big Smoke Monster that represented the Evil Force of Procrastination.)

But wait! There’s more! You can visualize web logs, too. (It works with Apache; I have to see if I can make it work with our nginx logs, as visualizers could actually be very beneficial with the kind of complex data you get in something like a web log):

Check it out!

NOTE: To view WebGL, you’ll need to use a Mozilla trunk nightly build (starting September 18th). Once installed, switch webgl.enabled_for_all_sites to true in about:config. Your OpenGL drivers will also need to be up-to-date. A video of the experiment is included in this post if you’re having trouble installing.

A couple weeks ago, Vladimir Vukicevic posted the first working sample of WebGL, which gives us a glimpse of how JavaScript will access OpenGL ES 2.0. Since then, we’ve seen a number of impressive demos, and it seems GPU accelerated 3D in the browser is finally becoming a reality. This technology deserves attention.

Of course with WebGL in its infancy (even though they’ve been working on it for years), and the spec still in flux, no documentation exists. Learning the API involves scouring sample source and the nsICanvasRenderingContextWebGL interface reference. My OpenGL programming knowledge being limited, I had to use iPhone OpenGL ES 2.0 guides to get me up to speed with the spec. Jeff LaMarche has a fantastic series on the subject, and I decided to use his icosahedron example as a starting point.

It’s a simple low poly test, but I’m rather impressed with the result. And really, who can complain about finally having access to the GPU? Hopefully there aren’t too many roadblocks (ahem, Microsoft/DirectX) in the way before this spec becomes official. Next step for me will be texturing.

Wish granted!

Think 3D in the browser will never catch on? Think again. The folks at Google Labs have built an incredible-looking 3D API called O3D. It does just about everything you want, and then some:

• It’s multi-platform: Mac + Windows + Linux.
• It can render to both OpenGL and DirectX render pipelines.
• You can write your own vertex and pixel shaders. You have to use O3D’s own language for doing this, but that actually enhances compatibility, as frustrated shader coders may already know. (See the FAQ)
• It’s a scene graph, so managing complex 3D scenes isn’t a chore.
• It has powerful built-in functions like viewports and pickers (plus custom pickers), so you can actually get something up and running in a reasonable time.
• It has an import workflow with COLLADA, an open standard for 3D assets (and which, incidentally, has support in Google’s own SketchUp).
• You code in JavaScript, using the powerful V8 engine (developed for Chrome).
• Gears lets you run offline.

There are already some complaints about “another standard,” but to me, putting together a whole package here and employing other, lower-level standards (JavaScript, COLLADA, OpenGL, DirectX) makes a lot of sense.

http://code.google.com/apis/o3d/

I expect the folks working on Java and JavaFX are busy thinking about the fact that Sun just got bought by Oracle – something I’m hopeful, at least, ensures Sun’s future and is ultimately a good thing. But I hope someone on those teams is starting to get the message: 3D isn’t just something that’d be “nice to have.” It’s essential. And while even most developers likely don’t have a clue about things like custom shaders, having access to customize the graphics pipeline is likewise something ultimately benefits all developers – even if they just wind up relying on someone else’s code. I really do hope this is a priority with the coming development of Java and JavaFX, which could have the power to do these sorts of things. (Heck, Java could even benefit from the code Google just posted.)

On the proprietary side, this to me is a big blow to Microsoft’s WPF and Silverlight and Adobe’s Director. Unlike those products, O3D looks simple, powerful, flexible, open source, and directly programmable with JavaScript.

That’s not to say there aren’t some questions here – and the Java/JavaFX comparison is especially relevant:

• Another plug-in: You do have to install a plug-in to work with O3D, something that actually isn’t necessary with JavaFX (when it finally does 3D) or right now with JOGL and Java3D.
• Mobile, or just desktop? My big question I have is what this means for mobile. I’d love to have O3D work with OpenGL ES on, say, Google’s own Android platform.
• Not Just JavaScript? It would be nice if eventually you could use other languages like Java to program O3D.
• Sound? Oh, yeah, that. 3D sound is an ideal complement to this sort of scene, and the browser may be a bit constrained in that respect. I’m curious whether O3D might eventually include an audio API. (And yes, that’s where something like Director is still unparalleled.)
• Making it actually work: Okay, there’s also the fact that I haven’t successfully installed it just yet. Working on that.

(I’ll try to get answers to those questions.)

Oh yeah, and then there are details like the necessity to write your own custom shaders just to add more than one light to a scene – I think this will initially appeal only to folks with some real 3D experience.

But am I excited? Ohhhhh, yes, indeed. O3D itself looks fantastic, and I think this is a sign that 3D times ahead are going to be really fantastic.

And as long as you have the plug-in working and a browser in full-screen mode, you could literally set up an O3D project as a performance / installation tool. O3D visualists? Absolutely. Enjoy.