The Yahoo! Mail Visualization [yahoo.com], designed by Periscopic, aims to show of the network processing power required to run a vast emailing service by revealing the streams of keywords that are sent around the globe in real-time.

The geo-located circles on the world map represent the activity and volume of processed emails, totaling to about 5.6 billion emails a day. Current and predicted email traffic is show at the bottom of the screen. A streamgraph shows the top 10 keywords used in email subject lines over the last 5 minutes (at least from those countries that have no strict privacy laws). Both views allow for zooming into specific continents, and for filtering the spam from the non-spam email (which actually occur to a ratio of 1 to 4). Additional infographic illustrations further explain the email filtering processes.

The interactive infographic The Evolution of the Web [appspot.com] simply shows what the title denotes: the progress, through time of the most popular web standards (unique ribbons) and browsers technologies (horizontal), such as HTML5, CSS3 and WebGL.

The dynamic flowing of the differently colored ribbons in this visualization represent the interaction between browsers and standards, which reflect the many powerful web apps that have been used. The numerical values on the horizontal axes describe the subsequent browser releases.

Thnkx Bram.

See also HTML5 and CSS3 Readiness.

Natural Gas [geblogs.com] designed by Information is Beautiful (slash David McCandless) is the latest addition to G.E.'s already impressive collection of interactive infographics, which already includes works like Pentagram's CFO Outlook and Contrasting the Drivers of Innovation.

The Natural Gas infographic illustrates various factual statistics about the scarcity of this natural resource, and includes a view on how many cubic meters or years that are left before the availability of gas will run out, an overview of the top 7 countries with reserves, yearly producers and biggest consumers, and contrasts the availability of natural gas against that of oil and coal.

In his TEDMED talk, Thomas Goetz, executive editor at Wired Magazine and author of the book "The Decision Tree: Taking Control of Your Health in the New Era of Personalized Medicine"
discusses the issues surrounding medical data, and how it has the inherent power to drive behavior change.

More particularly, he analyzes how we tend to communicate messages about health to people (e.g. mainly through conveying images of fear) and makes the bold call to make such health information more relevant and personal (e.g. by unpacking personal choices). Soon he focuses on the current practice of conveying lab test results by way of numerical tables, and explains how this is the worst information presentation possible. He then presents a compelling visual redesign of these reports, which aim to provide more insight to lay patients, and ultimately should empower them to act on the information shown.

Watch the talk below.

The project Visual Budget - an Interactive Guide to the Federal Budget [kickstarter.com] is raising funds on Kickstarter in order to build a cutting-edge data visualization website which will explain all the complicated ins and outs of the US Federal Budget, showcasing interactive charts and motion graphics. The money will be spent to develop an interactive prototype, which then will be used to raise additional money from philanthropies and foundations to fund the complete website.

The website will contain 'polished designs' of charts and pages, 'visual effects' that make charts come to life on screen, 'animated transitions' between charts with related information, and 'presentations and voice-overs' turning complex charts into digestible pieces.

While this project has already acquired more than the original $10,000 treshold after less than 2 days, you can still support this initiative with as little as $1. For $25, you will become a beta-tester.

(Note: Have little money and still wish to support something else that is worthwhile: participate in our online study!)

Do horoscopes really all just say the same thing? We scraped & analysed 22,000 to see.

See our completed meta-horoscope chart and make up your own mind.

We’ve also created a single meta-prediction out of the most common words..

---

How we did it

How do you gather 22,000 horoscopes? Obviously you could manually cut and paste them from one of the many online Zodiac pages. But that, we calculated, would take about a week of solid work (84.44 hours). So we engaged the services of arch-coder Thomas Winnigham to do a bit of hacking.

Yahoo Shine kindly archive their daily predictions in a simple and very hackable format (example). Thank you! So Thomas wrote a Python script to screen-scrape 22,186 horoscopes into a single massive spreadsheet. Screen-scraping is pulling the text off a website after it’s displayed. Python is a programming language. You can use it to write scripts that only gather the specific text you want. Then you run it multiple times so it mines an entire website.

Well, it’s not quite that easy. Big sites like Yahoo have ‘rate-limiting’ on their servers. That means if you access a page too many times too quickly, it thinks you’re a hacker and deploys all kinds of anti-hacking counter-measures. Initially, Thomas set his scraping speed too high (once every 10th of a second) and his IP got instantly banned from Yahoo for 24 hours. After some experimenting (and more bans), he found that a two second delay between scrapes prevented the defense mechanisms from kicking in. The script was set to run in the background (while we smoked cigars and discussed the empire). 12 hours later, we had our 22,000 horoscopes in a single file!

We can’t share the 9.5MB spreadsheet with you because it’s Yahoo’s copyright. But here are the Python scripts should you feel like recreating the experiment.

Filtering it down

So every different type of horoscope got sucked up – career, teen, love, daily overview. Who knew there were so many? It was felt, though, that career & love predictions would have their internal biases i.e. lots of mentions of work, career, love, marriage etc. So we opted to just analyse the generic daily horoscopes for each sign. A total of 4,380 (365 per star sign).

Word Analysis Version 1

We used an online tool called TagCrowd to find the most common words. I prefer it to Wordle. You’ve got better control over any ‘noise’ in the signal, because you can not only filter common words (“and”, “for”, “is” etc) but also a special ‘stoplist’ of words you’ve chosen.

So we broke down the most common 50 words to see if there are any patterns of unique words. This is what was revealed:

You can see the full data in a Google spreadsheet here.

Word Analysis 2

It struck me that several words in the top 50 – like “someone”, “really”, “quite” – were just qualifiers and not really that revealing. You’d find them in any English word analysis.

So we stripped those kinds of words out (see our stoplist). And lo! A fresh set of unique, revealing and more accurate words appeared in the top words per sign.

Can I just say that I have no personal interest in horoscopes. I don’t know what the various characteristics of each star sign are meant to be. So you’ll have to tell me if any of this corresponds to folklore.

This was the data we used to create our meta-chart. Check out the final image. Or see all the data in this Google spreadsheet.

Meta-Prediction

One more thing though. This analysis appears to reveal something. The bulk of the words in horoscopes (at least 90%) are the same. That’s not a full, proper statistical analysis. (If you are a statistician and you want to do a proper analysis, please get in touch)

The cool thing is, once you’ve isolated the most common words, you can actually write a generic, meta prediction that would apply to all star signs, every day of the year. Here it is.

The Future

As ever, I’ve laid out my whole process and all the data here: http://bit.ly/horoscoped.
That way it’s all balanced and you can make up your own mind. Typical Libran!

---

---

Kamel Makhloufi developed a simple but powerful pixel diagram [flickr.com] that highlights the past human death toll during the Iraq war. Each pixel has a color, where blue stands for "friendly", green represents "host", orange are "civilians" and grey denotes "enemies". The left diagram is ordered by category, whereas the right one lists the casualties by time.

Via DataPointed. See also CNN's Home and Away.

"Infographic". See also full version below.

Do you know what he means?

Sync/Lost [3bits.net] is a multi-user data visualization installation that allows users to become immersed in the history of electronic music. Starting from a complex timeline, user interactivity allows rhythms and sub-rhythms to merge into new sounds. The user interaction is accomplished with wiimote controls, wireless handheld pointers which are relatively easy to use and learn. The audible feedback is provided via wireless headphones.

The project's objective is to create an interface where people can explore all the connections between the main styles of electronic music through both visual and audible feedback. The interaction with the resulting network graph leads to "a collective consequence in the spatial visualization of information".

Via Visual Complexity.

The webpage Visualizing Prime Numbers [primalchaos.yolasite.com] does exactly that. It contains a number of consecutive visualization experiments titled "Maps of Factors". Each "map" reveals the factors for each number, so that when only 2 circles intersect the horizontal axis at their right quadrants, a prime number is revealed (highlighted by green vertical lines in the graphs). Since factors in general are geometrically chaotic in nature, prime numbers are also geometrically chaotic in nature, as the resulting image seem dependent from initial conditions (much like "atoms in Quantum Mechanics").

If I understand this correctly, in an interesting twist, the author attempts to find large prime numbers in a more efficient way through analyzing and predicting the resulting visualizations, instead of working the other way around, that is to visually represent prime numbers which were derived purely algorithmically.

See also:
- The Visualization of Randomness
- Data Signals (also of prime numbers)
- Symmetrical Mathematics
- Ulam Prime Numbers Spiral
- Prime Number Spiral