Over the holiday period, the physics arxiv blog is re-running the most popular blogs (by page views) of 2007.

Breaking the Netflix prize dataset
27 November

Hell, this is good work. In October last year, Netflix released over 100 million movie ratings made by 500,000 subscribers to their online DVD rental service. The company then offered a prize of $1million to anyone who could better the company’s system of DVD recommendation by 10 per cent or more.

Of course, Netflix assured everybody that the data had been anonymized by removing any personal details.

That turns out to have been a tad optimistic. Arvind Narayanan and Vitaly Shmatikov at the the University of Texas at Austin have just de-anonymized it.

Here’s how: turns out that an individual’s set of ratings and the dates on which they were made are pretty unique, particularly if the ratings involve films outside the most popular 100 movies. So it’s straightforward to find a match by comparing the anonymized data against publicly available ratings on the Internet Movie Database (IMDb).

That’s exactly what Narayanan and Shmatikov have done. And get this, once the match is made, it immediately links the user to the any private ratings on the Netflix database.

“Given a user’s public IMDb ratings, which the user posted voluntarily to selectively reveal some of his (or her; but we’ll use the male pronoun without loss of generality) movie likes and dislikes, we discover all the ratings that he entered privately into the Netflix system, presumably expecting that they will remain private.”

So what, I hear ya ask.

Here’s what the dynamic duo have to say about one person whose data they outed:

“First, we can immediately find his political orientation based on his strong opinions about “Power and Terror: Noam Chomsky in Our Times” and “Fahrenheit 9/11.” Strong guesses about his religious views can be made based on his ratings on “Jesus of Nazareth” and “The Gospel of John”. He did not like “Super Size Me” at all; perhaps this implies something about his physical size? Both items that we found with predominantly gay themes, “Bent” and “Queer as folk” were rated one star out of five. He is a cultish follower of “Mystery Science Theater 3000”. This is far from all we found about this one person, but having made our point, we will spare the reader further lurid details. “

So Netflix may have inadvertently revealed the political affiliation, sexual orientation, BMI and God-knows-what else of 500,00 of their subscribers. Way to go!

Next up the mobile phone datasets we talked about a coupla weeks back

Ref: arxiv.org/abs/cs/0610105 : Robust De-anonymization of Large Datasets (How to Break Anonymity of the Netflix Prize Dataset)

Over the holiday period, the physics arxiv blog is re-running the most popular blogs (by page views) of 2007.

Invasion of the jivin’ nanoshrooms
24 August

Convertin’ a constant force into an oscillatin’ one is a useful trick. Ya’ll seen em: gravity-powered pendulums and wind-powered turbines for example, them both set machines a-spinin and a-swingin by exploitin’ a constant force.

Them machines might work sweetly at macroscopic scales but ain’t nobody cracked it on the nanoscale even though nanobods are a-chompin at the bit to reproduce this trick. The trouble is that gravity ain’t strong enough at this level and as for wind, who you kiddin?

That leaves only tricky-dicky forces from the dizzy world of electrostatics and magnetics and these are so poorly understood on tiny scales that them nanobods are still a-wondrin and a-ponderin over how to harness them.

But Hyun “Mighty” Kim and his crew at the University of Wisconsin-Madison say they cracked it.

Their device is a kinda nano-mushroom that stands between the plates of a capacitor, in a constant DC field.

Give the mushroom a push and it leans towards the source electrode where electrons tunnel across into the mushroom head. The DC field exerts a force on this extra charge on the ’shroom, pushing it towards the drain electrode where the electrons jump ship. The force disappears and the mushroom’s stiffness sends it swinging back to the source again like metronome, and the process starts again.

Voila! A nanomechanical oscillator that converts a a constant force into an oscillation.

Them nanobods are gonna be cockahoop over this one, betcha!

Ref: arxiv.org/abs/0708.1646: Self Excitation of Nano-Mechanical Pillars

Over the holiday period, the physics arxiv blog is re-running the most popular blogs (by page views) of 2007.

Cellphone records reveal new patterns of human activity
29 October

Switch yer mobile phone on and it checks into the local network giving your location and the time you were there. The network also records the calls you make, their frequency, duration and to whom you make them plus wherever they happen to be too.

Multiply that by the entire popualtion (mobile phone penetration approaches 100 per cent in many western countries) and you’ve got a data set that can give an unprecedented insight into the links between people and the way they move and behave.

Albert-Laszlo “Bar” Barabasi at Northeastern University in Boston and a few pals have been a-grindin’ and a-crunchin’ the data from several million cellphones and are now revealing what they’ve found.

Turns out the data can be used to identify friends and family (from the frequency and duration of calls and whether they are reciprocated), they can show how social groups evolve and how they fall apart.

The data can also suggests how to monitor the way people behave in emergencies in realtime. For example, a pile up on the freeway causes lots of rapidly moving phones grind to a halt, a few call the emergency services while others call the office/spouse/lovers. Spot that pattern and its a pretty good indication that an event has occurred. Location information might even help to determine exactly where the accident took place.

Barbara Rasi has also found previously unknown patterns in human behaviour. For example, although the number of people making calls varies hugely during the day and night, the percentage who are on the move (ie who make consecutive calls from different lcoations) is always roughly the same. And the average distance they travel between calls in a half hour period is also stable at about 6 km. He says this is just the beginning of what will be possible with this kind of data.

What ya’ll want to know is how anonymous is it? The message is: Big Brother is watching…but he only has access to anonymized data.

But ah don’t buy it. It wasn’t so long ago that the raw data from search engines was thought to be anonymised if a person’s name was removed from it. But that myth was exploded by journalists from the New York Times who tracked down one individual using only her AOL search records.

How long before we see a similar expose with supposedly anonymized mobile phone records?

Ref: arxiv.org/abs/0710.2939: Uncovering Individual and Collective Human Dynamics from Mobile Phone Records

Over the holiday period, the physics arxiv blog is re-running the most popular blogs (by page views) of 2007.

The frightening prospect of flu
23 November

Bird flu may get all the headlines but the number of deaths it causes each year is currently measured in hundreds. The real killer, the one that should set yer spine a-shiver, is ordinary fly which kills hundreds of thousands each year.

With winter nearly upon us up here in the northern hemisphere, the spectre that we’ll be a-shakin and a-sweatin our way through a flu pandemic in the coming months is raising its ugly head again.

So what can science tell us about the way pandemics spread? Not enough, according to Gerardo Chowell from Arizona State University in the US and his friend Hiroshi. They’ve posted a comprehensive review of epidemic science dating back to the 1918 Spanish flu epidemic that killed as many as 100 million people by some estimates. We’ve learnt an awful lot since then about the way flu spreads but the frightening thing about Gerardo’s review is how much more we need to understand.

It’s not just small things that are up in the air but sizable pieces of the jigsaw. We still don’t know some basic probabilities associated with infection. For instance, given that an individual is infected with flu, what are the chances that the disease will manifest itself clinically? And given that the disease has manifested itself clinically in an individual, what are the chances of that person dying. And if a virus can be caught from a number of diffferent host species (as it might eventually be with bird flu) what is the probability of transmission?

Without a good understanding of these kindsa factors, it’s gonna be difficult for authorities to plan an effective response to a flu pandemic.

Mah advice? Stock up on food and water.

Ref: arxiv.org/abs/0711.3088: Quantifying the Transmission Potential of Pandemic Influenza

Over the holiday period, the physics arxiv blog is re-running the most popular blogs (by page views) of 2007.

Game theory and the future of Adwords
8 September

Ain’t Google Adwords a miracle o’ modern science? Here’s a system that searches your web page for keywords, hunts for advertisers who wanna have their message displayed next to these keywords and then auctions the advertising space to the highest bidder. All in the twinklin’ of an eye. Adwords is so good that it’s made Google millions or billions or zillions (who’s countin’?)

What can game theory tell us about the future of this digital auction house? According to Sudhir “Secrets” Singh, an electrical engineer at the University of California, Los Angeles and his buddies, game theory suggests that everybody would benefit by the emergence of a new kind of online business that exploits the inefficiencies of the Adwords system.

Here’s the thinkin’. There is a limit to the number of advertising slots on each page and this leaves advertisers a-scramblin and a-scufflin for spaces next to the most popular keywords. Inevitably, the advertisers who miss out are left a-sobbin and a-wimperin at the end of the day. The sobbers and wimperers are perfect fodder for the new businesses.

According to Secrets Singh, these businesses are gonna buy up popular keywords and then resell them to advertisers who want to ensure they don’t miss out on a slot. Secrets and his crew have built a model of this new market and calculated the revenues and payoffs for all parties. The model assumes that this game is a symmetric Nash equilibrium, meaning that everybody plays in the same way (or at least that nobody has anything to gain by unilaterally adopting a different strategy) .

Under these circumstances everybody wins–the auctioneer, the reseller and the bidders.

Ain’t that a bee-yoo-tiful story? So ya’ll gonna see resellers springin up all over the place soon. Secrets Singh and his buddies even have a start up called Ilial.com and there ain’t no prizes for guessing what that’s gonna be doin’.

They’ll already be a-dreamin and a-wondrin about the zillions they gonna make.

I know ya’ll lurv a happy endin. So ya’ll keep quiet about Adwords not being a symmetric Nash equilibrium. Y’hear?

Ref: arxiv.org/abs/0709.0204: Capacity Constraints and the Inevitability of Mediators in Adword Auctions

Over the holiday period, the physics arxiv blog is re-running the most popular blogs (by page views) of 2007.

The incredible galactic foxtrot
30 August

Ya’ll know the laws of physics are symmetrical–they have no preferred direction. The speed of light (and more or less everything else) is the same whichever way it is pointin’, right?

So it’d be surprisin’ if the universe turned out to be asymmetric, shockin’ if it had a prefered axis and stunnin’ if it had a direction in which things were different.

But it’s beginning to look that way. Two years ago, you could hear astro-jaws dropping all over the planet when a couple of upstarts suggested that data from a giant space-thermometer called the Wilkinson Microwave Cosmic Anisotropy Probe or WMAP looked as if it were aligned in a certain direction.

A-babblin and a-brayin, critics pointed out that the alignment matched the galactic co-ordinate system and so was probably an imprint of the Milky Way. Subtract this properly and the alignment would go away.

But it didn’t. And that left theorists a-scratchin their heads and a-rubbin their chins in amazement.

Now they gotta do a whole lot more a-broodin and a-worryin. Michael “Bongo” Longo at the University of Michigan has spotted another kind of alignment by a-meterin and a-measurin the orientation of 200,000 elliptical galaxies. That’s a lotta galaxies and it turns out they’re all aligned in the same direction too. We just ain’t noticed before now.

That’s put a ‘gator in the pool.

So what’s been causin this cosmic foxtrot? Ain’t nobody sure but we’re gonna be hearin a lot more about it in the weeks and months to come. Betcha!

Ref: arxiv.org/abs/0708.4013: The Axis of Opportunity: The Large-Scale Correlation of Elliptical Galaxies

This week’s posts:

 Why MHD propulsion won’t work

The puzzling presence of DIBs

The sunset on HD 189733b

Tune into the snowflake channel

Black holes may convert dark matter into cosmic rays