Big Data’s Big Moment

Good post on "big data" from a speaker at the upcoming Money:Tech conference in New York:

At present, much of the world’s Big Data is iceberg-like: frozen and mostly underwater. It’s frozen because format and meta-data standards make it hard to flow from one place to another:  comparing the SEC’s financial data with that of Europe’s requires common formats and labels (ahem, XBRL) that don’t yet exist. Data is “underwater” when, whether reasons of competitiveness, privacy, or sheer incompetence it’s not shared: US medical records may contain a wealth of data, but much of it is on paper and offline (not so in Europe, enabling studies with huge cohorts).

Yet there’s a slow thaw underway as evidenced by a number of initiatives:  Aaron Swartz’s, Flip Kromer’s infochimps, Carl Malamud’s, as well as Numbrary, Swivel, Freebase, and Amazon’s public data sets.

More here.