Dark Matter in the Information Universe

For people who haven’t yet read my Harvard Business Review piece last year on feeds and syndication technologies, let me summarize a theme that I find myself talking about to many groups: Dark matter — of the information type.

One of the great wonders of syndication technologies is that they free up and expose informational dark matter. I argue that organizations and individuals throw off ten bytes of dark matter (read: informal and often transient) for every byte of “lit” matter (read: formal and findable) that they issue.

But most of that information is latent and invisible, whether it is data about organizational processes, like a shop-floor controller, or environmental, like water levels in Mission Valley, or financial, like credit-card transactions, or technical, like referrer-log data, or meta, like the status of my Feedburner feed on an hourly basis (sorry guys, couldn’t resist). The list is endless and largely invisible.

All that information is out there and useful to someone, even if it’s someone waaaay out in the long tail (you knew I was going to say that). But it’s not been cost-effective to expose it. Until now. Syndication technologies make it possible to cost-effectively and usefully expose informational dark matter, which is more than press releases and blog articles, but is just about every kind of change of state you can imagine in our physical, social, and organizational environments.

What would you (or your organization) monitor if you could? What would like to know about that you don’t? What happens on a daily basis that would like to track but you can’t? What if you could search all that information prospectively and retrospectively? It’s a world-changer.

[Update] By way of response, Stephen Castellano gives an intriguing example of this sort of thing from the world of investment research.


  1. Using Keyword Searches to Improve Market Intelligence and Investment Research

    The night prior to the announcement of S

  2. I’d enjoy reading that Harvard Business Review piece on feeds and syndication technologies you mentioned you wrote. Any idea where I can a pennyless MBA student can get a pdf copy of it? If not, what issue is it in? Thanks…
    your spam filter isn’t letting me spell correctly!

  3. Based on your own examples, dark matter is such not because of access, but because it needs a layer of analysis to become lit matter. A database containing one million of credit card transactions is perfectly useless, until you perform some sort of statistical analysis. The process of turning dark matter into lit matter is all about information stores.
    Syndication technologies are information pumps: they are all about flow – rather than stores – of information. An article is posted, a book becomes available, and syndication technologies pump that information to users. Syndication technologies help find the right pumps, but whatever went through each pump in the past is largely history, and (maybe) accessible through means other than syndication.
    Syndication technologies are also all about collection. The information is already out there (“lit”), but it’s scattered in many places and formats. Syndication helps users conveniently collect that information in one place.
    You can use syndication to pump dark matter over time from somewhere on the Net onto your hard drive rather than downloading it all at once from a database, but that does not turn it into lit matter.

  4. Paul Kedrosky on ‘Dark Matter in the Information Universe’

    UCSD fellow and Web 2.0 pundit Paul Kedrosky gets to the heart of the staggering value of publish-and-subscribe technologies like RSS in the enterprise…

  5. Charlie Wood leaves NewsGator to start new enterprise RSS company

    Moonwatcher Personal: Starting Up Again

    I’ve left my position at NewsGator Technologies as Vice President of Enterprise Solutions to start …

  6. Website changes via RSS

    Wouldn’t it be great if we could get changes in websites (changed pages and new pages) via RSS. Producing RSS feeds for websites can be done technically by external web site monitoring services (change detection services) or even more naturally by inte…

  7. Came across a company called Brightplanet that searched “deep web” which may be somewhat related to this interesting dark matter concept.