The Problem with Parabots

Let’s coin a term: Parabots. That’s my new word for software spiders — bots, as they’re sometimes called — that leech content from your site in a way that is more parasitical than useful. I’ve been thinking for some time about ill-behaved bots that suck bandwidth without returning traffic, but Tom Foremski made my thinking a little more empirical with the following aside in a recent post:

…my web site is spidered by 17 bots every day (a number that is increasing.) The bots suck up one third of my bandwidth and deliver about 5 per cent of my traffic. More than 90 percent of my readers come direct, they know where I live, which is a great position to be in and not have to rely on search engines for traffic. The bots slow down my server, so that means they must be slowing down the internet experience for millions of people as they visit sites like mine. I wonder how long that situation will last.


  1. Paul, I don’t know if you’ve seen yet, but it’s the blog of a large content site webmaster who writes regularly on this subject (and does a pretty good job of ‘bot blocking.) Profane but educational – I recommend him.