Do we need a crawler roster?

The other day I was working on my file server after a couple of hard drives failed within almost an hour of each other. Bizzare and unfortunate as it may sound, I was not in so much of a panic as I had the two drives mirrored onto another large, hard drive daily using Delta Copy. While potentially I could have lost nearly 200GB of data on the two drives, the fact that they were mirrored meant that I only lost around 10GB of files that were mainly DVD rips (will have to rip them again).

To cut a long story short, I was waiting for Windows to install on my server when I noticed that the response time for my web server seemed to have increased considerably. While it is not the fastest web server around, it has been doing its job well for nearly five years with a few reboots in between. Usually it responds to web requests quite speedily, but for some reason it was taking its time. For a very brief period I felt a little panicked about the possibility of the server’s main hard drive failing, but after seeing half of my website load I figured something else was up.

I looked at the netstat and it showed there were quite a few connections open and on reading the server’s log file life (via tail -f) I realised that not only I had Googlebot crawling my site, I also had Slurp from Yahoo and another bot from Webalta in a Russian domain all crawling all over my web server at the same time. This kinda crawling gave my web server the creeps.

Can’t these bots at least have a roster about who was going to crawl which site at any given time? What is the point of being indexed by the search engines if no one can reach your site without patiently waiting for nearly a minute?

Let the better format lose!

The HD DVD is looking like it’s on its last legs at the moment. Toshiba has quietly admitted defeat to Sony’s Blu-Ray and is currently preparing to cease further production of hardware for the platform at the cost of hundreds of million of dollars and probably thousands of jobs worldwide. Having been dumped by the movie studios and then retailers such as Netflix and Walmart, the future of the format looks a lot less than rosy. Certainly without the support from the movie studios the format will be without or very limited entertainment content which makes it less desirable for the consumers.

The problems that always faces a new format in the market is that it cannot get support from the movie studios unless it is popular and it cannot be popular until the movie studios supports it. Well it is just one of those dilemmas, I guess. So while I am not looking forward to having a Sony-only entertainment format choice (could this be called Buckley’s choice?), I am kind of glad that the format war (was there ever a war like VHS vs Betamax, anyway?) is now almost over. It can be now definitively said that the better format lost the way (I am not sure if Blu-Ray or which one is better I am just saying that because the looser is always the better format).

One thing that HD-DVD could do is to direct the platform more towards data storage or software delivery platform. This might be where HD-DVD will end up, rather than in the living rooms of consumers. Well, an HD-DVD player is always welcome in my living room!