[ILUG] MSN Search Beta

[John McCormac]

On Thu, 11 Nov 2004, Colm Buckley wrote:

> On 11 Nov 2004, at 09:19, Chris Higgins wrote:
>
> > The MS alternative to google - but look at the URLs that are
> > generated - MS track every search you make !
>
> In fairness, the links from Google are *also* redirect-based.  However,
> it's not for the purpose of building up a profile or whatever, it's
> basically so that we can see which results get the clicks and use those
> data to tune our index.

The MSN Search has, from looking at logs here, some serious database
corruption. (Trying to look for .asp pages that are not present on servers
here.) The msnbot is so badly written that it does not use 304s and puts
excessive loads on webservers. So many webmasters have complained about it
that Microsoft even introduced its own robots.txt entry so that webmasters
can use a delay between pages being fetched by its scrapers.

> Has anyone noticed the increase in our index size yet?  Tee-hee.

Nearly 8 billion? I'm surprised Microsoft doesn't claim 10.  :)

Regards...jmcc