Tired of trawling 10000 hits? Try 1 billion!

[U-96]

For anyone who relies on search engines for work or play and heard the news even the best only cover 16% of the web, listen up...

A new (okay so I hadn't heard of it http://www.sysopt.com/forum/smile.gif ) search engine makes the ambitious claim to index the entire web within 12 months.
For the details:

http://news.bbc.co.uk/hi/english/sci/tech/newsid_410000/410251.stm

For the engine:

http://www.alltheweb.com

For another engine mentioned in this article (apparently scores well on relavence)
http://www.google.com

For a report on "Accessibility and Distribution of Information on the Web"
http://wwwmetrics.com/ (report by return email)

For a very good site which keeps tabs on latest search engine developments:
http://www.searchenginewatch.com/

Comments on successes/failures are encouraged http://www.sysopt.com/forum/smile.gif

U-96

[socalgal]

www.dogpile.com (http://www.dogpile.com)

I'll check out those others. How are the relevancy hits (just wondering...)

[U-96]

OK I tried both Alltheweb and Google with similar searches: here are some results; draw your own conclusions.
If anyone wants to add scores for regular engines like altavisa and yahoo, that would be good for comparison (i'm too lazy).
Note: "relevance" is relative by its very nature. I wasn't too harsh in applying this, but from the search string you can guess which kind of site I was looking for http://www.sysopt.com/forum/smile.gif

Google:
String sent: System optimization
Hits: 971 in 0.33secs
Relevance: High (www.sysopt.com is first - other sites on similar subjects)

String sent: "System optimization"
Hits: 3744 in 0.09secs
Relevance: High (ditto)

Alltheweb:
String sent: System optimization
Hits: 23273 in 0.65secs
Relevance: Poor - the sites contain the words, that's it

String sent [exact phrase selected]: System optimization
Hits: 8554 in 0.296secs
Relevance: Poor - not much better than the first, but getting better

String sent: "System optimization" PC
Hits: 5999 in 0.45secs
Relevence: Good - www.sysopt.com 5th, other sites on similar subjects

Conclusion
If you know what you are looking for, and have the time to trawl, the AlltheWeb really does seem to be indexing the sites it claims. To get the best out of it, I believe making the search term as narrow as is practicable is the way to.
However, if you want a result from a simple word or phrase, then Google's reputation for relevance seems justified. I got what I was looking for, so I'm a happy user.

Not particularly scientific, but it demonstrates the quality vs. quantity issue quite well.
If anyone can suggest a way to benchmark search engines, I would be glad to give it a go.

U-96

PS yup socalgal I'm a dogpile convert too - that one has saved me soooooo much time http://www.sysopt.com/forum/smile.gif

PPS One day I shall write a short post http://www.sysopt.com/forum/wink.gif

[socalgal]

Too lazy? Could have fooled me!

BTW, I've been looking all over for... JK http://www.sysopt.com/forum/smile.gif