Analysis of search engine market shares is very inconsistent and varies by up to 50% from source to source. Some claim Yahoo to make up 1/4 of all searches, and place Google just slightly ahead. Others accredit the entire market to Google. Some bloggers have tried to perform their own analysis, but the numbers closer to the truth are in the comments – how many hits do webmasters actually see? It seems that Google attributes to 75~85% of incoming search engine traffic, Yahoo at about 10~15% and MSN trailing at about 5%.

Interested to find out for myself, I went looking closer through my access logs.

Over the last two months, CompSci.ca domain has recieved 20,000 unique visits, 4000 of which were organic. Google accounted for 95.3%, while Yahoo made the remaining 4.0% (0.7% unaccounted for.. possibly MSN). Feeling under appreciated by the alternative search engines, I went to research further.

How much do search engines index?

Using the site: metasearch, I was able to see how many pages each engine actually has indexed – that is, how many different pages can actually be found.

site:compsci.ca

The results seem to relate to the incoming traffic, possibly offering a first clue to this market.

• Google – 26,300
• Yahoo! – 2,623 (10% that of Google)
• MSN – 130 (0.5% that of Google)

Between 12,170 forum threads (10 posts per thread average), 4300 registered users, and all the Wiki pages and various blogs, Google seems to have a pretty good coverage of the pages available. Others cover just a fraction.

How do indexes compare to other sites?

Though for a more empirical comparison, another related question must be answered – “How often does my website come up in search results?”. Since each engine has a vastly different indexed set size, there needs to be a way to compare them against themselves – against pages each search engine has indexed from other related websites.

CompSci.ca’s unique niche content is the Turing programming language. I wanted to know what percentage of such niche searches result with one of my pages. A total of 6 searches were performed with each engine:

computer science canada
site:compsci.ca computer science canada

turing programming tutorials
site:compsci.ca turing programming tutorials

“hacker dan”
site:compsci.ca “hacker dan”

Starting out with a general topic, moving on to niche, and finally a very specific keyword (this site’s co-admin’s alias).

How do search engines compare?

/	Google	Yahoo!	MSN
computer science canada	50,900,000	25,900,000	3,481,737
site:compsci.ca computer science canada	2,970	322	264
Percent:	0.0058%	0.0012%	0.0076%
turing programming tutorials	1,230,000	208,000	15,917
site:compsci.ca turing programming tutorials	594	291	217
Percent:	0.048%	0.140%	1.363%
“hacker dan”	14,300	2,370	2,255
site:compsci.ca “hacker dan”	4,000	375	89
Percent:	27.97%	15.82%	3.95%

What does this all mean?

Google indexes more. Much, much more. Though it is interesting to note that while Yahoo! has indexed just 10% of compsci.ca webpages that Google has, the ratio is not the same for all of the web. On computer science canada topic, Yahoo! has 50% of Google’s results, and MSN has 7%. Decending into a more specific topic of turing programming tutorials, Yahoo! shows just 17% of Google’s indexed pages, but assuming that compsci.ca is indeed the most accurate source on the topic, it’s interesting to note that Yahoo! results in 3 times the accurate (mine!) matches. Better yet, MSN’s tiny 1% index size results in the best match for what I’m really looking for – MSN has 28 times better concentration of niche content results than Google!.

“hacker dan” is a very peculiar case to consider. Once again Yahoo! shows just 17% of Google’s indexes, and so does MSN. And while 1/3 of Google’s results point back at compsci.ca, one should keep in mind that Dan accounts for 3807 of the forum posts, and since Google has indexed just about every thread, it’s natural that his signature would come up this much. Yahoo! and MSN, now on par in results, display a much broader variety of content.

Bottom line of search engine comparison

Google will index every page it can get to. Your content will be stored and accessable, but likely lost in the web of every other page out there. A fine-tuned search query is a powerful tool.

Yahoo! index size is smaller. They try to be Google, but are falling behind. Not much more to be said here for now.

MSN resulted in a surprise. MSN offers a tiny index in comparison, but it might be making up with the quality and variety of results. MSN offered the best concentration of results on a topic, and a far better concentration of good results for a specific niche topic. Very interesting to note that in search for a very specific keyword, MSN resulted in the least hits to a specific site, but a much broader variety of choices.

My conclusion to search engine traffic

Yahoo! and especially MSN appear to put variety of resources over the raw quantity. Since every forum thread looks similar, a large portion of them gets neglected. In order to increase the appearance of your website in index, I would recommend offering unique content from page to page. A closer look at the index of this blog specifically site:compsci.ca/blog supports this theory, as Google, Yahoo! and MSN have indexed 56, 56, and 55 pages respectively.

I would love to hear what kind of search traffic others have been getting, and what I can do to tap into those 15% that Yahoo! supposedly offers.
– Tony

Read more

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.