When Big And Small Become Statistically Equivalent

If simple math helps create an "all things being equal" situation among providers of usage data, whom do we decide to trust, and why? Even Omniture's Matt Belkin admits that small samples, such as Janco's and others, have their place in the overall evaluation of Web trends. That's especially the case, as Belkin pointed out, "if you're doing a really deep dive, and there's some really interesting things you're trying to figure out about agents. Even a sample of one customer could be really interesting, if you're specifically trying to isolate agent versus non-agent activity." However, in his view, he would stop short of declaring such small samples as "statistically representative of the entire Internet population."

Sharon Fisher is an independent computer industry consultant, and former analyst with a major firm. We gave her the market share numbers from WebSideStory's StatMarket, Janco, and OneStat.com, and asked her to assess their relative value, given her own experience at tallying similar numbers herself. She told us a key gauge of the reliability of such numbers is the makeup of the analysts and their directors: "Who are they, who funds them, what's their methodology?" Looking at StatMarket, she said she believed they could render an unbiased opinion; but Janco and OneStat.com, she said, revealed too little about their own internal structures. Such knowledge could help alleviate skepticism about a firm's bias and motive.

But besides a firm's background, Fisher said, it's important to determine what it truly means by the terms it uses. "How are they defining 'market share?' How do they know?" she asked. She pointed out that WebSideStory and OneStat.com - which reported Firefox's market share at 8.45% last March - both use the term "usage" without defining how they apply that term in their research. StatMarket's Geoff Johnston told us his company prefers the term "usage data" as being typically more accurate than "market data."

Many Web sites are publishing their own log file analysis, which openly disagrees with major firms' market share numbers. This has been fueled by the growing interest in traffic measurement prompted by such industry leaders as Sun Microsystems' Tim Bray - whose personal studies sparked the latest round of discrepancy speculation. One log file analysis produced by the freeware package "awstats," and published by the popular blog site BoingBoing, reports over one million unique users thus far during the month of July 2005. Among those, 39% have identified themselves as Firefox users, versus 34.4% for Internet Explorer.

The growing interest in the success of Firefox - a product without a major marketing division or even a corporate structure to back it up, managing nonetheless to capture a substantial share of traffic - is fueling renewed skepticism about how we've assumed the Internet works thus far. How Web sites and service providers react to this skepticism may either increase their understanding of it, or quite possibly, renew their fear of it.