I recently noticed that almost all the cameras they reviewed were "highly recommended". So I looked at all the reviews they have done since 1999 and an interesting trend emerges.
The graph show the average of ratings awarded each year using 5 for "highly recommended" to 1 for "below average". 14 of the 19 cameras reviewed in 2009 were "highly recommended".
This kind of 'grade inflation' is understandable; the site relies on equipment makers loaning them the cameras for (pre)review and doing so early so as to attract eyeballs to the site. Manufacturers are understandably more likely to lend equipment to sites that give them high marks. So getting hold of new cameras is easier if you give the makers better ratings.
However the value to consumers declines as ratings are compressed into the top category. Telling one from another is no longer a case of looking at the bottom line, but now requires making a subjective judgment (often based, in no small measure, on someone else's subjective judgment).
Ratings inflation also compromises the site's brand for integrity and objectivity. If the raters are basing their assessments on something other than their response to the product, namely the desire to get product to review and eye-balls to the site, how much faith can one place in what they write? (And to my eyes, some of what they write bears absolutely no relation top the test images they comparing.)
Of course, this kind of compromised objectivity applies not only to reviewers of digital cameras; the financial instruments that were the building blocks of the "normal accident" (Perrow, 1984) that befell our economic system last fall, were rated by supposedly impartial third parties - and look how well that worked out.
