Are Review Units Better Than Store Versions? 407
Anonymous Howard writes "Every now and then you hear about hardware manufacturers optimizing their hardware for certain tests or games to make their hardware look superior. I was surprised to hear of a new controversy brewing over reviewer units sent to hardware reviewers. This article claims that Samsung is sending LCD monitors with a contrast ratio of 700:1 when the consumer version of the same monitor has a contrast ratio of 450:1. Various sites list different specs for the same model, so it's somewhat confusing to know for sure which is correct. I don't doubt this happens, but I'm surprised that it would be this blatant. Has anyone heard of other stories of manufacturers being deceptive so that they could get better reviews?"
That's why Consumer Reports (Score:4, Informative)
buy their stuff off the shelf to use in reviews. Otherwise companies will send the cherries to reviewers.
I worked for a couple of electronic manufacturers that had a standard operating policy to do this very thing.
If it looks like a duck... (Score:2, Informative)
Quack II, anyone?
Reviewers (Score:5, Informative)
Unfair tweeking is part of the reason why Consumer Reports never accepts review units from companies, but rather buys them from retail stores, just like anyone else would. The other reason is that receiving free stuff creates a potential conflict of interest which is why they also do not have any advertizing in their magazine or their website. This means that you won't have reviews out before products are released, and operating this way is more expensive, relying on subscribers to run, but it is worth it. I don't always agree with CR's subjective descriptions of products (cars especially), but the hard numbers they provide are the most usefull I have found, and have saved me plenty of money.
I really wish that there was some site equally trustworthy in the computing world. For providing informative analysies there are usefull sites (I have always been impressed with anandtech). But for reviewing components, I have yet to find one I trust.
It gets even worse - Best Buy for Example.... (Score:5, Informative)
Take a look real hard at that stereo reciever before you buy it....
Benchmarking developers have known this for years (Score:2, Informative)
More often than not you could catch this stuff and even the playing field when reviewing hardware.
The video card hardware vendors were even more creative.
A story (Score:3, Informative)
What they told me was quite strange at the time, they said their review unit had a different refresh rate and that they checked with Samsung, but that there was no definite answer as to how this could have happened. All in all, they gave me a 19" for free for the trouble (which they apparently had no part of.)
This happened in Toronto, Canada in 1998.
It is good to know SlashDot picks up on such small things.
Consumer Reports (Score:3, Informative)
Review sites that take donated hardware and advertizing from those same hardware vendors should always be held somewhat suspect until you verify the quality through another source. Few sites are willing to give a bad item "both barrels" because they would be essentially slashing their own throat/revenue stream.
It's called a "Golden Sample" (Score:3, Informative)
Perhaps I'm just overly cynical, but I tend to trust reviews where the reviewer went out and purchased an off-the-shelf retail copy of X rather than those where the company sent something. Of course, this is hard to do in print publications, because of the time-lag that magazines run through (ie, two months after it's released on the shelves, they have a review of it), but I see no reason (aside from money, which is a big reason) that online reviewers can't do things such as this. I also tend to look towards user-reviews and give those a pretty good weigh-in when I'm making a purchase decision. This is the first instance that I can recall where products are blatently better when given to reviewers than those that are store-bought, but I get the feeling that it's been done in the past.
The above paragraph reflects what I do for my personal buying choices and should in no way construe that that's the optimal/correct/whatever way for large corporations/organizations/whatever to buy-in-bulk
Consumer Reports (Score:3, Informative)
Consumer Reports will not accept donations of vehicles or products from manufacturers or vendors just for this reason. They will discreetly send someone out 'under cover' to go acquire the products in an "off the lot" or "off the shelf" state.
This is good, and commendable.
However, i see a lot of times they will end up mis-matching the cars and trucks they compare. Usually it is simply a matter of trim levels on similar classed models. This *will* have an impact on the final outcome. Obviously it's difficult to do things *exactly*.
Less often, but still wrongly, they will compare vehicles from incompatible classes- things like Buick Century vs. E-class Mercedes vs. Toyota Camry. Or the classic truck comparisons with the 3/4 ton, V8 powered Dodge and Chevy fullsize trucks, against a V6 F150, against V6 Toyota Tundra and Nissan.
Consumer Reports might do this to other product reviews too, but i only pay attention to their auto ads for `entertainment'.
I guess that no matter what, *any* test can be flawed.
Samsung 955DF CRT switcharoo (Score:3, Informative)
Now the "Samsung 955DF" has controls on the front, the screens are much more reflective and oily-looking, and black appears grey even when the brightness is all the way down. More recent Samsung 955DF [balta.pl]
Re:Well (Score:2, Informative)
Most likely because doing so would be somewhat cost-prohibitive.
my experience as a reviewer (Score:3, Informative)
A lot of the manufacturer reps and pr reps I worked with would hand-select or pre-screen review units, but I never ran into any where I thought I was being given something better than what would ship just to get a better review.
Re:That's why Consumer Reports (Score:2, Informative)
CR's sports car reviews always make me laugh a little. It seems like in every one of them they always complain about the stiff ride, engine noise and fuel economy.
Re:Car and Driver (Score:2, Informative)
A 0-60 time difference of half a second could easily be attributed to a natural difference in each car. Cars differ more than you think from one to the other and I have read more than once about discrepancies in cars within the same model.
In fact (I was trying to find it) but I believe Car and Driver (or some other car mag) said exactly this in a "letters to the editor" reply in a recent issue. Sometimes discrepancies of over a full second will also happen depending on where the car is in the life cycle. I agree that this isn't the case here since they received a "lifetime" car but it just shows you the number of variables a car goes through. By the way, sometimes they start fast and slow down and sometimes they are slow and speed up.
At any rate, I agree that it is odd, but I don't believe this is proof of deception.
Used to happen a lot with graphics... (Score:2, Informative)
For vendors that did this chronically we switched to getting boards through other channels -- but we needed the hardware as soon as it was released, so we'd usually have pending orders with the retail arm of a board manufacturer. They got wise to this and started doing the same thing with retail boards being sent to us.
Then we switched to straw buyers. Since there were only a few preorders made in my state (AZ) they started doing it to all boards destined here, which was pretty entertaining. We'd wait a month and buy the board from a storefront and it'd be clocked 10-20% slower.
I won't even begin to go into what we saw happen with drivers...
Are you kidding? (Score:3, Informative)
In a previous job, my employer had a special team of people called "Product Managers" - but their job was to go visit magazine reviewers, ensure that they got top of the line grade A technical support during the review process, including onsite support, and coded patches directly from the developer's desktop to the reviewer's. Additionally, there was wining and dining, and talk of strippers and lapdances (though I never witnessed that). In that sense, what was reviewed in no way bore any resemblance to the shring-wrapped package some poor sucker paid $699 for.
I'm no longer working in that sector, but for my 10 years, the practice was commonplace. Which is why I never read reviews.
Re:Well (Score:3, Informative)
Re:Reviewers (Score:3, Informative)
I'm not vouching for them, but Legit Reviews bought retail memory for a recent review [legitreviews.com]. I also liked Anand's recent test [anandtech.com] of OCZ memory, comparing pre-production and retail parts.
Oh Please. Unit mix-up? Domestic vs Int'l? Typo? (Score:3, Informative)
The contrast specs on the Samsung USA site show the following:
172T [samsungusa.com] - 700:1
173T [samsungusa.com] - 450:1
The specs on the Samsung Canada site say:
172T [samsung.ca] - 500:1
173T [samsung.ca] - 700:1
Perhaps he got a Canadian unit although I don't know why they would be any different.
Test Engineering (Score:2, Informative)
The last company I worked for engineered a benchmark test before the company's new RAID SCSI host card was completed.
The card was benchmarked against a RAID Ultra SCSI LVD card from a German competitor named ICP Vortec. Even though the new card was in development, they expected it to beat the Vortec. It didn't...until they removed most of the Vortec's components and replaced them with parts from a much slower (and inexpensive) card that wasn't even considered a serious competitor. These benchmarks were used in advertisements, entire phrases stating the superiority of the product were co-opped from press releases by lazy magazine columnists (which is sadly common in most journalism nowadays), and sent to companies requesting further information. On a somewhat related note, a claim on the product sheets for the aforementioned product line that the product had been Windows certified ended up costing the company a $10K fine from MS and had to be destroyed.
I guess that ultimately it doesn't matter. The owner sold the company to the largest competitor.
Re:Ah, the Sad Effect of Technology (Score:4, Informative)
Yes, great price, 15% to 30% below everyone else. Then once you order it, they call and mention that the camera doesn't have a warranty, that's extra. You know, the plastic lens mount is of lesser quality, you'd be much better of with model with the metal mounting ring. Of course the battery charger and battery is not included in that model.
Whoops, the price is now 25% above everyone else, and you have the standard package that everyone sells. Those other models you've upgraded from don't exist.
Of course, if you stick to your guns and insist on the advertised price, it's mysteriously never in stock or gets lost in the shipping system.
BTW, there are 3 or 4 genuine web shops in NY, check DPReview forums, etc to find them.
Re:Well (Score:2, Informative)
Effectively making the manufacturer pick up the retailer's profit margin bill. The samples manufacturer's usually send out are direct from the warehouse/factory, so the cost is significantly smaller.
Here's a classic example (Score:3, Informative)
http://www.valentine1.com/lab/MikesLabRpt5.asp
While you're there, please check out a Valentine One. Mike Valentine makes by far the best detector on the planet, and he's a heck of a nice guy!
--