Are Review Units Better Than Store Versions? 407
Anonymous Howard writes "Every now and then you hear about hardware manufacturers optimizing their hardware for certain tests or games to make their hardware look superior. I was surprised to hear of a new controversy brewing over reviewer units sent to hardware reviewers. This article claims that Samsung is sending LCD monitors with a contrast ratio of 700:1 when the consumer version of the same monitor has a contrast ratio of 450:1. Various sites list different specs for the same model, so it's somewhat confusing to know for sure which is correct. I don't doubt this happens, but I'm surprised that it would be this blatant. Has anyone heard of other stories of manufacturers being deceptive so that they could get better reviews?"
Well (Score:5, Funny)
Re:Well (Score:2)
Re:Well (Score:5, Insightful)
Consumer reports had the right idea, that is why they have been so successful.
Re:Well (Score:2)
Re:Well (Score:3, Insightful)
It's nice to know that the data comes from people who also read the magazine, and are interested enough in the data to report it without any sort of incentive. I would be more worried about getting real data if any sort of incentives were offered.
For those of you who are not
Re:Well (Score:3, Insightful)
Re:Well (Score:2, Informative)
Most likely because doing so would be somewhat cost-prohibitive.
Re:Well (Score:3, Informative)
Re:Well (Score:2)
Re:Well (Score:5, Interesting)
It definitely isn't an objective model, but one that allows multiple people to review the same product. Ultimately, you have to make two really shaky assumptions in such a model.
1. The vendor is sending the reviewer a consumer level product (or nearly so).
2. The reviewer is objective and honest enough to verify the capabilities match that of the consumer available product and disclose where the product came from.
For the most part the model works. Point 1 is completely out of the consumer's control and cannot even be known to the consumer without point 2. As a result, there is only one thing that consumers can do, and that is learn to spot the honest reviewers from the frauds, fanboys, and sponsored reviewers. For example, Anandtech [anandtech.com] and Tom's Hardware Guide [tomshardware.com] do a pretty good job. They clearly indicate where the hardware comes from, identify any differences between it and the shipping hardware, and do their homework as best they can to prevent getting duped. Contrast this with many reviewers who seem to be simply paying lip service to the vendors so they can get quoted in an advert and continue to get free hardware to play with. Researching the product also means researching the reviewer if you don't wanna get burned. It's like taking advice from the Gartner Group without seeing who paid for their latest study.
Re:Well (Score:2)
why aren't the reviewers performing their tests with retail purchased equipment for integrity sake anyway?
Because integrity is secondary to ad revenues. Good review = more ad revenue.
Are Review Units Better Than Store Versions? (Score:4, Insightful)
Yeah, in my review version of the Ep IV DVD (Score:5, Funny)
Re:Yeah, in my review version of the Ep IV DVD (Score:2)
FARK : OBVIOUS (Score:3, Funny)
Ah, the Sad Effect of Technology (Score:3, Funny)
Re:Ah, the Sad Effect of Technology (Score:2, Funny)
-r
Re:Ah, the Sad Effect of Technology (Score:2)
Re:Ah, the Sad Effect of Technology (Score:4, Informative)
Yes, great price, 15% to 30% below everyone else. Then once you order it, they call and mention that the camera doesn't have a warranty, that's extra. You know, the plastic lens mount is of lesser quality, you'd be much better of with model with the metal mounting ring. Of course the battery charger and battery is not included in that model.
Whoops, the price is now 25% above everyone else, and you have the standard package that everyone sells. Those other models you've upgraded from don't exist.
Of course, if you stick to your guns and insist on the advertised price, it's mysteriously never in stock or gets lost in the shipping system.
BTW, there are 3 or 4 genuine web shops in NY, check DPReview forums, etc to find them.
pressure on reviewers (Score:5, Insightful)
Like in "here's a free expensive item for review that you get to keep. We'll be watching the review to see if you get anything else to review? Oh, it's still happening, but sending the reviewer a item that isn't the same as the crap they intend to sell you and me is just a little added insurance.
You can pretty much see this in a lot of reviews that are written too. The only reviews that merit much trust are the independent ones where the reviewer actually went out and got an off-the-shelf item to review; but this is an almost dead pratice. No only does the reviewer not get neat fre stuff then, but his review may be months after the reviews by the company shills come out, and he ends up with the same crap you and I get rather than the free good working versions.
Re:pressure on reviewers (Score:4, Insightful)
Now what do you suppose happens to her free trips if she publishes unflattering copy about her "hosts?"
You don't have to be bribed to be beholden.
She could, of course, simply take her own trips on her own dime. .
And thus the media is corrupted without any application of coercion at all. No threat to remove advertising or anything. Just a loose understanding by everyone involved as to what's in their own personal interest.
KFG
KFG
Re:Ah, the Sad Effect of Technology (Score:2)
works for me, at least.
Wait wait... (Score:2, Interesting)
Re:Wait wait... (Score:5, Interesting)
Bah. They had the same model number on two different models. That's a lie.
Is the difference between an outright lie and a deception really that important here anyway? They were expected to send the same product real consumers get. They didn't. That's enough to condemn them in my book, whether there's an outright lie there or not.
Re:Wait wait... (Score:2)
The model may be the same, but some low level qualifier (version, revision, manufacturer ID, etc.) is very likely unique between the units. Otherwise the manufacturer can't tell them apart either.
Re:Wait wait... (Score:2, Interesting)
3com Ethernet cards, when they were the most popular (1997?). They came out with new chipsets that required different drivers under the same product name.
Started doing the model "a", "b", crap.
At least LinkSys puts "Version x.x" on their boxes!
Re:Wait wait... (Score:2)
Yes! (Score:2, Funny)
Yes. Thank you for asking.
That's why Consumer Reports (Score:4, Informative)
buy their stuff off the shelf to use in reviews. Otherwise companies will send the cherries to reviewers.
I worked for a couple of electronic manufacturers that had a standard operating policy to do this very thing.
Re:That's why Consumer Reports (Score:5, Interesting)
CR tries to distill down all sorts of subtle performance parameters into a box score that ranks easily against competitive products, and in the meantime, miss the value of those parameters. Quite honestly, I'd be surprised if CR could accurately determine if they had a cherry LCD display or not, given the "rounding error" of their review/comparison methodology.
Tim
Re:That's why Consumer Reports (Score:2, Informative)
CR's sports car reviews always make me laugh a little. It seems like in every one of them they always complain about the stiff ride, engine noise and fuel economy.
CR's two biggest problems (Score:3, Insightful)
This would be easily mitigated if the product reviews were serious articles with a lot more commenta
Re:CR's two biggest problems (Score:3, Insightful)
Re:That's why Consumer Reports (Score:3, Insightful)
That doesn't invalidate their methodology of buying review units off the shelf, though. Good magazines learn this lesson early. Car and Driver (which *ahem* does have good sports car reviews) has written about an early radar detector revi
Re:That's why Consumer Reports (Score:3, Interesting)
I heartily agree with this. Consumer Reports reviews ae pretty close to useless in many areas. Yes, they tell you which product they liked better. But they never seem to give you any of the numbers or any information on how they decided which product was better. They will tell you that one receiver performs better than the others, but they don't tell you how they te
Re:That's why Consumer Reports (Score:3, Interesting)
So the review is incomplete, at least to me.
For example, in the (aged) review on vaccuum cleaners, I beleive that I have purchased one of the better (if not the best) models on the market. Yet CR has never reviewed any machine from the company.
Also, they do not update their product ratings. If they reviewed a series of toasters in 1997, and that's the latest
Re:That's why Consumer Reports (Score:3, Insightful)
Cherries are one thing -- but did your employers really send out demo models with fundamentally different capacities than in the specs? I'm surprised this is the first time a reviewer noticed that, say, the 250 cc motorcycle he was reviewing looked suspiciously like a 600.
Re:That's why Consumer Reports (Score:2)
> say, the 250 cc motorcycle he was reviewing looked suspiciously like a 600.
Oh, I'm sure that would get caught... the real question is, would he notice the port'n'polish job on the head, the blueprinted motor, carb tuning or EFI mapping (throwing it out of EPA spec), degreed cams (ditto), etc, etc?
Of course, the manufacturer would have to be careful not to do anything to the motor which increased its octane requirement, or the reviewer
Re:That's why Consumer Reports (Score:2)
In any manufacturing line there are units that are built more carefully, look better, perfom to closer tolerances, more polished, that end up performing better/more reliably than a standard unit.
Technically they are the same as every other unit. But due to their careful manufacturing would operate better than any unit pulled off the line. There are other specifications that are not mentioned in the literature that are equally important to operation. So it's not really a question of different operating spec
Scandalous! (Score:5, Funny)
But only by the fact that Samsung have never sent me [dansdata.com] any such thing.
Dammit, I got into this business for the corruption. But do I get over-spec high-dollar hardware, automobiles or prostitutes? No, I do not. It's a bloody swindle, I tell you.
Look, Samsung. 20 inch diagonal, 1600 by 1200, 700:1 contrast ratio, 16ms response time. Is that too much to ask?
Delivery address provided on application. Favourable review [dansdata.com] guaranteed.
Re:Scandalous! (Score:2)
hey wait, isnt it like some ungodly hour Down Under right now?
Re:Scandalous! (Score:5, Funny)
I'd really like to know what's an over-spec prostitute. I think I can figure out what's a high-dollar one.
Re:Scandalous! (Score:5, Funny)
Well, you know how sometimes you'll get one with a little extra equipment "down there" . . .
Re:Scandalous! (Score:4, Funny)
(but (question-do I (get (or (is hardware (and over-spec high-dollar)) automobiles prostitutes))))
See how clear everything is now?
Re:Scandalous! (Score:2)
Hell, I'd buy one without question.
If it looks like a duck... (Score:2, Informative)
Quack II, anyone?
Re:If it looks like a duck... (Score:4, Funny)
Re:If it looks like a duck... (Score:2)
Reviewers (Score:5, Informative)
Unfair tweeking is part of the reason why Consumer Reports never accepts review units from companies, but rather buys them from retail stores, just like anyone else would. The other reason is that receiving free stuff creates a potential conflict of interest which is why they also do not have any advertizing in their magazine or their website. This means that you won't have reviews out before products are released, and operating this way is more expensive, relying on subscribers to run, but it is worth it. I don't always agree with CR's subjective descriptions of products (cars especially), but the hard numbers they provide are the most usefull I have found, and have saved me plenty of money.
I really wish that there was some site equally trustworthy in the computing world. For providing informative analysies there are usefull sites (I have always been impressed with anandtech). But for reviewing components, I have yet to find one I trust.
Re:Reviewers (Score:3, Insightful)
Think about it...a good graphics card roundup should review cards from all of the companies that make a card based on a particular chipset. If there are 8 companies making that card, at $200+ retail each, that's $1600+ per review.
Of course, I don't think it's a bad idea. Just one that will take a little bit of ingenuity. A good method MIGHT be to sell advertising space not to hardware manu
Re:Reviewers - I'll do it... (Score:2)
Re:Reviewers (Score:3, Informative)
I'm not vouching for them, but Legit Reviews bought retail memory for a recent review [legitreviews.com]. I also liked Anand's recent test [anandtech.com] of OCZ memory, comparing pre-production and retail parts.
Re:Reviewers (Score:3)
CR just gives the facts.
Some see cheap details like that being missing as indicitive of a cheap product with every corner cut. If they're too cheap to spring for a $0.0025 LED, what else might be missing?
It's up to you whether or not you care if the eMac has a HDD activity light. CR is just reporting what's there.
Well.. (Score:2)
It gets even worse - Best Buy for Example.... (Score:5, Informative)
Take a look real hard at that stereo reciever before you buy it....
Re:It gets even worse - Best Buy for Example.... (Score:2)
News to me. Do you have any specifics?
I thought the usual game was for these kinds of retailers to sell model numbers that don't exist anywhere else, but have the same specs as a commonly available one, so that they can "honestly" deny any compettitive pricing claims. ;-)
BRe:It gets even worse - Best Buy for Example.... (Score:5, Interesting)
Having worked at best buy, in the audio department, I can tell you that this is a fact. We sold products that other people did not sell; in fact, it was fairly rare to find products from well known manufacturers that had the same model number at Circut City and at Best Buy. It was done so that we could a.) say that we had exclusive products, and b.) say that we weren't doing competative pricing, but the much bigger one was 3.) when the models had different features, some people want the one from CC, some want the one from best buy. If they had Identical features, people would just go the place with the lower price, but because one may have an extra S-video input or what-have-you, they're willing to shell out the extra $20 for the extra stuff. The trick is for the corporation to get the model that looks more desireable to consumers.
But, yeah, the main reason for the similar-but-different model numbers for similar-but-different products is to keep people from being able to compare identical products, and thus, simply wait for the sale. It's perfectly legal, and when you think about it, pretty smart.
~Will
Another reason - no bugus returns... (Score:3, Interesting)
An example of
of course this happens (Score:3, Insightful)
car companies used to do this all the time. they would send a 'ringer' to the review magazines. you would then get your car, put it on a dyno or take to a track and not be able to match the numbers.
just one of those buyer beware things.
Ethics (Score:5, Insightful)
-Shadow
Re:Ethics (Score:2)
maybe that has something to do with it.
Re:Ethics (Score:2)
The immediacy of todays news, people with personal weblogs, and web sites that can stir up interest in a little news item probably means that companies are just being exposed more.
I'm not suggesting that corporations are filled with crouched over profit dwarves, physically drooling over the prospect of sucking some sap dry, it's just that the little decisions that 'sounded good at the time' are be
Re:Ethics (Score:2)
As long as there is success to be had from sketchy processes, there will be sketchy processes. Ethics are nothing more than a form of PR. If you believe otherwise, by all means start your own company. See how far you get.
Re:Ethics (Score:2)
Money has the power to let people forget the difference between good and evil. --paraphrased from the movie Sneakers
At Business schools they try to teach business ethics. In fact with the recent stink of corporate scandals, many schools are finding it necessary and not an elective to have ethics traning. I heard an NPR interview with an ex-con who went to prison for fraud on a coporate level. He now works by speaking to MBA students about his experience with ethics and how not to fall into the same tr
Benchmarking developers have known this for years (Score:2, Informative)
More often than not you could catch this stuff and even the playing field when reviewing hardware.
The video card hardware vendors were even more creative.
A story (Score:3, Informative)
What they told me was quite strange at the time, they said their review unit had a different refresh rate and that they checked with Samsung, but that there was no definite answer as to how this could have happened. All in all, they gave me a 19" for free for the trouble (which they apparently had no part of.)
This happened in Toronto, Canada in 1998.
It is good to know SlashDot picks up on such small things.
Not Surprising (Score:2)
Re:Not Surprising (Score:2)
Yeah, but it least it's random then, much like the actual consumer. When you get it directly from the manufacturer, there is little reason to believe they picked a random unit. Furthermore, we're talking a difference in specifications here (wrt Samsung). It's pretty hard to get, say, 2 HDs off the assembly line and one 'randomly' only ha
You know... (Score:2)
Consumer Reports (Score:3, Informative)
Review sites that take donated hardware and advertizing from those same hardware vendors should always be held somewhat suspect until you verify the quality through another source. Few sites are willing to give a bad item "both barrels" because they would be essentially slashing their own throat/revenue stream.
Is it really that surprising? (Score:2)
Hell, I work with commercial billing systems, and I can tell you nightmare stories of benchmarks being run on "special" data.
Reviewers shouldn't tell them they are reviewing! (Score:2)
In a perfect world reviewers would be able to pay for the items they review without letting the manufacturer know. But unless your reviewing technology that's 5 years old the price is just to insane. Not to mention that most review sites online aren't anywhere near a real "money maker" and plaster ads everywhere trying to ma
It cuts both ways (Score:5, Insightful)
Car and Driver (Score:5, Interesting)
Now, ever car enthusiast knows that 0-60 times and such the like are subject to various conditions, but that's a pretty large inequity in the difference between the two cars. They said they must have just gotten a lucky hot car, but I believe that perhaps they got a cherry that didn't have to last as long as the car on the long-term test. If they were only going to have the car for a few weeks, then it didn't matter if it was as reliable as a longterm car, so they upped a few things and gave it to them for review. Same thing with the monitors, I guess. Since its just for the flat panel review, they might as well spice it up. These companies base a ton of business on "independent" reviews, so I suppose its worth it to fix the results.
Re:Car and Driver (Score:2)
Re:Car and Driver (Score:2, Informative)
A 0-60 time difference of half a second could easily be attributed to a natural difference in each car. Cars differ more than you think from one to the other and I have read more than once about discrepancies in cars within the same model.
In fact (I was trying to find it) but I believe Car and Driver (or some other car mag) said exactly this
It's called a "Golden Sample" (Score:3, Informative)
Perhaps I'm just overly cynical, but I tend to trust reviews where the reviewer went out and purchased an off-the-shelf retail copy of X rather than those where the company sent something. Of course, this is hard to do in print publications, because of the time-lag that magazines run through (ie, two months after it's released on the shelves, they have a review of it), but I see no reason (aside from money, which is a big reason) that online reviewers can't do things such as this. I also tend to look towards user-reviews and give those a pretty good weigh-in when I'm making a purchase decision. This is the first instance that I can recall where products are blatently better when given to reviewers than those that are store-bought, but I get the feeling that it's been done in the past.
The above paragraph reflects what I do for my personal buying choices and should in no way construe that that's the optimal/correct/whatever way for large corporations/organizations/whatever to buy-in-bulk
Blatant example of review Frod (Score:2, Interesting)
Well, how about this [motherjones.com]?
Search for the word 'Canada' to get to the falsification bit. Yes, this is a very old example, and no, it's not computer-related, but it still seems pretty relevant.
This is not a hard problem to fix (Score:2, Insightful)
Yes, I have. (Score:2)
I've heard a lot lately about car manufacturers flat out lying about their horsepower ratings, and every electrical engeneer and audiophile I know swears that Watt ratings on speakers mean absolutely jack shit. So what else is new?
Re:Yes, I have. (Score:2)
Consumer Reports (Score:3, Informative)
Consumer Reports will not accept donations of vehicles or products from manufacturers or vendors just for this reason. They will discreetly send someone out 'under cover' to go acquire the products in an "off the lot" or "off the shelf" state.
This is good, and commendable.
However, i see a lot of times they will end up mis-matching the cars and trucks they compare. Usually it is simply a matter of trim levels on similar classed models. This *will* have an impact on the final outcome. Obviously it's difficult to do things *exactly*.
Less often, but still wrongly, they will compare vehicles from incompatible classes- things like Buick Century vs. E-class Mercedes vs. Toyota Camry. Or the classic truck comparisons with the 3/4 ton, V8 powered Dodge and Chevy fullsize trucks, against a V6 F150, against V6 Toyota Tundra and Nissan.
Consumer Reports might do this to other product reviews too, but i only pay attention to their auto ads for `entertainment'.
I guess that no matter what, *any* test can be flawed.
Hardly new (Score:2)
The problem with reviews with hardware is
In some cases, the opposite is true (Score:5, Interesting)
That said, there's no way for reviewers to know with pre-production units whether they are getting what will eventually be on the shelf - and it may not be a case of the manufacturer trying to get away with something. A processor in a pre-production unit may be faster, or an LCD screen have a greater contrast ratio, than what ends up at retail, but the reason often is that design changes are made at the last minute related to cost or part availability. In fact, sometimes the product may be less powerful in pre-production than what is finally delivered to buyers. This was particularly true in the days of falling RAM prices - I'd get review PCs with 128 MB of RAM, and when they shipped they'd have 256 MB.
172t vs 173t (Score:2)
Caveat emptor also comes to mind.
Samsung 955DF CRT switcharoo (Score:3, Informative)
Now the "Samsung 955DF" has controls on the front, the screens are much more reflective and oily-looking, and black appears grey even when the brightness is all the way down. More recent Samsung 955DF [balta.pl]
Re: (Score:2)
my experience as a reviewer (Score:3, Informative)
A lot of the manufacturer reps and pr reps I worked with would hand-select or pre-screen review units, but I never ran into any where I thought I was being given something better than what would ship just to get a better review.
The author is full of shite. (Score:2)
That's obviously not true. If DesignTechnica has a monitor with a 700:1 contrast ratio, and the boxed retail product, the monitor you would purchase in your local store has a 450:1 contract ratio, then they are not reviewing boxed retail products.
If they don't go to a retail store and buy the product in its retail package, they are not reviewing boxed retail products.
We know these revi
In my experience... (Score:2)
The common situation involved getting sent a final beta while
Well Duh (Score:2, Insightful)
Wouldn't you, if in Intels shoes?
I was burned by this when I bought the first run of the Asus P4S8X motherboard. Review sites like toms were talking about this board being the cats ass, stable as a rock.
However, the first runs of the retail version were garbage.
They had a different clock gen, a different stepping on
Are you kidding? (Score:3, Informative)
In a previous job, my employer had a special team of people called "Product Managers" - but their job was to go visit magazine reviewers, ensure that they got top of the line grade A technical support during the review process, including onsite support, and coded patches directly from the developer's desktop to the reviewer's. Additionally, there was wining and dining, and talk of strippers and lapdances (though I never witnessed that). In that sense, what was reviewed in no way bore any resemblance to the shring-wrapped package some poor sucker paid $699 for.
I'm no longer working in that sector, but for my 10 years, the practice was commonplace. Which is why I never read reviews.
Oh Please. Unit mix-up? Domestic vs Int'l? Typo? (Score:3, Informative)
The contrast specs on the Samsung USA site show the following:
172T [samsungusa.com] - 700:1
173T [samsungusa.com] - 450:1
The specs on the Samsung Canada site say:
172T [samsung.ca] - 500:1
173T [samsung.ca] - 700:1
Perhaps he got a Canadian unit although I don't know why they would be any different.
Here's a classic example (Score:3, Informative)
http://www.valentine1.com/lab/MikesLabRpt5.asp
While you're there, please check out a Valentine One. Mike Valentine makes by far the best detector on the planet, and he's a heck of a nice guy!
--
Damn lie! (Score:2, Funny)
Little Different. (Score:2)
Re:Car reviews (Score:4, Interesting)
This is far more certain than testing every vehicle to find the "good" ones, which will never, ever, be quite as good as one assembled specifically to be good.
Even well heeled amatuers with access to a dealer's or distributer's parts bins do this. Hence classes like Star Mazda and Legends where the motor can only be touched by an official builder and has a seal affixed to it to prevent tampering.
This practice was first started in the 60's by the official Austrian Formula Vee team ( a class where every engine part must be absolutely box stock). Jochen Rindt simply ran away from the international field with a perfectly legal engine whose parts had all been individually cherry picked.
But the engine was completely stock.
KFG
Re:Car reviews (Score:2)
Just a case-in-point.
Re:What about the other way around? (Score:2)
Re:I've been thinking about this for a while (Score:2)