Futuremark Delists Samsung and HTC Android Devices for Cheating 3DMark 188
MojoKid writes "Benchmarks are serious business. Buying decisions are often made based on how well a product scores, which is why the press and analysts spend so much time putting new gadgets through their paces. However, benchmarks are only meaningful when there's a level playing field, and when companies try to 'game' the business of benchmarking, it's not only a form of cheating, it also bamboozles potential buyers who (rightfully) assume the numbers are supposed mean something. 3D graphics benchmark software developer Futuremark just 'delisted' a bunch of devices from its 3DMark benchmark results database because it suspects foul play is at hand. Of the devices listed, it appears Samsung and HTC in particular are indirectly being accused of cheating 3DMark for mobile devices. Delisted devices are stripped of their rank and scores. Futuremark didn't elaborate on which specific rule(s) these devices broke, but a look at the company's benchmarking policies reveals that hardware makers aren't allowed to make optimizations specific to 3DMark, nor are platforms allowed to detect the launch of the benchmark executable unless it's needed to enable multi-GPU and/or there's a known conflict that would prevent it from running."
And why do you think they are? (Score:3, Insightful)
On iOS benchmark scores do not change when you change the executable name...
When you ship with a fast enough system you don't need to cheat to look good on benchmarks.
Re:And why do you think they are? (Score:2, Insightful)
Funny how a Slashdot article on Samsung and HTC attracts Apple haters like yourself. You might want to seek professional help for your rage.
Re:And Apple are still listed why? (Score:4, Insightful)
If you believe Apple aren't doing precisely the same thing, then I have a shiny white featureless brick to sell you
That is only speculation. Prove me that Apple does cheating. For Samsung and HTC their cheating has been proven.
Re:3DMark cheats, so no wonder it is cheated (Score:4, Insightful)
Creators of 3DMark do not have a clue how to test modern multicore smartphones, but they do not care and release their product.
The real problem? People use this shitty benchmark and judge product basing on the meaningless score it produces.
Why should Samsung LOSE customers because 3DMark lied to them?
It's better to 'cheat' this crappy software into being at least a bit more FAIR in judging their products.
Sigh ... if a phone identifies that its running a benchmark application and changes its behaviour then the benchmark is of the maximum hardware performance rather than that available to a normal application. In doing so its not giving a real world measurement of the performance of the device.
By your argument all of the single threaded apps that run slowly on the S4 are at fault for slow performance because the programmer hasn't optimised their application for the S4 instead preferring to be compatible with all Android phones out there.
So, whilst the rigged S4 may be faster in raw power, its not what the end user is going to see. Which is cheating?
Here's the RDF at work. (Score:5, Insightful)
When Apple brought out Retina Display, that was 100% FAN BLOODY TASTIC according to Apple fans. Absolutely the best thing EVAR, and PROOF Apple are "innovative" by making displays finer in resolution than any other smartphone.
Nokia didn't count, since they were ~12ppi lower resolution! SHUT UP!
But now resolution is higher than Retina Display, higher resolution and better pixel count is BAD. Which, yet again, PROVES Apple are BEST EVAR because they don't waste time trying to get uselessly higher resolutions!
7" tablets were too small when the iPad was only 10".
But when the iPad mini comes out at 7", it's the best size for many many tasks!
Phones were too big if they had a 4.3" screen. Until Apple brought out a bigger screen, then they had many uses!
And so on.
Benchmarks (Score:5, Insightful)
A benchmark measures the performance of a machine while under that particular benchmark.
Otherwise, it's pretty useless. No benchmark has been able to be used for comparison purposes for more than a few months after release (and things like this are re-released once a year or more). Even back in the days of Dhrystones and Whetstones and all that crap - at best it benchmarks one particular run of code, and that's it. And in terms of general performance, it can do no better than guess.
Fact is, if anyone buys because of a 10% increase in a certain benchmark they are an idiot, unless the code they want to run *IS* that benchmark (to all intents and purposes). This is why the best "benchmarks" are things like how many FPS you get in the game you want to play. Because then you'll know exactly how many FPS you'll get in the game you want to play...
We haven't had highly-determinstic computer systems in our PC's for many, many, many years. Caches, bus speeds, interactions, multi-processors, etc. all throw benchmarks in the bin. And everyone's use case is different. Personally, I'd prefer 8 2GHz cores to any other configuration you could imagine at the moment, other people will have different ideas.
Benchmarks are a waste of time. It's like having stupid logic questions on a job interview. All that gets you is people good at answering those stupid logic questions, not at the job, or at worst someone who *appears* good at answering those logic questions.
Benchmarks on smartphones? It makes even less sense. I'm more shocked that Samsung think that anyone gives a shit.
1080P Phones (Score:5, Insightful)
That's funny because Apple seems to be the only smartphone maker paying attention to such laws, not building needlessly dense displays that suck power like a kid with a juice box.
I bought the Nexus 5 which comes with premium 5" 1080P screen and is half the price of the bottom end iPhone. There are phones that come with similar screens to the iPhone like the Moto G for instance which is a sixth of the price of the iPhone.
Re:And Apple are still listed why? (Score:4, Insightful)
Then you'll have no problems naming many examples, then. No, showing benchmarks that show your product to be superior while...not publishing those that don't, does not count. When and where has Apple actively cheated.
Re:And Apple are still listed why? (Score:5, Insightful)
Above 300 PPI, you are just wasting battery life and hurting performance to display pixels the human eye can't even resolve.
This is a myth often repeated by Apple fans, but Apple themselves offer you proof that it is not true. Find some screenshots taken from Retina displays and zoom in on the text. Notice how it is still anti-aliased? If the resolution was high enough to be impossible for the human eye to resolve there would be no need for anti-aliasing. I don't think you can turn it off in iOS but you can make an image on your computer with both and try viewing it on the phone.
The simple fact is that the human eye does not work the way you think it does. It is particularly good at picking out edges and uses spatial and temporal over-sampling to increase the effective resolution. It is an analogue sensor, not digital. I can see the different between a Retina display and a similarly sized 1080p display, even if you claim you can't. Then again I'm one of those super-human freaks who can see a difference between 1080 and 4k, despite needing to get another prescription in the next few months.
Re:1080P Phones (Score:3, Insightful)
The Nexus is an outlier, as comes with a near-zero profit margin for Google; that's not sustainable. The Moto G is much, much more interesting because Motorola's devices are still supposed to turn a profit.
No, they're not (Score:2, Insightful)
Benchmarks are serious business.
For a tiny segment of the population, maybe. For the rest of the world, raw benchmarks don't matter at all. It's all perception and other features over raw framerate. Normal humans can't really detect anything above 50 or 60 FPS. So if you are proud your phone gets 150FPS, congratulations! You got that going for you, which is nice. I guess.
Re:1080P Phones (Score:5, Insightful)
The Nexus is an outlier, as comes with a near-zero profit margin for Google; that's not sustainable. The Moto G is much, much more interesting because Motorola's devices are still supposed to turn a profit.
The Moto G sells for $179 unlocked with no contract. It might not be as powerful as the Nexus 5 but it is 95% of the way there for most people. Chinese devices with similar specs are in the $140-$170 range on Aliexpress. I don't see how Motorola is making much of a profit on the G either. I don't see it as being "much more interesting".
Android phones are based on a high volume, low profit model. Apple is working under a high profit model. Neither is wrong but if I want a phone that isn't overpriced, I'm not going to buy the Apple one.
Re: Actually, Apple is not cheating ... (Score:5, Insightful)
Re: 1080P Phones (Score:2, Insightful)
Re: 1080P Phones (Score:4, Insightful)
Anon forgot to read TIA: Samsung cheats (Score:4, Insightful)
Samsung cheats at benchmarks by changing the hardware behavior when it detects certain apps with specific names are running. Change the name, and the cheating stops.
That's the fundamental difference between Samsung and most of the rest of the Android devices. The name is the key to the cheat.
Funny how you're ignoring lots of detailed information about that. This has nothing to do with Apple, this has everything to do with Samsung/HTC/Android.
Great attempt to troll, though!