Futuremark Delists Samsung and HTC Android Devices for Cheating 3DMark 188
MojoKid writes "Benchmarks are serious business. Buying decisions are often made based on how well a product scores, which is why the press and analysts spend so much time putting new gadgets through their paces. However, benchmarks are only meaningful when there's a level playing field, and when companies try to 'game' the business of benchmarking, it's not only a form of cheating, it also bamboozles potential buyers who (rightfully) assume the numbers are supposed mean something. 3D graphics benchmark software developer Futuremark just 'delisted' a bunch of devices from its 3DMark benchmark results database because it suspects foul play is at hand. Of the devices listed, it appears Samsung and HTC in particular are indirectly being accused of cheating 3DMark for mobile devices. Delisted devices are stripped of their rank and scores. Futuremark didn't elaborate on which specific rule(s) these devices broke, but a look at the company's benchmarking policies reveals that hardware makers aren't allowed to make optimizations specific to 3DMark, nor are platforms allowed to detect the launch of the benchmark executable unless it's needed to enable multi-GPU and/or there's a known conflict that would prevent it from running."
End the PPI race (Score:3, Interesting)
Part of the problem is that many of the latest 1080p phones are slower in games than their 720p predecessors such as nexus 5 vs nexus 4. When you double the resolution, you need to quadruple the pixels rendered. Consumers want longer battery life and games to run smoothly but the manufactures are pushing for these useless 1080p screens and cheating in benchmarks to make up for loss in performance. On 4" screen 720 is more than enough for normal eyesight.
Re:And Apple are still listed why? (Score:3, Interesting)
Apple doesn't need to cheat because the last phone that was slower than its predecessor was the iPhone 4. Ever since then, every successor has had a faster gpu while rendering the same number pixels and therefore outperforms on the benchmarks and battery life. Above 300 PPI, you are just wasting battery life and hurting performance to display pixels the human eye can't even resolve. I wish more android manufactures had the guts to follow Apple's engineering wisdom here.
Re:And why do you think they are? (Score:3, Interesting)
Because under iOS, all binaries are encrypted and cannot be changed without creating a non executable.
The name is irrelavant
The combination of DRM key and code identifies to the OS the precise application and whether or not it's allowed to run, not the mere executable's name. While samsung have been caught with their pants down by listing using executable/task names, Apple need only boost applications according to a mathematical model that surprise surprise only includes benchmark apps.
Funny how iPhone battery falls 20-30% faster and the phone runs substantially hotter when running 'official' benchmark apps, but xcode apps by lone developers that try to hit the hardware as hard don't have anywhere near the effect. Apple betrayed by the only thing they can't control, the laws of physics.
Re:And why do you think they are? (Score:5, Interesting)
The name is irrelevant
Not to Samsung phones! Which was kind of my point.
The combination of DRM key and code identifies to the OS the precise application and whether or not it's allowed to run
And anyone who has jailbroken a phone can easily re-sign an application bundle if they chose, renaming it or doing whatever else they like with it.
Or if you are compiling the benchmark yourself you just change what you like.
Apple need only boost applications according to a mathematical model
Yeah, you see that's called "compiler optimization" and applies to all applications, not just benchmarks.
Again, when you ship with a fast enough processor you don't need to waste time scanning for benchmarks.
Funny how iPhone battery falls 20-30% faster and the phone runs substantially hotter when running 'official' benchmark apps
Funny how I've never noticed that at all in my own testing. And in fact sometimes the system will get hot when playing commercial games.
Oh wait, it's not funny at all - you're an Apple-Hater AC so we know not to believe anything you post.
Apple betrayed by the only thing they can't control, the laws of physics.
That's funny because Apple seems to be the only smartphone maker paying attention to such laws, not building needlessly dense displays that suck power like a kid with a juice box.
Just went over this in the Texas anti-evolution BS (Score:4, Interesting)
Buying decisions are often made based on how well a product scores,
That is an unproven hypothesis. Null hypothesis: Buying decisions are often NOT made based on how well a product scores on benchmarks. Evidence: iDevices. The burden of proof is on the claimant to provide GREATER evidence than the null hypothesis, otherwise the claim can be dismissed as confirmation bias, even if you find evidence in support of the orginal hypothesis: Stepping on cracks does not break backs, even if you observe it happening a couple of times. Nerds checking benchmarks before buying gadgets happens. Is this frequent enough to warant use of the word "often"? If so, where's the evidence? You haven't any.
Try this on for size: The niche market segment of geeks who care enough about benchmark scores and use Futuremark as a source for statistics occasionally purchase products based on those scores. It's hypocritical to hold Creationists to a higher standard of evidence than you do yourself.
Re: And why do you think they are? (Score:5, Interesting)