Forgot your password?
typodupeerror
Hardware

Futuremark Delists Samsung and HTC Android Devices for Cheating 3DMark 188

Posted by Unknown Lamer
from the serious-bar-charts-people dept.
MojoKid writes "Benchmarks are serious business. Buying decisions are often made based on how well a product scores, which is why the press and analysts spend so much time putting new gadgets through their paces. However, benchmarks are only meaningful when there's a level playing field, and when companies try to 'game' the business of benchmarking, it's not only a form of cheating, it also bamboozles potential buyers who (rightfully) assume the numbers are supposed mean something. 3D graphics benchmark software developer Futuremark just 'delisted' a bunch of devices from its 3DMark benchmark results database because it suspects foul play is at hand. Of the devices listed, it appears Samsung and HTC in particular are indirectly being accused of cheating 3DMark for mobile devices. Delisted devices are stripped of their rank and scores. Futuremark didn't elaborate on which specific rule(s) these devices broke, but a look at the company's benchmarking policies reveals that hardware makers aren't allowed to make optimizations specific to 3DMark, nor are platforms allowed to detect the launch of the benchmark executable unless it's needed to enable multi-GPU and/or there's a known conflict that would prevent it from running."
This discussion has been archived. No new comments can be posted.

Futuremark Delists Samsung and HTC Android Devices for Cheating 3DMark

Comments Filter:
  • End the PPI race (Score:3, Interesting)

    by Anonymous Coward on Tuesday November 26, 2013 @04:23AM (#45523965)

    Part of the problem is that many of the latest 1080p phones are slower in games than their 720p predecessors such as nexus 5 vs nexus 4. When you double the resolution, you need to quadruple the pixels rendered. Consumers want longer battery life and games to run smoothly but the manufactures are pushing for these useless 1080p screens and cheating in benchmarks to make up for loss in performance. On 4" screen 720 is more than enough for normal eyesight.

    • But we must have MOER 'P'. And with 4K being the new 1080p, you can in fact expect more of them to be here soon.

      Incidentally, Skyfall would have been a better movie if it were about a plot to switch all smartphone manufactures to power draining 1080p displays without them realizing until it was too late.

      • Re: (Score:3, Funny)

        by Anonymous Coward

        In a pissing match you always need MOAR P

      • Apple - making significant disadvantages of their iDevices sound like good things (tm).

        The old "perfect size / one size fits all" 3.5" display comes to mind...

        • by Sockatume (732728)

          That's a marketing point for their four inch displays. They didn't need to spin having a 3.5-inch display because until about 2010 it was one of the largest displays you could get on a phone.

          • Well it's what happened with the old 3.5 inch screens, with the awkward 3:2 aspect ratio. Don't forget that the iPhone 5 (with the 4" screen) was not released until march 2013.

            http://gizmodo.com/5847981/this-is-why-the-iphones-screen-will-always-be-35-inches [gizmodo.com]

            This time it's the screen resolution... Apple has always bragged about their high resolution retina displays, and now that they're lacking in that department, all of a sudden high resolution is a bad thing and Apple's retina are the "perfect resolution".

            • by Sockatume (732728)

              I don't think that Dustin Curtis counts as "Apple". And I don't think you'll find many remarks from Apple that "high resolution is a bad thing" either.

              (Fact is, they're not going to upgrade resolution until they can do an integer multiple or iOS goes resolution-independent. It's not a question of "lacking", it's a question of them making a tradeoff that suits their particular product line, in the same way that Samsung's shipping Pentile displays rather than RGB.)

    • by AmiMoJo (196126) *

      Part of the problem is that many of the latest 1080p phones are slower in games than their 720p predecessors such as nexus 5 vs nexus 4.

      Most benchmarks support off-screen rendering at a fixed resolution. Even so this just highlights why benchmarks are pointless - different GPUs will perform differently in a variety of games, and manufacturers have been providing drivers with tweaks for specific games since the late 90s.

      Basically benchmarking is a waste of time when trying to make generalizations about a device.

      • by Sockatume (732728)

        The whole idea of a benchmark is to provide a transferable baseline; any benefits from "tweaks" (which simply do not exist on mobile devices) would appear over and above that.

    • by citizenr (871508)

      Part of the problem is that many of the latest 1080p phones are slower in games than their 720p predecessors such as nexus 5 vs nexus 4

      I wonder why arent they rendering to smaller resolution window and then rescaling? M$ does this in X180 rendering games in 720p and upscaling to pretend its a next gen full HD console.

  • Buying decisions are often made based on how well a product scores,

    That is an unproven hypothesis. Null hypothesis: Buying decisions are often NOT made based on how well a product scores on benchmarks. Evidence: iDevices. The burden of proof is on the claimant to provide GREATER evidence than the null hypothesis, otherwise the claim can be dismissed as confirmation bias, even if you find evidence in support of the orginal hypothesis: Stepping on cracks does not break backs, even if you observe it happening a couple of times. Nerds checking benchmarks before buying gadgets happens. Is this frequent enough to warant use of the word "often"? If so, where's the evidence? You haven't any.

    Try this on for size: The niche market segment of geeks who care enough about benchmark scores and use Futuremark as a source for statistics occasionally purchase products based on those scores. It's hypocritical to hold Creationists to a higher standard of evidence than you do yourself.

    • by Sockatume (732728)

      Then these phone companies are wasting perfectly good time and money by cheating on the benchmarks, and there's no harm in 3DMark delisting these phones.

      (I'd say that if nothing else, these benchmarks generate news stories promoting the new, allegedly-faster device [google.com].)

  • Benchmarks on buying decisions are for CPUs and GPUs. They are for people building high end machines, or people trying to get the best processing bang for buck.

    These are phones. What sells is screen size, phone style, and feature list. No one cares how many points a phone has in benchmarks except for some reviewers. People want to know if it takes good photos, how well the hover features work, if it's 3G or 4G, hell most recently buyers have been more interested in if it comes in white or black rather than

  • Benchmarks (Score:5, Insightful)

    by ledow (319597) on Tuesday November 26, 2013 @06:34AM (#45524505) Homepage

    A benchmark measures the performance of a machine while under that particular benchmark.

    Otherwise, it's pretty useless. No benchmark has been able to be used for comparison purposes for more than a few months after release (and things like this are re-released once a year or more). Even back in the days of Dhrystones and Whetstones and all that crap - at best it benchmarks one particular run of code, and that's it. And in terms of general performance, it can do no better than guess.

    Fact is, if anyone buys because of a 10% increase in a certain benchmark they are an idiot, unless the code they want to run *IS* that benchmark (to all intents and purposes). This is why the best "benchmarks" are things like how many FPS you get in the game you want to play. Because then you'll know exactly how many FPS you'll get in the game you want to play...

    We haven't had highly-determinstic computer systems in our PC's for many, many, many years. Caches, bus speeds, interactions, multi-processors, etc. all throw benchmarks in the bin. And everyone's use case is different. Personally, I'd prefer 8 2GHz cores to any other configuration you could imagine at the moment, other people will have different ideas.

    Benchmarks are a waste of time. It's like having stupid logic questions on a job interview. All that gets you is people good at answering those stupid logic questions, not at the job, or at worst someone who *appears* good at answering those logic questions.

    Benchmarks on smartphones? It makes even less sense. I'm more shocked that Samsung think that anyone gives a shit.

    • by Sockatume (732728)

      That's why you perform use-case-specific benchmarking. 3DMark, for instance, is closely modelled on the kinds of calculations actually involved in 3D graphics on mobiles. You'd use an entirely different benchmark protocol for something like a server, or a scientific supercomputer.

  • by buddyglass (925859) on Tuesday November 26, 2013 @09:26AM (#45525351)
    How hard would it be for Futuremark to disguise their benchmark app so as to fool the device? If it just looks for the package name it should be easy. If Samsung reverse engineered the exact workload being done in each benchmark then micro-optimized for that workload...that's harder to fix.
  • No, they're not (Score:2, Insightful)

    by Enry (630)

    Benchmarks are serious business.

    For a tiny segment of the population, maybe. For the rest of the world, raw benchmarks don't matter at all. It's all perception and other features over raw framerate. Normal humans can't really detect anything above 50 or 60 FPS. So if you are proud your phone gets 150FPS, congratulations! You got that going for you, which is nice. I guess.

1 Dog Pound = 16 oz. of Alpo

Working...