Forgot your password?
typodupeerror
Intel Hardware

PCMark Memory Benchmark Favors GenuineIntel 298

Posted by kdawson
from the not-what-you'd-call-neutral dept.
javy_tahu writes "A review by Ars Technica disclosed that PCMark 2005 Memory benchmark favors GenuineIntel CPUID. A VIA Nano CPU has had its CPUID changed from the original VIA to fake GenuineAMD and GenuineIntel. An improvement of, respectively, 10% and 47% of the score was seen. The reasons of this behavior of FutureMark product are not yet known."
This discussion has been archived. No new comments can be posted.

PCMark Memory Benchmark Favors GenuineIntel

Comments Filter:
  • Money (Score:5, Insightful)

    by TheRealMindChild (743925) on Friday August 01, 2008 @08:32AM (#24432213) Homepage Journal
    The reasons of this behavior of FutureMark product are not yet known

    Easy. Intel paid them to make it that way.
    • Re:Money (Score:5, Funny)

      by Shadow Wrought (586631) * <shadow.wroughtNO@SPAMgmail.com> on Friday August 01, 2008 @08:38AM (#24432319) Homepage Journal
      Yep. They increased the L2 Cash size.
    • Re:Money (Score:5, Interesting)

      by SimonGhent (57578) on Friday August 01, 2008 @08:40AM (#24432353)

      Easy. Intel paid them to make it that way.

      If anyone can come up with a better explanation I'd be interested to hear it.

      • Re:Money (Score:4, Insightful)

        by Lord_Frederick (642312) on Friday August 01, 2008 @08:44AM (#24432435)

        Even if this is an unintentional error, they have certainly lost some credibility.

      • Re:Money (Score:5, Informative)

        by ArcherB (796902) on Friday August 01, 2008 @08:47AM (#24432481) Journal

        Easy. Intel paid them to make it that way.

        If anyone can come up with a better explanation I'd be interested to hear it.

        TFA offers the following:

        At the very least, this suggests some incredibly sloppy coding on Futuremark's part, as the company may be enabling or disabling CPU optimizations based on a processor's vendor name in CPUID instead of actually checking CPUID for SIMD support. In this case, PCMark 2005's memory subsystem test doesn't appear to be aware that Nano supports SSE2 and SSE3, and is instead running a decidedly less-optimized code path. There are two factors, however, that make this explanation a bit difficult to swallow.

        First, there's the issue of timing. PCMark 2005 was released (obviously) in 2005, and was obviously coded with an eye towards supporting current and future processors. This is standard operating procedure for Futuremark, which always builds benchmarks designed to last for at least a year, and often two. VIA's C5N-T (Nehemiah) core may have only supported MMX and 3DNow!, but the C7 launched in 2005, and that processor supported SSE2 and SSE3 from day one. Even if proper extension support wasn't built into the first version of PCM2K5, we tested version 1.2.0, and that patch was released on or around 11-29-2006.

        Second, there's the issue of performance when Nano is identified as AuthenticAMD. If performance between the AMD and Intel CPUIDs was identical, there wouldn't really be a story here, but it isn't, and that's curious. Futuremark could plausibly argue that VIA's C3/C7 processors weren't exactly on the radar back in 2004-2005, but AMD and K8 certainly were, and K8 launched with full SSE and SSE2 support, with SSE3 added in 2005.

        There's more, but I don't want to quote the entire article.

        • Re:Money (Score:5, Insightful)

          by ShieldW0lf (601553) on Friday August 01, 2008 @09:08AM (#24432893) Journal

          Moral of the story is, when you're dealing with code like this, where it has the capacity to influence who receives billions of dollars and who doesn't, well, you can't trust it if it's closed source and not subject to public scrutiny.

          Closed source test suites cannot be trusted, shouldn't even be considered by potential purchasers, and have been misleading the public for years and years. This is mute evidence to the fact.

          • you got a point there which is important to the discussion, if the source is closed, how can we know if the test is fair?

          • Re:Money (Score:5, Interesting)

            by Fumus (1258966) on Friday August 01, 2008 @09:46AM (#24433693)
            What I don't get is why game developers don't release freeware benchmark versions of their engines.
            Saying that a config has 9000 points is pretty much useless. Saying that it gets an average of 40FPS in the UT3 benchmark at high details, and 1680x1050 is much more informative.

            Unfortunately, this also is a little bit more complicated, and as we know everything simpler is more popular with the dumb masses.
            • Re:Money (Score:4, Insightful)

              by ShieldW0lf (601553) on Friday August 01, 2008 @09:53AM (#24433799) Journal
              It doesn't have to be complicated. I can see a business case for a few large game developers to collaborate on creating a modular open source test suite that would allow a user to load, run and score game-based benchmarks. The modules themselves wouldn't have to be open source for it to be effective in gauging the performance of this game on this hardware. Then, if Intel pays the publisher a fortune to make this game run faster on their hardware than the others, that wouldn't corrupt the integrity of the test, just the game.
            • Re:Money (Score:5, Insightful)

              by Goaway (82658) on Friday August 01, 2008 @10:00AM (#24433947) Homepage

              What I don't get is why game developers don't release freeware benchmark versions of their engines.

              Because that would require a non-trivial amount of work for no substantive payoff?

            • Re: (Score:3, Insightful)

              by Splab (574204)

              Because the drivers will be optimized for these tests, rather than the game - they did it before, they will do it again (Nvidia and ATI sure wasn't above it).

          • Re: (Score:3, Insightful)

            by SirShmoopie (1333857)

            Ok then, point me to an open source benchmarking program that's as complete, and I'll use it.

            Might it just be that they got the software done as cheaply as possible, marked it as ready for release as soon as they could, and never bothered to fix what was obviously a glaring flaw?

            Anyway, as an open source developer myself I don't really buy this 'open source will always be better' deal. It can only be better if the project is fortunate enough to attract quality coders and designers. There are a lot more open

            • Re:Money (Score:5, Insightful)

              by SanityInAnarchy (655584) <ninja@slaphack.com> on Friday August 01, 2008 @10:53AM (#24434813) Journal

              Ok then, point me to an open source benchmarking program that's as complete, and I'll use it.

              glxgears.

              Seriously, when they are changing the results based on the vendor name, it makes any result suspect -- which makes it pretty much useless as a benchmark. At least with glxgears, while it may not be a particularly accurate benchmark, it's at least guaranteed to be fair.

              Anyway, as an open source developer myself I don't really buy this 'open source will always be better' deal.

              That's not the point of this exercise.

              Open source will not always make a better game, or a better office suite, or even a better text editor.

              But there are some kinds of software which you need to trust, and which are difficult to verify without the source. Benchmarks are one example. SSH clients are another. For these, I would not even consider a proprietary version -- it's not about features or relative quality; open source is a necessity.

            • Pick me, pick me! (Score:3, Informative)

              by pxc (938367)

              The Phoronix Test Suite [phoronix-test-suite.com].

              It's Linux only, but a CPU that performs better on Linux will perform better on Windows.

        • Re:Money (Score:5, Informative)

          by Eponymous Cowboy (706996) * on Friday August 01, 2008 @09:09AM (#24432917)

          I'll give 10:1 odds that Futuremark simply compiled their benchmark with Intel's C++ compiler.

          I wrote a detailed explanation [slashdot.org] back in 2005 about how the Intel C++ compiler generates separate code paths for memory operations to make AMD processors appear significantly slower, and how you can trick the compiled code into believing your AMD processor is an Intel one to see incredibly increased performance. See this article [slashdot.org] for additional details.

          • Re:Money (Score:5, Interesting)

            by JamesP (688957) on Friday August 01, 2008 @09:22AM (#24433219)

            Yes, I remember that...

            But why would icc make AMD better than "no name" beats me.

          • Re: (Score:2, Insightful)

            by DrWho42 (558107)
            If the authors of this benchmark test were competent they would have written the code for low-level tests like memory bandwidth in assembly language, so compiler choice would not impact them.
            • Re: (Score:3, Interesting)

              by CastrTroy (595695)
              That's what I would think they are doing. If you write a processor benchmark in C, you don't really know what it gets assembled into, you don't know if it's actually even using the SSE/SSE2/SSE3 optimizations. It's hard to know what path the program is taking unless you only give it one path. You'll still get some discrepancies between processors that don't support the same features, because the one that doesn't support SSE will take longer than the one that does, but at least you won't show one processo
              • Re:Money (Score:5, Interesting)

                by Bert64 (520050) <bert@slaSLACKWAR ... com minus distro> on Friday August 01, 2008 @11:04AM (#24435027) Homepage

                If your benchmark tool is going to use multiple code paths, then they should be configurable, so that you can benchmark different systems both using the same code and more optimal code. That way you'd get an idea of how much speedup various features provide.

                As an example, john the ripper's SSE2 support for cracking DES - on a core2 the SSE2 version is considerably faster and compiling the C version never comes anywhere close regardless of compiler and flags, but on an AMD compiling the generic C code with appropriate flags and a modern version of gcc produces slightly faster code.

                Running the SSE2 ver on a 2.3ghz core2 quad achieves about 2mil c/s per core, while a 2.3ghz amd phenom yields about 1.6, but compiling the C source with various flags and gcc versions makes amd slightly faster, while the core2 is nowhere close.

        • Re:Money (Score:5, Insightful)

          by Kamokazi (1080091) on Friday August 01, 2008 @09:40AM (#24433573)

          And this is partly why I generally ignore benchmark scores, and look at real-world performance. It's possible for the benchmark or the hardware being benchmarked to 'cheat' or at least behave very differently and produce bogus scores. If i'm looking for a new video card, I don't look at 3DMark scores, I look at framerates in games that I play (or that use the same engine). If I'm looking for a CPU, I'll look at RAR compression times or video encoding speeds. If I'm looking for a storage solution at work, I look at file copy speeds of similar file quantities and sizes, or I/O performance of a similar database.

          • Re: (Score:3, Informative)

            by CastrTroy (595695)
            You have to still be pretty careful when looking at game benchmark scores. ATI has cheated [techreport.com] before with popular games. I wouldn't put it past a lot of companies.
      • Re:Money (Score:5, Funny)

        by Chrisq (894406) on Friday August 01, 2008 @09:04AM (#24432817)

        Easy. Intel paid them to make it that way.

        If anyone can come up with a better explanation I'd be interested to hear it.

        OK, far-fetched it maybe but what if VIA paid them to do it so that the expose would generate a lot of free advertising and ram home the information that the Nano is faster.

        Alternatively the US military could have engineered it to distract us from the possibility that they are working for aliens and have files full of UFO data on their systems. Gosh, i'd better hack in and take a look....

    • Re:Money (Score:5, Funny)

      by Chrisq (894406) on Friday August 01, 2008 @09:00AM (#24432725)
      It is just what you'd expect with "Intel" inside ..... even inside another manufacturer's processor!
  • by Plantain (1207762) on Friday August 01, 2008 @08:32AM (#24432219)

    I'm a GenuineIntel, mod me 47% higher!

  • Money? (Score:4, Insightful)

    by IWantMoreSpamPlease (571972) on Friday August 01, 2008 @08:35AM (#24432265) Homepage Journal

    Seems obvious, but follow the money trail, does PCMark get backing from Intel?

  • by dlgeek (1065796)

    A VIA Nano CPU has had its CPUID changed from the original VIA to fake GenuineAMD and GenuineIntel. An improvement of, respectively, 10% and 47% of the score was seen.

    It sounds to me like this could possibly be explained by some kind of conditional optimization that the compiler puts in for various chips, to take advantage of differences in their designs that can improve performance.

    Then again, probably not.

    • Re: (Score:3, Insightful)

      by Plantain (1207762)

      This could all be explained if they compiled with something silly like ICC

      http://www.theinquirer.net/en/inquirer/news/2005/07/13/intel-compiler-nobbles-amd-chips-claim [theinquirer.net]

      • Re: (Score:2, Insightful)

        by Anonymous Coward

        Yeah, except that ICC Intel optimizations frequently improve AMD scores as well (over generic optimizations). Not always as much as it helps Intel processors, but it does help some.

    • Re: (Score:3, Interesting)

      by k_187 (61692)
      Exactly, what happens when you run an AMD chip under both IDs? or an Intel? If the Intel optimizations are faster on all three, then there might be a problem, as it stands the via chip might just benefit more from the Intel optimizations than the AMD ones.
      • by MBCook (132727) <foobarsoft@foobarsoft.com> on Friday August 01, 2008 @08:51AM (#24432581) Homepage

        You can't. That's why it was discovered now. Intel and AMD don't let you change the CPUID results on their CPUs. Via DOES let you change it. (You could hack the benchmark to change the checks, but then your results are invalid because you changed the benchmark code)

        Either way, that's not an excuse. As Ars points out, if it is just checking for something like SSE2 the Nano has that. If you want to make an optimized code path it should be based on if a feature is reported as present or not, not who made the CPU.

        It's just really REALLY fishy.

        • by k_187 (61692)
          I'll give you that its fishy, but never attribute to malice what can be explained by sloppy coding.
      • by brunascle (994197) * on Friday August 01, 2008 @08:52AM (#24432585)

        Exactly, what happens when you run an AMD chip under both IDs? or an Intel?

        As TFA mentions, we cant test it. AMD and Intel lock the CPUIDs on their chips. VIA doesnt. I do think AMD should do some testing in-house though, as I'm sure they could change the CPUID themselves. Though I wouldnt be surprised if they'd already tried this long ago. I know I would have. And if there were major discrepancies, we probably would have heard about it by now.

    • by mikeabbott420 (744514) on Friday August 01, 2008 @08:40AM (#24432359) Journal
      Code that only used SSE3 or some such on the basis of the CPU ID might explain it but conspiracy is more fun to believe. Lies, Damn Lies and Benchmarks.
      • by Bert64 (520050)

        Well, it might not be SSE2 based at all...
        It could be that it recognises the nano chip as an older variant VIA chip, and thus follows a codebase optimized for *that* chip...
        Seeing an Intel it follows a codebase optimized for whatever chip Intel had out at the time, and thus performs better.

        As an example, cyrix processors used to have very weak floating point support, to the extent that it was sometimes quicker to run software floating point emulation. The older VIA chips may have had a similar issue perhaps

    • by LWATCDR (28044)

      That is actually a pretty good guess.
      An extreme example is the Intel compiler which used to do great optimization on Intel CPUs but really bad optimization on AMD.
      The VIA chip is really new and the code the compiler generates treats it as the a P3 or P2.

    • by neokushan (932374) on Friday August 01, 2008 @08:47AM (#24432487)

      In all likelihood, this probably IS the case, but that still goes a long way to discredit Futuremark as it shows their benchmarks were certainly NOT fairly tested.

    • It sounds to me like this could possibly be explained by some kind of conditional optimization that the compiler puts in for various chips, to take advantage of differences in their designs that can improve performance.

      People are trusting closed-source benchmarks? Well, golly gee, who'd'a thunk there'd be errors, oversights, or shenanigans?

      If this was used for anything more than entertainment value, any methodical person would have at least compared multiple closed-source benchmarks. If that proved to be

    • by Kegetys (659066)

      If this is the case then wouldn't changing the cpuid give you better "real-life" performance too because similarily compiled (non-benchmark) apps would also be utilising the optimisations? TFA doesn't seem to mention testing that...

    • Re: (Score:3, Insightful)

      by poot_rootbeer (188613)

      It sounds to me like this could possibly be explained by some kind of conditional optimization that the compiler puts in for various chips, to take advantage of differences in their designs that can improve performance.

      I suppose it's possible that a VIA chip running code optimized for what the benchmark believes is an Intel CPU might perform better than the same chip running the benchmark's unoptimized code path, but as I understand it the VIA Nano is pretty entry-level; any optimizations present in it shou

  • Is this like changing the user agent in a browser?

  • by davidwr (791652) on Friday August 01, 2008 @08:39AM (#24432345) Homepage Journal

    This definitely requires clarification from the creator of the benchmark.

    It is possible that the benchmark uses the CPUID to change how the benchmark works, for example, to work around known flaws in a given chip. If this is the case, then the problem is not "omyghoshitplaysfavorites" but rather lack of full disclosure that the benchmarks are not directly comparable across different chips. In the most benign scenario, this could be someone at the benchmark creator's shop forgetting to tell the documentation team. This is still a very serious issue, but it's not fraud.

    • Re: (Score:3, Interesting)

      by SimonGhent (57578)

      That's an insightful explanation, but IMO the benchmark is then only valid is the operating systems people use make the same allowances for the different chips.

      One thing that doesn't seen to have been investigated is the permutations of the test - marking an Intel chip as a VIA, etc. If the differences are the same drop in performance as the improvement in marking a VIA as an Intel then your explanation has effectively been disproved.

      • by DJProtoss (589443)
        Unfortunatly, intel/amd chips don't let you set cpuid
      • by MBCook (132727)
        Try reading the article. You can't do that. Only VIA lets you change what the CPU reports. Intel and AMD lock it down so it can't be changed making those tests impossible to run.
    • by Loibisch (964797)

      They probably just use code compiled with optimizations for each given chip family...if they didn't then people would be shouting how not using special features of a certain cpu would be unfair, etc.

      So what now if Intel was the biggest desktop CPU manufacturer in the market (I know it's a stretch, but bear with me for a minute) and profiles for their CPUs (either through ICC or maybe even GCC) were just better optimized because more people put time and effort into them?

      I can certainly see this being true fo

      • by MadKeithV (102058) on Friday August 01, 2008 @09:26AM (#24433289)
        But nothing changes in the CPU operation if you only change the reported CPUID. In the best case that means the 3DMark developers have spent a lot more optimizing for Intel specifically, applying a number of techniques that would have been just as valid for an AMD or VIA processor. They spent that effort without bothering to check the effect on the AMD and VIA specific paths, thus they did not get the same treatment as intel.
        And that's simply Intel favoritism.
        • by MadKeithV (102058)
          Reply to myself: another possibility is that the calculation results with GenuineIntel turned on in a VIA processor are actually wrong. Maybe the VIA doesn't support those instructions after all, defaulting to a no-operation, which is of course pretty fast.
          It will be interesting to see how this pans out. It FutureMark playing favourites with Intel? Or did they actually try to avoid some substandard implementations or bugs in the VIA and AMD processors?
        • by Loibisch (964797)

          Well, not really

          Let's say (for simplicity) they used gcc to compile the different code paths. If all they told the compiler was to "optimize this one for Intel, optimize this one for AMD and optimize this one for VIA" and at the end you had results as those in the article then it wouldn't really be a fault of the PCMark guys.

          This gets even worse if they optimized the different code paths using the respective compiler and libraries of the CPU vendor (e.g. ICC for Intel).
          If ICC's optimizations were a lot bett

  • MMX/SMD Extensions (Score:5, Insightful)

    by Cassini2 (956052) on Friday August 01, 2008 @08:41AM (#24432377)

    Could it be that FutureMark uses the GenuineIntel and AMD flags to enable processor specific extensions? and then does a whole bunch of math with those extensions and never bothers to check the result?

    This would indicate some really terrible code on FutureMarks part, and VIA should be flagging those op-codes as illegal op-codes, but it might be possible that something like this could happen. It is even possible that the CPUID checks are duplicated in some library somewhere that actually gets the correct code sequence right, and the main FutureMark code disables the advanced functions of the library whenever the GenuineIntel and AMD flags are missing. Thus FutureMark may feature both code sequences that work and those that don't, and the resulting incompatibilities are what causes the issues.

    • by AcidPenguin9873 (911493) on Friday August 01, 2008 @09:55AM (#24433837)
      The thing is, the x86 CPUID instruction [sandpile.org] gives you many different things in EAX/EBX/ECX/EDX depending on what you put into EAX before you execute it. The "GenuineIntel"/"AuthenticAMD" strings are what you get when you put 0 in EAX. When you put 1 in EAX, you get various feature bits in ECX and EDX, such as support for SSE/SSE2/SSE3. Any code worth its salt will base any decisions on those feature bits from CPUID function 1, not on the string you get from CPUID function 0.
  • by Ed Avis (5917) <ed@membled.com> on Friday August 01, 2008 @08:41AM (#24432379) Homepage

    Why would you even consider running a benchmark program you don't have source code for and cannot compile yourself? (If you are worried about random compiler differences messing up the results, you can check an MD5 sum of the final binary against the published one, but it is important that you can reproduce the binary from source and you can read the source to find out what it does.)

    If compilers like ICC cripple their code depending on CPUID, that will just lead all manufacturers to set CPUID to GenuineIntel, just as moronic websites (with help from Microsoft) ensured that all browsers call themselves 'Mozilla'.

  • Benchmark (Score:4, Insightful)

    by Rinisari (521266) on Friday August 01, 2008 @08:41AM (#24432389) Homepage Journal

    Well, PC Mark 2005 is no longer good for testing processors against processors of another maker, i.e. only good for intra-AMD, etc.

  • by LiquidFire_HK (952632) on Friday August 01, 2008 @08:43AM (#24432427)

    That should be AuthenticAMD, not GenuineAMD.

    But that would be expecting editors to actually, you know, edit.

  • Numerology. (Score:4, Funny)

    by cp.tar (871488) <cp.tar.bz2@gmail.com> on Friday August 01, 2008 @08:51AM (#24432563) Journal

    V+I+A == 224
    G+e+n+u+i+n+e == 715;
    Genuine+A+M+D == 925
    Genuine+I+n+t+e+l == 1223

    The bigger the number, the faster the processor. And you get 20% extra when you pass 1000.

  • CPUIDs (Score:5, Interesting)

    by bestinshow (985111) on Friday August 01, 2008 @09:09AM (#24432925)

    VIA's is "CentaurHauls"
    AMD's is "AuthenticAMD"
    Intel's is "GenuineIntel"

    There's no "VIA" nor is there "GenuineAMD".

    Clearly PCMark2005 is buggy (at the best) and cannot be used to compare different CPU families in this test. At the worst it is intentionally flawed, and shouldn't be used at all.

    It's a shame that not one VIA Nano review benchmarked the built-in Padlock functionality. Not one OpenSSL benchmark.

  • by clickclickdrone (964164) on Friday August 01, 2008 @09:11AM (#24432955)
    It's a 3+ year old benchmark being let loose on 2008 vintage CPU's and making mistakes on it's optimisations. I wouldn't expect anything else. It's going to have a 3 year old view on the kind of things these CPUs can do and will act accordingly.
  • by tucuxi (1146347) on Friday August 01, 2008 @09:20AM (#24433143)

    If I were an evil fraudster at PCMark, paid for Intel to deliver worse scores to rivals, I would make sure that these rivals had no easy way of uncovering the fraud. Testing for an ID looks much more like bad code paths than like "sneaky fraud".

    There is no shortage of alternative quirks that can be used to see whether a given processor belongs to one family or another. Should enough of these quirks be combined, it would be *very* hard to discover an evil-related cause.

    Of course, choosing the 'bad' path given an ID may just be blatant enough to provide plausible deniability for the developers that "messed up". However, being a firm proponent of Hanlon's Razor, I would rather call it a bug than a "sponsored feature".

    On the other hand, kudos to the guys at Ars who thought of changing the ID and, when the numbers did not add up, make further tests to nail down the argument. Instead of just forgetting about the problem and performing a "review as usual", which would have doubtlessly required less effort. Yay for inquisitive hacker - reviewers.

  • Given Intel's track record involving anti-competitive practices, I have no doubt in my mind that Intel paid off PCMark.
  • by Ihlosi (895663) on Friday August 01, 2008 @09:27AM (#24433311)
    if(cpuid == "GenuineIntel")
    {
    Run_really_fast();
    }
    else if(cpuid == "AuthenticAMD")
    {
    Run_no_so_fast();
    }
    else
    {
    Run_slow();
    }
  • ... and synthetic benchmarks.

If builders built buildings the way programmers wrote programs, then the first woodpecker to come along would destroy civilization.

Working...