PCMark Memory Benchmark Favors GenuineIntel 298
javy_tahu writes "A review by Ars Technica disclosed that PCMark 2005 Memory benchmark favors GenuineIntel CPUID. A VIA Nano CPU has had its CPUID changed from the original VIA to fake GenuineAMD and GenuineIntel. An improvement of, respectively, 10% and 47% of the score was seen. The reasons of this behavior of FutureMark product are not yet known."
Do I understand this correctly? (Score:2, Interesting)
Is this like changing the user agent in a browser?
Re:Compiler Optimization? (Score:3, Interesting)
Re:Money (Score:5, Interesting)
If anyone can come up with a better explanation I'd be interested to hear it.
Re:Possible semi-benign explaination? (Score:3, Interesting)
That's an insightful explanation, but IMO the benchmark is then only valid is the operating systems people use make the same allowances for the different chips.
One thing that doesn't seen to have been investigated is the permutations of the test - marking an Intel chip as a VIA, etc. If the differences are the same drop in performance as the improvement in marking a VIA as an Intel then your explanation has effectively been disproved.
CPUIDs (Score:5, Interesting)
VIA's is "CentaurHauls"
AMD's is "AuthenticAMD"
Intel's is "GenuineIntel"
There's no "VIA" nor is there "GenuineAMD".
Clearly PCMark2005 is buggy (at the best) and cannot be used to compare different CPU families in this test. At the worst it is intentionally flawed, and shouldn't be used at all.
It's a shame that not one VIA Nano review benchmarked the built-in Padlock functionality. Not one OpenSSL benchmark.
The more obvious answer (Score:3, Interesting)
Re:Money (Score:5, Interesting)
Yes, I remember that...
But why would icc make AMD better than "no name" beats me.
troll? really? mod up again! (Score:2, Interesting)
you got a point there which is important to the discussion, if the source is closed, how can we know if the test is fair?
Re:Money (Score:5, Interesting)
Saying that a config has 9000 points is pretty much useless. Saying that it gets an average of 40FPS in the UT3 benchmark at high details, and 1680x1050 is much more informative.
Unfortunately, this also is a little bit more complicated, and as we know everything simpler is more popular with the dumb masses.
Re:Money (Score:3, Interesting)
Re:Money (Score:3, Interesting)
Fine, testing is one reason for two code paths; you want to make sure the generated code works on all x86 CPUs, but you don't want to thoroughly test on all of them. But that means that icc isn't suitable for compiling benchmarks. because different code is being run depending on the CPUID. The comparison is not a fair test; you have two variables (the code and the CPU) instead of one (the CPU).
Two questions:
- Why didn't Intel tell benchmark writers not to use icc? Obviously the results will be unfair if an Intel CPU is being compared against a non-Intel CPU, and Intel will be accused of cheating.
- Why not call the memcpy in the C library instead of doing a one-byte-at-a-time memcpy? Amateur coding? Or professional evilness masquerading as incompetence?
Re:Money (Score:2, Interesting)
Re:Money (Score:5, Interesting)
If your benchmark tool is going to use multiple code paths, then they should be configurable, so that you can benchmark different systems both using the same code and more optimal code. That way you'd get an idea of how much speedup various features provide.
As an example, john the ripper's SSE2 support for cracking DES - on a core2 the SSE2 version is considerably faster and compiling the C version never comes anywhere close regardless of compiler and flags, but on an AMD compiling the generic C code with appropriate flags and a modern version of gcc produces slightly faster code.
Running the SSE2 ver on a 2.3ghz core2 quad achieves about 2mil c/s per core, while a 2.3ghz amd phenom yields about 1.6, but compiling the C source with various flags and gcc versions makes amd slightly faster, while the core2 is nowhere close.
Re:Money (Score:2, Interesting)
Intel crossed the line when they made their compiler do optimizations based on the CPU vendor-ID "GenuineIntel" instead of the CPU feature-flags.
The feature-flags are what Intel documented as the correct way to check if advanced features are supported by the CPU.
And that is why our shop won't use Intel's ICC compiler.
Re:Money (Score:1, Interesting)
"I'll give 10:1 odds that Futuremark simply compiled their benchmark with Intel's C++ compiler.
I wrote a detailed explanation back in 2005 about how the Intel C++ compiler generates separate code paths for memory operations to make AMD processors appear significantly slower, and how you can trick the compiled code into believing your AMD processor is an Intel one to see incredibly increased performance. See this article for additional details." - by Eponymous Cowboy (706996) * on Friday August 01, @10:09AM (#24432917)
Good job man, & that's completely 'no sarcasm' on my part - congrats/kudos/salutations.
That sounds COMPLETELY logical, & reasonable + "right on the mark", about futuremark & other benchmarks... especially, as others state here, IF you "follow the money trail" & look @ the motivations of billions of potential profit.
Hey - I'll even "go out on a limb", & say YOU are probably right here - even though "the great arstechnica" (NOT) apparently couldn't figure that out, & you basically DID, long ago (3++ yrs. back)...
----
HEY - between the processor errata that INTEL has, that has folks in security circles 'terrified' to a big extent, since hacker-cracker types are out to exploit them, per this news:
Researcher to demonstrate attack code for Intel chips:
http://www.infoworld.com/article/08/07/14/Researcher_to_demonstrate_attack_code_for_Intel_chips_1.html [infoworld.com]
SALIENT/PERTINENT EXCERPT:
----
"Kaspersky says CPU bugs are a growing threat, with malware being written that targets these vulnerabilities... Security researcher and author Kris Kaspersky plans to demonstrate how an attacker can target flaws in Intel's microprocessors to remotely attack a computer using JavaScript or TCP/IP packets, regardless of what operating system the computer is running."
----
I am GLAD I have an AMD here, by all means, because still, to date? AMD has not been shown as vulnerable to the above, either... not yet @ least. So far, so good.
(I would have told anyone that INTEL CPU's were better/faster, until I saw this article of yours AND the one I point out above - regarding CPU errata exploitation being possible on INTEL cpu's, where so far, no such thing has surfaced on AMD stuff!)
----
Ordinarily?
I'd say that is an accomplishment on YOUR part, in figuring out YEARS AGO, what all of arstechnica apparently cannot!
Then again - Comparing yourself vs. arstechnica's general populace in computing know-how isn't saying much man (it's like comparing a world-class athlete vs. a newborn child, in a footrace - not exactly a fair comparison (& most anyone could win)).
No HUGE trick, getting the better of them & their "experts" (like the "no degree, no certification, no years-to-decades of professional hands-on experience in the trenches in the art & sciecne of computing" Jeremy Reimer, for example. No contest).
APK
P.S.=> Good job man, good job... &, I liked your article as well - I must have missed it years ago, thanks for bringing it up again too! apk
Agree; but could we formalize auditing more? (Score:4, Interesting)
I agree with you.
I was wondering if there is some way we can get code audited by the community on a more formal basis, perhaps with a bounty system and a reputation system, so that one might donate to get the KDE4 code audited by me ($10), or some KDE contributor ($300), or Linus Torvalds ($10000). Then these people could develop a formal reputation system, like + or - votes on SourceforgeAuditVoting.org. They'd use their PGP signature to sign the audits.
Or something. I would view this as the next phase of the open source economy. Eventually companies might hire people with good reputations, to audit their own intra-company code.