Become a fan of Slashdot on Facebook

 


Forgot your password?
Close
typodupeerror
×
AMD Intel Hardware

AMD Rips 'Biased and Unreliable' Intel-Optimized SYSmark Benchmark (hothardware.com) 174

Posted by samzenpus from the worst-practices dept.
MojoKid writes: AMD is making a stink about SYSMark, a popular benchmarking program that's been around for many years, and one the chip designer says is not reliable. Rather than provide meaningful results and information, AMD claims SYSMark unfairly favors Intel products and puts too much emphasis on strict CPU performance above all else. John Hampton, director of AMD's client computing products, explained in a video why SYSMark itself is an unreliable metric of performance. He even brought up the "recent debacle" involving Volkswagen as proof that "information provided by even the most established organizations can be misleading." Salinas says SYSMark's focus on the CPU is so "excessive" that it's really only evaluating the processor, not the system as a whole. In comparison, PCMark 8 probes not only the CPU, but graphics and subsystems as well. In an attempt to drive the point home, AMD ran a set of custom scripts it developed based on Microsoft Office and timed how long it took each system to complete them. The Intel system took 61 seconds to finish the benchmark versus 64 seconds for the AMD platform, a difference of about 6-7 percent and in line with what PCMark 8 indicated, though Sysmark shows a stark delta of 50 percent in favor of Intel with comparable CPUs.
This discussion has been archived. No new comments can be posted.

AMD Rips 'Biased and Unreliable' Intel-Optimized SYSmark Benchmark More

AMD Rips 'Biased and Unreliable' Intel-Optimized SYSmark Benchmark

Comments Filter:
  • If you're buying an AMD processor, it's for price. If you're buying an Intel processor, it's for performance.

    • Re:Here's my benchmark... (Score:4, Insightful)

      by DarkOx ( 621550 ) on Monday January 18, 2016 @03:57PM (#51324693) Journal

      The reality is for desktop and gaming workloads AMD's upper shelf products really are a better value. You will get a lot more value putting the savings on the processor/chipset into other system components like a little faster SSD or another 4GB of memory. The performance delta between Intel and AMD looks bigger in some of the benchmarks than it is in real world applications. The vast majority of desktop/laptop PC users don't benefit from Intel's premiums and lets call it superior "strait line performance" as for most applications the situation is I/O bound.

      Which is NOT at all to say that if you have more specific workloads like you do a lot video processing, certain types of simulations, etc not to pick Intel.

      If you are just getting a PC to do "generically everything and anything" with the A-series systems are a great value. If you have little more to spend FX + discrete video is still probably a better value than an equivalent i3/5 + video on the Intel side. If you really need an i7 for something than by all means get an i7.

      • Re: (Score:2)

        by Z00L00K ( 682162 )

        It depends on your application what's best. If you have a solution where you can spread out over multiple cores and multiple processors where each core can run pretty much independently then you may want to look at AMD, and a motherboard like the TYAN S8812.

      • I'm not so sure (Score:3, Insightful)

        by rsilvergun ( 571051 )
        I like AMD and all, but we're talking about a company that ships water coolers with their i7 competitors ( the 9xxx line). My brother's running his i7 with a stock fan... I'd love to be proven wrong but right now AMD just doesn't seem like they can hang. My A10-5800k is nice and all but in games it's about the equivalent of a mid range i3, but I can replace that i3 with a 5 or a 7... With the AMD the best I could do is an 8350...

        • Re:I'm not so sure (Score:5, Informative)

          by alvinrod ( 889928 ) on Monday January 18, 2016 @05:30PM (#51325365)
          To some degree that's because most of AMD's CPUs are still being fabbed on an older 28nm (or even 32nm) node, which Intel moved beyond almost four years ago. Right now Intel has such a leg up on them due to process differences that there's no way AMD can legitimately compete at the high-end. The best they can do is crank up the clock speed and use a more powerful cooler, but Intel is shipping chips on a 14nm node right now. That comes out to a 4x difference in transistor density.

          AMD still provides really good value for a regular consumer machine, and with them finally having some CPUs on a newer process node (14nm supposedly based on Samsung tech) later this year they might even be able to compete with Intel at the higher end of the charts, though I doubt they'll nab any performance crowns.

        • I have a FX-6300 3.5Ghz and a i5 3.2Ghz they are not really even as far as specs or cost but neither require an extra cooling system the FX blows the i5 out of the water and the i5 still cost considerably more. You can make want you want of that but I'll buy an FX-8350 before I'll spend more on an i7.

          • Re: (Score:3)

            by armanox ( 826486 )

            Oddly enough in every case that I run my i5-3570K is much faster then my FX-8120.

            • It's probably a difference in use since I'm not really a gamer but also a difference in board and the resources installed my fx 6300 system has a better graphics card and more/better and ram still cost less. I'm sure if I upgraded the ram, hard drive, and graphic card on the i5 it would run a hell of a lot better but then again we are talking the difference in a pre-built expensive i5 system and less expensive custom built system.

      • I watched a lot of videos and perused a lot of benchmarks before I went ahead and went AMD anyway, but I have to say that while I agree with you in principle, the gap between AMD and Intel is actually pretty big if you measure pessimistically. If you're counting dollars per maximum FPS, then AMD comes out well ahead. If instead you count dollars per minimum FPS, look at the worst cases instead of the best-case or the average, then Intel doesn't look so bad. The minimum frame rates are where you get your ass

        • Re: (Score:3)

          by EEPROMS ( 889169 )
          I read an article lately where they tested a whole pile of games against different CPU's. The results showed beyond your average 2-3 year old 4 core i5 there is little if any advantage for gamers for spending more money on a CPU. Your average gamer would be better of buying a better graphics card (or two), SSD or motherboard than buying an expensive/new cpu.

          • A relatively snazzy quad-core i5 beats the performance of any AMD FX-series CPU, surprisingly remaining true even when running heavily multithreaded games (e.g. ports from Xbox 180.) So yes, I would say that's the sweet spot in relatively cheap gaming right now. With, say, a 950 or thereabouts, and a SLI motherboard so you can add another later. With that said, my 8350 and 750 can play pretty much anything at decent quality at 1920x1200. Not top quality with antialiasing, but high resolutions are better th

      • I have been happy with my circa 2012 FX-8350 for a while now. Was planning to skip it as my Phenom II X6 was still very viable, but the A10-5700 changed my mind. With 16GB of DDR3-1866 and a 480GB Mushkin SSD it is as fast as any other machine I've used for 99% of what I do. An i7 might give me a boost in ripping video, and I'm certain that should I upgrade my Nvidia 770 the CPU will be the bottleneck, I can still max out most games.

      • First off does anyone use synthetic benchmarks anymore? I mean they were proven to be cheating like a decade ago. Anyway I'd agree with AMD that it isn't a good measure of performance.

        Having said that, there are different types of performance for different things. However there are limits. So while generally speaking, saving some money on a CPU, and spending it elsewhere may make sense for gaming, there are limitations. I think generally speaking, you're wrong about AMD being a "better value" for gaming. Th

    • Re:Here's my benchmark... (Score:5, Informative)

      by NoNonAlphaCharsHere ( 2201864 ) on Monday January 18, 2016 @04:08PM (#51324775)
      I hate to break it to you, but Intel's been cheating on benchmarks since the late 70's. There have been countless stories through the years about how their compilers and processors have detected that they're running Whetstone, or SPECmark or whatever, and done something out-of-band as a result. There have also been numerous attempts by them to game the benchmark standards themselves. Think the Volkswagon diesel cheat on steroids -- Intel is famous for it.

      When you say you're buying Intel for performance, you're saying that you're really buying Intel for brand-name recognition.

      • Think the Volkswagon diesel cheat on steroids -- Intel is famous for it.

        When you say you're buying Intel for performance, you're saying that you're really buying Intel for brand-name recognition.

        Actually what you're saying is that we're buying it for performance. How we gain that performance is irrelevant as long as it translates to performance for the end user. Just like the diesel cheat produced a cheaper car with more performance for the user. It's only when other externalities are considered that the problems start to affect choice, e.g. if you bought a VW based on its emissions.

        While Intel has been cheating for a long time to varying degrees, it doesn't change the fact that for many work loads

    • I've always thought of SYSMark as just a tool for testing the CPU anyway.
      Other than that, another person whining about benchmarks not being fair. So what year, heck, what decade should we queue up for more deja vu statements?
      Everyone in the industry has been complaining about this or that benchmark being unfair to this or that product since the first benchmark program was released.
      The solution is to know what each benchmark is good for testing, and weigh the results appropriately.
      It reminds me of an adult c

      • It reminds me of an adult complaining that they have to take twice as much of the kids gummy vitamins.

        Since when has anyone ever complained about taking gummy vitamins?

        I have to force myself not to eat those things by the handful...

    • Re: (Score:3, Insightful)

      by Blaskowicz ( 634489 )

      Well not really, since Intel has put cheap dual core, dual thread CPU on the market that have an incredibly high single thread performance. e.g. Celeron G1820 is cheaper than A6-6400K, and Pentium is cheaper than A6-7400K (or same price)

      AMD is good somewhere at the mid range, and if you're not gaming.

    • Generally I've bought AMD on price-point for various machines, but I've also played a bit with their APU's in cases where space was more of a consideration and I didn't want to run a discreet card (and Intel's onboard graphics weren't very good).

    • Think of it this way- if AMD is competing against those intel cpus that do PcMark 8 8% less then that it just not true. There are always I5 systems that have parity with the AMD, and once you factor in power consumption the cost is same or worse for AMD

    • Actually no. I buy AMD processors for *intregity*

      I haven't bought an Intel processor since they said : If you can show us you *need* it you can have a fixed Pentium (without the FDIV bug).

      https://en.wikipedia.org/wiki/... [wikipedia.org]

      Publicly, Intel acknowledged the floating-point flaw, but claimed that it was not serious and would not affect most users. Intel offered to replace processors to users who could prove that they were affected. However, although most independent estimates found the bug to be of little importa

      • Don't forget that where most amd users cripple their system by buying a cheap ass motherboard. People complain amd is slow then you realize they went from a cheapo 50 dollar motherboard to a 250 dollar one....

    • Re: (Score:2)

      by Trogre ( 513942 )

      Not necessarily. AMD lose out in anything requiring high FPU performance ever since the brain-dead Bulldozer design but for integer intensive work AMD opterons with their large number of cores are still the better option.

    • Re: (Score:2)

      by mjwx ( 966435 )

      If you're buying an AMD processor, it's for price. If you're buying an Intel processor, it's for performance.

      It wasn't even 10 years ago when you said the reverse.

      20 years ago it was "you buy Intel and pay through the nose". Personally I'd rather not return to a CPU monopoly.

    • As someone who has both AMD and Intel chips -- AMD Phenon II 965BE @ 3.5 GHz (and soon an FX-8350), and Intel i7 4770K @ 4.0G Hz, couldn't have said it better myself !

      Why isn't there room in the market for both?

  • oh yay hothardware.com (Score:1, Interesting)

    by Anonymous Coward

    I just upgraded from an FX-8350 to an i7-5930k

    night and fucking day

    • No shit! Your new processor costs about 3 times the price if your old one, never mind the difference in cost of the motherboard.

      Everyone knows AMD don't compete in the 550 pound desktop processor segment. That doesn't mean that Intel isn't massively faking benchmarks for the lower cost ones where AMD does compete.

  • Hey, it is not cheating, it is not compliance checking, so it is not like lying like VAG or Bosch. I understand that this is a synthetic benchmark that favours Intel CPUs. However, there are a variety of benchmarks out there. For example, Oracle posted some benchmarks where Sparc CPUs run faster than Intel CPUs, for a given application. Curiously enough, Oracle did not bother benchmarking agains AMD CPUs.

  • Just follow the money. (Score:3)

    by xxxJonBoyxxx ( 565205 ) on Monday January 18, 2016 @03:36PM (#51324515)

    Intel's a member of BAPco, the SYSmark organization, and AMD isn't.
    https://bapco.com/about/ [bapco.com]

    On the other hand, if it's really a big deal to AMD, they should be able to find $100K or so to join BAPco and tilt the deck in their favor - total annual budget only seems to be $400K.
    http://www.faqs.org/tax-exempt... [faqs.org]

  • Maybe (Score:2, Insightful)

    by Billly Gates ( 198444 )

    Just maybe if AMD got off their butts and made unbiased reports and reliable/fast chips and graphics they would not be in this predicament.

    • Re:Maybe (Score:5, Interesting)

      by TomR teh Pirate ( 1554037 ) on Monday January 18, 2016 @04:17PM (#51324847)
      I used to run AMD's consumer benchmark group during the K6, K7, K8 days. I'm not sure what you mean by "unbiased reports", but I can tell you that the process the company went through to create and execute benchmarks that were unbiased was remarkably fair. In the time I was there, the company ran benchmark results for any application that met three key requirements:

      1) repeatable results
      2) relevant software
      3) practical to benchmark

      So this meant that using canned benchmarks from applications such as Winstone for MS Office applications was a great option to look at office productivity software. We spent a lot of time trying to figure out how PC Magazine was weighting the application between the various MS Office applications, and I hit upon a way to do this by changing core frequency during benchmark runs so that we could create a multi-dimensional array of scores vs. frequencies to determine that Word was x%, Excel was x+5%, etc. We came up with a likely weighting scheme, although I don't recall what became of that work. In the consumer space, the other big hitter is obviously games. At the time of my tenure, AMD used many or most of the same gaming applications that were en vogue with Firing Squad, Toms Hardware, Anand Tech, Sharkey Extreme, etc. There was nothing nefarious about the work we did, nothing unbiased. We looked at these applications with equal weighting and determined that for a given frequency of relevant, competing Intel CPUs, there was an AMD offering that on balance, performed equally or better at a lower frequency. This processor was then given a model name such as 1800+ that was meant to convey it compared favorably to an Intel 1.8GHz CPU. In the days that my group did this work, AMD made a point of publicizing this process and went so far as to have the process vetted via direct supervision of a 3rd party auditing company who was one of the big-4 industry auditors. It was painstaking work to demonstrate that software load order and procedure was identical for AMD and Intel parts. When a benchmark completed, we showed the score to the auditor. Sometimes benchmarks returned imperfect scores because of a stray hard disk latency event and would throw the score off for either product. We would work with the auditor to show that the result of the otherwise repeatable values was an outlier and subsequently toss it in favor of another run.

      Others in this Slashdot post have complained of heat dissipation. My team was solely concerned with instructions per second and performance per watt was not a concern for us. I do vaguely recall that this may have been a factor for the server team. My guess is that based on reading the occasional tech article here and there, AMD has made some important progress on power management.

    • When AMD was KING lntel locked them out of big parts of the market and that give intel time to get back on top. And now that they are on top they are ripping people off and lacking on pci-e lanes on sky lake and cutting the number of lanes on the on higher end chips unless pay more then the last gen when the lowest one had it all.

  • Known for a while (Score:5, Insightful)

    by gman003 ( 1693318 ) on Monday January 18, 2016 @03:39PM (#51324547)

    SYSMark has been known to be not particularly representative of actual performance for quite some time. In particular, they seem weirdly sensitive to memory latency, way beyond its actual impact, yet they deliberately evade caching even in benchmarks measuring something where caching is normally useful. And they do seem to unreasonably penalize AMD chips, although I'm not sure if that's malice or simple incompetence.

    The review sites I frequent tend to use PCMark for the general-overview synthetic benchmarks, along with some actual-program benchmarks (usually compression, crypto, and video encode). I of course prefer the latter - nobody runs synthetic benchmarks in production, it's always some actual application. The closer you can get to benchmarking that actual app, the better.

  • Microsoft Office (Score:5, Insightful)

    by Anonymous Coward on Monday January 18, 2016 @03:47PM (#51324619)

    The real outrage should be that operations in Microsoft Office are measured in seconds and minutes instead of nanoseconds and milliseconds.

    • Re:Microsoft Office (Score:5, Funny)

      by Korbeau ( 913903 ) on Monday January 18, 2016 @05:31PM (#51325373)

      The real outrage should be that operations in Microsoft Office are measured in seconds and minutes instead of nanoseconds and milliseconds.

      Well every time you type a key it must send it online for anonymous data collection, match it against a dictionary for instant grammar correction, save a copy of your modified file to your OneDrive online free storage space, run a few ticks into Clippy's neural network, send your typing statistics to Cortana, pass through 10 layers of automation & scripting support interfaces & abstractions ...

      Ah and yes, eventually also update the output buffer with actual letter symbol to be displayed on the screen!

  • Yes and no. (Score:3, Insightful)

    by Anonymous Coward on Monday January 18, 2016 @03:53PM (#51324673)

    I agree with AMD that most of the market is slanted towards Intel.

    But I don't want a benchmark score that is dictated by a graphics card and it's driver set. I want a cpu score that is based on CPU performance, only CPU performance and perhaps taking into account the effects of memory memory bandwidth. Plenty of tools on the internet for that they could have showed instead.

    What I want from AMD is a cpu in the 150$~ range with a performance equal or exceeding my old overclockedi5-2500k (2011 vintage) and be capable of gnarly overclocking. If they can deliver that, they have me for an entire socket generation.

    • But I don't want a benchmark score that is dictated by a graphics card and it's driver set. I want a cpu score that is based on CPU performance

      That's like someone in the 486 era saying they don't want a benchmark that's dictated by the floating-point coprocessor. Face it, there's no such thing as a "graphics" card anymore; there's just a coprocessor that's very good at very parallel workloads.

      • Re: (Score:2, Interesting)

        by Blaskowicz ( 634489 )

        It's also fairly irrelevant if a) you're not running specific productive software under MS Windows (that may mean running a particular plug-in for Photoshop) or b) you're playing a game and the CPU doesn't keep up (e.g. you're stuck at 20 fps even at lowest details and resolution)

        AMD : the CPU performance of 2009, today.

  • It gets worse... (Score:5, Funny)

    by msauve ( 701917 ) on Monday January 18, 2016 @04:03PM (#51324745)
    If I benchmark a common office task, like typing War and Peace into Microsoft Word, I find that these new processors run no faster than an old IBM PC/XT with a 4.77 MHz Intel 8088.

    • Re: (Score:1)

      by Anonymous Coward

      Here in the future, some of us are working with terabytes of data, teaching computers how to think, and manipulating millions of pixels simultaneously in real time (among other things)

      Hardware performance is kind of important.

    • No no no. You have to be fair and benchmark how long it takes Windows 10 to boot and office to open too.

      I'm running this benchmark right now. I hope to publish the results sometime in 2022.

    • Re:It gets worse... (Score:4, Insightful)

      by epyT-R ( 613989 ) on Monday January 18, 2016 @05:31PM (#51325377)

      Well, yes, because each keypress in current versions of ms word triggers several megabytes worth of code to execute, probably inside a CLR (because somehow this makes everything better). 'Progress' is now defined by getting gigaflop spec cpus to mimic the same laggy performance we got from 286s back in the day.

  • I like AMD... (Score:5, Insightful)

    by MetricT ( 128876 ) on Monday January 18, 2016 @04:26PM (#51324925)

    My work desktop is AMD, my home fileserver is AMD, and both my parent's desktops are AMD. That's because in those use cases, AMD is "good enough". Web browsing and email don't require a lot of horsepower.

    That said, my gaming/transcoding PC is an Intel i5-4690, because AMD's top line CPU can barely compete with Intel's I3 line. CMT didn't pan out, and they've been held hostage by TSMC/GloFo's failure to produce a sub-28nm lithographic process.

    I love AMD's engineers, they have some impressively smart people working for them, and I hope Zen + 16nm heralds a new beginning for them. But today, they aren't "competitive", merely "good enough".

    • Out of interest what games are you running into where CPU is the bottleneck? I have a Phenom II X6 1055T which is paired with a GTX580 and I don't seem to have any issues playing anything. In passmark it loses out to an i5 but if the application is well threaded the 1055 will give an i7 a run for its money. When you look at futuremark it scores 4530 vs 4600 for an i5-4570

      • Re: (Score:3)

        by MetricT ( 128876 )

        I bought the Intel i5-4960 because, having done high-performance computing for over a decade, current Intel CPU's absolutely maul AMD CPU's when it comes to numerical work. It was my first new computer in almost a decade, and I wanted a "no excuses" gaming box. Games *can* be good on AMD, but many top tier games require various trade-offs.

        And I'm not an AMD hater. Once upon a time, we had 100's of AMD Opteron's in our compute cluster. But it's been several years since they have put out a competitive ch

        • Fair enough. My machine happily plays fallout 4 or anything else I have thrown at it so I'm not getting a bottle neck as far as I can tell. No question the intel chips are higher performance I just am not seeing a difference on a day to day basis.

          That said, for a long time now I've found little need to upgrade regularly as nothing seems to strain even semi modern pcs.

        • Re: (Score:2)

          by sl3xd ( 111641 )

          +this. I've also done High performance computing for over a decade, and claiming that one company's benchmark is unfair is crap when every other open benchmark tells a similar story.

          It broke my heart to see AMD fall the way they did, but when you can compile and run the benchmarks yourself, with whatever compiler you choose, and see, with perfect clarity, how badly AMD gets mauled by Intel's chips. Those of us working in High Performance Computing don't care who makes the part, we just need the part that's

      • That want an i5 or better. Arkham Knight, Just Cause 3 and some of the Ubisoft games (Far Cry 4 et al) have troubles with lower end CPUs. There's strong evidence that the reason is that the problems are due to the newer DRM encrypting the entire game and decrypting it on the fly but I haven't heard a 100% confirmation of that from anyone.

        Intel's next CPUs are suppose to have hardware support for decryption. I think you're going to get stuck with a powerful CPU at some point to run games for just that re

    • Re: (Score:1)

      by phorm ( 591458 )

      My transcoding machine is an FX-8350. It may not has as much raw per-core power as the i5, but on a multi-threaded transcode job (or multiple simultaneous transcode jobs) it does very nicely. It does get hot as shit during a heavy workload though, so it's got a big case with lots of fans and room for air moment.

  • Honestly, most of these benchmarks have LONG outlived the point where they provide any sort of useful information for anything.

    They're basically masturbatory devices whereby people who have nothing better to do flaunt their e-peen.

    They try to tell you that they correlate back to specific tasks in the system. Unfortunately this has been so utterly abstracted by now that the synthetic counterpart is utterly meaningless.

    The only thing that matters is "Does the system do what you need it to do at a reasonable

    • Honestly, most of these benchmarks have LONG outlived the point where they provide any sort of useful information for anything.

      I agree but so has CPU speed and number of cores but we need some way of comparing apples and oranges to decide which CPU a person should buy. I find AMD's approach interesting. A benchmark written in excel should be about as good as any for testing desktop work. Many video games like minecraft and even some first person games can be accessed via scripts so it would be simple enough to create benchmarks based on actual games too. This seems like the better approach. Build the benchmarks inside the actu

  • Compile for the architecture (Score:3, Insightful)

    by GreatDrok ( 684119 ) on Monday January 18, 2016 @06:39PM (#51325861) Journal

    While an AMD chip can run x86-64 code compiled for an Intel processor, it isn't surprising that the code doesn't perform as well since a lot of optimisations relate to features of the specific chip. You can't use a precompiled binary across all chips and expect them to be useful other than to say one chip can run that binary quicker than another. I remember years back having some code that was optimised for the Intel PIII and when that same code ran on the AMD Opteron it was slower despite the Intel running at a clock speed of 1.4Ghz and the AMD running at 1.7. Once I went in and had a look at the ASM I could see why - the AMD had a 64 bit bus and the code was using instructions which weren't as efficient on AMD's chip as a result of this. Once I realised that, I rewrote that section of code to account for this and the AMD ended up being 30% quicker than the old code when I rewrote four lines of C. Compiler optimisations only go so far but you still have to be aware of the underlying chip if you really want to get the most out of it.

  • Remember the Rules Of Benchmarks (Score:4, Insightful)

    by SoftwareArtist ( 1472499 ) on Monday January 18, 2016 @09:38PM (#51326725)

    Rule 1 (good to a first approximation): All benchmarks are meaningless.

    Rule 2 (for experts only): Every benchmark measures something very specific. A benchmark is only meaningful if you know exactly what it is measuring, and the thing it measures is something you actually care about.

    • Re: (Score:2)

      by sl3xd ( 111641 )

      Rule number 3: If you make a product, don't point to benchmarks that test something entirely unrelated. ie. if you make CPU's don't argue that your product is better because of a third party hard drive.

  • Okay, so AMD is in the business of manufacturing and selling CPUs. Along comes a tool to qualitatively analyze CPU performance. AMD doesn't like that. What are they really trying to say?

    P. S. I'm fully aware that there are all kinds of backdoor deals and benchmark fudging in the market, but as other posters have noted, you want a CPU score based on the performance of the CPU.

Slashdot Top Deals

Ocean: A body of water occupying about two-thirds of a world made for man -- who has no gills. -- Ambrose Bierce

Close