Examining Benchmarking 95
VL writes "Benchmarks exist to determine how a particular piece of hardware performs in relation to itself, and to others. Question is, are readers getting the information they really need?"
This discussion has been archived.
No new comments can be posted.
Examining Benchmarking
Related Links Top of the: day, week, month.
A committee takes root and grows, it flowers, wilts and dies, scattering the seed from which other committees will bloom. -- Parkinson
science studies (Score:5, Insightful)
Re:science studies (Score:5, Insightful)
Sometimes you get the feeling that the whole thing is a cut/paste job from someone else's paper, with your own study objects and figures pasted in here and there.
Other times, you know it's made by a weatherwane who will write up whatever is the most popular opinion at the time -- use of fad words like "vetting" and "triage" is a prime example of this, and enough that you should take the whole study with an ounce of salt.
Then there's the studies where you instantly see is made by a bureaucracy, where the documentation and amount of paperwork is much more important than whether the tests and figures actually makes sense. That's the studies where you read whole chapters on methodology, and still haven't figured out just WHAT they're testing, and WHY.
Then there's the tests that are overly consumer friendly, and try to produce one single big nice number that symbolises everything, while dumbing down the language so much that you have no idea what is really tested. Unfortunately, those seem to be the tests that people LIKE the most, although the value of them is moot.
Finally, there's the obviously paid (by money or by enthusiasm) studies, that will skew the results one particular way. Abundant use of graphs, and especially non-linear or cut-off graphs is a telltale sign here, as is the absense of any explanation for just why THAT particular test was emphasized, while other tests appear to be missing or downplayed. Use of deltas instead of hard numbers is also revealing -- you are told that going from video card A to B will give you 300% more increase than going from A to C, but if you analyse the raw figures you'll find that the real difference is going from 100fps to 104fps versus going from 100fps to 101fps.
All in all, I can't say that I've seen many benchmarks and benchmark studies that aren't biased or skewed one way or another, or just plain irrelevant.
Regards,
--
*Art
yes... (Score:2, Funny)
They are irrelevant. (Score:3, Funny)
I don't care if one can get 900fps in Unreal Tournament while the other can only get 880.
As for bias, did you know that my Timex Sinclair is the best computer there that has ever been made or will ever be made? The salesman said so, so it must be true.
Re:They are irrelevant. (Score:4, Funny)
The guy on eBay said it's scientifically proven as the fastest computer on earth*.
* In 1947, that is.
Re:They are irrelevant. (Score:2, Redundant)
Goedel says benchmarks are inherently flawed. (Score:5, Interesting)
Goedel's Incompleteness Theorem [stanford.edu] states that you can't define a system entirely in its own terms, and that any system needs to be defined by terms outside of it.
So, how can you accurately rate hardware based on similar hardware? To meet the GIT (Goedel's Incompleteness Theorem), you would need to compare the hardware with something outside of the system, so you have an external reference point. For example, if you're benchmarking graphics cards, you need to also compare them to something outside of that area of hardware.. so.. say, a graphics tablet, or an iPod.
So, say that the first graphics card is 0.7% compared to the iPod, we now have an external reference to use with the other graphics cards.. so a better card might be 10% compared with the iPod, or a few percent compared to the graphics tablet, which proves that the second card is better than the first, due to the respective ratings compared to the external objects.
This is just regular math. I have to say, it's pretty amazing what you can apply regular math to.. yes, even benchmarks!
Re:Goedel says benchmarks are inherently flawed. (Score:5, Insightful)
especially when pcworld or whoever compares entire PC packages... so much can change depending on what background software is being run, system tweaks/factory settings that could be off... no one should really buy a pc package based on those comparisons alone.
Re:Goedel says benchmarks are inherently flawed. (Score:1, Interesting)
When you buy somthing to perform job X,how well it does at performing job x is one of the reasong for buying it.
Ipods don't run quake..so the point is maximum silliness.
The Dynamic application of intellect is what defines real intellegence..not theorys..thats just memorization.
Re:Goedel says benchmarks are inherently flawed. (Score:4, Informative)
Software is an external reference point, its somthing outside of the system. is itself iffy.
We know that the video cards are designed, in part, to benchmark well. Some manufacturers have even gone so far as to write drivers that inflate framerate at the expense of accuracy, under certain benchmark like conditions. (Quake.exe v. Quack.exe, anyone?). Apple inflated its spec results by using a unrealistic single threaded malloc library. Intel's icc is rumoured to detect, and optimize for SPEC.
The Dynamic application of intellect is what defines real intellegence..not theorys..thats just memorization.
Theories? Theories are meant to be proven as an exercise for the student, not just memorized.
Re:Goedel says benchmarks are inherently flawed. (Score:5, Funny)
Re:Goedel says benchmarks are inherently flawed. (Score:2, Funny)
Re:Goedel says benchmarks are inherently flawed. (Score:5, Insightful)
Godel's incompleteness theorem is about complex mathematical systems and the essense of proof. You don't need an external object to compare to, anything will do. You just choose some graphics card as your fixed point and then compare everything to that card.
Identify the truths ... Re:Goedel says benchmarks (Score:2)
One of the key things in Godel's conception is that there are certain truths, i.e. for example that the math on earth and the math on mars will have something similar to prime numbers, etc.. That is why the two maths created on earth and mars, where certain common symbols are used in both the earth-maths and mars-math, have some similarities. The similarity is not that they are using some of the same symbols, but that they using math to describe the same reality ...
the equivalent of the benchmarks de
Re:Goedel says benchmarks are inherently flawed. (Score:2)
*measuring*. Note that Goedel's incompleteness
theorem does not necessarily imply that ALL
propositions are unprovable! It merely states that
there is at least one proposition that is true but
cannot be proven (Goedel constructed this
proposition in an ingenius manner[1]). It suffices
to build ONE such proposition to derive incompleteness.
MANY other propositions can be proven, including,
possibly the fact that card A is better than card B
(if you can call this thing
alpha geeks (Score:2)
that said, people should demand accurate/unbiased benchmarking becuase of all budding nerdlings who end up with junk in there system and helping some bloated crap company stay on top
the whole unbiased thing: i personally have a VERY hard time believing almost any scientific study (be it benchmarking or dieting) to be unbiased... whether or not there may be a
The only way benchmarks could work (Score:3, Insightful)
As for benchmarking results posted by benchmarking companies and sites. Well they have to eat too
Second problem are known tools, leads to driver tampering, ok that's not related with food on the table of benchmarkers
Re:The only way benchmarks could work (Score:3, Insightful)
Unfortunately, the "in secret" part applies far too often. Take a look at the tiny print of the license that came with your latest piece of hardware (and in some cases software), and chances are that you're agreeing to not publishing any benchmark results.
Hardware manufacturers, of course, only wants to see favourable benchmarks, and pays quite well to get them, and to supress others. Fair comparisons are all well and fine, but it won't make their investors any mone
Slanted and biased (Score:3, Informative)
Prefer multipl e benchmarks, or your own 'problem' (Score:5, Interesting)
So, if you want to know about your hardware, you better run more than one benchmark, and more importantly, your 'problem code'. Yes, you want hardware that performs well for you problem. Something that can be good in general, is ratrher rare.
Re:Prefer multiple benchmarks, or your own 'proble (Score:2)
Here's a nice example of how benchmarking can give non-applicable "information":
Say one's got a system that acts as a file-server, NAS, or something, and one backs-it-up using DVD-RW's, and one's victi^H^H^H^H^H users complain about the system being intermittently hammered, whenever you're doing the backup, calling-up some program to tell you how many context-switches are happening would show you that when you've got the DVD-RW loopbacked, and are diffing ( niced to 19 ) the ISO with the DVD-RW, you're su
Not all of us (Score:2, Informative)
Re:Not all of us (Score:2, Interesting)
Ivan Sutherland cites "the wheel of reincarnation" whereby the graphics co-processor becomes more and more powerful until it is a stand-alone general purpose computer which in turn gets its own graphics co-processor starting the cycle again.
we have a long way
Re:Not all of us (Score:3, Informative)
Video card power is only used in DESIGNING and PREVIEWING the scenes...
Video card is good but not as good as raw math over some minutes (or even hours)
Hard to care. (Score:5, Interesting)
My favorite Operating Systems haven't been the ones with the best selection of software.
My favorite games haven't been the ones with the best graphics.
The reviews I find most valuable don't have the most complete set of numbers of why something's the best or worst.
It's interesting that the goal of benchmarks is to be objective as possible, when it's the subjective that makes me want to buy or not buy something. But meanwhile the more the objectivity of the benchmark tests are in doubt, the less important the tests become. So I guess that means benchmarks don't mean anything to me one way or the other, huh?
Alex.
Re:Hard to care. (Score:2)
Does it really matter? (Score:1, Insightful)
Benchmarks and science (Score:4, Insightful)
Its very hard to get a good benchmark (Score:5, Interesting)
For less widely used benchmarks, its possible to do one offs in the lab and include the false results in the marketing material. The primary examples of this are spec, drhystone, and whetstone. For awhile Intels compilers had recognition routines just for these benchmarks. Apple has always done tuned versions of the benchmarks.
Once a benchmark gets into the wild and is in a form that anyone with a website can just load without too much trouble on a machine, you get manufacturers actively moving to cheat the benchmark. Best examples are Nvidia and ATI's optimizations that are specific to 3dmark and quake III.
I don't know of anyone who would buy a piece of hardware solely on a benchmark, However salesmen when they can't sell are without peer in inventing excuses and shifting blame. So as long as you have sales goals that are unrealistic and salespeople that are good at inventing excuses, you will have engineering departments forced to cheat the benchmarks.
Its true money changes everything.
Re:Its very hard to get a good benchmark (Score:1)
Actually, this reminds me of the "teaching to the test" controversy. This should lead to good results when the test is carefully designed, but may lead to
best scientific benchmark is Hint (Score:1)
Hint is designed to fulfill the following goals.
As most other ben
benchmarking gripes (Score:4, Insightful)
My issue with benchmarking is this... When people read benchmarks, aside from the bias occurring with someone using a favored product, people will often have to take benchmarking as nothing more than an indicator for the following reasons: People will not have access to all the equipment used in a benchmark trial, hardware/software, so they're often going to have to rely on someone else's OBSERVATION. Information can be tweaked easily, and someone who has say a favored product can often tweak it to perform better than the competition, or make the competition's product behave worse.
Also as stated on an above post, who is sponsoring the benchmark testing, and why. Often you will see that %99.99999 of the companies sponsoring benchmarking tests come out with gleaming reviews. Has anyone here seen an MS sponsored test prove unfavorable to MS. It just doesn't happen. Independent studies should post all information concerning why they're doing benchmark tests including any sponsors, this way those reading the published results can get an overall VIEW of the results and use them as nothing more than in indicator and not solid fact.
The best benchmark is the app you're using (Score:5, Insightful)
When it comes pointless is when a single simple benchmark is taken alone. If that were the case then a machine like a 1GHz G4 would own everything else looking at just RC5-72 benchmarks. 10 million keys/sec? no problem, quicker than any other machine like it on the market.
Look at that as just one benchmark among dozens and you form a better picture, that the G4 has a vector unit that performs exceptionally well, and you can get an idea how the rest of it performs.
Add up enough of those simple numeric benchmarks and all you get is one huge mess in mind with no REAL idea of how a machine will perform other than theoretically. Best combine them all together and go back to running the app(s) you're likely to use most.
Re:The best benchmark is the app you're using (Score:1, Troll)
Wouldn't it be better to then analyse what is making it run worse on the machine which should be capable of X+1 GB/sec and then improve the software to suit.
Just going with the best working solution
Re:The best benchmark is the app you're using (Score:1)
What you mean to say is that you haven't tried it yourself and just wish to parrot what the world says about it.
A little more maturity and you would see it is an ideal tool.
Re:The best benchmark is the app you're using (Score:1)
Last time I tried it, it didn't. And without CMYK it's completely useless for prepress.
Re:The best benchmark is the app you're using (Score:2)
I play games (Score:2)
The answer is no. (Score:3, Interesting)
Here's the problem: I don't really want to put together any more systems - at least, not from scratch. My time is worth more now, and the savings from DIY are worth less.
But neither do I want to buy (or recommend) a system that's a stiff: one that uses an unreliable motherboard, or an older chipset, or flakey power supplies.
The site I need would:
- take the systems sold by everybody from Wal-mart, Dell, HP, etc
- find out what components they contain
- then use the review data from places like Tom's Hardware
- Pass judgement and explain why
For example, they might say something like "The Dell Excavator uses an obsolete chipset. For $10 more, buy the E-Machine X321 - but beware the reliability."
I've owned so many video cards (Score:5, Interesting)
The bottom line is that you really can't put much trust in benchmarks. Well... Thats not exactly true, but think - of those games and apps that you always see the same people run over and over again, how many of those do you use on a daily basis? Personally, i've read so many reviews that I don't even have to think about what a pixel shader is anymore, so it probably will come as no suprize that I skip through the mumbo jumbo they tell you about the card and go straight to the benchies. And its always the same ones.
Thats all well and good, and I guess it gives you a VERY generic view at how those particular things work, but how about real life performance? How about a screenshot in the HL mod Natural Selection when there are 15 turrets firing at bile bombing aliens with the show_fps set to 1? Can we get something like that? I guess that would consitute in there with fill rate, and before you tell me thats an arcane game. Let me direct you to the little X on the top right of your browser. I don't care.
You can get a very good idea about the speed of a card, but you have no idea what the card will have trouble with until you load up your copy of Star Wars : Pod Racer just to be greeted by a big white screen when the race starts. Thats one thing I really miss about 3dfx. Thier cards worked. Always. Well, at least they did at the time.
I used to do this... (Score:5, Insightful)
The problem is that every workload will have a different I/O and instruction mix. Each instruction has a different execution time, and the performance of I/O devices is frequently a function of the access patterns to data.
As a result, a synthetic benchamrk may be a poor indicator of the result from the actual execution of your individual workload. These benchmarks are intended to provide guidance, and potentially identify platform performance bottlenecks. That's all. Reading any more into them is the fault of those that use the results improperly.
Hardware compared to itself? (Score:5, Funny)
Re:Hardware compared to itself? (Score:1)
No big deal, but since everybody is concerned with numbers, I just
Re:Hardware compared to itself? (Score:1)
Re:Hardware compared to itself? (Score:1)
Seriously... this is not helpful at all.
Re:Hardware compared to itself? (Score:2)
Re:Hardware compared to itself? (Score:1)
You're scaring the children.
Re:Hardware compared to itself? (Score:2)
Re:Hardware compared to itself? (Score:2)
You fascists are all alike.
Re:Hardware compared to itself? (Score:2)
Re:Hardware compared to itself? (Score:2)
Frames per second benchmark idiocy (Score:5, Insightful)
What matters is how much stuff you can draw per frame time, not how many times you can redraw it during a single frame time. 3D benchmarks should gradually increase the scene complexity until the frame rate drops. Often, there's a huge performance drop when the onboard memory of the graphics board fills up. Running old games at huge frame rates won't show that.
Scene complexity is the limiting factor for game developers. Artists are always saying "I need a bigger poly budget". If benchmarks focused on scene complexity, we'd have gigabyte graphics boards, and "wow, you can see every eyelash" scene complexity.
We also need more intensity depth in graphics boards, to clean up that murky look so typical of games. Rendering really should be done into at least 16 bits of intensity, then sent to the screen through a film-like gamma conversion. That's how it's done in offline renderers for film.
Re:Frames per second benchmark idiocy (Score:2)
What interests me... (Score:4, Insightful)
The other bad thing about benchmarks is you will probably not have the same motherboard/ram/cpu as the test system.
Identify the need of person looking at benchmarks (Score:4, Insightful)
I think the writer made very relevant points. Generally the universe expands faster than our own speed, and so we never get to the edge of the universe. But in the world of technology, sometimes technology grows faster than the real world it is supposed to inhabit, and so the real world gets left behind ... The writer has identified just such a crossover, and hence his call to update benchmarks is very valid ...
But I would like to add another dimension and that is the eye of the beholder ... If all the person is looking at the benchmarks for, is to quickly sell it to his unsophisticated boss, or another unsophisticated boss who will get his employees to use it, then what he needs are simple and clear cut benchmarks - and more important, time tested benchmarks. Generally the powers-to-be with the moolah do not like the messiness that inherently comes with trying to "realify" the models .... From my experience this is not how it should be, but I have found this is how it is ....
I am not saying that the writer was in anyway wrong .. just that he must also look at the consumer of his benchamrks ... is it someone who is going to use the technology him/her self or is someone who is going to sell a technology to someone who will have someone else use the technology ....
Proper Method for Benchmarking (Score:5, Funny)
2. Hold the equipment 3 to 5 feet above the bench surface
3. Release. Gravity will take care of the test
4. Measure the mark left in the bench by the equipment. Bigger mark = better equipment.
Re:Proper Method for Benchmarking (Score:1)
Given the amout of heat modern chips generate and the size of the heatsinks needed to dissipate the heat, this is not far from the truth.
Re:Proper Method for Benchmarking (Score:2)
I'm trying to find a good home stereo amp, but I can't stand the modern stuff (usually identified by being all black with a giant volume knob). I like the vintage hardware, with actual transistors rather than integrated circuits. So the way I shop is (mostly) by weight. If it was manufactured prior to 1980 or thereabouts, I simply go for the heaviest thing I can find. All else being equal (which it never is, of course), the amp with the biggest, baddest set of heat s
imho (Score:3, Insightful)
Video Card Benchmarks (Score:3, Interesting)
Re:Video Card Benchmarks (Score:1)
I've seen a buttload of synthetic benchmarks (all the way from the 80's), and invariably they never predict anything useful. They're actually _less_ meaningful than just running your favourite game or app or whatever.
First of all, computers are too complex to put everything into a single number. Graphics cards too. The exact mix of instructions (e.g., which shaders are used), the exact data set (e.g., overd
Sound Cards (Score:2, Interesting)
what they should compare the most is the sound quality, this i find very lacking in sound card benchmarks... such things as Signal/Noise ratio, frequency response, bass/
The "study" seems to be fairly biased... (Score:1)
They never actually showed how the Kyro II actually "out-performed" the GF2 in real life applications.
I could just say that I don't believe there's people being killed in Iraq, it's all a goverment conspiracy. If I had no proof, I *should* be laughed out of town.
Synthetic benchmarks are not *inheriently* bad. A
Benchmarking is an inexact science (Score:5, Insightful)
1, Choose a test/workload that is representative of what *you* will be doing. There is no point in looking at SPECINT200 if you are going to be running an I/O intensive application like a RDBMS. Try and run or study tests that are relevant to the intended use of the system/component you are benchmarking.
2, Take note of things like compiler flags etc. These are important in tests like SPEC, as your results can vary wildly according to things like optimisation level. Some compilers produce faster code on certain CPU families and not on others. This is a reason why a lot of vendors will build their own compilers and test with them (e.g. SGI, SUN, DECPAQ).
3, Look at the full disclosure notice in the benchmarks. Take a look at the system configuration used. This is particularly, IMHO, on tests like TPC-C. The score you see might be based on a really whacky config, like most of the figures at the top of the list. For example, look at the Proliant figure (709k) and look at the config: 32 x 8 way servers to run a single database. Then compare it to a 64-way SuperDome or 32-way p690. Which comfig makes more sense? For a database, I would likely go with the single system for simplicity's sake. On another application, maybe the cluster would make sense.
4, Compare apples to apples. This is the hardest part, as CPU's, OS's, I/O, Apps. Compilers etc etc all vary across platforms. I like to to try and compare one variable if possible. To take the TPC-C again, I try to compare DB against DB, Cluster against Cluster, SMP against SMP etc etc. There is nothing to be gained, IMHO, from comparing MS-SQL server in a cluster on Xeon with Win2k3 to Sybase on a SF15k running SPARC Solaris. How do you properly compare these two results? Maybe the solution would be to look at SQLServer on one system against another or Sybase vs Oracle on a similar Unix system.
5, YMMV. Benchmarks are only ever an indicator of performance, not a guarantee. I tell my customers this all the time. They represent a result with a particular system, data set, O/S, tuning settings etc etc at a point in time. Other people's results with a similar config might differ considerably.
I could go on forever, but the above are my 2c
Well... (Score:4, Insightful)
He didn't even mention 3DMark2003, which does a more comprehensive job testing modern GPU features and is included on any benchmarks of 'modern' (aka DX9 supported) cards.
Think about it, in 2000 when they were working on the 3dmark 2001, directx8.1 wasn't even done; to my knowledge, most of DX8 wasn't even used in 3dmark 2001se.. Since then, cards came out with tons of new feature sets (directx9, AGP 8X, etc) and there was simply a lag time between good benchmarking software.
Now, I do agree with charting performance over time. This would be much more handy when doing comparisons of AMD and Inel processors. I get the same over-all frame rate with my AMD 2400 as an Intel @ ~2.6gig. But, the Intel w/ a faster bus will likely not be getting those split second ticks where the AMD is 100% occupied or the FSB is flooded.
I'm not knocking AMD at all. I can just tell a difference in the overall smoothness of a CPU intensive game. When I bought mine, I spent about 3/4 of what I would have spent on an Inel rig and got around 3/4 of the performance.
It all works out once you stop paying attention to a marketing department. People always say you can't trust advertising, but act so suprised when a company is exposed for making a false claim of some minor sort.
you get what you pay for.
Also, instead of complaining about poor benchmarks in real-world situations, you should write the various game developers and request they add, or consider adding, a benchmark to their game engines. Having to 'devise' a way to test game performance probably isn't going to result in wide-spread adoption of that particular benchmark. ID Software's engines have always come with built-in benchmarks (timedemo), thus making them very easy to test. That's why you always see the games that use ID's engines in benchmarks.
That brings me to my final point, he mentions that StarWars game should be tested instead of Q3, yet it uses the same engine. Sorry, more copies of Q3 exist, and since any game using that engine doesn't bring anything new to the table, might as well stick with it. eh?
Speedy (Score:3, Insightful)
Various BIOS settings are also able to be changed on the fly, checking all these values whilst the benchmark is running will alter the results of the benchmark, but the difference they can make requires any true benchmarker to monitor them...
Studying Benchmarking? (Score:5, Funny)
There's Lies, there's damn Lies and finally there are benchmarks.
Robert
Better definitions (Score:2, Funny)
Edelstein's First Law of Benchmarks: Every commercial product has its best performance on standard benchmarks.
Edelstein's Corollary: If the system you wanted to win didn't, the benchmark wasn't fair.
"in relation to itself"??? (Score:2, Funny)
That should prove to be an interesting technical comparison.
"We were surprised when Hardware A managed to score a 975 on the TurboMaxQuad Doohickey test, but we were shocked when Hardware A blasted out of the gates and scored a whopping 975 on the TurboMaxQuad Doohickey test..."
Good Benchmarks (Score:4, Insightful)