Slashdot Log In
Not All Cores Are Created Equal
Posted by
kdawson
on Mon Dec 22, 2008 08:56 PM
from the working-out-the-kinks dept.
from the working-out-the-kinks dept.
joabj writes "Virginia Tech researchers have found that the performance of programs running on multicore processors can vary from server to server, and even from core to core. Factors such as which core handles interrupts, or which cache holds the needed data can change from run to run. Such resources tend to be allocated arbitrarily now. As a result, program execution times can vary up to 10 percent. The good news is that the VT researchers are working on a library that will recognize inefficient behavior and rearrange things in a more timely fashion." Here is the paper, Asymmetric Interactions in Symmetric Multicore Systems: Analysis, Enhancements and Evaluation (PDF).
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
unsurprising. (Score:5, Interesting)
Anyone who thinks computers are predictably deterministic hasn't used a computer. There are so many bugs in hardware and software that cause it to behave differently than expected, documented, designed. Add to that inevitable manufacturing defects, no matter how microscopic, and it's unimaginable to find otherwise.
It's like discovering "no two toasters toast the same. Researches found some toasters browned toast up to 10% faster than others."
Re:unsurprising. (Score:5, Funny)
Parent
Re:unsurprising. (Score:5, Funny)
Parent
Re:unsurprising. (Score:5, Funny)
I had a Pentium that DEFINITELY went to 11.
Parent
Re:unsurprising. (Score:5, Funny)
Parent
Re:unsurprising. (Score:5, Funny)
I am sure you mean to say; Wow, a joke from 1994.995994999.
Parent
Re: (Score:3, Funny)
That joke is so badly done it's not even funny.
1994.995994999
If you look carefully at this number, it's clearly one constructed by a human. The first '5' might be random, but the proceeding numbers do not have any specific reason to be weighted towards higher digits!!!
Thus, a more realistic semi-random number would be:
1994.995974983
Re: (Score:3, Funny)
Actually, your number looks more like random number string by a human as human try to avoid using long chains of same numbers when writing random numbers. But you are right, my number was made by randomly punching multiple number keys on my keyboard and those happened to register. I did then edit it so that the first digit after to dot was 9.
Re: (Score:2)
Those things were HAWT!
Re: (Score:3, Insightful)
Re:unsurprising. (Score:5, Interesting)
Parent
Re:unsurprising. (Score:5, Funny)
Oh. So that's what's been doing it.
Parent
Re: (Score:3, Interesting)
There is a very interesting channel on youtube called googletechtalks. There, you can find a lecture called "We have it easy, but do we have it right" about performance measures that really made me worry. Basically you can't just easilly compare performance by measuring the cpu time, because there are a lot of factors that determine performance. E.g.: by adding a environment variable before running a program, this can cause page allignments to change (even if the environment variable isn't used by the progr
Re: (Score:3, Interesting)
Re: (Score:3, Interesting)
Re: (Score:3, Informative)
Processor affinity is still a nasty corner of OS design. It was one of the outstanding issues with the BeOS kernel that was not resolved before the company tanked.
Re: (Score:3, Interesting)
Processor affinity is even harder on modern CPUs. You often have 2 or so contexts sharing execution units and L1 cache in a core, then a few cores sharing L2 cache in a chip. Deciding whether to move a process is tricky. There's a penalty for moving, because you increase the cache misses proportionally to the distance you move it (if you move it to a context that shares the same L1 cache, it's not as bad as if you move it to one that shares only the L2 cache, for example), but there's also a cost for not
Re: (Score:2)
Actually, the PC was designed to be non-deterministic. No software bugs, hardware bugs or manufacturing defects needed.
On the other hand, many early home computers were quite deterministic. In fact the Atari 2600 game machine was deterministic down to a single CPU cycle. Many 2600 games would not have worked if it were otherwise.
Re:unsurprising. (Score:5, Interesting)
Actually, (sorry no link) there was a researcher that was using FPGAs and AI code to create simple circuits, but the goals was to have the AI design it. What he found is that due to minor manufacturing defects, the code that was built by AI was dependent on the FPGA it was tested on and would not work on just any FPGA of that specification. After 600 iterations, you'd think it would be good. One experiment went for a long time, and in the end when he analyzed the AI generated code, there were 5 paths/circuits inside that did nothing. If he disabled any or all of the 5 the overall design failed. Somehow, the AI found that creating these do nothing loops/circuits caused a favorable behavior in other parts of the FPGA that made for overall success. Naturally that code would not work on any other FPGA of the specified type. It was an interesting read, sorry that I don't have a link.
Parent
Re:unsurprising. (Score:4, Informative)
A simple Google search for "fpga genetic algorithm" shows up references quite quickly - e.g.
http://biology.kenyon.edu/slonc/bio3/AI/GEN_ALGO/gen_algo.html [kenyon.edu]
The only part of the GP story I haven't seen before (and can't find a reference for) is the bit about the design not working on other FPGAs of the same specification. The closest story is that of Adrian Thompson at the University of Sussex who got a circuit with unconnected elements which nonetheless seem to be needed in order for the whole thing to achieve its goal. Nothing about the design only working on specific instances of the FPGA.
Parent
Re:unsurprising. (Score:4, Insightful)
They probably put in the if(1) lines because they were testing various aspects of the program, or maybe some like to turn off various aspects of the program, but don't want to be arsed to write the proper code to select options. I commonly do that in POVray (3d raytracing) scripts when testing, so I don't have to wait for long renders--fog, radiosity, lots of light and such take orders of magnitude more time.
As for the AI adding crap, it is probably more trying random code than truly thinking about how the code should work. This leads to the useful code intertwined with lots of crap code. Unfortunately, there are programmers who write like this too... (cue funny mod)
As for the code not working on other FPGAs, maybe the researcher should not use real chips to check the iterations. A simulated one which conforms to the spec exactly and upon where quirks and such are expected, dies or sends a signal back to the AI program. Testing after the fact on real chips to verify the AI didn't exploit bugs in the simulator would be more proper procedure.
Maybe I have too much of a background in theory, but I am not completely sure why the FPGAs would be so different. Is it race time conditions? Or is the FPGA being used in some analog way? Or does the circuit depend on the exact timing of some input, so the speed / capacitance of each component make a huge difference? Or was the poster talking about FPGAs with different specs?
Crazy things happen when you enter the real world. I remember back when I was in electronics assembly. One would first assume all the solder would wick onto the metal, but the boards would always have tonnes of solder bridges, and we had to carefully examine every component and correct them. Friggin' microprocessors had countless tiny legs too!
Parent
Reminds me of an issue... (Score:3, Interesting)
...I had with an Asterisk VOIP server. Under certain conditions, calls transferred from one of two receptionist's phones were bouncing back and ending up at the wrong voicemail. Since only two phones had a problem I suspected it was something specific to these phones. After checking the configuration and even hardware on the phones, I checked the server. I narrowed the problem down to one macro (a macro in asterisk is basically a user-defined function) that allows a "fallback line" to ring if the first is b
Re: (Score:3, Funny)
who would've guessed... (Score:4, Insightful)
Re:who would've guessed... (Score:5, Insightful)
Parent
multicore dev is fun... much like prison rape! (Score:4, Interesting)
Re: (Score:2)
Could you point me at some direction for more information about the problems of developing for the N64? I knew developers didn't liked the Sega Saturn or whatever it was which had multiple cores but I don't remember reading anything about N64.
Re: (Score:2, Informative)
http://en.wikipedia.org/wiki/N64#Programming_difficulties [wikipedia.org]
The amount of video memory for textures was way too small.
Re:multicore dev is fun... much like prison rape! (Score:5, Interesting)
Parent
Re:Yup (Score:4, Informative)
How about a "parallel foreach(Thing in Things)" ?
That is easy. If your application can be parallelized that easily, then it is considered embarrassingly parallel. OpenMP exists today and does just this. All you have to do (in C) is add a "#pragma" above the for loop and you have a parallel program. OpenMP is commonly available on all major platforms.
The real problem is that most desktop applications just don't lend themselves to this type of parallelism and so the threads have lots of data sharing. This data sharing causes the problem because the programmer must carefully use synchronization primitives to prevent race conditions. Since the programmer is using parallelism to boost performance, they only want to introduce synchronization when they absolutely have to. When in doubt, they leave it out. Since it is damn near impossible to test the code for race conditions, they have no indication when they have subtle errors. This is what makes concurrent programming so difficult. One researcher says that using threads makes programs "wildly nondeterministic".
It is hard to blame the programmers for being aggressive in seeking performance gains because Amdahl's Law [wikipedia.org] is a real killer. If you have 90% of the program parallelized, the theoretical maximum performance gain is 10X no matter how many cores you can throw at the problem.
Parent
Close (Score:3, Interesting)
But you have to think about it too much.
How about:
Things.ParallelEach(function(thing){
Console.Write("{0} is cool, but in parallel", thing);
# serious business goes here
});
There are lots of stupid loop structures that are used in desktop apps that are just begging to be run in parallel, but the current crop of languages dont make it braindead easy to do so. Make it so every loop structure has a trivial and non ugly (OpenMP pragmas) way of doing it.
Also, IMHO, not enough languages do stuff lik
Linux and Windows (Score:4, Insightful)
I don't know if Linux or Windows has an automatic mechanism to schedule task priority based on processor caches, but the study didn't even mention Windows. Seeing that the scheduling and managing the caches are OS problems this seems kind of important.
The other thing that seems odd is they were using a 2.6.18 Kernel and in 2.6.23 they added the Completely Fair Scheduler which could potentially change their results. It doesn't seem logical to base a cutting edge study on stuff that was released years ago.
Re:Linux and Windows (Score:4, Informative)
They mentioned this in an ESX class I took. I seem to remember it in the context of setting a processor affinity or creating multi-CPU VMs and how either the hypervisor was smarter than you (eg, don't affinity) or that multi-CPU VMs could actually slow other VMs because the hypervisor would try to keep multi-CPU VMs on the same socket, thus deny execution priority to other VMs (eg, don't assign SMP VMs because you can unless you have the CPU workload).
Parent
Linux schedules better than this (Score:4, Informative)
Re: (Score:3, Interesting)
Helpful reading list :)
http://www.google.com/search?q=linux+%22scheduler+domain%22+%22multi+core%22 [google.com]
Re:Linux schedules better than this (Score:5, Interesting)
Possibly... but it appears an SMP kernel treats each core as a separate physical processor.
Take an Intel Core2 Quad machine and start a process that takes 100% of one CPU. Then watch top/htop/gnome-system-monitor/etc where you can watch the process hop around all four cores. It makes sense that the process might hop between two cores -- the two that share L2 cache -- but all four cores doesn't make sense to me. Seems like the L2 cache is wasted when migrating between each core2 package.
Parent
Re:Linux schedules better than this (Score:4, Interesting)
Parent
Re: (Score:3, Informative)
I thought that Intel specifically did that, that if one core were loaded it would overclock that core and downclock the others to get a speed boost...
Yup, I thought I remembered correctly [tomshardware.com].
NUMA NUMA (Score:4, Informative)
Linux can already deal with scheduling tasks to processors where the necessary resources are "close". It may not be obvious to the likes of PC Magazine, but its trivially obvious that even multithreaded programs running on a non-location aware kernel are going to take a hit. This is a kernel problem, not an application library problem.
This isn't news (Score:5, Informative)
Anyone who has been doing performance work should have known this. The tools to adjust things like core affinity and where interrupts are handled have been available in Linux and Windows for a long time. These effects were present in 1980s mainframes. DUH.
Re:This isn't news (Score:5, Insightful)
80s mainframe tech is NEW and EXCITING to a depressing number of tech people, look at how excited everyone got when someone remembered and re-implemented virtualization.
Parent
Re: (Score:2)
I don't know if they've been in the default kernel for "a long time", but they are there now.
read: http://www.alexandersandler.net/smp-affinity-and-proper-interrupt-handling-in-linux [alexandersandler.net]
it's the affinity (Score:2, Informative)
not a surprise (Score:5, Insightful)
Here's an exercise: Take 2 brand-new systems with identical configurations and start them at the same time doing some job that takes a few hours and utilizes most of the hardware to some significant degree. Say, compiling some huge piece of code like KDE or OpenOffice. System administrators who do exactly this will tell you that you'll almost never see the two machines complete the job at precisely the same time. Even though the CPU, memory, hard drive, motherboard, and everything else is the same, the system as a whole is so complex that minute differences in timing somewhere compound into larger ones. Sometimes you can even reboot them and repeat the experiment and the results will have reversed. It shouldn't come as a surprise that adding more complexity (in the form of processor cores) would enhance the effect.
Re:not a surprise (Score:5, Interesting)
We have this problem at work.
We have a render farm of 16 machines. 12 of them are effectively identical but despite all of our coaxing one of them always runs about 30% slower. It's maddening. But "What can you do?". Hardware is the same. We Ghost the systems so the boot data is exactly the same... and yet... slowness. It's just a handicapped system.
Parent
Well known problem (Score:4, Insightful)
The problem is a complex one. Every possible scheduling decision has pluses and minuses. For example, keeping a process on the same core for each timeslice maximizes cache hits, but can lose if it means the process has to wait TOO long for it's next slice. Likewise, if a process must wait for something, should it yield to another process or busy wait. SHould interrupts be balanced over CPUs or should one CPU handle them?
A lot of work has gone in to those questions in the Linux scheduler. For all of that, the scheduler only knows so much about a given app and if it takes TOO long to 'think' about it, it negates the benefits of a better decision.
For special cases where you're quite sure you know more than the scheduler about your app, you can use the isolcpus kernel parameter to reserve CPUS to run only the apps you explicitly assign to them.
You can also decide which CPU any given IRQ can be handled by (but not which core within a CPU as far as I know) wilt /proc/irq/*/smp_affinity.
Unless your system is dedicated to a single application and you understand it quite well, the most likely result of screwing with all of that is overall loss of performance.
This isn't hardware (Score:3, Informative)
Why is this article labeled as hardware? Sure they talk about different procs being ... well, different. Duh! The article is about the software Tom and others developed to run processes more efficiently in a multi-core (an possibly heterogenous) environment. Big energy savings as well as performance boost. Green computing. HELLO! Did you read page two?
Re: (Score:2)
And this is useful info because?
Isn't most of the point of using -j parameter that your machine can carry on compiling something else while whatever it did earlier get the resources it needed from disk or similar? Will it really help out with cache usage?
Should more processes mean better or worse cache performance? Worse because cache is shared between them, better because if something is missing some other instruction can be done while the needed data is fetched from RAM?
Re: (Score:2)
Re: (Score:3, Funny)