Not All Cores Are Created Equal 183
joabj writes "Virginia Tech researchers have found that the performance of programs running on multicore processors can vary from server to server, and even from core to core. Factors such as which core handles interrupts, or which cache holds the needed data can change from run to run. Such resources tend to be allocated arbitrarily now. As a result, program execution times can vary up to 10 percent. The good news is that the VT researchers are working on a library that will recognize inefficient behavior and rearrange things in a more timely fashion." Here is the paper, Asymmetric Interactions in Symmetric Multicore Systems: Analysis, Enhancements and Evaluation (PDF).
unsurprising. (Score:5, Interesting)
Anyone who thinks computers are predictably deterministic hasn't used a computer. There are so many bugs in hardware and software that cause it to behave differently than expected, documented, designed. Add to that inevitable manufacturing defects, no matter how microscopic, and it's unimaginable to find otherwise.
It's like discovering "no two toasters toast the same. Researches found some toasters browned toast up to 10% faster than others."
Re:unsurprising. (Score:5, Funny)
Re:unsurprising. (Score:5, Funny)
Re:unsurprising. (Score:5, Funny)
I had a Pentium that DEFINITELY went to 11.
Re:unsurprising. (Score:5, Funny)
Re: (Score:2, Funny)
Wow, a joke from 1995. It's true, Slashdot is at the forefront of cutting-edge humor.
Re:unsurprising. (Score:5, Funny)
I am sure you mean to say; Wow, a joke from 1994.995994999.
Re: (Score:3, Funny)
That joke is so badly done it's not even funny.
1994.995994999
If you look carefully at this number, it's clearly one constructed by a human. The first '5' might be random, but the proceeding numbers do not have any specific reason to be weighted towards higher digits!!!
Thus, a more realistic semi-random number would be:
1994.995974983
Re: (Score:3, Funny)
Actually, your number looks more like random number string by a human as human try to avoid using long chains of same numbers when writing random numbers. But you are right, my number was made by randomly punching multiple number keys on my keyboard and those happened to register. I did then edit it so that the first digit after to dot was 9.
Re: (Score:2)
Well, yeah. But his LOOKS more random because we have an implicit assumption that randomness will make things different, rather than select the same thing every time. We're hard-wired as humans to recognize patterns ;)
Re: (Score:2)
Yes but the probability of a string of random numbers looking similar to the first is far less then that of a string of numbers looking similar to the second. Just like it may be the same chance to have 9999999 as to have 8675309, one is very consistent and can be seen to have a pattern while the other may not. Unless you're Jenny anyway.
Re: (Score:2)
Those things were HAWT!
Re: (Score:2)
Hell yeah, that one was a bargain.
I had mine clocked at at 400MHz and iirc saved about $200 over an equivalent "real" PII.
Re: (Score:2)
I had an Abit motherboard (VP6) that went to 11. Unfortunately it ended with a little fireworks show. :( Stupid bad caps, lousy Abit QC.
Re: (Score:2, Funny)
The review for "Not All Cores Are Created Equal" was merely a two word review which simply read "Shit Sandwich".
Re: (Score:3, Insightful)
Re:unsurprising. (Score:5, Interesting)
Re:unsurprising. (Score:5, Funny)
Oh. So that's what's been doing it.
Re: (Score:2)
Oh. So that's what's been doing it.
Yeah, Vista says my proc should actually be a vacuum tube.
Re: (Score:2)
Oh, and mod the comment above me up as well - that was just funny.
Re: (Score:2, Insightful)
If overclocking is the cause of so many of these problems, why hasn't Intel or AMD got a mechanism to tell the OS that the hardware's being run out of spec? The blame for these crashes should be directed where it belongs - with the -funroll-loops ricers.
Reporter bias (Score:2, Insightful)
Often, an issue presents that isn't reproducible in the presence of a tech support person who knows what he's doing.
Sometimes it's a user error they don''t want to admit, and so they won't reproduce it in front of somebody who knows they should not have done that.
Sometimes it's just a glitch. Regardless, the best thing to do is smile and say "The bug must be afraid of me" and close the ticket.
Re: (Score:3, Interesting)
There is a very interesting channel on youtube called googletechtalks. There, you can find a lecture called "We have it easy, but do we have it right" about performance measures that really made me worry. Basically you can't just easilly compare performance by measuring the cpu time, because there are a lot of factors that determine performance. E.g.: by adding a environment variable before running a program, this can cause page allignments to change (even if the environment variable isn't used by the progr
Re: (Score:3, Interesting)
Re: (Score:3, Interesting)
Re: (Score:3, Informative)
Processor affinity is still a nasty corner of OS design. It was one of the outstanding issues with the BeOS kernel that was not resolved before the company tanked.
Re: (Score:2)
Re: (Score:3, Interesting)
Processor affinity is even harder on modern CPUs. You often have 2 or so contexts sharing execution units and L1 cache in a core, then a few cores sharing L2 cache in a chip. Deciding whether to move a process is tricky. There's a penalty for moving, because you increase the cache misses proportionally to the distance you move it (if you move it to a context that shares the same L1 cache, it's not as bad as if you move it to one that shares only the L2 cache, for example), but there's also a cost for not
Re: (Score:2)
Actually, the PC was designed to be non-deterministic. No software bugs, hardware bugs or manufacturing defects needed.
On the other hand, many early home computers were quite deterministic. In fact the Atari 2600 game machine was deterministic down to a single CPU cycle. Many 2600 games would not have worked if it were otherwise.
Re: (Score:2)
What we need is a toaster with an IQ of around 4000.
Re:unsurprising. (Score:5, Interesting)
Actually, (sorry no link) there was a researcher that was using FPGAs and AI code to create simple circuits, but the goals was to have the AI design it. What he found is that due to minor manufacturing defects, the code that was built by AI was dependent on the FPGA it was tested on and would not work on just any FPGA of that specification. After 600 iterations, you'd think it would be good. One experiment went for a long time, and in the end when he analyzed the AI generated code, there were 5 paths/circuits inside that did nothing. If he disabled any or all of the 5 the overall design failed. Somehow, the AI found that creating these do nothing loops/circuits caused a favorable behavior in other parts of the FPGA that made for overall success. Naturally that code would not work on any other FPGA of the specified type. It was an interesting read, sorry that I don't have a link.
Re: (Score:2, Insightful)
Damn it, get one!
At least a name for christs sake!
Re:unsurprising. (Score:4, Informative)
A simple Google search for "fpga genetic algorithm" shows up references quite quickly - e.g.
http://biology.kenyon.edu/slonc/bio3/AI/GEN_ALGO/gen_algo.html [kenyon.edu]
The only part of the GP story I haven't seen before (and can't find a reference for) is the bit about the design not working on other FPGAs of the same specification. The closest story is that of Adrian Thompson at the University of Sussex who got a circuit with unconnected elements which nonetheless seem to be needed in order for the whole thing to achieve its goal. Nothing about the design only working on specific instances of the FPGA.
Re: (Score:2)
This article [susx.ac.uk] mentions how it won't work on only the FPGA it was developed on.
Re: (Score:2)
Sounds like John Koza ( http://en.wikipedia.org/wiki/John_Koza [wikipedia.org] ) or someone following his research
FPGA programming (Score:2)
One experiment went for a long time, and in the end when he analyzed the AI generated code, there were 5 paths/circuits inside that did nothing. If he disabled any or all of the 5 the overall design failed. Somehow, the AI found that creating these do nothing loops/circuits caused a favorable behavior in other parts of the FPGA that made for overall success.
The author took the unusual step of disconnecting the clock for the FPGA, taking advantaged of undefined behavior that depended on the unique electrical characteristics of the FPGA he used. Had he left the clock connected he'd likely have more portable results, however he may not have arrived at the same results since he'd be depending on discrete logic and not the unspecified, non-linear analog behavior.
Re: (Score:2)
That's correct. My mind was fuzzy last night. Rereading it makes it very appropriate to this story though as it points out the minute variations in silicon/computers that is ignored by most software etc. as used today because of clocks etc. If the clock is not quite right, weird things can happen. Skynet was a clock failure?
Re: (Score:2)
Re:unsurprising. (Score:4, Insightful)
They probably put in the if(1) lines because they were testing various aspects of the program, or maybe some like to turn off various aspects of the program, but don't want to be arsed to write the proper code to select options. I commonly do that in POVray (3d raytracing) scripts when testing, so I don't have to wait for long renders--fog, radiosity, lots of light and such take orders of magnitude more time.
As for the AI adding crap, it is probably more trying random code than truly thinking about how the code should work. This leads to the useful code intertwined with lots of crap code. Unfortunately, there are programmers who write like this too... (cue funny mod)
As for the code not working on other FPGAs, maybe the researcher should not use real chips to check the iterations. A simulated one which conforms to the spec exactly and upon where quirks and such are expected, dies or sends a signal back to the AI program. Testing after the fact on real chips to verify the AI didn't exploit bugs in the simulator would be more proper procedure.
Maybe I have too much of a background in theory, but I am not completely sure why the FPGAs would be so different. Is it race time conditions? Or is the FPGA being used in some analog way? Or does the circuit depend on the exact timing of some input, so the speed / capacitance of each component make a huge difference? Or was the poster talking about FPGAs with different specs?
Crazy things happen when you enter the real world. I remember back when I was in electronics assembly. One would first assume all the solder would wick onto the metal, but the boards would always have tonnes of solder bridges, and we had to carefully examine every component and correct them. Friggin' microprocessors had countless tiny legs too!
Re: (Score:2)
Its for debugging.The code works on release versions(if (1)),but for debugging people need the ability to turn on/off certain parts of code(if (!1)).
Re: (Score:2)
That's entirely incorrect.
Computers are predictably deterministic -- the problem is that the number of variables used is neither known nor accounted for.
Most code is crap, because most code isn't important. The stuff that is important is written to specific acceptable levels of error. The problem is when you get alphabet-soup diploma holders getting a little experience at a random startup then going off to write vital code. Then you get problems because you continue bad practices. The venerable K&R C bi
Re: (Score:2)
Reminds me of an issue... (Score:3, Interesting)
...I had with an Asterisk VOIP server. Under certain conditions, calls transferred from one of two receptionist's phones were bouncing back and ending up at the wrong voicemail. Since only two phones had a problem I suspected it was something specific to these phones. After checking the configuration and even hardware on the phones, I checked the server. I narrowed the problem down to one macro (a macro in asterisk is basically a user-defined function) that allows a "fallback line" to ring if the first is b
Re: (Score:3, Funny)
who would've guessed... (Score:4, Insightful)
Re:who would've guessed... (Score:5, Insightful)
Re: (Score:2)
Re: (Score:2)
Actually it is the job of the programmer to make sure his program is cache friendly, that should work on all architectures.
Also you should in a multi-core/-CPU environment make sure data needed is close to where you are, that means fetching it from whatever storage it is in (ram, hdd, other core) as early as possible and non-blocking if possible so you can complete other tasks while waiting.
While the OS can help you with some tasks, there is no way for the OS to know what data you need next, so if you want
multicore dev is fun... much like prison rape! (Score:4, Interesting)
Re: (Score:2)
Could you point me at some direction for more information about the problems of developing for the N64? I knew developers didn't liked the Sega Saturn or whatever it was which had multiple cores but I don't remember reading anything about N64.
Re: (Score:2, Informative)
http://en.wikipedia.org/wiki/N64#Programming_difficulties [wikipedia.org]
The amount of video memory for textures was way too small.
Re:multicore dev is fun... much like prison rape! (Score:5, Interesting)
Yup (Score:2)
The libraries and the languages currently make threading harder then it needs to be.
How about a "parallel foreach(Thing in Things)" ?
I realize there are locking issues and race conditions, but really I think the languages could go a some ways to making things like this more hidden. Oh wait, does that mean I'm advocating for making programming languages more user friendly? I guess so. You know why people use Ruby, C# or Java? Cause those are way more user friendly than C++ or COBOL.
The usability of a pro
Re:Yup (Score:4, Informative)
How about a "parallel foreach(Thing in Things)" ?
That is easy. If your application can be parallelized that easily, then it is considered embarrassingly parallel. OpenMP exists today and does just this. All you have to do (in C) is add a "#pragma" above the for loop and you have a parallel program. OpenMP is commonly available on all major platforms.
The real problem is that most desktop applications just don't lend themselves to this type of parallelism and so the threads have lots of data sharing. This data sharing causes the problem because the programmer must carefully use synchronization primitives to prevent race conditions. Since the programmer is using parallelism to boost performance, they only want to introduce synchronization when they absolutely have to. When in doubt, they leave it out. Since it is damn near impossible to test the code for race conditions, they have no indication when they have subtle errors. This is what makes concurrent programming so difficult. One researcher says that using threads makes programs "wildly nondeterministic".
It is hard to blame the programmers for being aggressive in seeking performance gains because Amdahl's Law [wikipedia.org] is a real killer. If you have 90% of the program parallelized, the theoretical maximum performance gain is 10X no matter how many cores you can throw at the problem.
But that is ugly (Score:2)
And OpenMP isn't "standard" as far as I'm concerned. Plus it makes you think about threading and it only works in low-level languages like C.
I'm talking about this highly useful code (which is written in a bastardized version of C#, Perl and Javascript for your reading pleasure):
List pimpScores = PimpList.ThreadedMap(function(aPimp){
# score how worthy this guy is at pimpin'
if(aPimp.Hoes > 10) {
return String.Format("Damn brot
Re: (Score:2)
what you're asking for is pretty much already that easy
foreach(Thing in Things)
new Thread(Thing.DoStuff);
Close (Score:3, Interesting)
But you have to think about it too much.
How about:
Things.ParallelEach(function(thing){
Console.Write("{0} is cool, but in parallel", thing);
# serious business goes here
});
There are lots of stupid loop structures that are used in desktop apps that are just begging to be run in parallel, but the current crop of languages dont make it braindead easy to do so. Make it so every loop structure has a trivial and non ugly (OpenMP pragmas) way of doing it.
Also, IMHO, not enough languages do stuff lik
Re: (Score:2)
Only less ugly :-) (Score:2)
And for those who say "what what about all the weird race conditions and stuff". I'm not a computer science major, so I'm jumping off an edge asking this, but what if we actually use some of this new CPU power in our IDEs and our JIT compilers, couldn't our languages watch out for most of the nasty ways we can shoot ourselves in the food? Like if I do a Array.ThreadedEach(function(element){}) and I'm changing some shared data, couldn't the compiler or IDE let me know at compile time or while I'm writing t
Re: (Score:2)
There's something called Turing completeness that blows "solve it with smarter compilers" idea out of the water in the general sense (even though it might work 95 percent of the time).
Threaded stuff isn't super hard. Getting threaded stuff to run FAST is hard. There's a billion tradeoffs handled by what are traditionally different parts of the system. In your silly parallelize loops idea (aka MapReduce) the challenge is clear (How many items do you need before setting up parallelization is worth the extra c
Re: (Score:2, Offtopic)
The N64 was killed?
Best "party game" system of that generation, easily.
4 controller capability out of the box, 007 Goldeneye, Perfect Dark, Mario Kart, all the good wrestling games (hey, they were fun at the time...) etc.
The PS1 was only good for racing games and RPGs, IMO. Oh, and Bushido Blade 1 and 2.
Kind of like the Wii vs. 360/PS3. Any time we plug in a PS3 at a get-together, it's to ooh and ah over the graphics and maybe take turns playing the single player mode of a cool game (Need for Speed or som
Linux and Windows (Score:4, Insightful)
I don't know if Linux or Windows has an automatic mechanism to schedule task priority based on processor caches, but the study didn't even mention Windows. Seeing that the scheduling and managing the caches are OS problems this seems kind of important.
The other thing that seems odd is they were using a 2.6.18 Kernel and in 2.6.23 they added the Completely Fair Scheduler which could potentially change their results. It doesn't seem logical to base a cutting edge study on stuff that was released years ago.
Re: (Score:2)
In a nutshell, we need more portable multicore sol
Re: (Score:2)
This is just an engineering trade off.
Re: (Score:2)
I don't know if Linux or Windows has an automatic mechanism to schedule task priority based on processor caches, but the study didn't even mention Windows. Seeing that the scheduling and managing the caches are OS problems this seems kind of important.
I'm not sure why this article isn't tagged "duh".
It's pretty obvious from looking at the CPU graphs of my VMware ESX servers that their code does some optimization to keep processes on the same core, or at the very least on the same CPU.
This data is from a dual-socket quad-core AMD (8 total cores), which means a NUMA [wikipedia.org] architecture, so running the code on the same CPU means you have faster memory access.
So, some commercial code that has been around for nearly 4 years takes advantage of the "discoveries" in an
Re:Linux and Windows (Score:4, Informative)
They mentioned this in an ESX class I took. I seem to remember it in the context of setting a processor affinity or creating multi-CPU VMs and how either the hypervisor was smarter than you (eg, don't affinity) or that multi-CPU VMs could actually slow other VMs because the hypervisor would try to keep multi-CPU VMs on the same socket, thus deny execution priority to other VMs (eg, don't assign SMP VMs because you can unless you have the CPU workload).
Linux schedules better than this (Score:4, Informative)
Re: (Score:3, Interesting)
Helpful reading list :)
http://www.google.com/search?q=linux+%22scheduler+domain%22+%22multi+core%22 [google.com]
Re:Linux schedules better than this (Score:5, Interesting)
Possibly... but it appears an SMP kernel treats each core as a separate physical processor.
Take an Intel Core2 Quad machine and start a process that takes 100% of one CPU. Then watch top/htop/gnome-system-monitor/etc where you can watch the process hop around all four cores. It makes sense that the process might hop between two cores -- the two that share L2 cache -- but all four cores doesn't make sense to me. Seems like the L2 cache is wasted when migrating between each core2 package.
Re:Linux schedules better than this (Score:4, Interesting)
Re: (Score:3, Informative)
I thought that Intel specifically did that, that if one core were loaded it would overclock that core and downclock the others to get a speed boost...
Yup, I thought I remembered correctly [tomshardware.com].
Re: (Score:2)
You have a point when it comes to cache locality. It can be somewhat mitigated by smart timing of the core switching. For example long time on each core (as you probably would notice with your system monitor), or doing something like switching on e
Re: (Score:2)
How do you have your SMP configured, and do you have NUMA and is it enabled?
I'm running a now older 2xx series dual Opteron 290, so dual sockets, dual-cores each, physically configured with four gig memory hanging off each one. The AMD 8xxx chipset has the rest of the system (all the PCI-X channels and AGP, it's pre-PCI-E) hanging off socket-0. In the kernel, I have SMP set, SMT (multi-thread, this would be closer affinity than multi-core but of course the AMDs don't use it) unset, SCHED_MC (multi-core, l
Re: (Score:2)
``And Motorola is probably still at 500MHz.''
Actually, they gave up on the desktop CPU market. They spun off their chip division into Freescale Semiconductor [freescale.com], which now makes embedded processors.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
void PAUSE(){ printf("\nPress any key to continue. .
Whats even worse is that this line of code was used in a fake cmd.exe [anthonycargile.info] I made for a prank on my friend's computer. Tricky to install due to having to point the COMSPEC env. variable to a backed up version of the real cmd.exe and tinkering with the dllcache directory, but it was priceless to see his reaction to the fa
Re: (Score:2)
Re: (Score:2)
As if simply giving each process affinity for a given core solves the problem. But then you have interrupt handling, job loads with more than one process per core, multi-threaded programs - all sharing memory space yet with different memory access patterns - and different processors with e.g. different cache architectures. The task-switching OS is 50 years old and we still haven't settle
NUMA NUMA (Score:4, Informative)
Linux can already deal with scheduling tasks to processors where the necessary resources are "close". It may not be obvious to the likes of PC Magazine, but its trivially obvious that even multithreaded programs running on a non-location aware kernel are going to take a hit. This is a kernel problem, not an application library problem.
This isn't news (Score:5, Informative)
Anyone who has been doing performance work should have known this. The tools to adjust things like core affinity and where interrupts are handled have been available in Linux and Windows for a long time. These effects were present in 1980s mainframes. DUH.
Re:This isn't news (Score:5, Insightful)
80s mainframe tech is NEW and EXCITING to a depressing number of tech people, look at how excited everyone got when someone remembered and re-implemented virtualization.
Re: (Score:2)
I don't know if they've been in the default kernel for "a long time", but they are there now.
read: http://www.alexandersandler.net/smp-affinity-and-proper-interrupt-handling-in-linux [alexandersandler.net]
Re: (Score:2)
Meh. :/
That doesn't excuse the *rest* of the entire industry forgetting *everything* the mainframe folks learned.
it's the affinity (Score:2, Informative)
not a surprise (Score:5, Insightful)
Here's an exercise: Take 2 brand-new systems with identical configurations and start them at the same time doing some job that takes a few hours and utilizes most of the hardware to some significant degree. Say, compiling some huge piece of code like KDE or OpenOffice. System administrators who do exactly this will tell you that you'll almost never see the two machines complete the job at precisely the same time. Even though the CPU, memory, hard drive, motherboard, and everything else is the same, the system as a whole is so complex that minute differences in timing somewhere compound into larger ones. Sometimes you can even reboot them and repeat the experiment and the results will have reversed. It shouldn't come as a surprise that adding more complexity (in the form of processor cores) would enhance the effect.
Re:not a surprise (Score:5, Interesting)
We have this problem at work.
We have a render farm of 16 machines. 12 of them are effectively identical but despite all of our coaxing one of them always runs about 30% slower. It's maddening. But "What can you do?". Hardware is the same. We Ghost the systems so the boot data is exactly the same... and yet... slowness. It's just a handicapped system.
Re: (Score:2)
Move processors around so you get a different boot proc, if you haven't tried that already.
Re: (Score:2)
Some people put together servers all day that way: swapping a bunch of intermittent crap in and out until the box runs long enough to install the OS
Re: (Score:2)
Re: (Score:2)
Check your power supply... that's almost always been the cause of any "weird" errors I've gotten. Jitter in power causes all kinds of fun, unpredictable stuff to happen.
Well known problem (Score:4, Insightful)
The problem is a complex one. Every possible scheduling decision has pluses and minuses. For example, keeping a process on the same core for each timeslice maximizes cache hits, but can lose if it means the process has to wait TOO long for it's next slice. Likewise, if a process must wait for something, should it yield to another process or busy wait. SHould interrupts be balanced over CPUs or should one CPU handle them?
A lot of work has gone in to those questions in the Linux scheduler. For all of that, the scheduler only knows so much about a given app and if it takes TOO long to 'think' about it, it negates the benefits of a better decision.
For special cases where you're quite sure you know more than the scheduler about your app, you can use the isolcpus kernel parameter to reserve CPUS to run only the apps you explicitly assign to them.
You can also decide which CPU any given IRQ can be handled by (but not which core within a CPU as far as I know) wilt /proc/irq/*/smp_affinity.
Unless your system is dedicated to a single application and you understand it quite well, the most likely result of screwing with all of that is overall loss of performance.
What if... (Score:2)
Re: (Score:2)
"You can also decide which CPU any given IRQ can be handled by (but not which core within a CPU as far as I know)"
With the usage of IOAPIC you can redirect the IRQ to any cores. We have a in-house-developed commercial OS for telephony applications and we use the IOAPIC with a simple round-robin fashion. I do not know why linux does not do this.
This isn't hardware (Score:3, Informative)
Why is this article labeled as hardware? Sure they talk about different procs being ... well, different. Duh! The article is about the software Tom and others developed to run processes more efficiently in a multi-core (an possibly heterogenous) environment. Big energy savings as well as performance boost. Green computing. HELLO! Did you read page two?
Re: (Score:2)
Did you read page two?
This is Slashdot
Did anyone else notice. . . (Score:2)
Re: (Score:2)
And this is useful info because?
Isn't most of the point of using -j parameter that your machine can carry on compiling something else while whatever it did earlier get the resources it needed from disk or similar? Will it really help out with cache usage?
Should more processes mean better or worse cache performance? Worse because cache is shared between them, better because if something is missing some other instruction can be done while the needed data is fetched from RAM?
Re: (Score:2)
Re: (Score:3, Funny)