Linux May Need a Rewrite Beyond 48 Cores 462
An anonymous reader writes "There is interesting new research coming out of MIT which suggests current operating systems are struggling with the addition of more cores to the CPU. It appears that the problem, which affects the available memory in a chip when multiple cores are working on the same chunks of data, is getting worse and may be hitting a peak somewhere in the neighborhood of 48 cores, when entirely new operating systems will be needed, the report says. Luckily, we aren't anywhere near 48 cores and there is some time left to come up with a new Linux (Windows?)."
Comment removed (Score:5, Interesting)
Re:Original Source and Actual Paper (Score:5, Interesting)
Re:Original Source and Actual Paper (Score:5, Interesting)
Yes it is lacking: it's too long for a /. "story". Editors want small, easily digested soundbites, not articles with actual information.
Only Linux? (Score:4, Interesting)
64 cores (Score:3, Interesting)
At my last job we had a bunch of Sun T5120s which housed 64 cores. So yeah, we are "anywhere near 48".
Re:Original Source and Actual Paper (Score:4, Interesting)
Scare piece.
Your submission wasn't scaring enough. From your submission, it seems that it's not that big of a deal and rather easy solution. This submission makes it sound like linux kernel needs a complete rewrite ground-up, as in starting from scratch.
Plus yours was a bit long and lots of details.
seeing as Linux does 10240 cores already, WTF? (Score:5, Interesting)
No kidding. SGI's Altix is a huge box full of multi-core IA-64 processors. 512 to 2048 cores is more normal, but they were reaching 10240 last I checked. This is SMP (NUMA of course), not a cluster. I won't say things work just lovely at that level, but it does run.
48 cores is nothing.
Obligatory xkcd reference (Score:2, Interesting)
My Ubuntu 10.04 system still can't play embedded youtube videos. At least Adobe provided a work-around by adding a "play on youtube" option in the right click context menu.
Re:Original Source and Actual Paper (Score:4, Interesting)
And it's worth noting that the most common application for that kind of machine is to partition it and run several different operating systems on it. Solaris has already had some major redesign work for scaling that well. For example, the networking stack is partitioned both horizontally and vertically. Separate connections are independent except at the very bottom of the stack (and sometimes even then, if they go via different NICs), and each layer in the stack communicates with the ones above it via message passing and runs in a separate thread.
However, it sounds like this paper is focussing on a very specific issue: process accounting. To fairly schedule processes, you need to work out how much time they have spent running already, relative to others. I'm a bit surprised that Linux actually works as they seem to be describing, since their 'change' was to make it work in the same way as pretty much every other SMP-aware scheduler that I've come across; schedule processes on cores independently and periodically migrate processes off overloaded cores and onto spare ones.
There are lots of potential bottlenecks. The one I was expecting to hear about was cache contention. In a monolithic kernel, there are some data structures that must be shared among each core and every tim you do an update on one core you must flush the caches on all of them, which can start to hurt performance when you have lots of concurrent updates. A few important data structures in the Linux kernel were rewritten in the last year to ensure that unrelated portions of them ended up in different cache lines, to help reduce this.
Even then, it's not a problem that's easy to solve at the software level. Hardware transactional memory would go a long way towards helping us scale to 128+ processors, but the only chip I know of to implement it (Sun's Rock) was cancelled before it made it into production.
Re:Original Source and Actual Paper (Score:5, Interesting)
The thing is eldavojohn practically *is* an editor for /. , just check out his submission page. Despite having such a high UID he's got a solid reputation, a good writing style, and offers good commentary on a wide variety of topics.
Re:Linux already runs on thousands of cores (Score:4, Interesting)
SGI has some awfully big single-system-image linux boxes.
Not really. SGI has big NUMA machines, with a single Linux kernel per node (typically under 8 processors), some support for process / thread migration between nodes, and a very clever memory controller for automatically handle accessing and caching remote RAM. Each kernel instance is only responsible for a few processes. They also have a lot of middleware on top of the kernel that handles process distribution among nodes.
It's an interesting design, and the SGI guys have given a lot of public talks about their systems so it's easy to find out more, but it is definitely not an example of Linux scaling to large multicore systems.
Comment removed (Score:3, Interesting)
Re:Windows 7 scales to 256 cores (Score:1, Interesting)
It would be interesting to observe the transition over 64 cores in terms of scalability.
article on physorg (Score:1, Interesting)
explains it rather well, imho.
http://www.physorg.com/news205050157.html [physorg.com]
"In a multicore system, multiple cores often perform calculations that involve the same chunk of data. As long as the data is still required by some core, it shouldn't be deleted from memory. So when a core begins to work on the data, it ratchets up a counter stored at a central location, and when it finishes its task, it ratchets the counter down. ...
As the number of cores increases, however, tasks that depend on the same data get split up into smaller and smaller chunks. The MIT researchers found that the separate cores were spending so much time ratcheting the counter up and down that they weren't getting nearly enough work done."
Re:Original Source and Actual Paper (Score:3, Interesting)
you are not in the club of liked submitters. Honestly the number of crap submissions that get picked over well thought out and very well cited ones is nuts to the point that I simply stopped submitting stories here. Its a waste of time.
Re:48 cores? (Score:3, Interesting)
OK, I just use it on this box because there are a couple of programs I like not available for native Linux yet but at least it's not Vista or ME bad.
One trick in business and politics is to offer a bad choice next to a worse one so it doesn't seem as bad by comparison. Every time I see or hear that comment the conspiracy theorist in me wonders whether ME and Vista were deliberately bad to soften the shock of adjustment to XP and 7.
Re:Original Source and Actual Paper (Score:4, Interesting)
You can also think of it as the difference between rooms and buildings. Multiple cores may exist in a single CPU just like multiple rooms may exist in a building. Getting around between rooms in the same building isn't such a big deal. But getting from Room A in Building 1 to Room B in Building 2 requires you to leave Building 1 and then enter Building 2, which takes more time. Some motherboards support multiple CPUs (buildings) but most do not. Those that do are usually more expensive than the ones that support only a single CPU.
Re:other kernels (Score:3, Interesting)
Already Problematic with 4 cores (Score:2, Interesting)
Re:Original Source and Actual Paper (Score:4, Interesting)
And multiple threads per core can be thought of as say, movable dividers in rooms. Yeah, it's really one room, but you can divide it into 2 "sort of", and it doesn't really mean you have twice as many rooms, but there are certain benefits you can get from doing so.
Re:Original Source and Actual Paper (Score:1, Interesting)
Re:Linux already runs on thousands of cores (Score:4, Interesting)
Re:based on a 1970s OS and language (Score:2, Interesting)
A language can change nearly overnight to add mechanism for threading.
Is that why the C and C++ people have spent so long at trying to come up with a memory model that will actually work correctly under concurrent execution? Is that why Java got it wrong the first time?
Re:OT - your sig (Score:1, Interesting)
Yabanjin is a better fit for "barbarian" and yaban is closer to "barbaric or uncivilized".
For example, "Yabajin desu. Yaban na kuni amerika kara yattekimasita."