Forgot your password?
typodupeerror
Operating Systems Hardware Linux

Linux May Need a Rewrite Beyond 48 Cores 462

Posted by CmdrTaco
from the it's-all-stacking-blocks dept.
An anonymous reader writes "There is interesting new research coming out of MIT which suggests current operating systems are struggling with the addition of more cores to the CPU. It appears that the problem, which affects the available memory in a chip when multiple cores are working on the same chunks of data, is getting worse and may be hitting a peak somewhere in the neighborhood of 48 cores, when entirely new operating systems will be needed, the report says. Luckily, we aren't anywhere near 48 cores and there is some time left to come up with a new Linux (Windows?)."
This discussion has been archived. No new comments can be posted.

Linux May Need a Rewrite Beyond 48 Cores

Comments Filter:
  • by Gaygirlie (1657131) <(gaygirlie) (at) (hotmail.com)> on Thursday September 30, 2010 @12:54PM (#33748936) Homepage

    It's not the case of not being able to do such, but instead about where there are performance regressions. Of course it's possible to run Linux on multiple hundreds of cores, but it seems that after 48 cores there is a performance regression and thus all those cores don't benefit as much as they could. That is the issue here.

  • by Dragoniz3r (992309) on Thursday September 30, 2010 @12:54PM (#33748940)
    Oh look, CmdrTaco published yet another story with a poorly-written, hypersensationalist summary! Par for the course.
  • by klingens (147173) on Thursday September 30, 2010 @12:55PM (#33748962)

    Yes it is lacking: it's too long for a /. "story". Editors want small, easily digested soundbites, not articles with actual information.

  • Only Linux? (Score:4, Interesting)

    by Ltap (1572175) on Thursday September 30, 2010 @12:57PM (#33748990) Homepage
    It looks like TFS was written by a Windows fanboy; why mention Linux specifically when it is a general problem? Why try to half-assedly imply that Windows is more advanced than Linux?
  • 64 cores (Score:3, Interesting)

    by hansamurai (907719) <hansamurai@gmail.com> on Thursday September 30, 2010 @12:58PM (#33749000) Homepage Journal

    At my last job we had a bunch of Sun T5120s which housed 64 cores. So yeah, we are "anywhere near 48".

  • by Skal Tura (595728) on Thursday September 30, 2010 @01:04PM (#33749100) Homepage

    Scare piece.

    Your submission wasn't scaring enough. From your submission, it seems that it's not that big of a deal and rather easy solution. This submission makes it sound like linux kernel needs a complete rewrite ground-up, as in starting from scratch.
    Plus yours was a bit long and lots of details.

  • by r00t (33219) on Thursday September 30, 2010 @01:05PM (#33749102) Journal

    No kidding. SGI's Altix is a huge box full of multi-core IA-64 processors. 512 to 2048 cores is more normal, but they were reaching 10240 last I checked. This is SMP (NUMA of course), not a cluster. I won't say things work just lovely at that level, but it does run.

    48 cores is nothing.

  • by zill (1690130) on Thursday September 30, 2010 @01:12PM (#33749232)
    Do they have support for smooth full-screen flash video yet? [xkcd.com]

    My Ubuntu 10.04 system still can't play embedded youtube videos. At least Adobe provided a work-around by adding a "play on youtube" option in the right click context menu.
  • by TheRaven64 (641858) on Thursday September 30, 2010 @01:12PM (#33749234) Journal

    And it's worth noting that the most common application for that kind of machine is to partition it and run several different operating systems on it. Solaris has already had some major redesign work for scaling that well. For example, the networking stack is partitioned both horizontally and vertically. Separate connections are independent except at the very bottom of the stack (and sometimes even then, if they go via different NICs), and each layer in the stack communicates with the ones above it via message passing and runs in a separate thread.

    However, it sounds like this paper is focussing on a very specific issue: process accounting. To fairly schedule processes, you need to work out how much time they have spent running already, relative to others. I'm a bit surprised that Linux actually works as they seem to be describing, since their 'change' was to make it work in the same way as pretty much every other SMP-aware scheduler that I've come across; schedule processes on cores independently and periodically migrate processes off overloaded cores and onto spare ones.

    There are lots of potential bottlenecks. The one I was expecting to hear about was cache contention. In a monolithic kernel, there are some data structures that must be shared among each core and every tim you do an update on one core you must flush the caches on all of them, which can start to hurt performance when you have lots of concurrent updates. A few important data structures in the Linux kernel were rewritten in the last year to ensure that unrelated portions of them ended up in different cache lines, to help reduce this.

    Even then, it's not a problem that's easy to solve at the software level. Hardware transactional memory would go a long way towards helping us scale to 128+ processors, but the only chip I know of to implement it (Sun's Rock) was cancelled before it made it into production.

  • by NevarMore (248971) on Thursday September 30, 2010 @01:12PM (#33749236) Homepage Journal

    The thing is eldavojohn practically *is* an editor for /. , just check out his submission page. Despite having such a high UID he's got a solid reputation, a good writing style, and offers good commentary on a wide variety of topics.

  • by TheRaven64 (641858) on Thursday September 30, 2010 @01:18PM (#33749324) Journal

    SGI has some awfully big single-system-image linux boxes.

    Not really. SGI has big NUMA machines, with a single Linux kernel per node (typically under 8 processors), some support for process / thread migration between nodes, and a very clever memory controller for automatically handle accessing and caching remote RAM. Each kernel instance is only responsible for a few processes. They also have a lot of middleware on top of the kernel that handles process distribution among nodes.

    It's an interesting design, and the SGI guys have given a lot of public talks about their systems so it's easy to find out more, but it is definitely not an example of Linux scaling to large multicore systems.

  • by Gaygirlie (1657131) <(gaygirlie) (at) (hotmail.com)> on Thursday September 30, 2010 @01:23PM (#33749390) Homepage

    Since the multicore chips share caches (level 2 cache is shared, level 1 cache isn't IIRC, but I could be wrong) it's actually cache memory where the issue lies.

    That's what I thought too, but after thinking it a bit more I'd dare to claim it's both a hardware and software issue. Too small cache of course does cause issues like the researchers noticed but it's mostly because the method how memory accesses and cache is handled in software that makes it such a big issue. Rethinking the approach how kernel handles such could very well minimize the impact even in cases where there is not all that much cache available.

    Of course, I'm not an expert in SMP or multi-core systems so I could have verily misunderstood it.

  • by Anonymous Coward on Thursday September 30, 2010 @01:31PM (#33749522)

    It would be interesting to observe the transition over 64 cores in terms of scalability.

  • article on physorg (Score:1, Interesting)

    by Anonymous Coward on Thursday September 30, 2010 @01:40PM (#33749694)

    explains it rather well, imho.

    http://www.physorg.com/news205050157.html [physorg.com]

    "In a multicore system, multiple cores often perform calculations that involve the same chunk of data. As long as the data is still required by some core, it shouldn't be deleted from memory. So when a core begins to work on the data, it ratchets up a counter stored at a central location, and when it finishes its task, it ratchets the counter down. ...
    As the number of cores increases, however, tasks that depend on the same data get split up into smaller and smaller chunks. The MIT researchers found that the separate cores were spending so much time ratcheting the counter up and down that they weren't getting nearly enough work done."

  • by Lumpy (12016) on Thursday September 30, 2010 @01:41PM (#33749710) Homepage

    you are not in the club of liked submitters. Honestly the number of crap submissions that get picked over well thought out and very well cited ones is nuts to the point that I simply stopped submitting stories here. Its a waste of time.

  • Re:48 cores? (Score:3, Interesting)

    by Abstrackt (609015) on Thursday September 30, 2010 @01:49PM (#33749854)

    OK, I just use it on this box because there are a couple of programs I like not available for native Linux yet but at least it's not Vista or ME bad.

    One trick in business and politics is to offer a bad choice next to a worse one so it doesn't seem as bad by comparison. Every time I see or hear that comment the conspiracy theorist in me wonders whether ME and Vista were deliberately bad to soften the shock of adjustment to XP and 7.

  • by jgagnon (1663075) on Thursday September 30, 2010 @01:52PM (#33749916)

    You can also think of it as the difference between rooms and buildings. Multiple cores may exist in a single CPU just like multiple rooms may exist in a building. Getting around between rooms in the same building isn't such a big deal. But getting from Room A in Building 1 to Room B in Building 2 requires you to leave Building 1 and then enter Building 2, which takes more time. Some motherboards support multiple CPUs (buildings) but most do not. Those that do are usually more expensive than the ones that support only a single CPU.

  • Re:other kernels (Score:3, Interesting)

    by Beelzebud (1361137) on Thursday September 30, 2010 @01:55PM (#33749986)
    Possibly, but they still have tons of work to do. I recently installed Arch Hurd http://www.archhurd.org/ [archhurd.org] just to get some hands on time with the state of the OS, and was kind of surprised at the status. Many things are in place and work correctly, but it's nowhere near something I could say I'd actually want to use on a daily basis.
  • by chocapix (1595613) on Thursday September 30, 2010 @01:55PM (#33749988)
    Using "cat /proc/cpuinfo" as a benchmark, I can see that my quad core is several times slower with an SMP kernel compared to a non-SMP kernel.
  • by Anpheus (908711) on Thursday September 30, 2010 @02:06PM (#33750182)

    And multiple threads per core can be thought of as say, movable dividers in rooms. Yeah, it's really one room, but you can divide it into 2 "sort of", and it doesn't really mean you have twice as many rooms, but there are certain benefits you can get from doing so.

  • by Anonymous Coward on Thursday September 30, 2010 @02:12PM (#33750270)
    [expletives redacted] "A whole new operating system?" [expletives redacted]. It has nothing to do with operating system. Are you guys too cool for school, or did you forget http://en.wikipedia.org/wiki/Von_Neumann_architecture#von_Neumann_bottleneck [wikipedia.org].
  • by Troy Baer (1395) on Thursday September 30, 2010 @02:36PM (#33750648) Homepage
    Um, no. The early Itanium-based Altixes (Altices?) could go up to 512 cores running a single copy of Linux. The new Nehalem-based Altixes can have up to 2048 cores in a single system image IIRC. We just finished acceptance testing on an SGI Altix UV 1000 with 1024 cores [tennessee.edu]. It runs one copy of Linux on it.
  • by EvanED (569694) <evaned@ g m a i l.com> on Thursday September 30, 2010 @03:17PM (#33751192)

    A language can change nearly overnight to add mechanism for threading.

    Is that why the C and C++ people have spent so long at trying to come up with a memory model that will actually work correctly under concurrent execution? Is that why Java got it wrong the first time?

  • Re:OT - your sig (Score:1, Interesting)

    by Anonymous Coward on Thursday September 30, 2010 @04:42PM (#33752498)

    Yabanjin is a better fit for "barbarian" and yaban is closer to "barbaric or uncivilized".
    For example, "Yabajin desu. Yaban na kuni amerika kara yattekimasita."

The universe is like a safe to which there is a combination -- but the combination is locked up in the safe. -- Peter DeVries

Working...