Forgot your password?
typodupeerror
Intel Microsoft Windows Hardware

Microsoft Advice Against Nehalem Xeons Snuffed Out 154

Posted by Soulskill
from the keep-that-under-your-hat dept.
Eukariote writes "In an article outlining hidden strife in the processor world, Andreas Stiller has reported the scoop that Microsoft advised against the use of Intel Nehalem Xeon (Core i7/i5) processors under Windows Server 2008 R2, but was pressured by Intel to refrain from publishing this advisory. The issue concerns a bug causing spurious interrupts that locks up the Hypervisor of Server 2008. Though there is a hotfix, it is unattractive as it disables power savings and turbo boost states. (The original German-language version of the article is also available.)"
This discussion has been archived. No new comments can be posted.

Microsoft Advice Against Nehalem Xeons Snuffed Out

Comments Filter:
  • Broken processors (Score:5, Insightful)

    by Anonymous Coward on Saturday November 28, 2009 @01:35PM (#30255780)

    The processors are clearly broken, and anyone who bought them should get a refund or an exchange. End of story.

    • Re: (Score:2, Interesting)

      by Anonymous Coward

      We use them with Oracle VM (Xen), and they work ok.

    • All they need is a sticker that says "Windows 2008 Server Ready."
    • by sznupi (719324)

      Well, at least it seems to appear that fines on the scale of recent EU one don't seem to bother Intel that much...

    • Re:Broken processors (Score:5, Informative)

      by Waynelson (1068550) on Saturday November 28, 2009 @04:49PM (#30256936)
      I don't know if anyone actually read the kb article on the Microsoft website, but it appears that you don't lose the power saving features and what not with the hot fix installation, the loss of those features only occurs when you directly modify the registry to disable some of the c-states in the apci system as a quick fix. Either that or i'm reading the kb article wrong.
    • Excuse me? I am not a 'Microsoft Hater' as many people are here on Slashdot but that does not mean that I will take their word as gosple either! If Microsoft felt so strongly that this processor bug was terrible they should have never yeilded to Intel after all it is their reputation on the line too.

      While the bug does sound serious it has been dealt with in a hot fix; a fix that may not be perfect but it works. I'm not a server engineer but if you disable power mgmt and lock the hypervisor into turbo doe

      • Re: (Score:3, Insightful)

        by countach (534280)

        So you've missed the entire trend towards power saving in the data center?

  • AMD is looking better and this is the type of stuff that intel worshipers say amd systems do and now what will they say about intel?

    • Re: (Score:2, Insightful)

      by Anonymous Coward

      amd is incapable of having bugs in the convoluted exception path?

      • FTFA:

        For the integrated hypervisor of Windows Server 2008 R2, Microsoft has bravely resorted to a timer function that they themselves had classified as unreliable for former processors: the timer of the Advanced Programmable Interrupt Controller (APIC). Unlike, for example, the CPU timer (Time Stamp Counter, TSC) - which by now is comparatively resistant to power-saving, SpeedStep and turbo-boost modes, but is also virtualised by virtual machines - the APIC timer can also trigger interrupts. Unfortunately, right now, the Nehalem has too many of those, so that the hypervisor falters and then stops, returning the message "Clock_Watchdog_Time-out".

        So yes, if you depend on something that generates an interrupt whose code path may be suspended in certain power-saving modes, don't be surprised if it doesn't get serviced promptly. It looks more like a bug in Windows Server.

        Back in the old days, when you issued a CLI instruction, you made sure your routine didn't do too much work before issuing an STI, because that code isn't re-entrant (it's directly modifiable by the hardware, which is why you have to use the "volatile" keyword to make sure that compilers didn't "optimize away" any loops, etc). Kind of hard to guarantee that if you're putting that portion of the hardware to sleep between interrupts. As the article points out, disabling those power-saving modes fixes the problem.

        • by AcidPenguin9873 (911493) on Saturday November 28, 2009 @02:30PM (#30256158)
          I don't think so. Here's the text from the Intel erratum:

          During a complex set of conditions, if the APIC timer is being used to generate interrupts, unexpected interrupts not related to the APIC timer may be signaled when a core exits the C6 power state. The APIC timer stops counting in C6 and as such isn't typically used to generate interrupts when the C6 core power state is enabled. Implication: Unexpected interrupt vectors could be sent from the APIC to a logical processor.

          Interrupts not related to the APIC timer being caused by the APIC timer is not a software problem, it's a hardware problem. I could understand your argument if the APIC timer was generating too many interrupts upon C6 exit, or something else related to messed-up APIC timekeeping near power management events, but this is unrelated interrupts being generated.

          I don't know the details, but I would assume Microsoft is using the APIC timer in its hypervisor for a reason. Maybe it's because the hypervisor is required to virtualize all the other timekeeping mechanisms for the guest.

          • Re: (Score:2, Insightful)

            by Anonymous Coward

            This article is gibberish. The TSC does not generate interrupts. As a clocksource, the TSC is unreliable because while the frequency is fixed within a socket, it can skew across sockets particularly when dealing with multi-node systems.

            • I have no idea why you are talking about the TSC. Nothing in the article states that the TSC is generating interrupts. The article states that the APIC timer (which is fully capable of generating interrupts) is the thing generating spurious random, unrelated interrupts (i.e. not APIC timer interrupts).

              The article does mention that the hypervisor might be virtualizing the TSC for the guest, but again, I have no idea how you got from there to your post.

      • No, it's more like [hardware manufacturer of your choice] AND [software manufacturer of your choice] are incapable of making products that are both complex, and bug-free.

        And for some reason, 'high performance' often equals 'complex'.

    • Re: (Score:3, Insightful)

      by CAIMLAS (41445)

      I wouldn't say "AMD is better", necessarily. I will say, however, that the Xeons seem to have been plagued from the very beginning with problems like this. They're just fringe enough to not get enough run-in testing, and the bugs don't get as quickly found as they do with the more mainstream/many users processors.

      • Re: (Score:2, Informative)

        by lukas84 (912874)

        Xeon is just a marketing name. The Xeon 3400 are identical with the i5-7xx, i7-8xx CPUs, the Xeon 3500 are identical with the i7-9xx CPUs and the Xeon 5500 CPUs are basically i7-9xx with two QPI Links.

        For example, this issue also affects als i5 and i7 CPUs.

    • by amorsen (7485) <benny+slashdot@amorsen.dk> on Saturday November 28, 2009 @02:21PM (#30256086)

      Read the link. 5 pages of errata, and that's just headlines. Modern processors are very complicated, and they will have bugs.

      The major difference between Intel and AMD when it comes to errata is that Intel learned its lesson about secrecy from the Pentium FPU fiasco. Since then they have had a very open approach to processor bugs. AMD hasn't had such a PR disaster and isn't quite as open. That doesn't mean they are particularly less buggy.

      • It's also worth noting that the interrupt delivery mechanism on the Core 2 and newer processors is horribly complex. I'm not sure if these CPU's support VT-d, but I assume that they do and that makes it even more complicated. Parallels couldn't even get interrupt delivery right on the Core 2 (causing kernel panics as a result of IPIs in the host OS unless you paid for the new version) and that was a simpler system. Given the complexity of the hardware, it's not at all surprising that there are errata. T

      • ...just look at the phenom TLB problems.
    • by Bengie (1121981)

      Intel i7(quad) @ 3.33ghz Idle: 117watts Full-Load: 247watts

      Phenom2 x4(quad) @ 3.2ghz Idle: 148watts Full-Load: 236watts

      Now, include Intel's cpu being 2x-4x faster(depending on type of work) and check your performance per watt and tell me which is better

      I'm trying to follow your logic of AMD being better (at least for now, bulldozer has a lot of promise)

    • by TopSpin (753) * on Saturday November 28, 2009 @05:14PM (#30257036) Journal

      AMD has also built parts with equally screwed up timers, particularly TSC clock skew on multi-cores. Timers are just messed up on x86 from either company. This nonsense goes back years. There are now at least four distinct general purpose clock sources that must be present on modern systems; tsc, apci_pm, hpet and pit (as labeled by the Linux kernel.) There will probably be further proliferation in the future as ALL of the existing timers are inadequate in subtle ways. Implementations from both manufacturers have been plagued with bugs that require nasty work-arounds; google "clocksource tsc unstable", "pm-timer bug" or "athlon x2 tsc" for some examples. This nonsense that Microsoft has stumbled upon is just the latest in a long and colorful history of failure that we'll now have to add to the list.

      Computers are supposed to keep time. Today that means high resolution clocks that work correctly regardless of power saving, concurrency, etc. Using these crucial timers is not suppose to cause spurious interrupts, bus contention or other subtle problems. People that must work with this stuff are thoroughly fed up with this ever growing pile of half-baked bullshit.

      • I agree wholeheartedly. I once spent a lot of time trying to get a virtualised windows machine to run in plain old vmware server without the clock galloping head at 40% faster than wall-clock time; I tried many different things on the linux host side as well as the vm and the vmware tools install.

        Will Intel and AMD please sit down like adults and come up with a standardised mechanism that virtualises and copes with dynamic clocking, multiple cores with/without hyperthreading and all the idle and sleep st
        • Do the timers in Freescale or TI (or IBM) processors have problems like this?

          • Depends on the timer. The equivalent of the TSC generally does. It's a cycle timer; it increments once for every clock cycle (actually, this isn't guaranteed. As I recall, the TSC just guarantees that every RDTSC instruction will return a number greater than the last RDTSC instruction). If you scale the CPU speed back, the TSC becomes slower. If you read the TSC on one CPU and then on another in an SMP system then you will get entirely unrelated values. Oracle used to use the TSC for transaction timin

  • by chebucto (992517) * on Saturday November 28, 2009 @01:44PM (#30255824) Homepage

    Maybe Xeons are what end up being used on the UESG Marathon. I mean, half of the terminal messages on that ship are subject to the same bug. Just look at this typical example:

    http://marathon.bungie.org/story/nawmanhesclose.html#M3.13.1.1 [bungie.org]

  • by Faizdog (243703) on Saturday November 28, 2009 @01:48PM (#30255858)

    This story is interesting and timely because I plan on buying a new desktop in the next 2 weeks, just waiting for the right deal to come out, hopefully on Cyber Monday. While not getting a server, I will be getting Windows 7. I had been planning on an i7, but now am hesitant. Is there a problem with these processors for home use/gaming purposes under Windows 7? Or would I better off going with a Quad Core?

    • by Viros (1128445) on Saturday November 28, 2009 @02:08PM (#30255988)
      I've got an i7 920 on my desktop and run Windows 7 for gaming/home use purposes and it works fine. Don't let the problems with the server software dissuade you from a very good processor for home and gaming use. The kind of stuff you're describing doing will never run into anything close to the problems from this article.
      • Second. Been running the same proc with Windows 7 since RC and RTM. No probs whatsoever. I have been running VMWare for XP and encountered no issues.

      • by Anpheus (908711)

        I've run Windows Server 2008 R2 as a hypervisor with my i7 to do some testing and it's ran fine. I'm guessing this issue is particularly rare and that's why this fix is coming out close to five months after RTM.

    • by the linux geek (799780) on Saturday November 28, 2009 @02:10PM (#30256002)

      No, this only applies to the Hyper-V component of Server 2008 R2. Normal people do not use Windows Server for "home use/gaming purposes" (cue a dozen replies of people talking about how cool they are because they use pirated copies for said purpose), so its not a big deal. Also, Core i5/i7 is already a Quad Core, I assume you mean Core 2 Quad.

      • Re: (Score:1, Funny)

        (cue a dozen replies of people talking about how cool they are because they use pirated copies for said purpose)

        I'm cool because I use a pirated copy of Windows Server for said purpose.

        (Looks around.) Hey, waitaminute! Where'd the other 11 guys go? I've been set up! Curse you Microsoft and your clever Slashdot traps!

      • by jonbryce (703250)

        Does it affect the XP Mode component of Windows 7? Obviously you aren't going to use that for gaming unless you don't mind having less than one frame per second in your games through it's internal RPD thingy, but you may well have some sort of XP program running in the background when you are playing your game, even if it is just your XP virus scanner.

    • Looks like it's only if you're doing some virtualization. It probably wouldn't affect games.
    • Re: (Score:2, Interesting)

      Be sure to avoid any of the HP i7 processor models. They have a major motherboard problem, as I have luckily learned myself by buying one. Check some googling with HP i7 crash freeze, etc. Or go here to see gobs of users with problems:

      http://h30434.www3.hp.com/psg/board?board.id=lockups [hp.com]

      However, I hear that home made PCs on i7 platforms are fine as well as recent Dells. I'm going the Asus motherboard route to rectify my problem. It's just grand though b/c they are way expensive, then I need a new case (b/c H

  • by bill_mcgonigle (4333) * on Saturday November 28, 2009 @01:49PM (#30255862) Homepage Journal

    Many of the benchmarking sites have also posted some poor results - I was thinking this might be a generation to skip, but now I wonder if a flaw has been discovered that could be fixed with a microcode upload. Might help the benchmarks too if it was a hidden variable.

    • Re: (Score:3, Informative)

      by Darkness404 (1287218)
      A generation to skip for servers (or move to AMD for a generation) but Core i7s are amazing for home/gaming use. For just about anything other than visualization and server-specific stuff, Core i7s and CPUs with the same architecture have no comparison with what AMD has to offer.
      • I'm cautiously optimistic that Nehalem-EX will be a decent server processor, at least in the 1-2 socket segment. It seems to handle multithreading quite well, and have decent FP performance. For now, though, the 6-core Opteron is king.
      • by bertok (226922)

        A generation to skip for servers (or move to AMD for a generation) but Core i7s are amazing for home/gaming use. For just about anything other than visualization and server-specific stuff, Core i7s and CPUs with the same architecture have no comparison with what AMD has to offer.

        "citation needed"

        You're saying that like Microsoft Hyper-V is the only virtualization platform, nobody ever uses a server without virtualization, and there's no way Microsoft could ever release a hotfix for the issue. Sell your Intel stock now!

      • by gbjbaanb (229885)

        No, its not broken for anything other than *Microsoft* virtualisation in Windows Server 2008R2, if you use VMware, then you're fine - it doesn't use the dodgy interrupt that Hyper-V does.

        The answer is just not to install Windows Server 2008R2 or use Hyper-V.

      • For just about anything other than visualization and server-specific stuff, Core i7s and CPUs with the same architecture have no comparison with what AMD has to offer.

        At risk of sounding like an AMD fanboy, I should point out that AMD's chips are so much more expensive than the Core i7s, that you might as well buy a top-end AMD system, and replace the CPU and motherboard a year from now with whatever is the reasonably-priced state of the art at that point. You'll likely end up spending quite a bit less, and won't be stuck with an obsolete system in the future.

        I won't contest that the Core i7s are impressive. However, they appear to be the CPU equivalent of the $400 vid

        • What about the possibility that saved power in the long run will save money - to be fair, on that theory we might just as well scrap server farms and everyone gets IBM mainframes or Sun's "big iron by any other name"

    • by lukas84 (912874)

      What benchmarks were you looking at? The Xeon 5500 CPUs are fast as hell. They no longer need slow FB-DIMMs. They use much less power and yet deliver more Performance.

      A Dual-Socket, Quad Core Xeon 5500 machine can beat a quad-socket, Six Core AMD machine (c't, a Magazine also bei Heise published a test regarding this some issues ago).

      • What benchmarks were you looking at? The Xeon 5500 CPUs are fast as hell. They no longer need slow FB-DIMMs. They use much less power and yet deliver more Performance.

        I don't have a handy link to give you, but the focus was on the cache, IIRC. They performed slower than the previous Xeons for certain workloads. The focus was work/cycle - agreed they're much better in work/Watt.

  • by Anonymous Coward on Saturday November 28, 2009 @01:50PM (#30255868)

    It sounds like microsoft should retract the advice and issue a warning that no OS should be run on a processor with such spurious interrupts?

    Or is this the sort of crappy hardware kernels are supposed to put up with in which case it should be Intel advising against running windows on it's hardware?

    Int€l bashing..check
    M$ bahing...check
    now i just sit back and watch the karma roll in

    • Re: (Score:3, Funny)

      by Anonymous Coward

      Uh, guy? That symbol you used is a "C" with two lines through it, not an "E". Get it right.

      • Actually, no, it isn't. In official bullshit-speak:

        Inspiration for the symbol itself came from the Greek epsilon () - a reference to the cradle of European civilisation - and the first letter of the word Europe, crossed by two parallel lines to 'certify' the stability of the euro.

        Straight from the horse's mouth [europa.eu].

        The single-stroke $-sign OTOH might just as well be an 8:

        That the dollar sign is derived from a slash through the numeral eight, denoting pieces of eight. The Oxford English Dictionary before 1963 held that this was the most probable explanation, though later editions have placed it in doubt.

        according to wikipedia [citation needed].

        If this was true it would herald a major crisis for derogatory spelling worldwide. I propose a conference to establish new and reliable standards (.) that provide sustainable ways to express our unstillable rage (..) and call attention to the seriousness of the offenses (...) committed by mega-corps (*gasping for air*) in one

    • It sounds like microsoft should retract the advice and issue a warning that no OS should be run on a processor with such spurious interrupts?

      Ehm, aren't these spurious interrupts a hardware feature, designed to test the code handling them?

      (ducks to avoid in-flight chair)

  • I've been experiencing problems with intermittent lockups under VMWare as well. DL370-G6 boxes. HP has given us BIOS fixes and is even shipping new boxes, but if there's a suspect problem
    with working with MS' hypervisor, I wonder if this is the same issue?

    • Re: (Score:3, Interesting)

      by Glasswire (302197)

      Is it in response to a documented problem with VMWare ESX that HP trying to remedy with a specific BIOS change or is HP just flailing around suggesting BIOS updates as a fix to a problem they don't yet understand? There are 100s of reasons why you're having VMWare lockup issues - the ONLY similarity to MSFT issue that you seem to have is they are both hypervisors running on Nelhalem procs. Pretty thin. What does VMWare think the problem is?

    • I'm running a brand new DL370 G7 with a pair of 2.93GHz Nehalams and Oracle VM Server 2.2 (specialized RHEL 5.3 Xen) and it seems to be working fine, except for a completely unrelated SAS backplane / SmartArray p410 failure I experienced before the box was a month old, but that was just a simple fluke of a hardware failure like any server can experience.

  • by Anonymous Coward on Saturday November 28, 2009 @02:14PM (#30256030)
    I read the article, I read the MS support report, and I read the Intel advisory. And I don't think that the summary is correct.

    The summary says that the hotfix disables power savings and turbo boost. But my reading of the MS report is that an affected system has two options, (1) a workaround, and (2) the hotfix. The difference is that the workaround disables advanced power savings and is known to be stable without side effects, but the hotfix actually fixes the problem with the vector table, presumably by following the instructions provided in the Intel advisory note.

    Said another way, the hotfix doesn't disable power savings and doesn't disable turbo boost.

    I expect that this is another fine example where Slashdot editors misunderstand a situation. Someone prove me wrong.
    • by RDaneel2 (533639) on Saturday November 28, 2009 @02:27PM (#30256134) Homepage
      I just saw your post as I was finishing researching mine... and I certainly agree with you that the summary is wrong.

      The Microsoft KB article is quite explicit that the workaround is what disables the sleep states, leading to higher power usage - the hotfix itself does not exhibit this problem.
    • by Anonymous Coward on Saturday November 28, 2009 @02:29PM (#30256156)

      Your explanation is exactly how I interpreted the KB article. I think Slashdot was going for some sensationalistic journalism. :-)

      Taken from TFA:
      You can disable the Advance Configuration and Power Interface (ACPI) C-states by using a BIOS firmware option on the computer. If the firmware does not include this option, a software workaround is available. You can disable the ACPI C2-state and C3-state by setting a registry key. To do this, follow these steps:

            1. At a command prompt, run the following command:
                  reg add HKLM\System\CurrentControlSet\Control\Processor /v Capabilities /t REG_DWORD /d 0x0007c044
            2. Restart the computer.

      Note The computer idle power consumption will increase significantly if the deeper ACPI C-states (processor idle sleep states) are disabled. Windows Server 2008 R2 uses these deeper C-states on the Xeon 5500 series as a key energy saving feature.

      To continue to benefit from these energy saving states, remove this registry key after you install the hotfix that this article describes. To do remove this registry key, follow these steps:

            1. At a command prompt, run the following command:
                  reg delete HKLM\System\CurrentControlSet\Control\Processor /v Capabilities /f
            2. Restart the computer.

  • Actual errata (Score:3, Informative)

    by crow (16139) on Saturday November 28, 2009 @02:30PM (#30256160) Homepage Journal

    From the pdf file linked from the Intel site, I think it's AAK36, as it's the only one that mentions the word "spurious." This has to do with writing to the interrupt vector table when a local interrupt is pending. That doesn't look terribly serious from my perspective. If I'm mistaken and it's a different errata, please reply with the correction.

    • Re: (Score:3, Informative)

      by crow (16139)

      AAK36 for the Xeon version. AAN31 is the code for the i7 and i5 version. It's the same errata, just a different code number for different chips.

    • Re: (Score:3, Informative)

      by YesIAmAScript (886271)

      I don't think it's either of them. The top one about changing vectors would be unlikely to happen in commercial software like Windows, because they would have handlers installed for all interrupts already.

      I think it issue really is the watchdog, MS is using the APIC during C6 state and as the 119 errata, the APIC counter stops during C6 state. So some interrupt that is supposed to fire to reset the watchdog doesn't fire and thus the watchdog goes off (as indicated by the error code).

      So the 119 errata is rel

  • Looks like it's a Microsoft coding problem if there is no problem in Xen or VMWare ESX Hypervisors (post on VMware above is far from useful).
    And poster didn't read the MSFT article very closely. The hotfix doesn't preclude the energy saving sleep states, it's the workaround that inhibits their use.

    • by lukas84 (912874)

      It couldn't possible be that VMware, Xen and Microsoft have different approaches to the whole Hypervisor thing, which could expose different bugs in Intel's Hardware.

    • Xen need not use the hardware virtualization, and in fact performs far better in "para-virtualization". So would any system that avoided so much of the hardware virtualization and used a customized kernel, more suited to use in a virtualized OS by speaking more gracefully with the virtual server's system. I find it wonderful, and dearly with that VMWare could be convinced to support that kind of guest environment.

      • Xen now tries to use a hybrid system if the hardware is available; they use the host CPU's virtualisation capabilities where available and PV code where it's faster. System call delivery is a good example of this. On an old CPU, the kernel will install an interrupt 80h handler using the Xen Hypercall. Any syscall instructions will be bounced slowly back from the hypervisor and any int 80h instructions will jump straight to the system call handler in the kernel. With a new CPU, the kernel will just set a
        • Excellent points. But the last time I used Xen, there was no evidence of the kind of mixed hardware virtualization/para-virtualization you refer to here. It was either/or, hard-coded into the XML configuration files by the setup tools or re-configurating the installed guest environment. Are you saying this is a run-time detected and enabled behavior? And given that no server class motherboards ship with hardware virtualization enabled, it was very important to be able to run para-virtualized enviornments on

          • You can use the CPUID instruction in Xen to detect when you're running in an HVM environment and then use a virtual PCI device to map the XenStore and bootstrap the PV mode. Intel had a Linux kernel that did this in early 2007 (and even did some binary rewriting to replace native code with PV code where appropriate). Given the massive NIH that Linux has with respect to Xen, I wouldn't be surprised if it's not part of the stock tree, but you can probably get it working easily.

  • There is no evidence Intel pressured MS into their wording of the fix/workaround. It's quite possible that after not finding a fix/workaround for it and writing an initial draft saying not to use the processors, MS developed a workaround/fix (perhaps with Intel's help) that actually does work and put that in instead of saying not to use the chips.

    To those are are suddenly concerned about Intel chips because they have an errata, every chip has errata, tons of them. AMD has them too, trust me.

    I've been runnin

  • by George_Ou (849225) on Sunday November 29, 2009 @03:26AM (#30259828)
    Folks, this is a very irresponsible headline at slashdot. The Microsoft articles does NOT say hotfix breaks power save and it doesn't even mention turbo, but that it is an either or solution. Microsoft always offers workarounds as an ALTERNATIVE to the hotfix for people who don't want to apply hotfixes. The Microsoft KB article even tells you if you want to keep using those power states, then run the hotfix and make a certain modification to the registry.

    This post makes it sound like some kind of cover up and that the fix causes major CPU slowdowns, and that it's on the level of the AMD Barcelona TLB bug where the fix actually did cause a significant performance drop. This does not appear to be true. The real story is that all CPUs have hundreds of errata, and it's the job of the software maker to work around it, and that is what Microsoft is doing with their hotfix and registry hack. They're also telling you if you aren't experiencing any problems, don't bother applying the hotfix.
  • KB Link (Score:2, Informative)

    by woan (1629447)
    I didn't see a link to the KB article in question. I assume this is the one: http://support.microsoft.com/kb/975530 [microsoft.com]

The reason that every major university maintains a department of mathematics is that it's cheaper than institutionalizing all those people.

Working...