Price of Power in a Data Center 384
mstansberry writes "Much like the rest of the country, IT is facing an energy crisis. The utilities are bracing companies for price spikes this winter and according to experts and IT pros, those prices aren't going to come down any time soon. This is thefirst article in a four-part series investigating the impact of energy issues on IT."
Hot Intel chips are big contributor (Score:5, Interesting)
P.S. The submitter has a nice fishing web site and is holding about a 12" trout on his main page. Nice catch ... but I'd
recommend he go on a
fishing charter
in Seward Alaska [komar.org] if he wants to catch some mongo fish.
This trip was a major slayfest and my brother was
Captain
Crudd [komar.org] who knows how to fish with a beer in his hand.
Energy price predictions (Score:5, Interesting)
From here: [thewatt.com]
"EIA expects energy expenditures will be 18% higher this winter compared to last winter, which will be 8.3% of the annual gross domestic product, a record since 1987 when it was 8.4%."
And for those of you who want to find a way to save energy: Here's 60 Tips To Save Energy This Winter [thewatt.com]
A solution to winter price spikes: (Score:3, Interesting)
Procedure: Place fans in datacenter. Tape duct to fans. Route duct to office spaces.
Results: Save money on heating and cooling bills.....
Energy is costing more in all areas (Score:2, Interesting)
Sask Power is running advertising imploring people to unplug their underused second fridge, turn off their computers when not in use, and upgrade to LDC screens to save about 66% the power expense over CRT technology. They claim savings of $50/year if you turn off your computer when it's not in use.
Re:Folding (Score:3, Interesting)
Trust me, the cost of a roomful of PCs running Seti is nothing compared to keeping a 20-ton Liebert running 24x7.
Re:Hot Intel chips are big contributor (Score:5, Interesting)
Their current power bill is 40,000$/mo. At their new facility (you can see a design of it in this document [ucar.edu]), it will be far more. Most of the building will be for computers and associated equipment; the building is being largely designed for dissipating all of the heat. I recall he said it was to consume about 3 MW, so at 0.8 cents/kWh, that would be about 175k$/mo.
As an aside, it was a really fascinating presentation. They showed *their* model of Katrina (which was presented to the White House as an "experimental product"); it was spot on. Very impressive stuff indeed. At one point I asked him about proposed methods to induce global cooling such as dumping iron into iron-deficient waters. He stated that while he hadn't modelled that, their models already take into account natural mineral influxes and their effects on bacteria populations (and thus, the effects of those bacteria on the environment), so they could model that if they needed to. He also pointed me to some newer Vostok core data
Any time soon? (Score:3, Interesting)
Let's be realistic, they won't come down...ever. If they can get another 20% (example) out of you this year, do you think they're going to drop it 20% next year after the "crisis"? 10% even? No way. Just like any other energy business that is at a near-monopoly level (gasoline), they can raise it whenever they feel like it and blame it on whatever they want. What are you going to do, go to the competition? In the area I live in (Midland, Michigan) and the surrounding cities (Saginaw, Bay City, Flint, etc) we get ONE choice for gas and electricity - Consumer's Energy. That's it. You don't like their service or prices? Tough shit. You're stuck. There have been "alternative companies" in the past, but all they do is resell energy for Consumers Energy - it's all going through the same pipes and wires.
It sucks, but that's the way it is.
Re: Linux and power management (Score:3, Interesting)
A little fiddling with the power controls of Linux would probably get it to the same power consumption as Windows. While you measured something real, it's probably a configuration issue more than a builtin Linux vs. Windows difference.
Re:Unctuous (Score:3, Interesting)
I'd love the government to step in and set things straight but I don't see that happening anytime soon. For now we'll either have to bare the pricing or start pushing alternatives which have been available for a good long time now.
Re:Hot Intel chips are big contributor (Score:4, Interesting)
Re:Hot Intel chips are big contributor (Score:5, Interesting)
Re:Unctuous (Score:5, Interesting)
What a bunch of bullcrap. The cost of pumping oil varies from well to well. Sure, it might cost Saudi Arabian Oil Company $15 per barrel, but if they only release enough oil for half the world's demand, other producers have to fill that supply. It can cost those other suppliers much more to pull oil out of the ground. And that high price is going to lift the market price.
but instead of the $25 price before we invaded Iraq, it's pushing $70+ as a "permanent high". Maybe Congress and the White Hosue can exercise some accountability for their totally failed energy policies (including sending us to war) by stopping the price gouging the oil corporations are abusing us with.
Oh really? So they're just going to tell Saudi Arabia or Venezuala to lower their prices? How are they going to force them to do that? Oh, you mean force American Oil companies. Well here's a clue: American oil companies are bench-warmers in the global oil market. The biggest American company, ExxonMobil, ranks just 16th in the world in total reserves [petrostrategies.org]. They control about 2% of the worlds oil. Hell, even Petronas, a Malaysian company, is bigger than America's biggest oil company.
And looking at the table [petrostrategies.org] you see that the market is dominated by state-owned, national oil companies like Saudi Arabian Oil Company, and Petroleos de Venezuela. The only way you're going to lower the price they charge for oil is to invade and force them. Otherwise they'll sell their oil to the highest bidder.
I know those corporations are their best bribers^Wcontributors, and their foreign sources are our best traitors^Wallies, but Americans will vote on the entire House of Representatives and 1/3 of the Senate in elections next year. We might be willing to put up with a lot of BS on faith, but there's no denying we're not getting the spoils of all of our "superpower" status.
So your complaint is that Bush hasn't invaded enough countries yet to lower oil prices. Interesting.
The fact is state-run foreign oil companies set the price for oil. There is very little the government of the USA can do about it aside from rushing in with tanks to take their oil fields. Any kind of price control on this oil would mean it would get sold to someone else at a higher price, like the Chinese, for example.
Leading Oil and Gas Companies Around the World
Rank by 2004 Oil Equivalent Reserves Company Worldwide Liquids Reserves, Million Barrels Worldwide Natural Gas Reserves, Billion Cubic Feet Total Reserves in Oil Equivalent Barrels, Million Barrels
1 Saudi Arabian Oil Company (Saudi Arabia) 2,3 259,400 234,500 299,485
2 National Iranian Oil Company (Iran) 2,3 125,800 940,000 286,484
3 Qatar General Petroleum Corporation (Qatar) 3 15,207 910,000 170,763
4 Gazprom (Russia) 0 988,892 169,041
5 Iraq National Oil Company (Iraq) 2,3 115,000 110,000 133,803
6 Abu Dhabi National Oil Company (UAE) 3 92,200 196,100 125,721
7 Petroleos de Venezuela.S.A. (Venezuela) 3 78,998 149,891 104,620
8 Kuwait Petroleum Corporation (Kuwait) 3 99,000 55 99,009
9 Nigerian National Petroleum Corporation (Nigeria) 2,3 35,255 176,000 65,340
10 National Oil Company (Libya) 2,3 39,000 52,000 47,889
11 Sonatrach (Algeria) 2,3 11,800 160,500 39,236
12 OAO Lukoil (Russia) 23,215 39,089 29,897
13 Petronas (Malaysia) 5,290 85,200 19,854
14 PetroChina Co. Ltd. (China) 10,941 44,554 18,557
15 Petroleos Mexicanos (Mexico) 14,803 14,807 17,334
16 ExxonMobil Corporation (United States) 8,395 31,843 13,838
17 BP Corporation (United Kingdom) 5,775 46,650 13,729
18 Egyptian General Petroleum Corp. (Egypt) 2 3,700 58,500 13,700
19 OAO Yukos (Russia) 10,950 7,800 12,283
20 Petroleo Brasilerio S.A. (Brazil) 2 9,945 11,247 11,868
Re:Energy price predictions (Score:3, Interesting)
How much does power cost really matter? (Score:1, Interesting)
Fuel accounted for 17% of Southwest Airlines' revenue, for example. A data center would have to be less than that, wouldn't it?
Virtual Servers (Score:3, Interesting)
More and more players are entering the virtual market (look at the success of Citrix over the past decade, which is a technology that comes from a similar paradigm) - and that means that more and more datacenters are converting. While the cost per kwh might be rising, the costs of running a data-center are coming back under control.
Re:Blame XML and Java (Score:3, Interesting)
Interesting question. Let us consider a simplified universal payroll system and see where this goes. I'll stipulate roughly 200 million US payroll employees and 52 pay periods. Lets say individuals require 200 KiB of storage (historical deductions, contributions, etc. necessary for YTD results,) and generate 1 KiB of storage each period. The necessary software doesn't exist to accommodate the payroll requirements for every conceivable employment situation, but we'll just pretend it does.
You'll need 41 TiB of storage on day one. Each day you'll need to perform 28.5e6 payroll calculations generating 29.2 GiB of new data.
The storage requirements alone are going to blow your hardware budget. It's understood the data will need reliable storage, so a rack of 82 500 GB SATA drives won't do (with a new drive added every 16 days.) Lets factor in 25% storage overhead for (poor) redundancy. We need just over a hundred 500 GB drives. Storage at this scale will inevitably result in SAN or some such technology, so you'll need some storage switches. The storage hardware will consume many times the power of your AMD64.
Data will have to arrive via some network. Payroll is an aggregate calculation of many different forms of detail data. Lets say the time sheets, vacation requests, etc. represent 10 times the volume of the result. You'll need to handle 292 GiB of inbound (we'll assume outbound is negligible) data per day. That's 2 DS1s running at capacity at all times. You'll need redundant switching hardware for this. Also, we're taking for granted that fraction of Internet capacity necessary to move this data; it's not really free, after all.
Computational load is much harder to estimate without an accurate model of typical payroll calculations. I'll use my experience with OLAP consolidations. 2.4GHz Xeon will compute an 8 GiB cube in 45 minutes. This time is mostly (95%) spent in the CPU. Its also far simpler than payroll, consisting mostly of simple aggregates and hash calculations. Lets throw a factor of 3 at this to cover the extra computation necessary for payroll. We'll need 8.2 hours of CPU time per day. That seems quite feasible for an AMD64 CPU.
There is no wiggle room in the above estimates. No backups, no test system, no fail over. In the real world batch computation is highly synchronous, so you need a lot of spare network and compute capacity. You're doing a lot of co-processing here; the CPUs running the SAN switches and routers are all essential to the process, so you can't give all the credit to the AMD64 CPU. The interesting thing is that communications and storage are the real power hogs.
Disclaimer: To call the above 'back of a napkin' is probably flattery.
hydrogen fuel (Score:3, Interesting)
People cite Hydrogen as a source but most the available Hydrogen production is a byproduct of fossil fuel refining
Actually Iceland is doing quite well in working with hydrogen, "Iceland launches energy revolution" [bbc.co.uk]. However Iceland has a big advantage over other countries, they have an abundance of geothermal energy they can use to generate hydrogen from water.
FalconRe:Unctuous (Score:1, Interesting)
Re:Folding (Score:3, Interesting)
Reversible Computing does not produce heat. (Score:4, Interesting)
Ever wondered what happens to bits that you erase out of memory or a register? They get dumped out of the chip and turn into heat.
Reversible logic reuses the electrical charge for your next computation, or for storing the next 1 that comes along.
On the downside, reversible hardware is much harder to design, but any addition of reverible logic on today's CPUs would decrease the amount of electricity needed and heat produced.
Electricity bills would be lower, and heat output would be smaller.
Laptops would last much longer, desktops wouldn't need a CPU cooler.
Even better, we could continue increasing the speed and diesize of CPUs.
One problem right now is that AMD, Intel, IBM, etc are perfectly able to produce a CPU that they have no hope of cooling. If reversible logic were used instead, you could have a 6GHz chip with the heat output of a 4.77 MHz 8086.
Re:Energy price predictions (Score:2, Interesting)
1) all forecasts are wrong. But the trend is your friend.
2) try and get real independent data, the EIA is OK, but the US is large and the markets vary wildly from region to region
http://www.bloomberg.com/markets/commodities/ener
the futures markets can be a good guide - if you are severely exposed to Energy prices hedge at least part of your load - by contract with supplier or however. The board will not like the excuse that something could have been done but wasn't.
3) if the cost of energy can stop you eating or slow your business, read your contract very carefully, watch out for stuff like - "market adjustment" and "pool" . If in doubt their are any number of consultants who will take your money to help you - perhaps find one who will take a percentage of the savings between old and new contract. If you are on a regulated tariff, sorry you are about to be screwed as the regulators almost always give pool price pass through.
4) natural gas (my specialty) - if you are relying on natural gas in winter to keep the lights on/datacentre humming - DON'T PANIC. Be prepared. Can your emergency generator take liquid fuels? If so has it been tested? Has the fuel been checked? What does your fuel contract say about resupply in the event of a shortage? What resupply priority do you have is a good question to ask your provider. There may or may not be a natural gas shortage in the US this winter, but if so who will get priority - houses or businesses? Where do you rate?
5) extreme weather will test the energy delivery system - if there isn't enough natural gas or the price skyrockets, the price of electricity (as gas is often the marginal supplier) will also go up. There also may not be enough transmission capacity in you area if everyone tries to turn on their own site facilities.
Re:How much does power cost really matter? (Score:3, Interesting)
How much do we pay for the cabinet and extra power circuit? $775. Right there, that's about 25% of the cost. Now, remember that they need to provide the cooling for 4 kilowatts of heat generation - which uses more electricity.
In a few months, we're going to lease another rack - not because we've run out of rack space, but because we've run out of electricity and cooling. The thought of using more electically-efficient chips like Via's offerings sound pretty attractive, but there are drawbacks: First, with as many new dual-CPU machines as we have to throw in every year to keep up with demand, it's going to take an awfully high number of Eden or Nehemiahs. Going from a dual-Opteron to single-CPU nehemiahs, I'd probably need at least 6 Nehemiahs, probably 8. That means that not only do I have to buy 8 times as many CPUs, chassis, and power supplies, I need 8 times as many hard drives, and 8 times as much memory - and considering that each machine uses several gigs of memory for lookup tables and disk cache, that's a looooot of cost.
That being said, I'd really, really, REALLY like to see some ultra-high density Nehemiah-based server equipment. Given the size of the mini-ITX format, you should be able to pack at least 12 - if not 16 - entire systems inside of a 1U chassis.
steve
Re:Canada (Score:2, Interesting)
Multi-cores not necessarily bad (Score:3, Interesting)
One of the selling points for Sun's Niagara is that a single Niagara processor can do the work of a bunch of single core servers - for about the same amount of power as one single core server.
Big Money... (Score:3, Interesting)
Energy Consumption works out close to 3x server power consumption, so 1kW of load is equal to 3kW total energy input. That comes close to $2,700 per year in energy costs for 1kW of server power.
This is simply stupid (Score:3, Interesting)
Good (?) accounting tends to highlight grand total costs of small things. Good Lord, we spend $27,000 per year on paper clips! Better control them under lock & key. The lost-opportunity cost of the contract bid missed because somebody was hunting for paperclips does not, of course, appear on and ledger.
Now somebody has summed up the electrical costs of a really large server room and come up with a sum close to a human salary. That always impresses people. (Man is the measure of all things.)
But what is it as a fraction of total operations and capital?
At 11 cents per kilowatt-hour (a common residential cost except in badly-gouged locales; but high for major consumers, at least until lately and those 27% increases) your rule-of-thumb for 7x24 consumption is:
a buck per watt per year.
500-watt average constant consumption from a basic 3u rack server = $500/year. Easy, no?
But that's a pretty serious machine, home machines don't commonly have over 400W power supplies - and certainly don't use the 400W all the time. So we're allowing for air conditioning power in the estimate.
But a serious server starts at $10,000 and you won't get five years out of it, so the capital cost alone is $2000/year and up.
All but the most automated shops surely have a salaried sysadmin (and/or DBA, backup specialty guy...) for every ten machines. And those guys all cost $50,000 dead minimum. So that's another $5,000 per machined per year for care & feeding.
So that's $7000/year, plus power at $500. Maybe skyrocketing to $700 and a full 10% of costs.
And of course I had to assume that the $10,000 included 5 years of vendor support to keep it that low. Never mind insurance, rent on the space, huge UPS's, fire systems, air conditioning (not the power for it, the machinery). In truth, I can hardly imagine power reaching 10% of the operations cost.
Also, I'm taking some place like NCAR as my site: gargantuan computing power at the service of a dozen professors and their retinue of grad students. Totally running their own programs, not million-dollar software packages like SAP on Oracle. In short, the normal "IT" costs of programmers, analysts, support techs, software vendors, don't exist.
Because when they do, they dwarf the cost of running the server room and power dwindles down to being 10% of 10% of your total IT budget. Which in most companies is 5%-9% of total operating expenditure.
Wailing about this cost - which springs out on the accounting spreadsheet because it is up a large percentage from last year - leads to classic penny-wise, pound-foolish decisions.
Perhaps: "we'll use less power if we consolidate a dozen servers down into one big one". A lot of this has been done by IT departments, whom I swear are pining for the days of the mainframe.
But at least where I work, business didn't move off the mainframe because it was such a high cost per compute cycle - often enough we were increasing our total computing costs to go PC and small server. We did it for the flexibility.
And loss of flexibility could cost a business big - for want of a paperclip.