Data Center Designers In High Demand

Hugh Pickens writes "For years, data center designers have toiled in obscurity in the engine rooms of the digital economy, amid the racks of servers and storage devices that power everything from online videos to corporate e-mail systems but now people with the skills to design, build and run a data center that does not endanger the power grid are suddenly in demand. 'The data center energy problem is growing fast, and it has an economic importance that far outweighs the electricity use,' said Jonathan G. Koomey of Stanford University. 'So that explains why these data center people, who haven't gotten a lot of glory in their careers, are in the spotlight now.' The pace of the data center build-up is the result of the surging use of servers, which in the United States rose to 11.8 million in 2007, from 2.6 million a decade earlier. 'For years and years, the attitude was just buy it, install it and don't worry about it,' says Vernon Turner, an analyst for IDC. 'That led to all sorts of inefficiencies. Now, we're paying for that behavior.'" On a related note, an anonymous reader contributes this link to an interesting look at how a data center gets built.

News at 11

Qualified Professionals in demand, news at 11

I don't see what's so hard about it..

Get some folding card tables, throw yer servers on there, then get yerself a extension cord and a couple of power strips to give ya enough outlets offa those two plugs in th' wall, and get yerself one of them fans from Walmart ta blow over 'em if yer feelin fancy. Voila. Them college kids think they're so smart, that wasn't hard at all. You can even get a bucket of water in case anything catches fire!

if the engineers in the datacenter

really want respect, all they have to do is send an urgent email to the ceo that the dilithium crystals are deteriorating, and that the antimatter containment fields are failing, and we can't take much more of this captain

Doing it for the glory?

'...these data center people, who haven't gotten a lot of glory in their careers, are in the spotlight now.'

Glory? If I wanted glory I would have become a firefighter or something. I got into the data center business to read people's email, plain and simple. That's reward enough!

Because the power grid has become very fragile

This is only a problem because the power grid has become very fragile.

Electricity generation hasn't grown ahead of demand due to government meddling, atom-ophobia, and environmentalist obstruction in the courts and on planning boards.

The rolling blackouts will be coming soon. It'll start with small ones. Then everyone will buy battery backups that draw a lot of power to recharge once power is restored. This will cause the duration of the periodic blackouts to go from a few minutes to a few hours in about 2 years.

Not long after that, we'll start building power generation capacity in the US again.

Re:Because the power grid has become very fragile

How many letters have you written to your congressman advocating that the government build a coal plant on your block? That's the fuel that America has the most of.

Re:Because the power grid has become very fragile

On the other hand, it might be the final push that people need to start making their homes and businesses as energy efficient as possible, up to and including home solar and/or wind; use of more energy-efficient appliances, low-power-consumption electronics, etc.

I would dare say that the future looks good for ARM and Via on that last account, at least.

Re:Because the power grid has become very fragile

No, it doesn't generate power -- but it prevents me from needing as much. The less I need, the more I can make myself. If I can cut my use from say, 200Mega Watts to 800Kilo Watts by proper deployment of insulation, energy star appliances, replacing a desktop PC with a Pico-ITX system, etc -- then if I can generate half what I need with solar panels or a wind mill (if I live in an area where I can fit one), then I'd no longer be that big of a draw on the grid, would I?

Of course, those numbers are all just pulled out of my ass for an example, but still -- you get the point. Cost saving measures at home are also going to lead to energy savings at large. With power prices going up ~30% next month, I think more people will start looking at the alternatives and where they can cut costs.

I agree that things need to be done on the supply side as well, but they should be done in a responsible manner. Building more nuclear plants, for instance. But by reducing consumption, we can then close down fossil plants instead of doing a 1:1 replacement.

Green IT

There is a growing area of interest in so-called "Green IT" (mostly due to inevitable regulations), and the first area being looked at is data center organization. It's always the first stat a consultant firm throws out, because it's relatively easy to show significant cost savings in such an environment (just by reorganizing the appliances to distribute heat in a different manner).

Virtualization in equally high demand.

Companies with full data centers and in need of more servers are turning to virtualization technologies to increase their server density, reduce their physical server deployment, and improve efficiency in cooling, hardware maintenance, and administration.

It's amazing to see the differences VMware has made in my career in just a few short years... going from deploying hardware servers in weeks to a virtual in seconds.

Re:Virtualization in equally high demand.

yes and no.

I've seen companies that turned to virtualization to solve their power and cooling problems. Yes, you can serve more OS instances with less hardware. That is good.

However, these places are generally not managing their infrastructure well in the first place. Now you start running into problems with server sprawl and storage management. The management costs are going up because you have more servers running more applications. That takes more management, not less.

I think it is great that we are seeing more specialization in this space. I think that SysAdmins need to look at how they want to specialize moving forward. Are you going to manage hardware & resources? Are you going to be more OS and application tuning? We can't expect one person to have enough breadth to go from HVAC/electric/network/storage/OS/application. I'd hate to see a tape ape get into Data Center design because "hey they're down in the Data Center anyway"

Don't get me wrong, I think virtualization (server & application & desktop) is the wave of the future. But I don't think a lot of firms see this yet. I think they are still trading one problem for another.

endanger the power grid?

I don't understand the peculiar emphasis the New York Times places on "endangering" the power grid. Even though a data center uses a lot of electricity, it's a high value operation that needs a stable power supply. What's wrong with the idea of paying more to insure that your power supply is sufficiently stable for your needs? The power company accepting those checks can then work on delivering that power. It's like saying that I'm somehow responsible for the stability of the oil production and distribution infrastructure because I drive a car. Perhaps, if I tweak my engine just so, I can engineer a democratic transformation of Saudi Arabia. I'll see if changing the oil does the trick.

At some point, you have to realize that the consumer, no matter how big, isn't responsible for the supply of resources by another party. If there's a problem with how those resources are supplied, be it fixed price (regardless of demand) power transmission lines, pollution, or deforestation, then that problem should appear as an increase in cost to the consumer. If it isn't, then it's a problem with how the resource is distributed, not a problem with the consumer.

There has always been a shortage

There has been a shortage of architectural engineers for the past two decades. I say architectural engineers because very few mechanical engineers go into HVAC, and very few electrical engineers do power systems. It doesn't seem quite as bad structurally.

It us a shame because it really has a lot of great career opportunities.

Data center work is just a subset of that-- it is hard to find people with the experience, but not impossible to train.

kinda offtopic

I use to have a few friends who worked for UUNET in Richardson TX. After Worldcom bought them and then the scandel happened their datacenter was reduced to a skeleton crew (including security). My buddy worked nights so some weekends I'd drive up to Richardson from Dallas with some beer and he'd sneak me into the datacenter through a door that the smokers used and we'd hang out, drink, and download movies/watch pron. Good times.

Their UPS was pretty impressive. It was about a 2 thousand square foot room full of what looked like car batteries. I didn't like to go in there, I don't like being around large, uninsulated, potential. (I was electrocuted pretty badly as a kid once)

incremental datacentre design ...

You hear a lot about the big datacentres that are being planned by the likes of Google, Yahoo etc. I realize this is probably an over simpification but it seems like they know ahead of time what the systems will be for the datacentre. They seem to know the apps they will run, the servers they like etc.

While I admit that these datacentres are huge and get a lot of publicity, thus a lot of pressure to design right and "green" I don't think that level of advanced knowledge is typical for SMBs and even most non-IT centric businesses regardless of size.

In practice a company has a few servers and one or two system admins, then they grow, staff leaves, they start thinking about different technologies, required software changes etc. What they end up with is a few vendors servers, a few vendors disk arrays probably a few flavors of networking etc.

In short the "real world" problem for the majority of companies/sys-admins isn't the very academic concept of building a single purpose datacentre, but handling growth and change. I'm yet to see a good reference for how to handle this. At best I see vendors showing how great there new server/rack combination is in isolation, Another popular thing is the ever popular look how low our power needs per FLOP are for a data centre based on our products. Yeah like we are likely to use identical systems for databases as we do for LDAP, and the same one for a fileserver as we use for a MPI cluster.

Anyways, does anyone know a good reference to deal with these "real world" problems?

How many jobs per data center?

Here's an interesting related issue: how many people does it take to operate a data center? Google always says that it will create 200 full-time positions at each of its new data centers. But an analysis of data center staffing [datacenterknowledge.com] for new Yahoo and Microsoft facilities in Washington State suggests that these companies can run a data center with 30 to 50 staffers.

Data center employment often comes up in discussions of economic development. Many communities are eager to attract data center projects, but struggle to define the economic benefits of these facilities. Jobs have always been the primary benchmark by which economic development projects are measured. Incentive packages offered by state and local governments are often based on the number of full-time jobs created by a new business. And do data centers really hire locally, or do trained data center engineers migrate from other existing data center hubs? In some cases, local officials try to stipulate local hires, which is a sticky wicket.

Video tour of data center design & constructio

On the "how a data center gets built" front, last week I had a tour of a new $250 million data center facility in Virginia that is getting ready to open later this month. The facility manager provided a walk-through of the power and cooling infrastructure, explaining the company's approach to designing these systems for energy efficiency and scale. I shot video, which is now posted online [datacenterknowledge.com]. The data center operator, Terremark, separated most of the electrical infrastructure from the IT equipment, putting them on separate floors and housing the generators in a separate facility. They have 11 generators now, but will have 55 Caterpillar 2.25-megawatt units when the entire complex is finished.

Re:I thought it would be higher

Does anyone else think this number is low?

It is low. The whole article is basically saying data center growth has been bottle-necked by the need for certain types of engineers and administrators.

That's probably part of the demand for the "data center in a box" concept. With a shortage of engineers to design centers it's an obvious move to try and start mass manufacturing them.

Re:I thought it would be higher

Yeah, except that mass-manufacturing data centers is like designing a 1999 Mazda or a 1956 Jaguar. There are so many factors that are changing regarding how data centers go together, that it's a moving target to get long-lasting designs. In the bad old days, you could get a 20 year life from a data center, and not a lot changed, year to year.

Now data centers can have lots of change-- as the servers themselves change along with equipment that's located in a data center. The -48vdc telephony equipment is now housed there, along with blade server chassis that breathe fire and suck power like an SUV-- let alone the heat generation problems.

Add in mandates of 5-9's availability (a new concept in the computer industry), earthquake, hurricane/tornado/flooding, power grid availability, the liablities of co-los, legal mandates and constraints, and data center design has become a discipline unto itself. It's not necessarily constrained by good personnel, rather it's constrained by the huge number of changes in the industry overall, and the numerous disciplines needed to bring asset life out of a data center investment.

Re:I thought it would be higher

Blade servers do end up sharing power supplies, and possibly switches/other gear for efficiency. But the CPUs burn, and the disks turn. Density means that a single rack when loaded up consumes voracious amounts of power and requisite cooling. It's a great idea in a lot of ways, but data centers weren't designed for either the power draw or the chilling needs, let along the weight. Add in the fact that denser instances mean more can die in a single chassis, and there are questions posed by blades that older data centers were just not designed for.

Re:OK - what do they do?

A designer needs to understand how cooling, power and building design affect each other. High density cooling and power management is a different beast. (And disaster management! Redundant power, fire suppression that won't destroy your computers, etc...)

I think that is the point here: Data centers have become large enough that you don't want to just stuff them into a random office building and hope everything will work out fine. Specialization is valuable in this case.

Re:OK - what do they do?

From TFA:

Mr. Patel is overseeing H.P.â(TM)s programs in energy-efficient data centers and technology. The research includes advanced projects like trying to replace copper wiring in server computers with laser beams. But like other experts in the field, Mr. Patel says that data centers can be made 30 percent to 50 percent more efficient by applying current technology.

At least Mr Patel is doing the expected. He and others are applying the current technology the way that it was meant to be applied. The article did not cover the wide array of companies that are addressing this problem. Data Center efficiency is all about applying the technology correctly. What was not covered explicitly in the "also linked" article is how one company is building data center 'cells' in order to minimize on the cooling costs, and create efficient compartmentalized units inside a huge warehouse.

Those of you who have been in data centers have seen forced air cooling that is not used correctly; cabinets not over vent tiles, vent tiles in the middle of the floor, cabinets over air vent tiles but with a bottom in the cabinet so no air flows.

When equipment is nearing end of life and hardly being used, it sits there and turns electricity into heat while doing nothing. There are often a grand mix of cabinet types that do not all make best use of the cooling system, undersized cooling systems, very dense blade style cabinets replacing cabinets that were not so dense unbalances the heat/cooling process in the whole data center. Not to mention what doing so does to the backup power system when needed.

There are hundreds of 'mistakes' made in data centers all over the country. Correcting them and pushing the efficiency of the data center is a big job that not many people were interested in paying for in years gone by.

If you are interested in what you can do for your small data center, try looking at what APC does, or any cabinet manufacturer. They have lots of glossy marketing materials and websites and stuff. There is plenty of information available. Here's a first link for you http://www.datacenterknowledge.com/archives/apc-index.html [datacenterknowledge.com]

Re:OK - what do they do?

It's civil engineering, intersecting with 'real world IT'. Off the top of my head:

• Power - redundancy, and resisiliency as much as 'just having enough'
• Cooling - air conditioning is a BIG deal for a data centre - you need good air flow, and it probably doubles your electric bill.
• Specialist equipment - datacentres are _mostly_ modular, based around 19" racks. But there's exceptions, such as stuff that is 'multi-rack' like tape silos.
• Equipment accessibility - you'll need to add and remove servers, and possibly some really quite big and scary bits of big iron - IIRC A Symmetrix is 1.8 tonnes. You'll need a way to get that into a datacentre, which doesn't involve '10 big blokes' - spacing of your racks might not help
• Putting new stuff in - a rack is 42U high. Right at the top of that rack, is going to require overhead lifting.
• Cabling. Servers use a lot of cables. Some are for power, some are for networking, some are for serial terminals. You've got a mix of power cable, copper cables, fiber cables. They need to fit, they need to be possible to manipulate on the fly, and they need to not break the fibers when you lay them. You also need to be aware that a massive bundle of copper cables is not perfectly shielded, so you'll get crosstalk and intereference. And every machine in your datacentre will have 4 or more cables running into it, probably from different sources, so you need to 'deal' with that.
• Operator access - if that server over there blows up, how to I get on the console to fix it. If I am on the console to fix it, how do you ensure I'm not twiddling that red button over there that I shouldn't be.
• Remote/DR facilities - most datacenters have some concept of disaster planning - things as simple as 'farmer joe dug up the cable to the ISP' all the way to 'plane flew into primary data centre'. These things are relatively cheap and easy to deal with on day one, and utter nightmares to retroengineer onto a live data centre.
• Expansion - power needs change, space needs change, technology changes and ... well, demand for servers increases steadily. It's something to be considered that you will, sooner or later, run out of space, or have to swap out assets.

That's what springs to mind off the top of my head. There's probably a few more things. So yes, civil engineering, but with a splattering of IT contraints and difficulties.

Re:Suggestion for Data Center Design(ers)

Sound proof booth!

Will it b-b-b-b-b-be w-w-w-w-warm in th-th-th-th-there?

Re:Use geography to solve the cooling problems...

While it may appear that you don't have to work hard to cool the data centers, you will have to work hard to humidify them if you do not want your equipment to die. This is a non-trivial cost and is the reason the "free cooling" (taking in outside air to cool a data center) is often not free.
One answer may be heat wheels, but they are fairly new and unproven in the data center space. Take a look at http://www.kyotocooling.com/ [kyotocooling.com]