AT&T To Replace 17,000 Batteries

An anonymous reader writes "After four fires in two years — see earlier Slashdot discussions for background — AT&T is going against its own independent lab findings and declaring that the Avestor batteries powering its U-verse network aren't safe and need to be replaced. This is the network that SBC was building out prior to acquiring AT&T. Following the latest broadband equipment cabinet explosion in Wisconsin, the carrier says it will swap out 17,000 batteries deployed in several states across its network."

The Catch:

[Icarus1919]

AT&T will be replacing them with batteries that explode MORE often. The current frequency of explosion is unacceptable

Re:

[robbiedo]

Are you referring to "Dr. Jacques von Hämsterviel?"

Test run?

[CustomDesigned]

Amen. Shouldn't they replace say 20% - 50%, and see if any of the problems are in the new set? Maybe the problem is the batteries environment, and not the battery itself.

Metric

[Stanistani]

"Sir, we've experienced an explosive growth in customer satisfaction!"

Houston incidents

[I8TheWorm]

Said one AT&T employee in Houston...

"I don't know nuthin' about fautly batteries. But Houston is dang hot, and it ain't no dry heat. Things just catch on fire all the time. It's hotter than a whore in Sunday church down here."

Re:

[Locutus]

From what I heard, there are Microsoft Windows based servers in these boxes. So I wonder if they under estimated how hot that runs. poof! ;-)

And how come there are "Microsoft Windows" stickers all over those boxes like they are on all the OEM PCs and boxes? ;-)

LoB

But... why?

[Amorymeltzer]

Maybe I just don't know, but why on Earth do these things explode? It seems to happen with alarming frequency given the ubiquitous nature of these things - how hard is to make batteries or wires that don't catch fire when using them? Something like this has been happening a few times a year, and recalls or replacements aren't enough - punishments are in due order for making shoddy, dangerous products.

Re:But... why?

[$RANDOMLUSER]

Because customers demand quick-charge batteries - it's a dicey proposition - you have to measure the current draw pretty carefully, or the battery temperature itself. Picture filling a bucket with a firehose without overflowing the bucket and you get the idea.

Re:But... why?

[LWATCDR]

Except these are batteries in network cabinets. They are not in end user equipment. I doubt that they are quick charge batteries.

Re:

[DerekLyons]

Why wouldn't they be? Quick charge means they constantly (or nearly constantly) have maximum capacity available - which is the ideal state for a backup system (because it makes system performance predictable and maximizes uptime in the event of power outage).

Re:

[LWATCDR]

Quick charge means a lower life on the battery and really doesn't give a huge advantage. It just isn't a good tradeoff.

Re:

[nacturation]

Why wouldn't they be? Quick charge means they constantly (or nearly constantly) have maximum capacity available - which is the ideal state for a backup system (because it makes system performance predictable and maximizes uptime in the event of power outage).

Think about what you're saying. These are backup batteries and will ideally never get used. What does it matter if your batteries charge in 30 minutes if they're only used two times a year? You could have batteries which take days to fully charge because at the time of an outage, what difference did it make that the batteries charged in 30 minutes and then sat there at full charge for half a year? Plus, quick charge batteries are likely more expensive, have shorter lifespans, and have worse performance

Re:

[DerekLyons]

What makes you think I didn't think about what I said? That I proposed a solution based on sound engineering rather than guesswork?

Re:

[nacturation]

What makes you think I didn't think about what I said?

It was the contents of your post that indicate you didn't think it through. Yes, you're correct that having a system always charged is the best state to be in. However, having batteries fully charged is independent of how fast they get charged. If the power is going out so frequently that the only way to fully charge the batteries between outages is by having a quick charge battery, then I'll agree with you.
 

Re:

[DerekLyons]

Ah, yes. Seeking to maximize uptime, system reliability, and performance predictabilty is a symptom that I don't think. You're an idiot.

Re:

[nacturation]

Ah, yes. Seeking to maximize uptime, system reliability, and performance predictabilty is a symptom that I don't think. You're an idiot.

Actually you did think of that. What it appears you're missing is that a slow charging battery achieves the exact same characteristics as a quick charge battery. Both will be fully charged at the point they're needed. And if slow charge and quick charge batteries deliver the exact same performance characteristics and longevity with the only difference being charge time, then of course you'd be right... go with quick charge. Since I'm not a telecom engineer nor do I design batteries, I'll shut up now. :

Re:

[ScrappyLaptop]

Think about what you are saying. Think about your boss. Think about your boss making the decision of which batteries to purchase. Consider that he knows next to nothing about technology. Consider that he knows everyone under him knows *everything* about technology. Now tell me which ones he'll choose: The old-fashioned, slow ones or the faster ones labeled as 'advanced'? C'mon, everyone knows that when it comes to technology, faster is *always* better. The marketing folks (with whom your boss identifies mo

Re:

[nacturation]

Think about what you are saying. Think about your boss. Think about your boss making the decision of which batteries to purchase. Consider that he knows next to nothing about technology. Consider that he knows everyone under him knows *everything* about technology. Now tell me which ones he'll choose: The old-fashioned, slow ones or the faster ones labeled as 'advanced'? C'mon, everyone knows that when it comes to technology, faster is *always* better. The marketing folks (with whom your boss identifies more than with you) say so!

Lucky for me, my boss isn't an idiot and appreciates solid engineering. Let's see... quick charge, but reduced cycles meaning more frequent replacements, higher cost, and decreased output which overall increases total cost of ownership. Or, slower charge, but longer life, lower cost, and increased output which reduces total cost of ownership. Either way, the batteries will have a full charge in the event of an outage.

Re:

[macdaddy]

Think about what you're saying. These are backup batteries and will ideally never get used.

The batteries are not just for backup purposes. They are in service 24/7/365. The cabinets themselves run off of -48VDC provided by the string of batteries. The cabinets do not run off of VAC or unregulated VDC. Therefore the batteries are always in production, not just when you VAC input is unavailable. I work for a telco.

Re:But... why?

[dgatwood]

Any time you store that much power in a confined space, you're taking a risk. If anything shorts out one of those batteries, whatever shorts it out will go up in flames because the battery dumps so much current through it. Usually, these fires are caused by an internal short triggered by dendrite growth, random metal fragments, impact, or chemical breakdown of the separator. When this happens, the little bit of shorting metal gets extremely hot. This starts a chain reaction, known as a thermal runaway in which the increase in temperature causes an increase in the chemical reactions in the battery, which, in turn, causes an increase in temperature.

http://ecsmeet3.peerx-press.org/ms_files/ecsmeet3/2007/01/03/00002421/00/2421_0_art_0_jbbdol.pdf [peerx-press.org]
http://en.wikipedia.org/wiki/Lithium_ion_battery [wikipedia.org]

Re:But... why?

[Locutus]

some have said that hot summer days were helping with the overheating of the batteries and the explosions. But, if you notice, the last explosion happened in late December in Wisconsin. Not likely to have been a very "hot" day. It could still be a heat issue if there's internal overheating do to "other" internal computer equipment, power supplies, and other equipment inside those boxes.

Don't expect AT&T to tell anybody what's really going on. After the dozens of images and stories that went out after the first explosion, AT&T is on top of these blown-up systems like white on rice.

Another interesting detail is that the company hired to examine the systems after the first two explosions said that the batteries and safety equipment were sound. They also said that they were likely better than most other batteries on the market. If this is the case, AT&T is going to have to start putting very large warning stickers on these boxes as they keep exploding. Maybe something like this:
http://www.flickr.com/photos/pluckytree/2186452007/in/pool-stickfiguresinperil/ [flickr.com]

or a version of this:
http://www.flickr.com/photos/blackloveinspace/494802125/in/pool-stickfiguresinperil/ [flickr.com]

LoB

Re:

[rjune]

I live in the Milwaukee area and there was absolutely nothing about this in the local news. We subscribe to the local newspaper (Journal-Sentinal), visit the Journal-Sentinal website on a regular basis, and listen to the radio. If the local media is ignoring the story, why should AT&T talk about it. By the way, guess who bought a boatload of advertising on the above mentioned local media to lobby for a change in Wisconsin cable law? The new law allows AT&T to bypass local municipalities to get a

Re:

[budgenator]

The Lithium itself is nasty stuff, but it's necessary to get the power densities needed, something that powerful is that powerful. The alternative is two have somebody try and call 911 and have a dead battery on the phone system.

Re:

[cheater512]

Why would they use lithium?
Here in Australia I believe they use lead acid.

Makes far more sense. Lithium batteries lose charge over time regardless of usage.

Re:

[budgenator]

The batteries go in metal boxes scattered around most towns and are about 1M wide, 2M tall and about 2m long and they contain equipment to convert from optical fiber to copper for voice and DSL. Most people in the US live too far from the central office for DSL so the teleco's made mini-CO. The enviro-Nazi's would have a field day over lead-acid screaming "OOOHH, heavy metal, poison the kiddies, think of the children" and the "proper disposal" fees might make them more expensive per watt/hr of capacity.

Re:

[cheater512]

Are you serious or is that just speculation?

If your serious then Wow! God Americans are stupid. ;)
(Enviro-Nazis should realize that their cars use lead acid)

And btw Australia has the same system where every suburb or so has a telephone exchange.
I dont think they have batteries though. The small telephone buildings have the lead acid batteries.
They send the power to the exchanges.

If they use Lithium Ion then they need to be replaced every 2 - 3 years.
Lead acid can go for decades in good conditions (which exc

Re:

[budgenator]

Are you serious or is that just speculation? Both, but these things are just huge ugly metal cabinets outside where they get too hot in the summer and too cold in the winter.
If your serious then Wow! God Americans are stupid. ;) yes indeed we've had some towns ban dihydrogen monoxide as a hazardous chemical!

Re:But... why?

[Ignis Flatus]

several years ago, i had a job where we installed batteries in telecom shelters for Qwest, AT&T, etc. it was lead-acid batteries at the time. some of the guys were working on production, but even on hourly pay, there's still a lot of pressure to do the work in a timely manner. and invariably, the things get busted up. it may have happened during shipping, or during assembly. maybe the guys putting putting the connections on overtorqued them. maybe it was even a factory defect. but whoever is at fault, it inevitably happens. with lead-acid batteries, it's often pretty obvious. you get a crack in a case, and the thing leaks all over the floor. we always carried baking soda with us for this reason. maybe with a lithium battery, the defects are just not immediately evident. but from what i've heard wrt to lithium batteries on hybrid vehicles, lithium is just hard to deal with. too much energy density makes for more mishaps, i guess.

not a mention of the recycling details

[Locutus]

I wonder if anybody is watching to see what they do with all these batteries?

LoB

Re:not a mention of the recycling details

[calebt3]

what they do with all these batteries?

Sell them to Sony.

Re:

[calebt3]

No. These batteries came from a company called Avestor. Unless that is the name of the brand under which Sony sells batteries.

Re:

[lhaeh]

I seem to recall reading a National Geographic article about kids in Mexico employed smashing open lead-acid batteries with hammers to extract the lead. I'm sure there are greener ways of recycling them, but the heaps of e-trash in Gungdong leads me to think a lot of old batteries end up there for that kind of recycling. Since these batteries are still good, I can see them ending up in a surplus auction. I tend to stock my UPS with lightly used surplus batteries and they are good for nearly as long at a fra

Re:

[ScrewMaster]

I wonder if anybody is watching to see what they do with all these batteries?

Probably send them to Iraq. Just let the terrorists steal them to power their equipment.

There's nothing good hearted about this

[ip_freely_2000]

Call me a cynic, but I'm sure they put a formula into a spreadsheet and discovered the liability issues outweighed the "do nothing" option. I'm sure there's a Ford Pinto kind of memo on a AT&T server somewhere.

Re:

[calebt3]

Your a Cynic [wikipedia.org]

Re:

[calebt3]

Ghaa! You're, not Your.

Re:

[Anonymous Coward]

Narrator: A new car built by my company leaves somewhere traveling at 60 mph. The rear differential locks up. The car crashes and burns with everyone trapped inside. Now, should we initiate a recall? Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one.
Business woman on plane: Are there a lot of these kinds of accidents?
Narrator: You wou

Re:

[petermgreen]

which works fine until one of the cases actually hits court and this behaviour is disconvered during the case.......

Re: Fight Club quote updated

[NFN_NLN]

which works fine until one of the cases actually hits court and this behaviour is disconvered during the case.......

Take the number of [soldiers] in [Iraq], A, multiply by the probable rate of [deaths], B, multiply by the average [veteran pay to the family], C. A times B times C equals X. If X is less than the cost of [buying proper bullet proof jackets and light armor], we don't do one.

Nope, looks like it doesn't matter even if people know.

NFN_NLN

Re:

[p0tat03]

Easy... estimate the dollar value of the public relations disaster you will face, times the probability of being discovered... if it's STILL less than the cost of recall...

Re:

[zippthorne]

Ah, but is this really an immoral policy?

If your ethical system involves doing the most good for the greatest number of people, you're going to have to do a lot of calculations like that. I'd say, depending on how the settlement process works, it's at worst, amoral. At best, it's actually the most ethical course of action possible. Coldly, calculatingly ethical.

Re:

[maxume]

The other way of looking at it is that it means the legal system is more or less working...

Re:

[VTMarik]

Still, it's a rare occasion when a company goes against the advice of a study that says "Everything is fine" instead of insisting that people trust the study and not worry about the potential explosive in their devices.

Sure, it's cheaper to swap it out then pay a lawsuit but when has that stopped a large company? It was cheaper to make sure the Ford Pinto didn't explode than to lose millions in sales when people just stopped buying them.

Re:

[DerekLyons]

It's not cynical - it's just recognition of good business practice.

Re:

[ScrewMaster]

That and earning a lot of good PR. Most of us don't mind so much when companies make mistakes (we all recognize that, well, sometimes shit happens.) It is when a corporation goes into deny/ignore mode that we get pissed off. This will cost AT&T a bit of cash, sure, but they deserve that for buying crap batteries in the first place. Furthermore, imagine the hot water they'd be in if this started happening again. Somebody upstairs decided not to take the risk.

Smart move.

Re:

[iminplaya]

Fuck Ford. I'm buying Japanese next time.

Yeah! Get a Mazda [ezinearticles.com]

Re:

[EmagGeek]

I can think of several reasons not related to civil liability to go ahead and replace the batteries

1) The cost of the VRAD equipment is astronomical when you consider the labor of sorting out hundreds upon hundreds of burnt copper and fiber cables and replacing them at union labor rates
2) The cost of losing customers due to the unreliability caused by failing VRAD modules
3) The indirect cost of investors worrying about future revenue, depressing share price

The batteries from Avestor were NOT cheap, as has b

They're getting 17,000 new batteries?

[Chris Mattern]

No wonder my drugstore is out of AAs!

Chris Mattern

Achmed the Terrorist Energizer Bunny

[Anonymous Coward]

Come on, you all want to see the commercial where the voiceover says: "Energizer Bunny keeps going and going and go- KABOOM! For fuck's sake, can't we find a bunny who hasn't turned suicide bomber?! Didn't airport security check the interior of his drum?"

Even better would be Achmed the Dead Terrorist dressed up in an Energizer Bunny outfit. "Hi, I am Achmed. I keep going and going and... I KEEEEEIIILL YOU!!". If you haven't yet seen it, check out Jeff Dunham's Achmed the Dead Terrorist act [youtube.com]. IMO, it's abso

Another sign of VRAD overheating observed?

[Lauren Weinstein]

Out here in the West San Fernando Valley (Los Angeles) -- and it can get *very* hot here during the summer -- where AT&T has widely deployed VRAD cabinets (though U-verse is not activated here yet), I've recently noticed another sign of possible overheating problems. On the side of the VRAD cabinets is apparently a large air intake with an exposed filter element. On several units I've observed recently, the filter element first vanished completely, and then was replaced shortly thereafter with what appear to be rather bulky external fan units. Interesting. --Lauren--

This is exactly why it is a bad idea

[neapolitan]

to offer 'integrated' [slashdot.org] batteries into devices.

Imagine what a recall of the iPod or Macbook Air battery would do to Apple's share price.

(Now smile to yourself, quietly.)

New Battery technology

[KD7JZ]

The Avestor batteries were a new innovation for powering telecommunications network elements. They were a lithium based chemistry (LiIon) instead of traditional lead acid gel cells. The advantage was supposed to be about twice the service life (6-8 years) instead of the traditional 3 to 5 years. Lithium is a reactive element and if they were having a problem where an internal short was causing thermal runaway, you have a reactive element combined with heat. Not a good thing...

Re:

[afidel]

Hmm, I wonder where I can get some of the pulled batteries. Stuff like that usually goes for a penny on the dollar or less. I know it would be plenty cool in my basement and would do wonders to extend the life of my Matrix 5000 =)

More info on WI explosion

[bigbadunix]

Being from Wisconsin, and not hearing about this, I did a little search and found this [freepress.net].

Funny thing is, it was a less than a mile from my house, and probably a week after 2 friends of mine in the area signed up for u*verse.

The article does a good job basically re-iterating TFA with a bit more detail.

Re:

[scottv67]

Funny thing is, it was a less than a mile from my house, and probably a week after 2 friends of mine in the area signed up for u*verse.

I hope you live a mile west or a mile south of 64th and North. A mile east would put you in the ghet-to. And a mile north of there ain't such a sweet place to live either. Can you post some photos of the new cabinet that ATT installed at that location? You mentioned that some of your friends ordered uverse. Are they happy with uverse? I have seen a million white ATT

Placing shields over a 2 meter vent...

[matrixghost1286]

Looks like the Empire decided to take preventative measures to ensure power generation.

I guess the Rebels will have to find a new way to blow up the Death Star.

Avestor went under in 2006

[EmagGeek]

I wonder why...

Needless to say, AT&T will not be able to recover anything from them. Hopefully none of the NiCd replacements explode - being incredibly toxic to the environment and all...