Database Glitch Grounds American/US Airways

An anonymous reader writes "According to numerous news sources, all American Airlines and US Airways flights were grounded for two or three hours this morning. Both problems were caused by a computer glitch in the systems hosted by EDS. Quote: The operating system that drives the airline's flight plans went down."

Re:Operating System (singular)

[Anonymous Coward]

use the 'yes' command in linux to take care of that.

Operating system?

[SlamMan]

Only one of the articles mention said anything about an "operating system." The rest called it a system problem. That does not necessarily mean an OS, or anything related to it. I think katu's reporter jumped to a conclusion.

Re:Great News!

[Vlad_the_Inhaler]

Typical Airline applications are Reservations, Check-In, Weight-and-Balance, Flight Planning (which route to take and how much fuel to carry) and Ticketing. Once you have left the terminal and are heading for the runway, software crashes cease to be relevant.

Once you head for the runway, you care about Air Traffic Control's software. The only exception I can think of is for flights to the US where the authorities want passenger lists.

I work for an airline and we host for other airlines. I feel sorry for whoever carries the can for this mess. As to the OS, those who said it will be MVS are almost certainly correct. AA and US Airways are/were IBM customers.

Re:Windows

[Anonymous Coward]

The following entities were NOT mentioned in the article you're linking to:

(1) American Airlines,
(2) US Airways,
(3) EDS.

So, what the hell are you talking about?
Why did you link to this article?
(I know, I know, because nobody will read it anyway)

Re:Windows

[ChatHuant]

Sounds like a troll. The article quoted by the parent is about a small regional airline (Atlantic Coast Airlines) that's doing its IT work internally. The article doesn't mention EDS at all. Moreover, browsing EDS's site, you can see that the solution [eds.com] they implemented for Continental Airlines is UNIX-based.

Not Windows, Unix

[JohnQPublic]

This is undoubtedly a problem with Sabre, which EDS runs on behalf of Sabre Holdings. Both American Airlines and US Airways use Sabre for much of their operations.

Sabre started it's life as an American Airlines internal system (SABER, slight spelling difference), running on a rare operating system (PARS, later called ACP and currently TPF) on IBM mainframes. In the last few years Sabre completed a lengthy migration to HP Unix on Non-Stop (i.e. ex-Tandem) hardware. The mainframe systems were rock solid, but software talent was hard to come by, so they decided the time had come to switch.

Sorry, no Microsoft to blame here!

Re:Operating System (singular)

[Vlad_the_Inhaler]

This is partially a question of cost, redundancy costs money and those airlines are rather short of the readies (although this crash will cost serious money).

For any *normal* 'extreme situation', a reboot should help.

Having just read that The operating system that drives the airline's flight plans went down, it might even be a Windows problem. A 'Flight Planning' application is a low volume application where you work out the optimum route for a plane based on the weather. That bit about the weather involves serious number crunching and the PC world has more of that kind of power to spare than the mainframe world. I helped write one of these apps 20-18 years ago and the central part has since been converted to run on PCs.

You probably won't hear it

[Anonymous Coward]

The systems that run the aircraft and the navigational and communication systems really are redundant. It's the law. It also means that usually there are two different ways to do something not just the same thing repeated twice.

Example 1 - The pilot and co-pilot can't eat the same meal. That way, only one of them can get food poisoning.

Example 2 - The hydraulic system fails and the wheels won't go down. There's a hand crank.

Example 3 - The communication systems at every tower I have worked at have two separate backbones. There are two of absolutely everything. If that fails, there are emergency radios under the desk. If the emergency radios don't work ... We used to joke that the controllers would climb to the top of the tower and wave fire extinguishers to warn the planes away. (I think it was a joke.)

Example 4 - You can't fly very far over open water in a single engine aircraft.

It used to be frustrating working on systems older than I was but we never had to worry about surprises.

Of course all of this redundancy is very expensive. You spend the money where people's lives are at stake. On the other hand, if the worst problem is that some planes will be late, perhaps you don't spend the big bucks.

Re:My guess ...

[sysjkb]

Sounds very unlikely to me. You will find weird custom S/360 derivatives in places like the Space Shuttle, but coordination and route planning doesn't sound a likely place for one.

Of the 360-based operating systems, IBM's TPF [ibm.com] has a major presence in the airline industry, but this probably isn't the system in question. TPF tends to handle ticketing and reservations. TPF stands for the Transaction Processing Facility; it's the descendant of the old Airline Control Program (ACP) developed for Sabre. Sabre in fact is still running TPF, although I believe they're busy transitioning away from the mainframe to Tandem's er I mean Compaq's er I mean HP's NonStop/UX.

Of course, it might not be an IBM mainframe at all; Unisys has a niche in the airline industry. But heck; given that this is route planning, just about anything from AIX to z/OS is a possibility. Even *shudder* Windows.

Re:EDS works with a variety of systems

[johnnyb]

Their reservation system is on Vax/VMS, if I remember correctly. I used to work in their midrange department, but I knew some of the VMS coverage guys. They have quite a diverse setup. The only operating system I _didn't_ see there was HPUX.

They actually have a data center that is underground, and has a retinal scanner to get in (for some reason, our group got in with keycards - I'm not really sure why). Their tape library is about three times the size of my house. It's a pretty massive operation. Travelocity, hosted in the same location (but on the ground floor, not downstairs), is a bunch of huge SGI machines (8 processors and more each - probably about 30 of them).

They run pretty much everything under the sun. I enjoyed being around the cool equipment while I was there, but absolutely hated the "big company" mentality, so I left after a year.

MOD PARENT DOWn

[WindBourne]

Folks, Not only does the link say nothing about windows, but AMR's flight res system (SABRE) is located in Tulsa in a silo and is absolutely not Windows(IIRC, IBM mainframes). Now, it is more likely not their flight system but some immeadiary system. While it is likely to be Windows (based on past history), there is so far no comfirmation of that. In addition, historically, AMR did not run windows as it was too expensive and too prone to crashes. Of course, that was when R. Crandel was there, which was a while ago.

Re:doesn't it say

[moby]

By the way, when's the last time you saw Microsoft.com go down?

Uhh ... like 2 days ago.

http://www.informationweek.com/story/showArticle.j html?articleID=12808118 [informationweek.com]

Sabre

[Anonymous Coward]

This is Sabre. EDS was contracted to run the Data Center. The shop has been mostly TPF/VM/MVS(ZOS) for some time, but they announced a couple of years back that they were going to shift most of the work onto servers (not sure what operating system. I do know that they were from around 20? mainframes(these are now the size of large refridgerators and IBM likes calling them servers) to about 4,000 servers. I am not sure how far they have gotten on this. As for this knocking out a couple of major carriers, I'm not surprised. Most of the domestic carriers are handled in 3 data processing centers, and lately those centers have gone down hill. I know at Galileo, they outsourced to IBM who are doing a very lackluster job of running it. So stand by for more of these type of outages.

You're not gonna DIE

[Old Telco Guy]

This [nasa.gov], ladies and gentlemen, is a flight plan. Now how the hell you gonna die because some FAA form can't get filled out right? All it was was a paperwork requirement. Planes still fly, pilots still know how to land them rubber side down.

Not "OS"

[Master of Transhuman]

When they said "operating system", they meant "operations system" - not the OS.

See this quote from one of the articles:

Wagner said a database malfunctioned that "basically runs every aspect of our client operations -- aircraft dispatch, crew scheduling (and) reporting weight, passenger load, balance."

This system is hosted by EDS, who only said it was a "systems issue".

So there's no evidence it was an OS problem. It could have been anything - OS, Oracle/DB2/SQL Server database, application code, upgrade, whatever.

Nothing to conclude here except that somebody screwed up - and even that isn't certain - could have been a bad memory board someplace, who knows.

Not having a backup is even irrelevant, since the "backup" might have taken three hours to bring up, when you're dealing with a production system like this. "Failover" is what you want, and they should have had, but if something got screwed there, it could still have been three hours.

Shouldn't have happened, but crap like this happens all the time because nobody can do their damn jobs.

More info about Sabre than you ever wanted...

[airbatica]

Sabre is a multitude of software products, for lack of a better definition. They include RES, DECS, TIM, BMAS and a couple of others that I can't remember.

All Sabre applications are text mode, no GUI whatsoever... think CLI from hell, with no command history if you fat finger an entry.

The system that went down was probably DECS (Dispatch Environment Control System), which is the system used by both American and USAir for generating flight plans, load planning, weight and balance, and various other flight operations functions.

RES is the Reservations system, which covers the spectrum from building reservations and selling tickets, to customer checkin, boarding and god knows what else. IIRC, it will even do car rentals and hotels.

TIM is also called Timatic. Its used for accessing information from the US State Department regarding internation travel to any country, from any country in the world. It covers entry and exit requirements, documentation, and pretty much anything you could want to know.

I don't remember what BMAS stands for, but it is a lost bag tracking and reporting system. When AA or US looses your luggage, this is what they use to find it.

Sabre is used by a whole variety of airlines and travel agencies, and is customised in modules to each particular user's needs.

Now you are probably wondering how I know all this... I work for a major airline that uses a majority of the systems listed above, with the exception of the Dispatch system. We were not affected by whatever snafu took down that portion of Sabre :)

Unfortunately, no

[DesScorp]

The airline's backend systems will continue to run on either old Tandem mainframes or port to new IBM mainframes (not running Linux, as of yet). Most of the airline's new IT investments are at the airport end.

Unfortunately, the Windows-everywhere trend seems to be winning here. My airport is going to a common-use terminal system, and it's Win2K based. All but one of the big common use vendors are selling Windows-based equipment. Northwest's CUSS (common use self service) terminals are Win2K based as well.

When I asked our vendor, who specializes in smaller airports, whether his company was doing any Linux development, he replied that nope, since most of the systems will never be on a public internet, it was easier and cheaper to get windows developers. No security concerns without the Internet, and 2K/XP/2K3 have become much more stable than older Windows platforms (his company still has older installations overseas that run NT 4 based systems, all due for an upgrade).

Re:Great News!

[keithcstone]

Unlikely the OS (which was TPF BTW) crashed at all. Title says "database curruption" not OS or hardware crash.