Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
×
Power United States News

Texas Was 'Seconds and Minutes' Away From Catastrophic Monthslong Blackouts, Officials Say (texastribune.org) 384

Texas' power grid was "seconds and minutes" away from a catastrophic failure that could have left Texans in the dark for months, officials with the entity that operates the grid said yesterday. Texas Tribune reports: As millions of customers throughout the state begin to have power restored after days of massive blackouts, officials with the Electric Reliability Council of Texas, or ERCOT, which operates the power grid that covers most of the state, said Texas was dangerously close to a worst-case scenario: uncontrolled blackouts across the state. The quick decision that grid operators made in the early hours of Monday morning to begin what was intended to be rolling blackouts -- but lasted days for millions of Texans -- occurred because operators were seeing warning signs that massive amounts of energy supply was dropping off the grid. As natural gas fired plants, utility scale wind power and coal plants tripped offline due to the extreme cold brought by the winter storm, the amount of power supplied to the grid to be distributed across the state fell rapidly. At the same time, demand was increasing as consumers and businesses turned up the heat and stayed inside to avoid the weather.

"It needed to be addressed immediately," said Bill Magness, president of ERCOT. "It was seconds and minutes [from possible failure] given the amount of generation that was coming off the system." Grid operators had to act quickly to cut the amount of power distributed, Magness said, because if they had waited, "then what happens in that next minute might be that three more [power generation] units come offline, and then you're sunk." Magness said on Wednesday that if operators had not acted in that moment, the state could have suffered blackouts that "could have occurred for months," and left Texas in an "indeterminately long" crisis. While generators rapidly dropped off the grid as the weather worsened, operators monitored the difference between the supply of power on the grid and the demand for that power. As supply dwindled and demand grew, the margin narrowed to more and more dangerous levels, forcing grid operators to enact emergency protocols to either increase supply or decrease demand.
Further reading: Texas Leaders Ignored Warnings A Decade Ago That Their Power Supply Was In Danger
This discussion has been archived. No new comments can be posted.

Texas Was 'Seconds and Minutes' Away From Catastrophic Monthslong Blackouts, Officials Say

Comments Filter:
  • Um, thanks (Score:5, Insightful)

    by maladroit ( 71511 ) on Friday February 19, 2021 @11:06PM (#61082070) Homepage

    ERCOT:
    We heroically averted the crisis that we created!
    Congratulate us!

    • Re: Um, thanks (Score:5, Insightful)

      by Your Father ( 6755166 ) on Friday February 19, 2021 @11:10PM (#61082078)
      To be accurate, deregulation is what fucked them. Regulation - even with privately owned utilities - ensures both a margin (which was the Texas issue that led to deregulation) and a reliability guarantee (which leads to the current Texas issue).
      • Re: Um, thanks (Score:5, Insightful)

        by AlexHilbertRyan ( 7255798 ) on Saturday February 20, 2021 @12:42AM (#61082288)
        Nonsense, deregulation is the american way, with regulation the ceos cant maximize profits and fucked be everyone else. This is a total success, im sure some ceo somewhere made a bonus and thats what counts. The american dream of paying bonuses and fucking everyone else must be kept alive, think of the CEOs and their mega millions.
        Havent you been paying attention for the past 200. years ?

        Didnt the American civil war teach you anything ?
      • Re: Um, thanks (Score:4, Informative)

        by minorproblem ( 891991 ) on Saturday February 20, 2021 @08:28AM (#61083008)

        There is nothing wrong with deregulation except you need a strong energy regulator who enforces minimum standards on the system operator and generators, this way liability can be enforced for anyone not meeting their connection conditions.

        ERCOT should have a system quality and supply standards (SCSS) and manage the grid to that standard. An example is ROCOF requirements (rate of change of frequency), if you are approaching the limit such as in an event like this, they should begin scaling back generators and bringing additional generators online if possible. That way when you have a large frequency swing your automatic load shedding kicks in, plus the generators that are still running can use their limited frequency mode settings (LFSM) to ramp up and meet the difference. All of these things require a competent system operator however.

    • "seconds and minutes" away

      What sort of English is that?

      "Seconds away from...."? Ok.
      "Minutes away from...."? Ok.

      "Seconds and winutes away from...."? Only a PHB would say that.

    • A lecce supply grid is ALWAYS seconds or minutes away from catastrophe. That is how it works. The more it is optimized for efficiency, the smaller the reserve becomes.
  • 3rd world (Score:5, Interesting)

    by Anonymous Coward on Friday February 19, 2021 @11:21PM (#61082110)

    The more I see, the more news, the more the US is presented as a third-world nation.

    - State-wide power outtages. A general inability to manage the power grid.
    - Hacker getting into water supply, setting chemical levels to toxic levels (fixed by operator)
    - Worst health crisis of any(?) country
    - Utterly corrupt leaders, every time
    - No, or terrible, public transit
    - Isolationism

    I don't know. Those are just off the top of my head.

    • well now trump, rush limbaugh, etc are gone the USA can go back to being not so for profit.

    • Re:3rd world (Score:5, Insightful)

      by Powercntrl ( 458442 ) on Friday February 19, 2021 @11:43PM (#61082148) Homepage

      Don't forget our terrible for-profit healthcare system, even when we weren't in the middle of a global pandemic.
      Shitty broadband speeds and telecom monopolies.
      Crumbling roads.
      Stagnant wages.

      America may not be a 3rd world country by the literal definition, but we're not far off.

      • Re: (Score:2, Offtopic)

        America has more homeless in the average town than all of australia. 90% of Australians have more holidays than 10% of americans.
  • Texas will be saved by all the hot air from the politicians and think tanks spouting: "small government", "keep the feds away from our grid" and "be self-reliant" tropes.

    • I wonder if this was a bellwether event though. Now that the freezing weather will be blown past in the next day or so, many will be spending a lot of money to handle broken pipes, recover their businesses, write their stories, and get over the shock of what happened. Even though the Texas former governor mentioned that it is better to be without power than to have power under a socialist system, not many are buying that. Even a Texas Mayor, Tim Boyd, was punted out of office when even the most conservat

      • I wonder if this was a bellwether event though.

        No. This same thing happened 10 years ago and we've seen how much change took place since then.

        As former governor Rick Perry said, Texans would rather freeze than have to follow federal government regulations. And he got his wish [newsweek.com].

  • Profitability (Score:5, Interesting)

    by burtosis ( 1124179 ) on Friday February 19, 2021 @11:31PM (#61082124)
    The up front costs to design gas pipelines, coal plants, wind farms, nuclear reactors, etc... that can withstand sustained freezing temperatures never seem to be spent when weather like this in Texas is uncommon but guaranteed to happen every few years to a decade. They don’t lose much revenue worst case and will never see a fine or repercussions. From someone in the upper Midwest, I can assure you that coal, nuclear, natural gas, and wind turbines are unfazed by a month below 0F if their profits rely on it.
    • they were able to keep the grid going for most of the wealthier neighborhoods (and the real well off have their own generators and plenty of fuel).

      All you really have to do is spread the blackouts in districts that you don't need to win elections. Easy peasy when the governor gets to appoint the heads of the quasi private/public 501c that oversees everything (ERCOT).
    • The funny thing is that the costs are likely not that much. Most places in the world have pipelines, coal plants, etc., all running without issue even in very cold temperatures. This is definitely a solved problem. There are rules for 100 year floods, it makes sense to have rules for 100 year freezes.

    • They don't lose much revenue worst case and will never see a fine or repercussions.

      Bonuses all round if they build a small proportion of the infrastructure to withstand this type of weather, while a significant proportion goes offline. Price spikes are amazing, if you can sell small amounts of electricity into the pool when the prices are 100x normal prices.

      The same people manging these plants learned from Enron: cause a shortage and profit from the price spike.

  • by Camel Pilot ( 78781 ) on Friday February 19, 2021 @11:33PM (#61082128) Homepage Journal

    Can anyone shed light on how this could have resulted in blackouts for months? I mean this wasn't like Fukushima where you needed emergency generators to pump water to an atomic pile.

    'if operators had not acted in that moment, the state could have suffered blackouts that "could have occurred for months,"'

    Sounds like someone is trying to find a way to create a new narrative where the grid operators are the heroes in this story. Maybe the post-truth era is still a thing in Texas.

    • The nuclear plant was never in danger. They just weren’t able to get more coolant water as the intake froze up. It just shuts down instead of having to spend a trivial amount of money to let it work full time cause that’s more profitable.
      • by Rei ( 128717 )

        I'm informed by a person who works at a different nuclear power plant in Texas that the coolant intake never actually froze up, it was a false sensor reading. I suspect (though have no evidence) that it might have been a dead leg sensor, where the sensor doesn't lie in the main flow but in a little dead-end branching path off of it. That would be much more vulnerable to freezing.

    • by Baloroth ( 2370816 ) on Friday February 19, 2021 @11:59PM (#61082190)

      I've not doubt they are stroking their own ego to look like heroes and shift blame, but there is probably some truth to what they say. Even in the best case scenario, restarting an electric grid from scratch is hard: plants need to be frequency synchronized, not all generators can cold-start with no external energy source, etc. But worse is the root problem: Texas plants were being damaged by the cold weather. A generator shuts down, it stops producing heat, and cold weather damage gets much worse. It might have gone from some natural gas plants failing to most of them failing. After that, it's months (or years) of repairs before the grid gets fully back online again.

      • Some 20 years ago there was a huge blackout in the NE USA that lasted several days. A tree had it a high tension power line.

        So what? you might think. Some area will go black for a few hours while they remove the offending tree.

        But it took them several days to get the power back on line. The reason was never clear to me. But I suspect really archaic control systems that prevented them from just bringing things back bit by bit and quickly.

        So there are problems beyond Texas.

        Anyway, the good news is that wi

      • by sjames ( 1099 )

        Synchronizing is necessary but not that hard. Modern plants can do it automatically. In the old days, farmers would synchronize a pair of generators using 2 light bulbs in series connected between the hot lines of the generators, then tweak the throttles until the bulbs were steady off.

      • by fgouget ( 925644 )

        I've not doubt they are stroking their own ego to look like heroes and shift blame, but there is probably some truth to what they say.

        You have to distinguish the people managing the grid in real time in the control room. They are probably competent and did the best they could given the circumstances. And then you have the C-level executives "managing" the grid by deciding how much should be invested in maintenance and weatherization, and those did a poor job. Now the latter are trying to attribute all the heroics of the former to themselves.

    • by ka9dgx ( 72702 ) on Saturday February 20, 2021 @12:47AM (#61082304) Homepage Journal

      When the grid goes down, a "Black Start" is required to get things going again. This is tricky, but can be done. If transformers experience overcurrent and fail, they can take YEARS to replace, but most have sufficient protection to prevent that from happening.

      You've no doubt seen what happens when pipes freeze in a house? Imagine this happening in a generating station with pipes more than a foot across everywhere.

      Electrical generators aren't allowed to stop turning, they have a "bull gear" to keep them turning, so they don't sag and bend, which would lead to vibration and potentially self destruction once they are turned on.

      There are a huge amount of things that have to work just right in order for power to be generated safely and profitably... a few days of freezing could break things that make that impossible.

      • What I find surprising is that a nuclear plant can't black start itself. Even in full scram, there is likely enough heat being produced which could muster enough energy to get the cooling system and other critical things back online.

        • by sjames ( 1099 )

          Interestingly, the original objective at Chernobyl was an experiment to see if the residual energy in the system would be sufficient to shut the plant down in an orderly manner should it have to scram without grid support.

        • What I find surprising is that a nuclear plant can't black start itself. Even in full scram, there is likely enough heat being produced which could muster enough energy to get the cooling system and other critical things back online.

          I'm afraid that no, they cannot black-start themselves. The power you could extract, even if there was some way to do it, from the core when shut down, isn't enough to run the cooling systems let alone all the ancillary equipment required to run the core at higher power (and raise it to higher power. Once running a nuclear plant can power itself, but they can't self-start. It takes around 50-100MW of power to run all the gear in a nuclear power station, and most of that is needed before you get any power ou

        • by LostMyAccount ( 5587552 ) on Saturday February 20, 2021 @07:56AM (#61082944)

          25 years ago I toured the Union Electric plant in Keokuk, Iowa, one of the oldest hydro plants in the US. At the time they still transmitted DC power to a lead smelter north of St. Louis.

          The tour was great because I was the only one on it and the tour guide was more or less just a plant guy. I got to go inside one of the spinning turbines and a bunch of other places I never expected. The water was high at the time or he said we could have walked out on the service walkway on the dam itself.

          He also showed me this hand crank about the size of a bicycle wheel, attached to a little generator. He said this was the "manual start" for the whole plant. I guess the generators used electromagnets, and the hand crank generator could excite a field for one turbine, and then that turbine would be used to excite the fields for the rest of them.

        • by eastlight_jim ( 1070084 ) on Saturday February 20, 2021 @08:26AM (#61083000)

          It's surprising how much power is used by a nuclear plant's systems. The status page [edfenergy.com] for the UK's nuclear fleet gives some indications.

          In summary, a planned shutdown (i.e. minimum requirement for a reactor) is about 14-16MW, and unplanned shutdowns (e.g. some types of trip) can have consumptions up to 30-40MW. I'd be suprised if this can be generated on-site without dedicated generators - something that would be uneconomical for all plants to have, even in a more heavily regulated system like the UK's.

    • by nicolaiplum ( 169077 ) on Saturday February 20, 2021 @05:43AM (#61082768)

      It's in the article, part way down:

      "The worst case scenario: Demand for power outstrips the supply of power generation available on the grid, causing equipment to catch fire, substations to blow and power lines to go down."

      "If the grid had gone totally offline, the physical damage to power infrastructure from overwhelming the grid could have taken months to repair..."

      If you have a grid of electricity lines and a lot of supply, especially supply in one place, goes offline then the grid may not be able to carry power from the remaining supply to all consumers. If any major transmission lines trip off then lesser transmission lines can easily become overloaded - there has been more than one cascading grid failure in the world because of this.

      If these overloads, or misbalanced loads, burned out any major switchgear or transformers then it would take months to replace them. The major components of large electricity substations and switching yards are made to order with long (months) lead times. A Congressional Report Service publication "Physical Security of the U.S. Power Grid: High-Voltage Transformer Substations" describes how they are custom-made and have long lead times: https://fas.org/sgp/crs/homese... [fas.org] If an HV transformer substation is damaged for any reason, it takes the same time to replace.

      Restoring large power lines damaged by anything, including overload, also takes time. It took many weeks to replace - not repair, much of them had to be replaced - the electricity transmission lines from Quebec to the US North-East after the 1998 ice storm. While the storm in Texas did not seriously damage power lines, had there been widespread by overloading (due to unbalanced grid supply and demand) then it would also have taken weeks to rebuild them.

      Major electricity plants have rotating parts, turbines and generators with complex magnetic fields that are only built to operate at certain frequencies (and therefore rotational speed). Overload causes overcurrent in the generator windings and problems with back-EMF as the generator slows down. If you burn out a power station generator, it takes months to get another one.

      The danger is real.

  • by Mspangler ( 770054 ) on Friday February 19, 2021 @11:34PM (#61082132)

    What do they have that can startup from nothing? As in no excitation power?

    The fine article was sort of doomsday oriented, if voltage drops too far the breakers are supposed to open before equipment is damaged. But restarting a completely dead grid is an issue.

    The boat had a battery sized to restart a shutdown and cold reactor. Hydro dams have a generator that can start first without power. Then you can bring up the rest when you have something to synchronize to. What does Texas have?

  • Its seems very strange that overloads would cause widespread physical damage rather than a cascade of shutdowns that could be brought back in a realatively short time. Are our electrical grids that easy to damage?
    • by K. S. Kyosuke ( 729550 ) on Saturday February 20, 2021 @12:18AM (#61082250)

      Are our electrical grids that easy to damage?

      You sure you want to hear the answer to that question? ;)

    • by dryeo ( 100693 )

      They're not that easy to bring online, especially if the heat goes out and the pipes freeze and burst. Even without that, it takes power to start a generator, actually they're alternators and it can be tricky synchronizing the frequency as they come back online.

    • by mysidia ( 191772 )

      very strange that overloads would cause widespread physical damage

      A rapid enough change in load above supply
      can occur faster than protection devices can react (Especially circuit-breakers or mechanical
      switches which are slow) is bound to cause minor damage in some cases - result to further reduce supply which can then spiral.

      Minor issues cannot necessarily be repaired instantly and might take weeks - for example certain expensive high amp value fuses or other electrical components (the protection

    • by whoever57 ( 658626 ) on Saturday February 20, 2021 @02:11AM (#61082382) Journal

      Are our electrical grids that easy to damage?

      They are in Texas, with insufficient winterization and insufficient interconnections to other grids that might have kept the lights on.

      All because they hate Federal regulation, apparently. Probably because that regulation would have prevented the massive profiteering that has been going on in Texas. Can't let ordinary people's lives and livelihoods get in the way of big corporations' profits. The sad part about that last statement is that it's not sarcastic: Texas politicians have openly expressed that sentiment.

    • Part of the problem is that the safety trip systems that protect the equipment, also make it very difficult to bring a tripped system back online, since it can trip again, or cause further cascading trips.
  • Hold On.... (Score:5, Informative)

    by Luthair ( 847766 ) on Friday February 19, 2021 @11:52PM (#61082174)
    Weren't we all told mere months ago that Texas was a panacea everyone was fleeing to?
  • Texas should have been able to source power from other areas of the country, right — it's a "grid", right? Well, that turns out not to be the case, because... Texas. There are three grids in the U.S.: the Western Interconnection, the Eastern Interconnection, and (yes, I set this up for you... wait for it...) the Texas Interconnection. Texas decided it was just fine to be its own grid, thankyouverymuch. And here we are. Quick reference: https://www.eia.gov/energyexpl... [eia.gov]

  • Damn those Russians and Chinese hackers, is there anything they cant do!
  • by h33t l4x0r ( 4107715 ) on Saturday February 20, 2021 @01:25AM (#61082352)
    This is climate change fucking with people who don't believe in it. They should consider it a warning because it's going to keep happening.
  • Grid blackout (Score:4, Informative)

    by stanbrown ( 724448 ) on Saturday February 20, 2021 @03:16AM (#61082462) Homepage
    ERCOT is what is refereed to as an Independent System Operator (ISO). These ae entities whose job is to manage (among other things) manage the balance between real time load, and connected generation. It is necessary to have a small amount of "rolling reserve" to pick up any added load. If they fail to achieve this a situation can occur whereby the frequency collapses because connected load exceeds the connected generating capacity. there have been 2 incidents in the Northeast US where this goal was not achieved.When this unbalanced condition occurs you will get into a cascading failure situation whereby most generation units will trip off and the transmission structure will fragment. Once you are in this position, the recovery requires starting with generating stations that are "black start" capable. Most generation stations need some power from the grid to do things like run boiler fans etc. The recovery from this would take (at most) a day or so.
  • WHY would that cause a something that causes months long blackouts. And WHAT exscrly would cause a blackout to last longer than "until we switch power back on"?

    Also: First word country ... my ass. ^^

"I am, therefore I am." -- Akira

Working...