A Flicker Away From a Blackout

washingtonpost.com

A Flicker Away From a Blackout
Canadian Engineers Say Rare Glitch Suggests Ongoing Threat to Power
Grid
By Doug Struck
Washington Post Foreign Service
Tuesday, June 21, 2005; A15

TORONTO -- At 4:15 p.m. on May 27, the lights flickered across Ontario.

Subway cars in Toronto rolled to a stop while safety signals were
reset. Pizza oven doors flew open on the ground floor of the city's
landmark CN Tower. Cement and steel plants paused while machinery was
restarted. Tens of thousands of computers automatically shut down and
rebooted.

Hydro One, the Toronto-based electric utility, quickly issued a press
statement seeking to reassure the public that the utility's
"equipment protection worked as designed to isolate the fault."

In fact, the situation was much more tenuous. The power blip involved
an extremely rare, still unexplained failure of two protection
systems, according to internal documents of the utility, reports to
oversight agencies and eight engineers. The eight are part of a group
that has been on strike since June 6. By their accounts, the failure
brought the region's power grid to the verge of a blackout like the
one that struck on Aug. 14, 2003, plunging 50 million people in the
United States and Canada into darkness.

"It was very bizarre, very disconcerting. And we may have a major
problem on our hands," said engineer Aaron Cooperberg.


He and other engineers said that because they are striking, the
precise cause of the malfunction has not been identified, and they
say it could happen again. "Fundamentally, we had a very close call,
and we don't know why," said another engineer, one of three who asked
not to be identified, saying they feared retaliation by the company.

The electrical grid handled by Hydro One, which has 17,000 miles of
high-power electric transmission lines throughout Ontario, managed
the blip "very well," said Peter Gregg, vice president for corporate
communications at the utility. Much of its cause is still not
understood and under investigation, he said, and "we continue to be
disappointed with the striking staff who seem to want to scaremonger
on this incident."

Striking engineers said they were not seeking to pressure the company
as part of their labor dispute. They said they were speaking out
because they were worried that a threat remains to a power system
they care about. "We're not pointing fingers," said one. "This isn't
about a strike. It's about reliability of the system."

The May 27 incident set power generators in New York, Massachusetts,
New Brunswick and Ontario swinging in wild oscillation, fighting one
another to try to cope, according to official reports filed with
oversight authorities in Canada and the United States.

The competition among generators produced what experts call a "ring"
on the grid. According to an internal company document submitted to
regulators, "the ringing lasted about 12 seconds." Computer
simulations showed that "the oscillations can grow with time and
result in the breakup of the power system in about 30 seconds," the
internal document noted.

The industry oversight agencies have not yet publicly drawn any
conclusions from the event. Edward Schwerdt, head of the Northeast
Power Coordinating Council, which reviews the system's grids in New
York, New England and eastern Canada, said the May 27 event was an
"extreme contingency" but that no additional precautions to prevent a
recurrence are being taken until an inquiry is complete.

An official of the U.S.-Canada oversight agency, called the North
American Electric Reliability Council, also said they are awaiting a
report. "We could see a change in the system frequency on the
continent. We knew that something big had happened," said Glenn
Brown, who heads a disturbance analysis group for the council. He
compared the irregularity to one that blacked out all of
Jacksonville, Fla., in April 2002.

Two outside experts cited by the strikers confirmed their concerns.
"This had the potential of cascading into a big blackout," said
Puttaveeraiah Prabhakara, an electrical engineering expert in Toronto
who has written standards for electrical grid protection that have
been adopted worldwide.

The symptoms "were indications of a weak system," said Claudio
Canizares, a power systems specialist at the University of Waterloo
in Ontario. "This is a serious concern."

Sixty-six striking engineers sent a petition to Parliament expressing
their concerns. Their affiliated union, the International Federation
of Professional and Technical Engineers, said it would appeal to
lawmakers in Washington on Tuesday to investigate dangers to the
electrical supply.

The system was "a mere 20 seconds from reaching the proportion of the
August, 2003 blackout," Gregory J. Junemann, the union president,
said in a letter to members of Congress.

The problem began after routine maintenance of a huge circuit breaker
at switching substation in Milton, Ontario, 30 miles west of Toronto.
The company, engineers and regulators agree that when the work was
completed, the breaker was inadvertently left switched to a grounded
"test" position. When an operator at the Ontario Grid Control Center
in Barrie threw the switch to resume flow of electricity, it was
diverted into the ground, causing a massive short circuit in a line
carrying 500,000 volts.

Immediately, the protection systems -- high-tech, computerized
circuit breakers -- stopped power to the line that was erroneously
grounded at Milton. That is supposed to happen within 50
milliseconds. Instead, it took about 150 milliseconds -- three times
as long -- leaving massive strain on the system, according to David
McGinn, 54, a senior protection and control specialist for the
utility who has joined the strike.

"We don't design the system for that," said McGinn. "It did survive,
but just by good luck."

It was a "significant shock to the system," said Schwerdt of the
Northeast Power Coordinating Council.

Then, the protection systems for two other massive power-carrying
lines, stretching eight miles to the Trafalgar substation and 20
miles to the Claireville substation, inexplicably tripped, causing
both those 500,000-volt superhighways for power transmission to shut
down, according to the engineers.

Within milliseconds, generators across New York, Massachusetts and
eastern Canada reacted, first accelerating due to the loss of load,
then automatically decelerating to compensate. There were four major
swings before the system stabilized, according to reports filed with
the Northeast Power Coordinating Council.

"We were lucky," said Emeka Okongwu, 61, an operating engineer. Had
the weather been hot that day, said Okongwu, who is also on strike,
the additional strain on the system would likely have produced a
domino effect of failing lines.

This type of cascading from an initial local problem at a power plant
in Ohio caused the huge 2003 blackout in just nine seconds, according
to a U.S.-Canada Task Force report. That failure stranded millions of
office workers and commuters from Pennsylvania to Boston and Toronto.
It left Cleveland without water, shut down 22 nuclear plants, caused
60 fires, required 800 elevator rescues in New York City and cost
businesses billions of dollars.

The May 27 glitch, Cooperberg said, has left him nervous. "If another
line had failed, we would have been in the soup."

"Obviously, that did not happen," said Gregg, of Hydro One. "The
system is robust and resilient and managed this fault very well. An
investigation certainly will give us and all interested parties a
more fulsome answer."