Monday, January 30, 2017

Death Of An ATS

At the first meeting our radio station's engineers had with our new boss, he told us, almost in so many words, “Woe betide if those very expensive emergency power systems ever don't work when we need them.”

We understood. In broadcasting, time is, literally, money. If you're not on the air, you have no listeners, and your advertisers are buying ads no one hears. So, broadcasters buy powerful generator sets (an engine tied to a generator), and rely on them when utility power goes out. In most cases, broadcast studios and transmitters are in different places, so you need “gensets” at both.

Usually powered by diesel engines, gensets have a voracious appetite for money, in the form of routine service (oil changes, keeping fuel fresh, having good starting batteries, and so forth). They usually have beefy “block heaters” that keep the engine warm 24/7, so it will start quickly if needed. (There are things slower than starting a cold diesel engine, but none comes to mind.)

The heart of an emergency power system is the Automatic Transfer Switch, or ATS. It has several jobs:

  • Detect that utility power has failed.
  • Wait to see if comes back by itself in a few seconds.
  • Decide that utility power really IS off and command the genset engine to start.
  • Wait for the engine to come up to speed and for the power from the genset to stabilize.
  • Throw the big switch that disconnects the building from utility power and onto generator power.

The drill is pretty much reversed when utility power returns:

  • Detect that utility power is back.
  • Wait to see if it stays on for a few seconds.
  • Transfer the building from generator power back to utility power
  • Wait for the genset engine to cool down, sometimes 30 minutes or more.
  • Command the genset engine to stop.

Not on either list is the most-important ATS function of all:

  • Make SURE that genset power never, ever, ever connects to the utility's wires.

This last is literally a matter of life-and-death. Utility power workers trying to repair outages have been killed by power fed back into what they thought were the “dead” circuits they were trying to repair.

Does it sound like a tall order? It is, and that's one reason emergency genset/ATS systems are hugely expensive.

When we retired to the country, we knew we'd need emergency power. Along with everything else electricity does for us, our home is supplied with water from our well – water which is pumped up to us by an electric motor. So, no power soon means no water, either.

Of course, one can get by with just a portable generator to run a few lights and appliances when the power is out. Sounds ok in theory, but getting the portable started (if it starts at all) and finding extension cords to plug appliances and lights into it (in the dark) is not that easy.

So, we had an expensive Onan genset/ATS emergency power system installed. It uses a propane-fired two-cylinder engine driving a 20,000 watt alternator. Our average-sized house has 200 amp utility service, so we included a “Service-Entry-Rated” ATS that provides the required utility disconnect switch as well as the other ATS functions.

As one might expect, the automotive jumper cables are not a normal part of the ATS!

The actual switch is the two black sections with brown covers.  It's actuated by a solenoid in the blue-and-white box.

Apart from the size, this one turned out to have a major difference from the big ones, both 208 and 480 volt three-phase ones.  On those, two solenoids pulled the switch between the NORMAL POWER and EMERGENCY POWER positions.  On this one, Onan cheaped out by using a single solenoid which alternately moved the switch between the two positions using a cam.  They saved the cost of a second solenoid, but the system proved unreliable.  In 2016, three years after ours was built, Onan redesigned the actuator.

As it turned out, the electric utility here in Rogue River is much MORE reliable than the one we had in suburban Lafayette California.  So, while the genset gets exercised by running the engine for 20 minutes each week, it's only had to actually transfer to emergency power a few times.  In December of 2016, we lost power for most of a day, and the transfer switch failed.

The symptom was this:  The ATS correctly recognized that utility power was off.  It sent a start command to the generator, waited for it to come up to speed, then tried to transfer the load to the generator.  But the solenoid jammed, leaving the switch stuck in the NORMAL position.  After that, the ATS shut down the genset, since it sensed (correctly) that the transfer switch had failed.

The normal backup is a manual handle one can use to move the switch by hand.  But we couldn't do that, either, because the actuator had jammed the switch.  Phooey,

So, with the caveat urging you not to try this at home, enter the jumper cables.  Remember, this is
a "Service Entry Rated" ATS, meaning that it includes the main 200A circuit breaker on the utility input to the house.  So, since we couldn't move the switch, we turned off the main utility breaker, then jumpered the GENERATOR side of the switch to the output contacts of the switch.  Then, with everything connected, we switched the genset itself from AUTOMATIC mode to RUN mode, so it would keep running without a command from the ATS.

It all worked.  

It was a hairy business, and not one that I'd normally choose.  The output of the genset is rated at around 80A or so, which is within (more or less) the  amperage ratings of auto jumper cables, but I was concerned about whether the insulation on the cables, meant for 12V DC would be adequate for 240V AC.  It was.

When our utility power came back on, we reversed the drill:  Stop the genset, remove the jumper cables, turn the main circuit breaker back on.  All was back to normal.

The very helpful folks at Cummins Northwest (Cummins, primarily a builder of diesel engines, acquired genset-and-ATS company Onan a few years back), sent out a repair person, who knew
his business inside and out, coming, as he did, from experience with emergency power systems in both the U,.S, Navy and the mining industry.

The eventual deal:  I agreed to pay for the service call, about $500, and Cummins agreed to supply me with the new version of the switch (also around $500) for free, even though my unit was one year out of warranty.  On the whole, an eminently fair deal, I thought.

In the course of replacing the switch, the tech broke a molded fiberglass insulator in the ATS.  Today, he came out and replaced it, at no additional charge.

So, what's the takeaway from all this?

First, it's worth having a genset and ATS, but be sure you give a a REAL test every year or so.  That means just shutting off the main breaker on the utility to see that all the steps discussed at the beginning happen, and do so in the right order.

Second, unless you've spent many, many years working with emergency power systems, don't try to do ANYTHING yourself.  It's bad enough that you could die, but you could also kill a utility worker if you do it wrong.

January 30, 2017