Are you making these backup power generator mistakes?

Posted on September 12, 2013

Summary Personal lessons from managing a critical facility.

A very common problem for facilities with critical loads is that the power generator doesn't start when it is needed. Fortunately[1] this can be remedied in the vast majority of cases.

When I say critical loads I am talking about computer data centers, hospitals, schools, stadiums, police stations, 911 centers, office buildings, or whatever you've deemed important enough to attach a backup power generator to.

The Situation

You, or your organization, has a critical load and has also gone to the trouble of spending the money on investing in a generator backup power source for it.

The Objective

This is simple enough: keep the power on.

The Problem

A local newspaper reported on a sewage spill in the county I live in:

About 20,000 gallons of sewage spilled from the California Men’s Colony prison at 4:10 p.m. Sunday when power was lost and an emergency generator did not start. The sewage flowed into Chorro Creek, which flows into Morro Bay.

The fault was apparently that the generator did not start after a utility power failure:

"The power failed and then our backup generator failed, so it was kind of like a double power failure," said Mike Minty, chief engineer at the prison’s waste water treatment plant. "It’s all fixed now."

If you operate a data center or critical facility that has a power generator, there are some very easy pro-active actions that can be taken to mitigate the most common problem I observe: the generator fails when the power goes out. For most, that's not the hoped for outcome of the capital they've invested in making their facility more resilient to utility power outages.

While there's always a possibility that shit this can happen even if various preventative actions are taken, the chances are far lower if a handful of items are paid attention to. I am not privy to the maintenance procedures at the California Men's Colony waste water treatment plant, so I'm just using their outage as the thought provoker and not judging them.

The Cause

When a generator "simply does not start", rarely is that the entire story. Rather than being the root cause of the outage, it's the manifestation of the maintenance and monitoring practices.

In my experience it's usually a symptom of a lack of a pro-active culture surrounding the backup power system. Sadly, some organizations that invest large sums of capital into their backup power systems (and, presumably, whatever the critical load is they are protecting), don't factor in proper operational costs and fail to implement appropriate procedures to see to it that appropriate preventative work is performed. This diminishes the return on investment on the capital invested in the entire system.

The failures then flow through to two areas:

  1. Maintenance
  2. Monitoring

The usual failure scenarios are one or more of:

  1. Generator fails to start (common)
  2. Generator starts, but fails under load (common)
  3. Generator starts, but no power reaches the critical load (less common)

The end result is the same: the critical load loses power.

The Solution

In the case of a generator, here's the practice I've learned to follow:

Weekly no-load automatic tests (usually this can be programmed into your automatic-transfer switch)

Monthly (or Bi-weekly) Manually Triggered Actual Load Tests

Yearly (or Quarterly) Dummy Load tests

Things This Solution Should Catch

Some of the causes of the inability to start, that would have been picked up under the above system, that I've observed are:

It's good to be aware of these so that the staff implementing the system can understand the specifics of the problems they're looking for.  They do require a checklist system -- something for a junior tech or maintenance person to perform/verify on a regular schedule. These items are easy and even "cheap" both in absolute and ROI terms.   I'm reminded of a couple quotes I recently wrote in my journal (credit to Robert Rosenthal for these two sayings):


  1. or unfortunately, depending on how you look at it 


If you enjoyed this, you're invited to subscribe to be notified when I post similar items. I also invite you to connect with me by email or on Twitter if you have a comment, idea, or question.