*The Importance of Penalty Testing*
By: Spitfire
27 December 2014

Systems designed for space have many thousands of parts and are extremely complex. Spacecraft are extensively tested on the ground to make darn sure everything is going to work before they blast off on a rocket. Invariably during testing something will break and need to be repaired. The very act of fixing it introduces the possibility of breaking something else so a lot of thought is given as to what level of "penalty testing" should be done to make sure that not only the original problem was fixed but that anything else that could have broken during the repair is also tested and proven to still work.

Even simple systems can have new failures crop up after routine maintenance. Firearms come to mind as we take them apart to clean and lubricate them and when we put them back together we could have left a part out or misaligned a pin or spring and not noticed it until we needed it to work. Most of us do penalty testing here whether we call it that or not. Who doesn't at least cycle the action after field striping and re-assembly?

I bring this up because last night I did an unplanned generator test to make sure the generator would still start and charge our new Alt Power AGM battery bank. I had already had a failure on this generator where it started up and ran fine, but didn't make any power. I didn't want that to happen again.

So the generator started right up and the new AC volt meters I installed to make sure it was actually making power showed good. I then went to the main breaker box to disconnect the grid and connect the now running generator so I could make sure it made power under load. We have a UL listed, Square D generator interlock bracket that does this job.

You simply flip the giant main breaker "Off", the metal interlock bracket falls down and allows the generator input breaker handle to clear the bracket and be turned "On", safely back feeding the panel. Or at least that's how it's *supposed* to work and has dozens of times before. This time, the bracket did not fall far enough to clear the generator breaker and the breaker could not be moved to the "On" position.

It turns out when I had the main breaker panel cover off to install a high power battery charger circuit for the new high efficiency mini-split heat pumps, I did not completely seat it when I put it back on. This caused it to 1/16" too high and since the interlock bracket is bolted to the cover, it too was 1/16" too high and caused the bracket to not clear the generator inlet breaker.

A simple adjustment to the lid seated it correctly and fixed the problem, but in an emergency where seconds count, spending the 10 minutes it took to find the problem, diagnose the cause and fix it could be bad.

What I should have done when I added the battery charger circuit was to do a penalty test to make sure I hadn't screwed something else up in the process. A simple one minute test to make sure all the breakers still turned on and off would have brought this problem into the light immediately.

Taking it a step further, a quick walk around the house to make sure all the grid power was restored would have been a good idea. (I actually did that part as the garage fridge is on grid power as are the overhead door openers)

For critical systems like a generator interlock, penalty testing should certainly have included firing the genny, powering the grid breaker box and actually powering something like the well pump to make sure everything is really still working.

Do you penalty test your systems after you've done maintenance or improvements?

www.alpharubicon.com
All materials at this site not otherwise credited are Copyright © 1996 - 2015 Trip Williams. All rights reserved. May be reproduced for personal use only. Use of any material contained herein is subject to stated terms or written permission.