Report on outage @ ISC Data Center (Redwood City)

Peter Losher Peter_Losher at
Tue Apr 28 23:39:18 UTC 2009

Hash: SHA1

ISC experienced a power outage at our Redwood City data center on April
27th from 1830 to 1945 UTC.  We were informed by the local utility (PG&E)
that there were some sparking power lines in the immediate area just
before the power loss.  When utility power was lost, the load was taken
by our UPS' until our on-site generator could engage.  The generator was
started, but reported a fault in the cooling system, and shut down.

We requested assistance from our generator service vendor, but by then
the UPS capacity had drained and we lost power to all of our data center
equipment.  Once power had been restored, we charged the UPS batteries,
then brought online the data center rooms.  In the meantime, we have had
the generator swapped out with a working replacement.

If you still have a system down and haven't already contacted us, send
mail to hands at, so we are aware of it.

For real-time updates, view ISC's status page at:

During the outage, we moved some core gear onto office power (which
stayed up) to keep essential services up for ISC staff.  That includes
the core router which we need to move that back under UPS/Generator
protection.  We have scheduled a 30 minute outage for 0200 UTC Wed.
morning.  This will cause a short network outage as the main transit link
is re-established.

If there are any questions, let hands at know.

Apologies for any inconvenience this outage may have caused.

Best Wishes - Peter
- -- 
Peter_Losher at | ISC | OpenPGP 0xE8048D08 | "The bits must flow"
Version: GnuPG v1.4.9 (Darwin)


More information about the noc-notice mailing list