All systems operational
Last incident 1 month agoUnexpected network outage: Atlanta
Feb 16, 21:00 UTC
|
Resolved after 1 day
|
Affected: Atlanta
The datacenter at 55 Marietta has informed us of a power event that has impacted the entire datacenter floor. QuadraNet are currently awaiting updates from the datacenter engineers.
Incident's timeline
Update
Feb 17, 08:04 UTC
At this time QuadraNet have restored network connectivity to their environment(s) and some customers have also been restored.
At this time they will begin the recovery of customers that are not currently restored.
The status of this situation is being changed at this time as QuadraNet has successfully deployed their disaster recovery playbook and they will continue efforts to restore customers who are still impacted by the original power failure.
Unfortunately, we at OVPN are still affected by this power outage and we are awaiting updates from QuadraNet as to when power will be restored.
At this time they will begin the recovery of customers that are not currently restored.
The status of this situation is being changed at this time as QuadraNet has successfully deployed their disaster recovery playbook and they will continue efforts to restore customers who are still impacted by the original power failure.
Unfortunately, we at OVPN are still affected by this power outage and we are awaiting updates from QuadraNet as to when power will be restored.
Update
Feb 17, 07:01 UTC
QuadraNet has not been successful at this time in restoring their edge routing equipment from the power failure.
At this time they are exercising all available options in their disaster recovery plan in order to restore service to us and all other customers.
At this time they are exercising all available options in their disaster recovery plan in order to restore service to us and all other customers.
Update
Feb 17, 06:09 UTC
At this time QuadraNet has not been successful in restoring their routing equipment and they are still troubleshooting this with the datacenter engineers.
Update
Feb 17, 05:26 UTC
At this time QuadraNet network engineers are working with local engineers to restore service to their routing equipment. Due to the hard power disconnect, it has resulted in the failure of several components in their edge routing equipment in the datacenter.
Update
Feb 17, 03:37 UTC
Engineers have been provided access to the building after being cleared by the Atlanta Fire Department. QuadraNet is currently working directly with the engineers onsite to restore service.
Update
Feb 17, 00:47 UTC
The datacenter has informed us of the following:
"A contractor working on the 2nd floor of the 55 Marietta building cut into a water main and flooded the 2nd and 1st level of the building. The flooding extended to the generator room which took down the datacenter generator and ATS (Automatic Transfer Switch). The Atlanta Fire Department is currently on the scene and is not allowing access to the building until it has been cleared for safety."
QuadraNet is currently working diligently with their datacenter partner in the Atlanta location to restore service. They will provide additional updates as they become available.
"A contractor working on the 2nd floor of the 55 Marietta building cut into a water main and flooded the 2nd and 1st level of the building. The flooding extended to the generator room which took down the datacenter generator and ATS (Automatic Transfer Switch). The Atlanta Fire Department is currently on the scene and is not allowing access to the building until it has been cleared for safety."
QuadraNet is currently working diligently with their datacenter partner in the Atlanta location to restore service. They will provide additional updates as they become available.
Issue identified
Feb 16, 21:01 UTC
The issue has been identified and a fix is being implemented.
Last update: 2024-12-23 06:45:02