Unplanned Downtime 2023-08-18

There has been some unexpected downtime today due to severe weather. The server is back up now, and no data was lost. I am currently working to ensure that it remains up as the storm continues. As always, please email me if you have any questions or concerns.

Incident details

What happened

  1. At 2023-08-18 00:43-0400, the electricity went out. One minute later, the internet also went out. They were restored at 02:45-0400.
  2. The internet went in and out until it stabilized at 12:00-0400.
  3. At that time, the Internet Service Provider (ISP) gave the server a new IP address. This occurs very occasionally, but I have systems in place to automatically detect this and update the DNS records.
  4. The server that tells FarkasCity its public IP address was incorrectly configured, so the automatic system did not work.
  5. When I got off work at 12:30-0400, I immediately went to the server to fix the problem. Roughly 15 minutes later, I discovered the IP address change and manually corrected the DNS records.
  6. I wrote this post.

Next steps

  1. The Uninterruptible Power Supply (UPS) that FarkasCity is connected to has run out of battery power, so I am going to connect the secondary one. This should last the remainder of the storm.
  2. I am going to update the the IP detection script to check with multiple other servers, and to alert me if one is unavailable.

#StatusUpdates #Downtime