Unplanned Downtime 2023-08-18
There has been some unexpected downtime today due to severe weather. The server is back up now, and no data was lost. I am currently working to ensure that it remains up as the storm continues. As always, please email me if you have any questions or concerns.
Incident details
What happened
- At 2023-08-18 00:43-0400, the electricity went out. One minute later, the internet also went out. They were restored at 02:45-0400.
- The internet went in and out until it stabilized at 12:00-0400.
- At that time, the Internet Service Provider (ISP) gave the server a new IP address. This occurs very occasionally, but I have systems in place to automatically detect this and update the DNS records.
- The server that tells FarkasCity its public IP address was incorrectly configured, so the automatic system did not work.
- When I got off work at 12:30-0400, I immediately went to the server to fix the problem. Roughly 15 minutes later, I discovered the IP address change and manually corrected the DNS records.
- I wrote this post.
Next steps
- The Uninterruptible Power Supply (UPS) that FarkasCity is connected to has run out of battery power, so I am going to connect the secondary one. This should last the remainder of the storm.
- I am going to update the the IP detection script to check with multiple other servers, and to alert me if one is unavailable.