Sign in to follow this  
Keenan

Multiple Server Outage 2021-05-03 - Postmortem

Recommended Posts

We had an outage today that affected the following game servers:

  • Xanadu
  • Affliction
  • Elevation
  • Harmony
  • Celebration

The outage lasted approximately 45 minutes.

 

The cause of the outage was a configuration issue. Our new hosts are configured with tools to match our needs, which means that we can more quickly recover should hardware issues crop up. A bug in the configuration tool lead to some hosts being improperly configured outside of a maintenance window. The resolution was to roll back the configuration on the hosts and restart the game servers.

 

Our provider has assured me that while most changes undergo testing, this was somehow missed and they are working on updating their procedures to ensure such an outage does not happen again.

 

Thank you for your patience.

  • Like 23

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this