Sign in to follow this  
Keenan

Post-mortem 2022-03-18: Server outages

Recommended Posts

Hi all,

 

We had a service interruption today that affected the following game servers:

  • Affliction
  • Celebration
  • Elevation
  • Harmony

 

The root cause of the outage was network-related, with the public-facing network link dropping. Our hosting provider was able to gain access to the machine and gracefully shutdown the four servers before restarting and resolving the network issue.

 

Timeline (All times in CET/Server Time):

  • 20:16 - Alert that four game servers went offline simultaneously.
  • 20:18 - Unable to ping the host, hosting provider notified.
  • 20:26 - Hosting provider confirms a problem and is looking into the situation.
  • 20:59 - Provider is still troubleshooting the issue, but indicates that it could be hardware failure.
  • 21:16 - Provider is able to log into the host and gracefully shutdown the game servers.
  • 21:23 - Host is restarted.
  • 21:30 - Ping! The host is now responding again.
  • 21:35 - Max begins starting up the game servers again, confirming their status.
  • 21:45 - Outage resolved.

 

Due to the outage, we will be awarding 5 hours of sleep bonus during the next maintenance restart on this coming Tuesday. This will be global as we are aware that folks were traveling at the time of the outage and therefore were impacted while not residing on the affected servers.

 

Thank you, and happy Wurming!

  • Like 26

Share this post


Link to post
Share on other sites
5 minutes ago, Keenan said:

Due to the outage, we will be awarding 5 hours of sleep bonus during the next maintenance restart on this coming Tuesday.

 

  • Like 1

Share this post


Link to post
Share on other sites

Will this be like the last time everyone was awarded 5 free hours of sleep bonus due to an outage? In that the 5 hours will tacked on to whatever existing sleep bonus you already?  or will we need to be at zero to get the full 5 hours?

  • Like 2

Share this post


Link to post
Share on other sites

@Keenan Thank you very much for sharing the postmortem!  - very helpful in creating understanding.  The consideration and recompense for the impact on players is appreciated as well.

Share this post


Link to post
Share on other sites
1 hour ago, Sinnjinn said:

Will this be like the last time everyone was awarded 5 free hours of sleep bonus due to an outage? In that the 5 hours will tacked on to whatever existing sleep bonus you already?  or will we need to be at zero to get the full 5 hours?

 

It will be up to the 10 hour hard cap.

  • Like 6

Share this post


Link to post
Share on other sites
7 hours ago, Keenan said:

 

It will be up to the 10 hour hard cap.

Awesome.  Thanks for the clarification....and the free sleep bonus!!

Share this post


Link to post
Share on other sites
On 3/19/2022 at 7:21 PM, Ulviirala said:

But where's the "what went wrong" part? :P

 

The only thing I can really think of is that we do need more communication with our provider when these situations arise. There was a decent gap of time where I wanted to provide something more than "We're looking into it". The flip side of this though is not bothering the people working on the problem and letting them do the work. There needs to be a balance in that. So that's something we'll be striving towards. :)

  • Like 2

Share this post


Link to post
Share on other sites
2 minutes ago, Russwoods said:

Does that apply to accounts only on those sever or to all 

 

Should have been applied to all accounts as of this morning. It adds 5 hours, with a maximum total of 10 hours.

  • Like 2

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this