Posted March 18, 2022 Hi all, We had a service interruption today that affected the following game servers: Affliction Celebration Elevation Harmony The root cause of the outage was network-related, with the public-facing network link dropping. Our hosting provider was able to gain access to the machine and gracefully shutdown the four servers before restarting and resolving the network issue. Timeline (All times in CET/Server Time): 20:16 - Alert that four game servers went offline simultaneously. 20:18 - Unable to ping the host, hosting provider notified. 20:26 - Hosting provider confirms a problem and is looking into the situation. 20:59 - Provider is still troubleshooting the issue, but indicates that it could be hardware failure. 21:16 - Provider is able to log into the host and gracefully shutdown the game servers. 21:23 - Host is restarted. 21:30 - Ping! The host is now responding again. 21:35 - Max begins starting up the game servers again, confirming their status. 21:45 - Outage resolved. Due to the outage, we will be awarding 5 hours of sleep bonus during the next maintenance restart on this coming Tuesday. This will be global as we are aware that folks were traveling at the time of the outage and therefore were impacted while not residing on the affected servers. Thank you, and happy Wurming! 26 Share this post Link to post Share on other sites
Posted March 18, 2022 5 minutes ago, Keenan said: Due to the outage, we will be awarding 5 hours of sleep bonus during the next maintenance restart on this coming Tuesday. 1 Share this post Link to post Share on other sites
Posted March 18, 2022 Will this be like the last time everyone was awarded 5 free hours of sleep bonus due to an outage? In that the 5 hours will tacked on to whatever existing sleep bonus you already? or will we need to be at zero to get the full 5 hours? 2 Share this post Link to post Share on other sites
Posted March 18, 2022 Thank you, @Keenan. You rock!! 1 Share this post Link to post Share on other sites
Posted March 18, 2022 @Keenan Thank you very much for sharing the postmortem! - very helpful in creating understanding. The consideration and recompense for the impact on players is appreciated as well. Share this post Link to post Share on other sites
Posted March 18, 2022 1 hour ago, Sinnjinn said: Will this be like the last time everyone was awarded 5 free hours of sleep bonus due to an outage? In that the 5 hours will tacked on to whatever existing sleep bonus you already? or will we need to be at zero to get the full 5 hours? It will be up to the 10 hour hard cap. 6 Share this post Link to post Share on other sites
Posted March 19, 2022 7 hours ago, Keenan said: It will be up to the 10 hour hard cap. Awesome. Thanks for the clarification....and the free sleep bonus!! Share this post Link to post Share on other sites
Posted March 19, 2022 But where's the "what went wrong" part? Share this post Link to post Share on other sites
Posted March 22, 2022 Thanks for the feedback Keenan 👍 Share this post Link to post Share on other sites
Posted March 22, 2022 On 3/19/2022 at 7:21 PM, Ulviirala said: But where's the "what went wrong" part? The only thing I can really think of is that we do need more communication with our provider when these situations arise. There was a decent gap of time where I wanted to provide something more than "We're looking into it". The flip side of this though is not bothering the people working on the problem and letting them do the work. There needs to be a balance in that. So that's something we'll be striving towards. 2 Share this post Link to post Share on other sites
Posted March 22, 2022 Does that apply to accounts only on those sever or to all Share this post Link to post Share on other sites
Posted March 22, 2022 2 minutes ago, Russwoods said: Does that apply to accounts only on those sever or to all Should have been applied to all accounts as of this morning. It adds 5 hours, with a maximum total of 10 hours. 2 Share this post Link to post Share on other sites
Posted March 23, 2022 Thank you for the great work. You all rock 1 Share this post Link to post Share on other sites