Sign in to follow this  
Keenan

Xanadu Outage - 2020-01-20 - Postmortem

Recommended Posts

At 03:50 server time an issue occurred that affected the skills tables for both creatures and players. This caused an issue with reading skill data and possibly saving skill data. The issue was brought to my attention at 04:07 server time when players complained about having no skill and being unable to complete actions as a result.

 

I had first attempted to repair the table while the server was running in order to preserve any data from skill gains that I could, but this only made the problem worse - the table seemed to have vanished due to the repair failing. Xanadu was brought down at 04:20 server time. At this point I was able to save the binary data of the skills table in order to preserve it and initiated a MySQL restart. When the database server came back online the table was restored and repaired.

 

I verified that skill data seemed intact by picking a few players who had complained originally and finding their skills to have appropriate values.

 

Xanadu was brought back online and available for connections at 04:50 server time.

 

Due to this issue there may have been a period of about 30 minutes where skill ticks were not saved.

 

I will be meeting with Samool in the morning to discuss compensation for the downtime and possible 30 minute skill loss. Most likely we will be doing 5 hours of sleep bonus on Thursday's restart. We will update to confirm the decision. Edit: Confirmed - 5 hours of sleep bonus on Jan 23 restart.

 

Thank you for your patience.

  • Like 28
  • Cat 1

Share this post


Link to post
Share on other sites

Shrimpiie did it I seen him do it. Thanks for fixing it so swiftly

 

 

 

 

I am not libila in the flesh...

  • Like 1

Share this post


Link to post
Share on other sites
3 hours ago, Keenan said:

Xanadu was brought down at 04:20 server time.

 

In order to better consider the situation no doubt.  🧠

 

  • Like 3

Share this post


Link to post
Share on other sites

Smells like a hardware issue

Edited by Sklo:D

Share this post


Link to post
Share on other sites

hip hip hooray? 

We do not always party when xana drops.. but sb's too good to not party for...😏

Share this post


Link to post
Share on other sites
18 hours ago, Keenan said:

At 03:50 server time an issue occurred that affected the skills tables for both creatures and players. This caused an issue with reading skill data and possibly saving skill data. The issue was brought to my attention at 04:07 server time when players complained about having no skill and being unable to complete actions as a result.

 

I had first attempted to repair the table while the server was running in order to preserve any data from skill gains that I could, but this only made the problem worse - the table seemed to have vanished due to the repair failing. Xanadu was brought down at 04:20 server time. At this point I was able to save the binary data of the skills table in order to preserve it and initiated a MySQL restart. When the database server came back online the table was restored and repaired.

 

I verified that skill data seemed intact by picking a few players who had complained originally and finding their skills to have appropriate values.

 

Xanadu was brought back online and available for connections at 04:50 server time.

 

Due to this issue there may have been a period of about 30 minutes where skill ticks were not saved.

 

I will be meeting with Samool in the morning to discuss compensation for the downtime and possible 30 minute skill loss. Most likely we will be doing 5 hours of sleep bonus on Thursday's restart. We will update to confirm the decision. Edit: Confirmed - 5 hours of sleep bonus on Jan 23 restart.

 

Thank you for your patience.

Thats some epic response time, Keenan! Glad to have you on board! Kudos for a job well done!

  • Like 2

Share this post


Link to post
Share on other sites

These things happen, but the ability of the devs to fix sudden, unexpected problems is what really matters in the end. And I agree... this response was epic, both in terms of speed and knowledge of the code, as well as the ability to diagnose a problem in real time. Well done indeed!

  • Like 2

Share this post


Link to post
Share on other sites

@KeenanAs an old school Wurm player who has been mostly away except for a few brief return sessions in the last few years, I look at your response above and am impressed.  Rapid, precise action and a comprehensible report of what happened afterwards.  That's so far from the Wurm I used to know, where we'd get a content update on Friday, with fun bugs, and not hear from the dev for a day or two because he took off for the weekend.  Then there was the Russian Roulette Deed Destruction Moon Crap on Epic which was the point at which I started playing Wurm only occasionally.

 

With all this sanity I'm seeing in the development direction now, I'm actually looking forward to returning to the game.  One thing that would be nice though.  Could you point me to somewhere on the website that lays out the planned direction for Wurm in the shirt and long term?  I'm seeing stuff about Steam, RMT, Player gods, WU no longer being supported, etc. etc. and it all seems to point towards some sort of significant shift in the game, but I haven't seen a big picture description of the plan anywhere that would seem like an appropriate place.  (A big, obvious link on the main Wurm webpage directing us to the future plans for the game, for example.)

Share this post


Link to post
Share on other sites
6 hours ago, Farmerbob said:

@KeenanAs an old school Wurm player who has been mostly away except for a few brief return sessions in the last few years, I look at your response above and am impressed.  Rapid, precise action and a comprehensible report of what happened afterwards.  That's so far from the Wurm I used to know, where we'd get a content update on Friday, with fun bugs, and not hear from the dev for a day or two because he took off for the weekend.  Then there was the Russian Roulette Deed Destruction Moon Crap on Epic which was the point at which I started playing Wurm only occasionally.

 

With all this sanity I'm seeing in the development direction now, I'm actually looking forward to returning to the game.  One thing that would be nice though.  Could you point me to somewhere on the website that lays out the planned direction for Wurm in the shirt and long term?  I'm seeing stuff about Steam, RMT, Player gods, WU no longer being supported, etc. etc. and it all seems to point towards some sort of significant shift in the game, but I haven't seen a big picture description of the plan anywhere that would seem like an appropriate place.  (A big, obvious link on the main Wurm webpage directing us to the future plans for the game, for example.)

 

It's a bit on the short side, but that post explains the current general direction of our work leading up to Steam - we're focused on making the game generally more approachable by everyone (UI changes, qol, early game stuff) and rethinking the more nonsensical things and balancing them.

There's not many plans we can share beyond the Steam release, as some of it depends on its success, but we'll share more as soon as we can.

We've not been great to sticking to roadmaps in the past, with Wurm development being full of surprises, and as such those will have to be a bit more carefully thought out :)

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this