Posted January 27, 2019 (edited) I would like to apologize for the extended downtime Celebration suffered this morning. We've brought the server back online without any data loss or roll backs. We will be awarding 5 hours of sleep bonus on all servers at the next restart on Thursday of this week as compensation for this downtime. Edited January 27, 2019 by Keenan Added when 14 Share this post Link to post Share on other sites
Posted January 27, 2019 Thank you for catching new hamsters and replacing the faulty ones -when should it happen if not on Sunday? 3 Share this post Link to post Share on other sites
Posted January 27, 2019 10 minutes ago, Jaz said: Thank you for catching new hamsters and replacing the faulty ones -when should it happen if not on Sunday? We'll enable it for Thursday's restart. That should give everyone time to use what they have. (Including me!) Share this post Link to post Share on other sites
Posted January 27, 2019 Just now, Keenan said: We'll enable it for Thursday's restart. That should give everyone time to use what they have. (Including me!) I meant the server crash that always happens to ruin your weekend forcing to work on the fix - but the restart timing is perfect too 1 Share this post Link to post Share on other sites
Posted January 27, 2019 1 hour ago, Keenan said: I would like to apologize for the extended downtime Celebration suffered this morning. Was the cause identified? Asking even though I'm sure many people did a certain thing at the exact second it crashed, I ruled out me digging peat, but perhaps this: [11:03] <Ulviirala> rip celebration [11:03] <Batolemaeus> it died exactly when my alt pressed the teleport option from the starter town to cele ? [11:04] <Ulviirala> YOU!@!! [11:04] <Batolemaeus> seriously, i went to the portal, confirmed i wanted to be teleported to cele, and bam ? Share this post Link to post Share on other sites
Posted January 27, 2019 2 minutes ago, Ulviirala said: Was the cause identified? Asking even though I'm sure many people did a certain thing at the exact second it crashed, I ruled out me digging peat, but perhaps this: ? The cause has been found and is not any single action. It's actually a rather rare situation that Celebration has been (un)lucky enough to encounter twice. I'm working on a fix now and will likely hot fix later today. 1 Share this post Link to post Share on other sites
Posted January 27, 2019 Multiple downs? I reckon those Cele peasants deserve an extra dose of sb. Share this post Link to post Share on other sites
Posted January 27, 2019 1 minute ago, Mataleao said: Multiple downs? I reckon those Cele peasants deserve an extra dose of sb. As my initial post states, 5 hours will be awarded on Thursday. No more, no less. 1 Share this post Link to post Share on other sites
Posted January 27, 2019 Thanks for the communique on this and the Sleep Bonus! 1 Share this post Link to post Share on other sites
Posted January 27, 2019 (edited) 6 hours ago, Keenan said: We'll enable it for Thursday's restart. That should give everyone time to use what they have. (Including me!) Thursday.... thats about 4 days from now.. let me see what my SB is at... https://imgur.com/a/gqAwPFh I think 10 or 11 days may be better for me. Maybe 14. Idk. edit: yay i got unbanned, should be able to also enjoy the SB ty Edited January 27, 2019 by Deathangel 1 Share this post Link to post Share on other sites
Posted January 27, 2019 26 minutes ago, Deathangel said: Thursday.... thats about 4 days from now.. let me see what my SB is at... https://imgur.com/a/gqAwPFh I think 10 or 11 days may be better for me. Maybe 14. Idk. https://imgur.com/a/H54TGM6 Share this post Link to post Share on other sites
Posted January 27, 2019 3 hours ago, Keenan said: It's actually a rather rare situation that Celebration has been (un)lucky enough to encounter twice. I'm working on a fix now and will likely hot fix later today. I'd love to read a post mortem once the cause has been fixed. Share this post Link to post Share on other sites
Posted January 28, 2019 17 hours ago, Batolemaeus said: I'd love to read a post mortem once the cause has been fixed. The cause of the crash had to do with a rare chance race condition. Essentially upon login, there was a chance a part of the player's data wasn't available yet when the journal code was attempting to view it. This has been safeguarded and fixed. The duration of the downtime had to do with security changes preventing our other server administrator from logging in while I was asleep and bringing the server online. This has also been addressed and fixed, as well as there being a direct line of communication between myself and Max. Essentially his phone number can now break through my phone's night time "Do Not Disturb". A hot fix is going out now to prevent the crashes, however the sleep bonus is still scheduled for a Thursday restart as I mentioned earlier. 6 Share this post Link to post Share on other sites
Posted January 28, 2019 Thanks a lot, Keenan, for these explanations. Just curious if you like to answer: Was the Xanadu problem today caused by a similar situation, or a new problem? Share this post Link to post Share on other sites
Posted January 28, 2019 Leaked footage of what caused the Cele server downtime: 8 Share this post Link to post Share on other sites