Keenan

Celebration's Extended Downtime

Recommended Posts

I would like to apologize for the extended downtime Celebration suffered this morning.

 

We've brought the server back online without any data loss or roll backs.

 

We will be awarding 5 hours of sleep bonus on all servers at the next restart on Thursday of this week as compensation for this downtime.

Edited by Keenan
Added when

Share this post


Link to post
Share on other sites

Thank you for catching new hamsters and replacing the faulty ones -when should it happen if not on Sunday?

Share this post


Link to post
Share on other sites
10 minutes ago, Jaz said:

Thank you for catching new hamsters and replacing the faulty ones -when should it happen if not on Sunday?

 

We'll enable it for Thursday's restart. That should give everyone time to use what they have. (Including me!)

Share this post


Link to post
Share on other sites
Just now, Keenan said:

 

We'll enable it for Thursday's restart. That should give everyone time to use what they have. (Including me!)

I meant the server crash that always happens to ruin your weekend forcing to work on the fix - but the restart timing is perfect too :P

Share this post


Link to post
Share on other sites
1 hour ago, Keenan said:

I would like to apologize for the extended downtime Celebration suffered this morning.

Was the cause identified?

Asking even though I'm sure many people did a certain thing at the exact second it crashed, I ruled out me digging peat, but perhaps this:

[11:03] <Ulviirala> rip celebration


[11:03] <Batolemaeus> it died exactly when my alt pressed the teleport option from the starter town to cele 😐
[11:04] <Ulviirala> YOU!@!!
[11:04] <Batolemaeus> seriously, i went to the portal, confirmed i wanted to be teleported to cele, and bam

?

Share this post


Link to post
Share on other sites
2 minutes ago, Ulviirala said:

Was the cause identified?

Asking even though I'm sure many people did a certain thing at the exact second it crashed, I ruled out me digging peat, but perhaps this:

 

?

 

The cause has been found and is not any single action.

 

It's actually a rather rare situation that Celebration has been (un)lucky enough to encounter twice. I'm working on a fix now and will likely hot fix later today.

Share this post


Link to post
Share on other sites

Multiple downs? I reckon those Cele peasants deserve an extra dose of sb.

Share this post


Link to post
Share on other sites
1 minute ago, Mataleao said:

Multiple downs? I reckon those Cele peasants deserve an extra dose of sb.

 

As my initial post states, 5 hours will be awarded on Thursday. No more, no less.

Share this post


Link to post
Share on other sites

Thanks for the communique on this and the Sleep Bonus! 

Share this post


Link to post
Share on other sites
6 hours ago, Keenan said:

 

We'll enable it for Thursday's restart. That should give everyone time to use what they have. (Including me!)

Thursday.... thats about 4 days from now.. let me see what my SB is at...

 

https://imgur.com/a/gqAwPFh

 

I think 10 or 11 days may be better for me. Maybe 14. Idk.

 

edit: yay i got unbanned, should be able to also enjoy the SB ty

Edited by Deathangel

Share this post


Link to post
Share on other sites
3 hours ago, Keenan said:

It's actually a rather rare situation that Celebration has been (un)lucky enough to encounter twice. I'm working on a fix now and will likely hot fix later today.

 

I'd love to read a post mortem once the cause has been fixed.

Share this post


Link to post
Share on other sites
17 hours ago, Batolemaeus said:

 

I'd love to read a post mortem once the cause has been fixed.

 

The cause of the crash had to do with a rare chance race condition. Essentially upon login, there was a chance a part of the player's data wasn't available yet when the journal code was attempting to view it. This has been safeguarded and fixed.

 

The duration of the downtime had to do with security changes preventing our other server administrator from logging in while I was asleep and bringing the server online. This has also been addressed and fixed, as well as there being a direct line of communication between myself and Max. Essentially his phone number can now break through my phone's night time "Do Not Disturb". :)

 

A hot fix is going out now to prevent the crashes, however the sleep bonus is still scheduled for a Thursday restart as I mentioned earlier.

Share this post


Link to post
Share on other sites

Thanks a lot, Keenan, for these explanations. Just curious if you like to answer: Was the Xanadu problem today caused by a similar situation, or a new problem?

Share this post


Link to post
Share on other sites

Leaked footage of what caused the Cele server downtime: 

 

tenor_1.gif

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now