Sign in to follow this  
Keenan

Server stability issues postmortem

Recommended Posts

Hi all.

 

Yesterday and early today we suffered from some server stability issues. Namely with Harmony, but also with Cadence, Deliverance, and Haven (tutorial server).

 

The underlying cause has been fixed and patched.

 

The technical details of the issue has to do with our database pooling. We discovered a connection leak in the code that purges 90-day inactive non-premium accounts. With the numbers we saw at launch on Steam and the fact that we've hit 90 days since launch, we're seeing a high number of inactive accounts from Northern Isles getting purged. This caused the database connections to spike over the max connections allowed for the database causing the first Harmony downtime.

 

To mitigate this before a patch was available, we increased the max connections on the database. This had an unintended result of hanging Harmony later on as there's a hard-coded max for connections on the server itself. The result was Golden Valley was hanging waiting on Harmony, and then player transfers on Deliverance and Cadence caused those servers to hang waiting on Golden Valley. A nice domino effect. The resolution was to shutdown Harmony and bring it back up on the patched server code. This restored the other servers to working order.

 

Later on Haven hit the same issue due to it being released shortly after launch. Haven was also brought up on patched code.

 

 

 

 

 

  • Like 11

Share this post


Link to post
Share on other sites
24 minutes ago, McGarnicle said:

What is haven? Is it the tutorial server?

 

Yes! Sorry for not clarifying that. :)

Share this post


Link to post
Share on other sites

-

Edited by Yserin
no longer playing, Per the mismanagment of AH update

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this