Posted October 23, 2020 Hi all. Yesterday and early today we suffered from some server stability issues. Namely with Harmony, but also with Cadence, Deliverance, and Haven (tutorial server). The underlying cause has been fixed and patched. The technical details of the issue has to do with our database pooling. We discovered a connection leak in the code that purges 90-day inactive non-premium accounts. With the numbers we saw at launch on Steam and the fact that we've hit 90 days since launch, we're seeing a high number of inactive accounts from Northern Isles getting purged. This caused the database connections to spike over the max connections allowed for the database causing the first Harmony downtime. To mitigate this before a patch was available, we increased the max connections on the database. This had an unintended result of hanging Harmony later on as there's a hard-coded max for connections on the server itself. The result was Golden Valley was hanging waiting on Harmony, and then player transfers on Deliverance and Cadence caused those servers to hang waiting on Golden Valley. A nice domino effect. The resolution was to shutdown Harmony and bring it back up on the patched server code. This restored the other servers to working order. Later on Haven hit the same issue due to it being released shortly after launch. Haven was also brought up on patched code. 11 Share this post Link to post Share on other sites
Posted October 23, 2020 What is haven? Is it the tutorial server? Share this post Link to post Share on other sites
Posted October 23, 2020 24 minutes ago, McGarnicle said: What is haven? Is it the tutorial server? Yes! Sorry for not clarifying that. Share this post Link to post Share on other sites
Posted October 23, 2020 I really like these tech notes! 1 Share this post Link to post Share on other sites
Posted October 24, 2020 (edited) - Edited August 8, 2021 by Yserin no longer playing, Per the mismanagment of AH update Share this post Link to post Share on other sites