To allow for easier spread and ensure players of all time zones can attend one, the Melody rift will be delayed by 12 hours

Keenan

Developer
  • Content Count

    2190
  • Joined

  • Last visited

  • Days Won

    41

Keenan last won the day on December 9 2020

Keenan had the most liked content!

Community Reputation

3376 Rare

About Keenan

  • Rank
    Villager
  • Birthday 07/02/1981

Profile Information

  • Gender
    Male
  • Location
    (final String _locale)

Accounts

  • Chaos
    Keenan
  • Exodus
    Xorith

Recent Profile Visitors

6959 profile views
  1. Today we suffered an outage of our login server Golden Valley during a maintenance restart of all servers. The host became unreachable and an investigation into the issue began immediately. It took roughly an hour and a half to discover the issue and another thirty minutes for our old hosting provider to address the issue. The issue has been resolved completely going forward and there was no data loss. This issue was unique in that it was an accounting error, not a technical one. We are still working on migrating Golden Valley to our new provider. As mentioned before, this is a complicated migration due to the shop existing next to the login server. That creates a few moving parts that need to work, and we lack a proper way to test the environment together as the test servers are not designed to work with an attached shop. We hope to have an update on this migration process soon. Thank you all for your patience.
  2. The issue should be resolved now. Thank you for reporting this to us.
  3. It's just getting bumped to me. I'll have a look. I'm not sure why it would think 'tar.gz' is plain/text.
  4. Hi all, The login server Golden Valley suffered two outages today. The first began around 5:12am CET and lasted approximately 5 minutes. The second outage began at 5:25am CET and we believe was a direct result of the first outage. The second outage was resolved at 5:38 am CET with all servers having login connectivity shortly thereafter. What we suspect happened was a network outage with our old provider, where Golden Valley runs still, caused a backlog that then caused Golden Valley to outright crash. The network outage also caused an outage in the tunnels we use to talk between the providers, which lead to some servers having login connectivity while others did not until those tunnels were restored. The solution is to move Golden Valley as soon as possible. If you were affected by the outage in-game, such as missed shop purchases or missing boats on crossing, then please submit a /support ticket and a team member will assist you. Thank you and happy Wurming!
  5. Yes! Sorry for not clarifying that.
  6. So a few things, as I wrote plot course. Plot Course does not use neighbors at all. It works based on the servers in your SERVERS table. That said, your neighbors table has a few errors that could be affecting things. 1) The 'SERVER' column should always be the ID of the local server. 2) Direction should not overlap, i.e. you should not have two WESTs or EASTs. You can give them arbitrary names up to 10 characters for servers not connected via cardinal direction. (e.g.: EASTSERVER)
  7. Hi all. Yesterday and early today we suffered from some server stability issues. Namely with Harmony, but also with Cadence, Deliverance, and Haven (tutorial server). The underlying cause has been fixed and patched. The technical details of the issue has to do with our database pooling. We discovered a connection leak in the code that purges 90-day inactive non-premium accounts. With the numbers we saw at launch on Steam and the fact that we've hit 90 days since launch, we're seeing a high number of inactive accounts from Northern Isles getting purged. This caused the database connections to spike over the max connections allowed for the database causing the first Harmony downtime. To mitigate this before a patch was available, we increased the max connections on the database. This had an unintended result of hanging Harmony later on as there's a hard-coded max for connections on the server itself. The result was Golden Valley was hanging waiting on Harmony, and then player transfers on Deliverance and Cadence caused those servers to hang waiting on Golden Valley. A nice domino effect. The resolution was to shutdown Harmony and bring it back up on the patched server code. This restored the other servers to working order. Later on Haven hit the same issue due to it being released shortly after launch. Haven was also brought up on patched code.
  8. This is not a bug. It is working as intended, even if the intention is a bit annoying at times. The suggestion to change them has been noted and we are aware of it.
  9. Hello all! As previously noted, we will be doing server migrations to Cloudnet today. The plan is to do three batches, two servers at a time. We will not be taking down any other servers except the ones being migrated. First batch: Deliverance Exodus Second batch: Release Chaos Third batch: Affliction Desertion We strongly recommend that you avoid mail and travel with the above mentioned servers until the migrations are complete. We will update this post and broadcast in-game with the status of the migrations. Expected downtime for each batch is about 1-1.5 hours. STATUS UPDATES Mon Oct 05 14:49:58 CEST 2020: Starting shutdown of Deliverance and Exodus. Mon Oct 05 15:39:12 CEST 2020: Copying data over for Deliverance and Exodus. Mon Oct 05 16:19:25 CEST 2020: Final checks before bringing servers back online. Will be a little over the 1.5 hour mark when we're back. Mon Oct 05 16:41:55 CEST 2020: Deliverance and Exodus migrated successfully and back online. Mon Oct 05 17:21:10 CEST 2020: Chaos and Release offline, data is being migrated. Mon Oct 05 17:58:19 CEST 2020: Chaos and Release are migrated successfully and back online. Mon Oct 05 18:05:11 CEST 2020: Shutdowns started for the final two servers: Affliction and Desertion Mon Oct 05 18:55:55 CEST 2020: Migration underway for Affliction and Desertion. Mon Oct 05 19:14:37 CEST 2020: All servers online! Migrations complete!
  10. There is apparently an issue with our Content Delivery Network (CDN) provider, which allows us to spread the pack downloads out globally for best speed. One work-around would be a VPN. Another would be a "hosts" file edit. On Mac and Linux it's /etc/hosts. On Windows it's C:\Windows\System32\drivers\etc\hosts. You will have to run Notepad as an administrator by right-clicking on the icon and clicking on Run As Administrator. Add this line to the hosts file and save: 185.59.223.26 updates.wurmonline.com This should be a temporary work-around and you will need to remove this if we decide to switch CDN providers!
  11. We don't currently have any APIs, though it's a wish list item to do so in the future. How public they are and what data they offer are yet to be determined.
  12. Hello all! Tomorrow I will migrate Pristine to our new hosting provider during our restart, which shouldn't impact downtime by too much. I have been migrating servers fairly quickly, so I expect this to take 30-45 minutes. I can bring all other servers back online after updating them with Pristine's new connection information as well. Then on October 5th I am planning on doing a major migration. I will be aiming to migrate Deliverance, Exodus, Release, Chaos, and Desertion (all other servers except Golden Valley) on that day, with a follow-up scheduled should anything arise that prevents me from completing them. These should be "live" migrations in the sense that I shouldn't need to take other servers down. This is quite tricky though, as it will impact those of you who travel or mail between these servers. We will post announcements in-game, on the forums, and in Discord to alert you on when this takes place. Finally, we will be planning the migration of Golden Valley and the Wurm Online shop. I'll make another post closer to October 5th as a reminder. Until then, Happy Wurming!
  13. Hello all! Due to the lag issue Elevation was experiencing, we decided to migrate it over to Cloudnet this evening. I'm pleased to say that the migration went smoothly and Elevation is now living happily on new hardware. Pristine is next on our list as it is also experiencing lag that we suspect to be related to hardware. Tomorrow I'll be prepping the new hardware to take our servers and then Wednesday/Thursday we plan to migrate Pristine. We are still deciding on the best time to do this migration, but it will either be close to noon server time or after midnight server time, hence the Wednesday/Thursday scheduling. I'm also working on a schedule for all other servers, and I hope to have that posted soon. Thank you all, and happy Wurming!
  14. I sent you a PM for a more real-time chat if possible. Can you try logging Kadminty in now?