Sign in to follow this  
Pandalet

Update on the lag issues - 4 Sept 2022

Recommended Posts

Hi folks,

 

As you will be aware, we have been suffering significant lag across the servers lately.  The development team have been looking into this over the past weeks, and we have made some progress:

  • We found an issue related to people moving items under certain circumstances
  • There was also an issue around cross-thread handling of one of the player buffers, where a buffer got some random data added in the middle; this was a side-effect of how the new CA chat channels were added that didn't play nicely between threads. 
  • Investigating the above issue also threw up a similar problem with how waystones and pathfinding works; both issues have been resolved, and the various threads told to straighten up and play nice with each others' buffers
  • We have added a significant amount of additional logging, which is showing up some other problem areas that need work

 

The out-of-synch restart last weekend was down to some extra logging that was making a lot more data than expected; to prevent disk issues, we needed a hot patch and restart.  Barring another oopsie, we should be back to regular restarts from now.

 

As we dig into the issues, it's becoming clear that there are a number of issues co-existing causing the problems.  Some of these have been around for a while, while some are recent additions.  This is an ongoing and high-priority investigation, and we are continuing to fix the problems.  Thank you for your patience and reports!  Things should be starting to improve now, and we're continuing to fix issues, so we expect the lag to decrease further back to the game being sensibly playable for everyone.

 

Pandalet

  • Like 30

Share this post


Link to post
Share on other sites

That was thorough! Thankyou!

Edited by SirEvo

Share this post


Link to post
Share on other sites

Thank you so much for the update and thanks to the entire development for your continued commitment to our community!

  • Like 2

Share this post


Link to post
Share on other sites

Hey, are there any updates on this?  While the game is technically playable and sometimes is even lag-free, the lag is becoming less and less pleasant to the point that other games are starting to look more appealing at times.  It's certainly causing me to wait on premium subscription decisions, which makes me a little sad.

 

  • Like 1

Share this post


Link to post
Share on other sites
8 hours ago, HerbalMentat said:

Oh forget it, not worth the effort.

Agree

Share this post


Link to post
Share on other sites

With today's update we addressed what looked to be one of the major causes of the recent lag spikes. Seems everyone truly has been out and about more as the most recent fix dealt with how we where sending clients info about catseyes in the world as you traveled around and it doing way more work then it really needed to on the server.

 

We will continue to monitor the situation and are waiting for the weekend to hit to so we can see how it helps at peak times vs weekdays before calling it resolved hopefully should that look good.

  • Like 5

Share this post


Link to post
Share on other sites

The lag is still here on cadence. ☹️

 

After the changes it's ok 1-2 days after restart but getting worse the longer the server runs. Currently it's hard to play. On lot of action stamina regeneration having lot of delay is the worst about it. The delay on queuing action and opening doors/gates takes lot of fun out of the game.

 

If you're unable to fix the issues maybe it's time to switch from weekly restarts to daily restarts...

Share this post


Link to post
Share on other sites
2 hours ago, timowi said:

The lag is still here on cadence. ☹️

 

After the changes it's ok 1-2 days after restart but getting worse the longer the server runs. Currently it's hard to play. On lot of action stamina regeneration having lot of delay is the worst about it. The delay on queuing action and opening doors/gates takes lot of fun out of the game.

 

If you're unable to fix the issues maybe it's time to switch from weekly restarts to daily restarts...

 

Independence is also starting to lag noticeably, although I think it's better than it was. I hope the source can be tracked down and fixed!

Share this post


Link to post
Share on other sites

We are still looking into it and have not closed this out yet, while we saw improvements after the last fixes we still clearly need to identify the cause of these remaining spikes before calling this resolved. Will update when we have more information.

  • Like 6

Share this post


Link to post
Share on other sites

Just wondering if there is any further update?  Have you guys been able to profile the code to find out where it's spending the most time during the lag events, or if nothing else, have informational debugging statements been able to tell you what's going wrong?

 

From the recent public dragon slaying event on Harmony, the lag felt like it did before the most recent changes.  It made me think one sticky point would be all of the "local" calculations among the hundreds of constantly moving and chatting players, so there is probably room for improvement there.  Maybe it's something else entirely, who knows.  Player position updates were exceedingly slow for the server to acknowledge, and I harvested a fruit tree while visually several hundred tiles away from it, just because I stopped there for a moment and gave it a whirl.

 

Any other progress or thoughts/ideas yet?

 

Share this post


Link to post
Share on other sites
On 10/4/2022 at 2:07 AM, lyhrinae said:

There's an update on this post, @Zakiah

 

That's not an update to the posts in this thread.  That post is over a month old.

 

 

 

Share this post


Link to post
Share on other sites

Sorry, I didn't realize it was on the same day. My bad. I just knew I saw it somewhere else. 

Share this post


Link to post
Share on other sites

@DarklordsLag is still terrible on Cadence. 9-10 second delay between actions and a long time for the stamina bar to refill. When can we expect a proper fix to this as this makes the game unenjoyable and unplayable at times.

Share this post


Link to post
Share on other sites
1 hour ago, BattleWall said:

@DarklordsLag is still terrible on Cadence. 9-10 second delay between actions and a long time for the stamina bar to refill. When can we expect a proper fix to this as this makes the game unenjoyable and unplayable at times.

We have been throwing fixes in for it week to week during resets but they are not seeming to have the impact we where hoping for. We are still investigating this and hoping some new tools soon will help us really narrow down the cause of the spikes. My apologies to everyone having to deal with this while we track it down and I appreciate your patience while we continue to work through the issues on Cadence this remains high priority for us.

  • Like 4

Share this post


Link to post
Share on other sites
20 hours ago, Darklords said:

We have been throwing fixes in for it week to week during resets but they are not seeming to have the impact we where hoping for. We are still investigating this and hoping some new tools soon will help us really narrow down the cause of the spikes. My apologies to everyone having to deal with this while we track it down and I appreciate your patience while we continue to work through the issues on Cadence this remains high priority for us.

Thanks for the update I am just going to inform you that it is back yet agian. It was fine for about an hour then it started up agian.

Share this post


Link to post
Share on other sites

You know this issue has been ongoing for more then a month now. It really hurts some actions such as imping, collecting resourses amongst others. How about until fix is found and implemented, turning off burning of Sleep Bonus? 

Share this post


Link to post
Share on other sites
16 hours ago, SouthPaw said:

You know this issue has been ongoing for more then a month now. It really hurts some actions such as imping, collecting resourses amongst others. How about until fix is found and implemented, turning off burning of Sleep Bonus? 

 

Several months.

Share this post


Link to post
Share on other sites

not sure if this is related to the lag.. but the other day i had my wagoner decide to go cross country instead of following the hwy..   it entered my towns center which only has regular roads..  it then decided to climb a 30 grade thru a fence and thru a building..  then it crossed the highway which it ignored .. to continue cross country.. thru a deed i know is locked out.. (no access except locked gates) then thru another's deed where i lost track of it..   again this may not be related to the lag.. but in some way it could possibly.. thus i am sharing this with you..   to day I see the wagoner wagons following the highway but only 2 tile at a time..   i suspect the lag issue is hidden in an odd place..  thus it may or may not be related to wagoner wagons deciding to travel cross country instead of following established highways..   and this is by no means a complaint.. just trying to help you guys out .. i love the game...  I just wish i was more knowledgeable in programming..   but don't let all this lag BS bring down your spirits.. we all support the work you do to make our experience better..   Oh if i recall , i read something where bandwidth helps increase data processing but i know very little on this..  just putting it out there from a guy that knows little but wants to help.. 

hope your day and weeks are positive..   thumbs up for all your efforts to fix this..    

Share this post


Link to post
Share on other sites

Any new updates on the lag issues? Cadence seems to be much smoother thus far, so the fixes seem to be starting to work!

 

I noticed that on the Post about Xanadu being down for 1hr 20mins that it was already said that they would be receiving 5hr of sleep bonus. I've not seen any word on if those of us dealing with lag issues on Cad for months now, taking up dozens of hours, will be receiving any form of SB or prem time credited to accounts for all the lost play time.

 

Thanks, and I look forward to all the new updates and changes!

  • Like 2

Share this post


Link to post
Share on other sites
5 hours ago, gnomegates said:

Any new updates on the lag issues? Cadence seems to be much smoother thus far, so the fixes seem to be starting to work!

 

I noticed that on the Post about Xanadu being down for 1hr 20mins that it was already said that they would be receiving 5hr of sleep bonus. I've not seen any word on if those of us dealing with lag issues on Cad for months now, taking up dozens of hours, will be receiving any form of SB or prem time credited to accounts for all the lost play time.

 

Thanks, and I look forward to all the new updates and changes!

The sleep bonus would be for everyone in general for the issues as of late and any upgrade pains we see as we continue to work on these issues. Glad to hear that they seem to be having a positive impact though.

  • Like 3

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this