Sign in to follow this  
Budda

Devblog: Movement & Desync

Recommended Posts

 

MvU90diHOaGadoZ9x8kPmRKdagrtHZDz7WBsMPOF

 

In this week's patch notes there was a small line that a lot of people would have glossed over, that being “Fixed a number of movement issues.” There’s a long story behind this one, with a few weeks total of working on it and something completely insane that I had to fix, so I wanted to write a bit about it - if you’ll forgive my rambling.

Note: Before we start this I should point out that in the context of this post, "desync" is referring to a creature or player appearing at one location from your view, but from their view they're in a different spot. This isn't related to the "Your position on the server is not updated." message that you can sometimes get when the server itself is lagging.

 

Hopefully by now some people have noticed a bit smoother movement in general (ignoring today's horses being a bit jittery on bridges), and a lot less (ideally none) movement desync that the game has had troubles with for a long time. These desync issues were magnified with the Fishing update and became a lot more apparent over the last few months. You might have encountered this on your boat by disembarking after a long trip and being a tile or two away from where you thought you were, or by leading a creature a long distance and it slowly getting farther and farther away from you.

 

The cause of the desync becoming more apparent was a change I did months ago for fish. Each fish in the game is technically a creature the same as any other and the server controls it as such. When you cast your line it decides when your fish will spawn, then tells it to move towards your line where it stops for a bit. Early on we ran into issues where the fish wouldn’t always stop right on your line, but sometimes off to one side or the other. To get into the reason for this issue and how I fixed it we have to dive a bit into how the movement updates in Wurm work.

 

Movement is calculated on the client based on your input which then sends your new position to the server. Server then checks that data against what you’re allowed and expected to actually do with that movement then if everything is okay it saves that as your new position and updates everything that needs to know where you are. This includes any other players that can see you, all the creatures within range of you, anything you’re riding or dragging or leading and a few other things. When doing these updates they’ve historically been done using position differences. The difference is passed along and calculations are done based on that instead of the exact new position. This is where the problem starts.

 

qnuljQM.png

 

For reasons of keeping data packets to all clients as low as possible, all movement difference values got packaged into byte values (which is a quarter of the data that sending the full position value would be), meaning a number in the range of -127 to +127. This means that when you’re moving from [1587.33, 903.19] to [1587.39, 903.81] the X and Y differences have to be converted into that -127 to 127 range, whole numbers only. To accomplish this the differences were multiplied by 10, which had the downside of ignoring any movement difference below 10cm (0.1) since that would get rounded down to 0cm when converted into a whole number byte. Here’s where we get back to the fish problem. Because the final movement of the fish to get to the hook was sometimes less than 10cm the update packet sent to the client was rounded down to 0cm and ignored. After some testing to figure out the actual movement values being sent to the client was never getting beyond the -7 to +7 range, I changed that x10 multiplier into a x100 multiplier. This gave us an extra degree of precision in movement, while keeping the movement difference values inside that -127 to +127 range.

 

Although this fixed the fishy issue that we were having at the time, it brought to light some new desync issues and made some existing ones worse. At that time the biggest issue was stamina drain problems because of this change - I had thought it was because I missed some x10 -> x100 change at some point, but after going through the related stamina drain code it didn’t make sense. I gave up some point after and tweaked the draining code to be as close to live as it was before the change and chalked it up to something weird that I might find later on. The last few weeks was that ‘later on’ where I found out that my initial thought of “this doesn’t make sense” was correct.

 

bWWDcwZ0-lpHCmPcPuibazPu0og0TYYP41435cxC

 

You may have noticed that I glossed over what happened in the old code when those difference values were rounded down to 0cm if they didn’t hit that 10cm (or 1cm after the x100 change) threshold. This is the source of the desync that the game has had in varying degrees since day one. When these values were rounded down to 0cm (or remainders from a whole number were rounded down), the movement would still be applied but the “update everything else” code would see it as no movement and not run. This means there could be some movement ticks where nothing was sent to players that were in range, or stamina would not drain at all if it was a player’s movement. But then how has everything still mostly worked all of this time if these movement differences weren’t updating properly?

 

Now we get to the meat of the issue. The code that transformed the new position values into a difference value was prone to a rounding error that every now and then would cause a movement difference that is usually under 1cm to get rounded up to 10cm and trigger the chain of updates to everything and stamina drain. When the difference code was working on 10cm intervals, the rounding error just happened to occur about as often as the “lost movement” from rounding down would have added up to 10cm, so it kind of balanced itself out. Changing it to 1cm intervals meant less data was lost to rounding down, but also meant when the rounding-up error did occur it was only triggering enough movement for 1cm, meaning it lost the ‘happy accident of balance’ that it previously had which lead to desync problems being slightly more pronounced over the last couple of months.

 

During my trek over the last few weeks into figuring out the desync issues Samool and I came across that piece code and recognised that it could cause some precision errors, so I changed it to something that wouldn’t. This lead me to a couple days of pulling my hair out over stamina no longer draining as expected before realising the rounding error was the reason it ever worked properly in the first place, and that the code down the chain relied on it to work 99% of the time for smaller movements. Without having this rounding error, slow movement will very rarely be above that 1cm threshold, meaning none of the drain code or update code was ever called, since the movement was rounded down to 0.

 

Fixing this new issue involved changing the entire chain of code to not use the byte value differences, and instead use the actual position data and calculating exact difference values when needed, eliminating any need for rounding down (or random errors causing rounding up). This also lead to a couple days of rebalancing stamina drain from movement to be as close to live as possible considering the old system relied on a random rounding error to work at all. I’m truly surprised that it worked as well as it did being based on an error for smaller movements like that.

 

The upside of all of this is that movement updates sent to players now use the exact position instead of difference values, meaning desync should be a thing of the past. The place that you see other creatures and players is the exact same position that the server sees that creature of player - and stamina draining should be a lot more precise for smaller movements instead of relying on a rounding error.

 

I hope I never have to touch movement code again.

 

  • Like 39

Share this post


Link to post
Share on other sites

Thank you for the detailed info! Desync issue was an awful thing we lived together with it is huge news to say goodbye to it. It is much more common than anyone would expect that complex systems work BECAUSE of multiple errors in the code which end up in an expected behaviour.

 

 

So how about jumping next? : P

 

Edited by Jaz
  • Like 2

Share this post


Link to post
Share on other sites

Does this mean xanadu desync is fixed its been bad for like 5 years. Noticed its been alot better lately :P

 

Glad indy had issues so something finally was done about xan lag :P

 

and thank you Devs

Share this post


Link to post
Share on other sites

now we need this fixed   

While mouselooking, you get randomly turned around. Therefore, you get disoriented

  • Like 3

Share this post


Link to post
Share on other sites
10 minutes ago, icenrns said:

now we need this fixed   

While mouselooking, you get randomly turned around. Therefore, you get disoriented

 

That one is fixed in the preview client already.

  • Like 7

Share this post


Link to post
Share on other sites

Wait a minute, I thought making video game engines was easy?

 

I like this dev blog.

 

 

Edited by Chakron
  • Like 2

Share this post


Link to post
Share on other sites

-snip-

 

IMO, I guess I expected it to be something like that. It's a perfectly ironic system, that Wurm itself relied on an error to function.

Edited by Ignacius
  • Like 1

Share this post


Link to post
Share on other sites

my appoligies for the language in advance, but HOLY ****!!!!

talk about WOGIC!

Share this post


Link to post
Share on other sites
7 hours ago, Budda said:

 

That one is fixed in the preview client already.

Fun thing is on Linux I never had that issue but the preview client has introduced a very similar behaviour on Linux. 

Share this post


Link to post
Share on other sites
7 hours ago, Budda said:

 

That one is fixed in the preview client already.

It still occurs with some mouses even in the preview client. I had the problems with a Logitech mouse and normal client, switched to preview client and it was fixed. Then like a month ago I bought a new mouse(Corsair Ironclaw RGB) and the issues came back.

 

Anyway, thanks for the devblog. Was an interesting read.

Share this post


Link to post
Share on other sites

I hope one day we can have carts and wagons that follow the terrain back as well...so perhaps you may have to touch movement code again...

 

Also, horses never unhitching in building walls and corners, and perhaps having a shorter rope so we can lead and breed horses into one tile areas.

 

Thanks for the hard work :) Wurm is a great game and despite relying on multiple errors to work most of the time, it is all part of the charm.

  • Like 4

Share this post


Link to post
Share on other sites

Uh, I hate to tell you, but desync still happens.  i was out yesterday collecting fragments and when I got back to my village my alt and I were desynced.  I saw him standing at the Fo altar, when he was standing at the Vynora altar, 1 tile west.  I guess you still have some work to do on that code.

Share this post


Link to post
Share on other sites
7 hours ago, Vroomfondel said:

Uh, I hate to tell you, but desync still happens.  i was out yesterday collecting fragments and when I got back to my village my alt and I were desynced.  I saw him standing at the Fo altar, when he was standing at the Vynora altar, 1 tile west.  I guess you still have some work to do on that code.

 

That shouldn't be possible anymore. Can you post a bug report in the server bugs section with some more info if you come across it again? Screenshots from both perspectives, what you were doing, where you were doing it etc.

The only thing I can think of is some mismatching range bug where you were just on the edge of local range of your alt where he wasn't removed from your local, but you also weren't receiving movement updates due to being out of local, then you came back in range and he didn't move at all after that.

Currently if there is some weird edge case like this, then any sort of slight movement from the alt (including moving the camera view at all) would force its proper position to be sent to anyone in range, moving him back to where he actually is for anyone in range and ending any possible desync.

  • Like 1

Share this post


Link to post
Share on other sites

Thanks Budda!

 

Do you know whether this fix can be included in the upcoming WU 1.9.0.0. beta update?  As it may be a while till the next WU update after this one.  

 

Is it an easy thing to patch into the WU code? I am very glad it got resolved as I recall severe "cart desyncs" in the tutorial update on the live servers (WO) that made the game frankly unplayable, and am hoping we won't get six months of the same in WU in case it gets "held over" till the next update down the road.  Would be lovely if it is something simple enough to pop into the code while we are still beta testing 1.9.0.0.

 

 

Edited by Brash_Endeavors

Share this post


Link to post
Share on other sites
5 minutes ago, Brash_Endeavors said:

Is it an easy thing to patch into the WU code? I am very glad it got resolved as I recall severe "cart desyncs" in the tutorial update on the live servers (WO) that made the game frankly unplayable, and am hoping we won't get six months of the same in WU in case it gets "held over" till the next update down the road.  Would be lovely if it is something simple enough to pop into the code while we are still beta testing 1.9.0.0.

 

The cart desync was adjusted fairly quickly, but this will be coming to WU in the beta, yes 

  • Like 2

Share this post


Link to post
Share on other sites

Does this mean PvP in boats will be less janky? It has been a long time since I have done any PvP in boats, but in the past, you could be right on top of someone and it would say you were too far away.

  • Like 1

Share this post


Link to post
Share on other sites
2 hours ago, Shannara360 said:

Does this mean PvP in boats will be less janky? It has been a long time since I have done any PvP in boats, but in the past, you could be right on top of someone and it would say you were too far away.

 

I would say try a bout out and see what happens, report back if it's wonky - because it shouldn't be!

Share this post


Link to post
Share on other sites
On 3/16/2019 at 12:38 AM, Budda said:

and a lot less (ideally none) movement desync that the game has had troubles with for a long time.

 

Sorry but it's still quite wonky.

 

Yesterday I was rushed by a few folks and on my screen I dropped down a big rock shaft and landed on the floor (enemies were above) though they still manged to hit me and kill me, my corpse however was still on the top of the rock slope.

Never had that happen to me before.

 

Another example is where me and @Tedzoghhad a 1v1 that he streamed, on my screen he was facing away and I hit him in the back, on his screen he was hitting me in the back.
We stood like that for a very long time.

 

Maybe Ted can find the video?

Edited by Sn00

Share this post


Link to post
Share on other sites

I didn't record that but should still be on twitch, i'll see if i can snip it out of there.

 

Btw, was realy scary while riding yesterday, my backup horse was just behind me all the time and not 19+tiles away, so he kept scaring me ?

Edited by Tedzogh

Share this post


Link to post
Share on other sites
13 minutes ago, Tedzogh said:

I didn't record that but should still be on twitch, i'll see if i can snip it out of there.

 

Btw, was realy scary while riding yesterday, my backup horse was just behind me all the time and not 19+tiles away, so he kept scaring me ?

 

Thanks for that, during the fight last night I was still ontop of the rock shaft when I died correct? (That's what my kingdom members tell me)

Share this post


Link to post
Share on other sites

iirc you did jump down Before you died on my screen, i will look for that aswell.

 

ok found it, at 9min in there. You did die mid falling.

 

Edited by Tedzogh

Share this post


Link to post
Share on other sites
9 hours ago, Tedzogh said:

iirc you did jump down Before you died on my screen, i will look for that aswell.

 

ok found it, at 9min in there. You did die mid falling.

 

 

Interesting.

 

On my screen I was at the bottom there, starting to walk away

 

@Budda

Share this post


Link to post
Share on other sites

The one at 9min just seems like good old lag to me, not movement desync. If the server is lagging slightly and a bit behind in processing the movements sent from the client (same thing as the cause of the "please move slower" message) then it can seem like your client is a bit away from where others see you at that exact moment. The rest of your movements were likely still processing when the final hit on you happened.

 

I'll have a look and see what I can do about those processing times, but can't promise anything in the short term.

 

I can't really see what you're pointing out at the 32:30 part of the video, is it something further on that just starts there?

 

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this