Sign in to follow this  
Sarcaticous

WURM's Lag Solution for Xan

Recommended Posts

Dear WURM,

 

Chaos


 

lag-day.png

 

Indy


 

lag-day.png

 

Xanadu


 

lag-day.png

 

What appears to be the lag solution is make the lag bad all of the time. That way it doesn't look bad because it's consistently lagging. Looking at the graphs I'd say Xan is worse than all of the other servers combined. The question is why? If you have a server four times larger than any other server should it not be given four times the resources? Granted, About 25% of the server is sea surrounding the land mass but still, more land, more resources. If it isn't server resource problem then exactly what is it? 

 

1. Trying to drive people off the server to close it?

2. Trying to drive people from the game to close it?

3. Not enough money to get the server resources to properly run the server?

4. A bug that is nigh unto impossible to trace?

5. No one knows what the problem is and there is no more hair to pull out?

 

I can't think of any other reasons. I do know that after weeks and weeks of lag problems it really wears thin. I don't expect any kind of communication regarding the lag problem but it would go a long way towards soothing frayed nerves if something could be said from time to time. Like maybe once a week. What I'd especially like to see is, "Eureka, we've squashed the lag".

 

Best regards,

Sarcaticous

Edited by Sarcaticous
  • Like 1

Share this post


Link to post
Share on other sites

I think that it's still being worked on, but the devs are definitely aware of Xanadu currently being a bit problematic.


Share this post


Link to post
Share on other sites

Its a server 4 times bigger than Chaos, with lots more population due to the hype. What did you expect?


The more assets the server will have to handle the worse it will become, the minor "stress tests" done were a nice indication to the level of seriousness the team had in mind when structuring this.


Now of course it will take time, because being that big a server it increases the complexity of everything and just unveils the frailties of the game server code. In the end we might come out with a more streamlined game, or simply come out empty.


Its wurm, though, so don't hold your breath for a timely best case scenario. Do remind yourself that even the smaller servers have been plagued by lag since ever, and constantly relapse to that, Xanadu once again is just exacerbating the issues due to its excessive size.

  • Like 3

Share this post


Link to post
Share on other sites

lag-month.png


(the monthly graph for Xanadu)


 


The long, flat periods happen because the lagometer breaks for Xanadu, sometimes, and restarts fix it.  The server isn't stuck at a fixed amount of lag, and when this happens it isn't representative of the server's current lag.


  • Like 1

Share this post


Link to post
Share on other sites

Kane, I agree with some of what you say. And, as I said in the OP, if you have a server four times larger you commit to having at least four times the resources for it. Xanadu's size is not excessive. More likely the resources committed are perhaps inadequate or, as you said, there is a coding problem. I also realize the lagometer might not represent the true nature of things because this morning the lag has been much worse than the graph shows. But that's probably my problem and not a server problem, right?


Share this post


Link to post
Share on other sites

I hope they can do something to alleviate the spikes. Those get frustrating. I can live with a split-second or so of continuous lag (in Wurm). A full second is a bit much.


 


This game does track a lot of items and puts very few restrictions on allowing players to leave stuff lying all over the place. It's one of the few things that really makes this game special. I wonder if the advancing age of Xanadu is growing the database ever, ever larger and more cumbersome.


 


It gives me the fears.


Share this post


Link to post
Share on other sites

Kane, I agree with some of what you say. And, as I said in the OP, if you have a server four times larger you commit to having at least four times the resources for it. Xanadu's size is not excessive. More likely the resources committed are perhaps inadequate or, as you said, there is a coding problem. I also realize the lagometer might not represent the true nature of things because this morning the lag has been much worse than the graph shows. But that's probably my problem and not a server problem, right?

Might be, idk how accurate that thing is even... Also it really doesn't show the server's CPU Usage which would also be a good indication of whether its the server being taxed by data or not.

Its not a simple problem, and not a simple solution... And at the same time, i don't think CC can afford to have a "special" server for Xanadu. They tried to make it something extreme and advertised it as bigger than any other permanent sandbox world in existence. Bold words, and probably true. But at the same time its also one of the most broken Permanent sandbox worlds in existence because its too huge.

Travel time is too big, animals are too sparse, ores are too sparse, and its laggier.

 

Personally, and i said this often, i feel no empathy towards anyone on Xanadu. The only appeal of it was that it would be large to the extreme, and it is. Anyone with a inkling of how Wurm works would have predicted it would suck like that.

Everyone knew that compared to the smaller servers Inde and Chaos have way worse hunting grounds, Xanadu is 4 times bigger, so probably 16 times worse, specially since even just looking at our pens we know that AI tends to bunch critters all in the same place. (which on inde usually seem to be on mountain slopes).

Etc, etc.

  • Like 1

Share this post


Link to post
Share on other sites

As of this reply 428 on Xan. 418 on the rest of WURM. That's over half of the player base on Xanadu. Just sayin.


Share this post


Link to post
Share on other sites

There are a number of other online games that have thousands of players on a server. Placing blame on the map size or population of Xanadu is only an excuse for poor hardware, hosting service or whatever. Bottom line is that this should have been resolved long ago, but instead there is little response from the team attempting a solution.


Share this post


Link to post
Share on other sites

There are a number of other online games that have thousands of players on a server. Placing blame on the map size or population of Xanadu is only an excuse for poor hardware, hosting service or whatever. Bottom line is that this should have been resolved long ago, but instead there is little response from the team attempting a solution.

Ok, i'll explain.

 

Lets take GW2 for an instance (because i know for a fact how many people can go into at least one of the maps) it has a 500 limit per server per WvW map, which brings it to 1500 players.

Now on that map there's mobs like Wurm, there's some stuff you can build wherever you want in any numbers you want. There's players running around and players fighting and whatnot, and its pretty awesome, and i'm also playing it instead of Wurm.

BUT

Wurm has all that plus 100% editable terrain that needs to be tracked by server, animals multiplay, they don't just respawn in the same numbers, players generate more, Xanadu is bigger than most permanently generated game maps that have fixed terrain and features.

 

Add to that the endless list of items a player produces, multiply it by the number of players, add trees, add crops, and buildings, and you'll start to realize that Wurm's servers do need to handle more stuff than most other games. And what's worse Wurm isn't very optimized client side at least.

 

All that doesn't excuse the devs, they decided they could make the server. Thing is, the players that also decided to move there aren't exempt of blame, at least not the experienced ones that KNEW how bad wurm behaves on the smaller servers, who knew how long travel is, who knew how bad hunt was on Inde and Chaos compared to Exo, Deli, etc.

They KNEW server size would impact that, and yet chose to move. Now its not working ideally, so yeah, i get, and i think that people should complain to the devs, but also please remember you made the choice of deserting the old servers. And yeah, i resent that, half the people on the old servers left, and half of those remaining ended up quitting because, like them or not, its interacting with others that makes this game better than Minecraft and other non Open World MMO Sandboxes out there.

 

I can't fault anyone for having moved, it was their decision as was mine to stay behind, i had to deal with being pretty much playing alone, and ended up quitting, was the way i had to own up to my choices. I'm not telling anyone to quit, but own up to your choices as well, and press the devs about improving, but you don't need to do it constantly by opening 10 different threads.

 

Specially not a thread titled "Solution to Lag" but that doesn't really present any.

Edited by KanePT

Share this post


Link to post
Share on other sites

 

They KNEW server size would impact that, and yet chose to move. Now its not working ideally, so yeah, i get, and i think that people should complain to the devs, but also please remember you made the choice of deserting the old servers. And yeah, i resent that, half the people on the old servers left, and half of those remaining ended up quitting because, like them or not, its interacting with others that makes this game better than Minecraft and other non Open World MMO Sandboxes out there.

 

Being butthurt about people "deserting" you doesn't change the fact that dev response time in handling a bad situation is very poor around here.

Telling players they deserve to deal with bugs for months, if not years, because they chose to leave one server for another is wrong. Let me also remind you that certain bugs persist in all server locations. The lag everyones crying about on Xan? Yeah, we had it for 3 years on Indy when the game was hosted on Swedish servers. 3 years of 1+2 extra seconds to gain stamina on every action...

Let me also remind you that it took the dev team almost 6 months to flesh out the altar influence problem on Xanadu.

Lastly, to prove my point about all servers being "buggy", let me bring up the animal spawning issue. Xanadu is not the only map in this game that has issues where most animals spawn/relocate to the north. That happens on every server. It's part of what made me leave Indy for Xan.

Edited by As_I_Decay

Share this post


Link to post
Share on other sites

This OP is silly, except the fact that Xanadu lags a lot more than other servers. It's very annoying, it can drive people from the server, I know for a fact that it does, I've seen plenty leave with for that reason.


 


Even so...


 


Improving performance is almost never about finding one bug or just adding more hardware. Two times the hardware doesn't mean double performance, not in complex implementations like game servers, databases or whatever. More hardware can be a good option if you know that your code already performs well and that it can scale well to make use of the hardware. If you know that you can handle 200 users with current hardware and adding one more processor means you can handle 150 more then sure, why not, but it is seldom the case. Often you don't know the exact gain from more hardware.


 


The more optimized your code already is the harder it is to find something that makes it even better. I don't know the server code of Wurm of course, but I can imagine that optimization has been going on from day one and is a constant thing they are working with. Every new feature needs performance optimization before release to the public. 


 


When you have done performance tuning for many iterations already and you can't really get it to run much faster you need to start thinking about completely other implementations for parts of the system or the system as a whole, but such solutions are expensive and time consuming. I know the devs are considering it for the client but I've never heard them say such things about server side stuff. Although I'm sure they rewrite parts of it now and then. 


 


Fixing the lag by just fixing one bug, while it can occasionally happen, I wouldn't count on that being the solution here. If it was that simple I think the other servers wouldn't work so well either. Xanadu is just huge and some tasks in the server probably grow exponentially with world size and item counts, making the demand on performance extreme.


 


Not saying the lag is unfixable. More hardware in combination with fixing some flaws in the server code could very well make it a lot better, but it's understandable if it takes time.


 


We should of course keep complaining to keep the devs working on it :)


Share this post


Link to post
Share on other sites

Hey, computer science student here. Hope it is ok to stick my nose in. Those graphs are pretty interesting in that they show a uniform bottleneck.


 


TL;DR at the bottom for my suggested solution.


Please try to give it a read anyway though. :D


 


So lets talk about some of the stuff that can cause lag. People like to argue about computers so lets reduce the scale of the term lag to any time you (the clientside user) have to wait for an action to be performed and get feedback from the game. I'm going to generalize a lot, but lag usually comes down to one of the following:


 


- Waiting for a thread to do work on. (You can think of it like waiting for a processor to be ready for work)


- Waiting for memory to become available in RAM. (Where we want to put the data the server is using to operate the game)


- Waiting for swap space to be used. (Where we would rather not put the data the server is using to operate the game, but if we need more RAM and it isn't important, so be it)


- Waiting for a programmatically mandated pause.


- Waiting for the internet connection to be in a ready state. (This includes waiting for the host to route the connection as well as latency to the target machine in general)


- Waiting to hear back from a client. (This generally is not an issue in modern software)


 


What those lag graphs show us is that the bottom two points are probably a pretty straight shot yes or no answer. Unless we as the playerbase did something silly like moving near each other in real life, I wouldn't expect to find a uniform line like that over the course of a day. And if we did, we probably would notice more turbulence and other oddities on our end. As Wurm's playerbase is pretty diverse, that one is probably out.


 


I went poking around those graphs and there is some pretty interesting stuff to see. For example, dbcreaturespos-month (Creature movements counted over the month) shows us something interesting:


 


Wild


dbcreaturepos-month.png


 


Indy


dbcreaturepos-month.png


 


Xanadu


dbcreaturepos-month.png


 


See those places where it flattens out and kind of forms a roof? Look around the end of week 7 and beginning of week 8 for wild, or the more pronounced ones on Xanadu at the end of week 5, middle of week 7 and middle of week 8. Those imply one of two things. Either the GM and/or Dev team have a button or command they can push that asks the server to throttle animal movement, or the server host is exercising their right to manually throttle processing time, and the animal movement loses out.


 


With this observation we can assume that animal movement is likely very processor intensive, even though we can't look at the code to check. Now, why can we do this? Well we have a bit of common ground in how computers work to go off of (more on this later), we know that the movement can be manually throttled (and by extension, we know the people who are familiar with the workings of the server find it to be a problem), and we also have more charts! Lets look at the creatures-month (Living creatures counted over the month) charts.


 


Wild


creatures-month.png


 


Indy


creatures-month.png


 


Xanadu


creatures-month.png


 


Wild and Indy see some pretty similar statistics. We can swing wide and say their population is probably soft capped somewhere between 25k and 27k. I would bet that Xanadu was initially capped at 100k and it got bumped up a bit due to complaints that animals weren't readily available? Dunno, I don't watch Xanadu discussion and I've been gone for months. In any case, the soft cap for Xanadu looks like 110k to 120k, which is about right for a server 4x the size of the others.


 


(Side note: We can ignore the blue bars completely for these because they are likely record highs for the server. Likely higher when it first opened or a bug in the spawn code caused lots of creatures to go unnoticed by the limiter and has since been handled. Spider pits, anyone?)


 


So, to review: We think that creature movement is intensive, possibly our bottleneck. We know that Xanadu is 4x the size of the other two given statistics and has figures representative of that, yet does not lag to the same extreme. We know there are approximately 4.5x the number of creatures on Xanadu that the others have. Most importantly, we know the latency is at exactly a second. What can we draw from this?


 


- Maybe the timings have been hard capped by staff to give the server more breathing room? (A bandage to keep the game running while a solution is implemented, if so. This at least tells you they are looking at the issue.)


- Maybe the load of animals doesn't scale linearly to the extent map handling does? (Due to pathing algorithm? Stuck critters no think gewd?)


- Game logic dictates constraints that say players movement and creature movement need to occur in order? (It wouldn't be much fun to fight a dragon that couldn't hit back while walking.)


- Creature handling only occurring on one thread? (Probably not, but still worth checking. If it was, it would be more likely that animals moved and reacted sluggishly, but the game itself was fine. Still might be better...?)


- Inefficient pathing algorithm? (Mo' tiles mo' problems?)


 


My suggested solutions:


 


- Chunked map units if they aren't in already. If a bear poops in the woods and nobody is around to smell it, then don't make the bear need to poop. As much "realism" as roaming creatures may add to the game, it doesn't need to be literal unless someone is occupying the space those creatures are in. Approximate the things that would have happened if someone had been there to see it, then update the zone during the work lulls. If you can't make time, then who cares where the creatures are moving if nobody can move enough to experience it? With this, you could unload unused areas completely until they are needed. You can't move what isn't in memory. Also, makes breaking the server up into shards easier (if it ever comes to that) and will scale better as multi-core processors continue to evolve.


 


- Treat groups of creatures as one pathing entity. This is the bigger one since it is simpler to implement with what is currently in place and doesn't require a major rewrite of the tick handling engine. If I have 90 scorpions in an area pathing about somewhere that may eventually reach a player, help them along a bit. Group them as 1 pathing entity, treat every x by y area of tiles (or chunks if you have them. :D) as a single unit for the purposes of pathing, and slowly do A* pathing until you find players in an adjacent unit. When you do, pick a spot in your current unit and distribute your monsters with random offsets from a random point in the current unit.


 


Congratulations, you just introduced a new game mechanic that gives the players reason to help reduce lag. You can bet that if there's an opportunity for 350 wolves to show up at the border of someone's deed every once in a while, players will organize to share the fun. At this point, you really don't care if a newbie dies to 75 giant spiders, they would have died to one.


Share this post


Link to post
Share on other sites

Rolf just needs to give the hamsters better living conditions.

Share this post


Link to post
Share on other sites

Throwing more horsepower at it may not fix the issue if there is something leaking processing time or if the server software isn't written to take advantage of more cores. Processors are going in the more cores working more efficiently direction instead of more power, same number of cores.


Edited by Zerocool

Share this post


Link to post
Share on other sites

Being butthurt about people "deserting" you doesn't change the fact that dev response time in handling a bad situation is very poor around here.

Telling players they deserve to deal with bugs for months, if not years, because they chose to leave one server for another is wrong. Let me also remind you that certain bugs persist in all server locations. The lag everyones crying about on Xan? Yeah, we had it for 3 years on Indy when the game was hosted on Swedish servers. 3 years of 1+2 extra seconds to gain stamina on every action...

Let me also remind you that it took the dev team almost 6 months to flesh out the altar influence problem on Xanadu.

Lastly, to prove my point about all servers being "buggy", let me bring up the animal spawning issue. Xanadu is not the only map in this game that has issues where most animals spawn/relocate to the north. That happens on every server. It's part of what made me leave Indy for Xan.

Maybe you should read my entire post. I simply stated that anyone experienced with Wurm should know better, not excusing the devs... In fact you should know by now i'm the last person excusing the devs.

But truth remains EVERYONE knew that Inde had issues due to its size, how do you think a server 4 times bigger than inde would fare?

 

Throwing more horsepower at it may not fix the issue if there is something leaking processing time or if the server software isn't written to take advantage of more cores. Processors are going in the more cores working more efficiently direction instead of more power, same number of cores.

Might not even be a CPU issue... We don't have data on that.

Share this post


Link to post
Share on other sites

Might not even be a CPU issue... We don't have data on that.

This is true, but we have data that would indicate a bottleneck. It can really only go 3 ways: too many packets going out, too much ram in and out of use, or not enough processor time available for work.

Share this post


Link to post
Share on other sites

Maybe you should read my entire post. I simply stated that anyone experienced with Wurm should know better, not excusing the devs... In fact you should know by now i'm the last person excusing the devs.

But truth remains EVERYONE knew that Inde had issues due to its size, how do you think a server 4 times bigger than inde would fare?

 

I did read your whole post. Did you think I randomly chose 5/6th through the post to pick out one part to read?

I agree with what you wrote, aside from the part where you say "the players are not exempt from blame." I wouldn't go so far as to pin the blame on all of the players... unless you want to really generalize the whole thing and say it's our fault for continually paying for a service we know has no immediacy or interest in getting fixed/upgraded.

We both know what kind of people each other are, we both know where each other's views stand on things. You know I am not confusing you with a brown noser. 

But seriously.... answer me this. If you think that the size of Xanadu has anything to do with it, then why did Indy lag for 3 years (exactly like Xan is doing now) while Chaos, which is the same size server (and has had a few more years to fluff the item count), didn't have a problem at all?

Share this post


Link to post
Share on other sites

Hi,


 


I may be completely wrong, but take this as a try to help.


 


I've seen a lot of tracert's that show how the lag is building up. Usually just before entering Hetzners net. Now Hetzner isn't known to be a low-quality provider - actually it's one of best of the big providers you can get in Germany.


The tracert's usually show INIT7, LEVEL3 or similar as last stations before entering Hetzners net, where big latency is building up.


 


The problem may be that Hetzner may be using a T-Com ("Deutsche Telekom) line to connect the Wurm servers.


 


T-Com is not an ordinary ISP. It's the heritage of "Deutsche Post" (German post), and does all it can to keep up it's privileges. Thus there's places in Germany at the edge of a town like Munich (I live there) where you don't have ANY mobile fone access, as well as Internet access limited to below 1Mbit/sec downloaded.


 


Besides, T-Com is not peering like any other tier-1 provider. They ask for money to do this. And thus other providers like INIT7 or LEVEL3 have poor peering with them.


 


Paying Hetzner to use a real service-provider might solve the problem.


 


Have fun!


 


 


 


Share this post


Link to post
Share on other sites

I did read your whole post. Did you think I randomly chose 5/6th through the post to pick out one part to read?

I agree with what you wrote, aside from the part where you say "the players are not exempt from blame." I wouldn't go so far as to pin the blame on all of the players... unless you want to really generalize the whole thing and say it's our fault for continually paying for a service we know has no immediacy or interest in getting fixed/upgraded.

We both know what kind of people each other are, we both know where each other's views stand on things. You know I am not confusing you with a brown noser. 

But seriously.... answer me this. If you think that the size of Xanadu has anything to do with it, then why did Indy lag for 3 years (exactly like Xan is doing now) while Chaos, which is the same size server (and has had a few more years to fluff the item count), didn't have a problem at all?

Share this post


Link to post
Share on other sites

I'm surprised nobody mentioned java as the problem.

Probably because it isn't.

 

[...]

In the end, all we can do is theorise about the actual code, and I'm sure it's fun. Contrary to playing on an unresponsive server, which gets annoying very quickly. Always waiting on the context menu, getting stuck on every door and gate because your position is not update quickly enough, and so on.

Everyone on Xanadu would probably very much prefer if Mr. Public Relations got some facts from the people who are actively working on the issue instead. It would probably help with all the new threads about Xanadu's lag too, I think :)

Share this post


Link to post
Share on other sites

Java would have little to no impact in this regard. Its more an issue of how much the supporting hardware can handle.


 


Granted the bottleneck could be anywhere in the setup.


Share this post


Link to post
Share on other sites

Hi,

 

I may be completely wrong, but take this as a try to help.

 

I've seen a lot of tracert's that show how the lag is building up. Usually just before entering Hetzners net. Now Hetzner isn't known to be a low-quality provider - actually it's one of best of the big providers you can get in Germany.

The tracert's usually show INIT7, LEVEL3 or similar as last stations before entering Hetzners net, where big latency is building up.

 

...

 

I live close to the servers in southern Germany and have an excellent connection to them, usually a ping of around 30-40 milli seconds. In most cases it's not network lag, it's the server itself that is lagged. You can also see that by the fact that in many cases it's only Xanadu that experiences the lag and other servers (using the same network) run just fine.

 

(There has been some issue with network connectivity in the past as well, but I didn't hear anything about it recently and anyway in those cases it would be all servers that are affected and not only Xanadu.)

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this