Recommended Posts

11 hours ago, Retrograde said:

The update will also touch on a few aspects of the journal, including crediting past legendary creatures slain, colossi built, and guard towers! This means less big building projects, lucky you.

 

[01:57:45] Congratulations, you fully completed The Road Less Traveled and earned the reward: Expert Title, 10% More Rarity Windows & 6hr Sleep Bonus

 

Oh well, more buildings on deed dont hurt, lol ?

  • Like 3

Share this post


Link to post
Share on other sites
2 hours ago, Ronnie said:

Honestly, an MMO that's almost 13 years old unable to support 1k players on the same server is pretty sad.

It's not the 1k players on the same server that hurts Xanadu, it's the fact the MMO is a living, breathing world around it. 

 

The server has to keep track of approximately 67 million tiles, 150,000 animals, over 10 million items that each decay, cook, produce in the world around them. 

 

There's a lot going on on any server, and the sheer size of xanadu simply increases that exponentially. 

 

We've done a lot of work refining how it works, it's got zero to do with strengthening a database or server end functions. It's simply how much is going on at any given second. 

  • Like 4

Share this post


Link to post
Share on other sites

omg that new journal sooo hyped i cant wait to reprem for that journal :D

Share this post


Link to post
Share on other sites
7 hours ago, Retrograde said:

It's not the 1k players on the same server that hurts Xanadu, it's the fact the MMO is a living, breathing world around it. 

 

The server has to keep track of approximately 67 million tiles, 150,000 animals, over 10 million items that each decay, cook, produce in the world around them. 

 

There's a lot going on on any server, and the sheer size of xanadu simply increases that exponentially. 

 

We've done a lot of work refining how it works, it's got zero to do with strengthening a database or server end functions. It's simply how much is going on at any given second. 

Just a quick question... wouldnt it hurt a lot less the server if its on smaller blocks and not whole server loading at once? and does the server load all trees 1 by 1 or does it load all at once?

 

Not saying i understand much of this stuff, but using common sense, i would say it would be better in blocks and not whole server at once.

 

Or is it possible to make houses so it wouldnt load those stuff inside it, unless a player goes near a window or uses a door? so it wouldnt have load the stuff unless have to.

Share this post


Link to post
Share on other sites
18 hours ago, Retrograde said:

It's not the 1k players on the same server that hurts Xanadu, it's the fact the MMO is a living, breathing world around it. 

 

The server has to keep track of approximately 67 million tiles, 150,000 animals, over 10 million items that each decay, cook, produce in the world around them. 

 

There's a lot going on on any server, and the sheer size of xanadu simply increases that exponentially. 

 

We've done a lot of work refining how it works, it's got zero to do with strengthening a database or server end functions. It's simply how much is going on at any given second. 

 

However there are ways to work around that by breaking down the map into linked nodes but one world like a lot of other single world MMO's have done to support similar system stress. So instead of the server being loaded in one chunk by one processor, you can have the map in sizeable chunks that can be ran and offlined when no one is within X range of that node/server chunk. Allowing smarter and less server stress with a larger player base on one map.

 

There are open world games that have over 300k mobs for players to kill, 20-100k players on one server at a time. That could be killings said mobs, other players, taking part in some more of guild/race versus guild/race warfare, gathering, and crafting all at the same time on one map. Sure they are not watching 67million tiles, most of which are never changed or goes weeks to months before being changed again.

 

Nodes or open world slicing has been a popular thing in MMO's and solo player games since around 2012.

Edited by Ronnie

Share this post


Link to post
Share on other sites
18 hours ago, Retrograde said:

It's not the 1k players on the same server that hurts Xanadu

Wait what?  When did Xanadu last have a population of 1K?  Last peak I saw was nearly 150 I think?

Share this post


Link to post
Share on other sites
4 hours ago, Etherdrifter said:

Wait what?  When did Xanadu last have a population of 1K?  Last peak I saw was nearly 150 I think?

 

1k I never saw since I play (which is 12/2017), but in excess of 500, yes. Mainly during public slayings.

Share this post


Link to post
Share on other sites

Thank you for putting this together Susie.

Share this post


Link to post
Share on other sites

The reason the game can't handle a world as large as Xanadu cleanly is due to a couple factors.

 

First of all, pretty much the entire server runs on one thread. That means any additional load that gets added on by new features that are computation-heavy during polling will cause everything to slow down instead of using another CPU core to handle the processing. When you add an item, like was mentioned, it is polled every now and again for decay. As each item is added, the "main" server loop has to handle that. When it polls that item, it has to wait until it's done doing that poll until the next "loop" can occur for processing. When you have several thousand, it's no problem. When you get a million items that need to be processed on every loop and checked for decay, you get in a situation where it's spending more time decaying items than handling the actions for players. This is why you can run into situations where the server will stop responding and everyone will be "Refreshing..." until the server catches up.

 

Secondly, there's a ton of really poorly implemented processes in that server loop. One of the most expensive operations in Java is handling exceptions. Exceptions are used in a ton of places in the code where they could easily be handled in a faster manner. For example, whenever a function needs to find an item on a creature/player it uses an exception. This runs when you are swinging in combat (to find your weapon), when you get hit in combat (to find your armour piece), and in some extreme cases, even when some creatures are moving. A prime example of that would be when you saddle/shoe a horse, it checks all the gear it has when it's moving so that it updates how fast it goes. Each one of these operations is an expensive check that, if it fails, can be extremely expensive. If I recall correctly, this check occurs on every single creature that's either mounted or hitched to a vehicle. Anyone who is using horses without gear on a vehicle is adding quite a bit to the processing required during server polling. This problem has been seeing some work recently and exception catches are becoming less frequent. However, they're so common that it still makes up a significant portion of the server loop.

 

Third, there's systems in the game that are designed in a way that makes them very difficult to process when the volume becomes too large. Decay is currently designed in such a way that it requires collecting and polling every single item, structure, and wall in the game constantly. These are by far the most difficult problems to solve. They're not impossible, but implementing the solution is extremely difficult and would take a significant effort to mimic previous behavior. As an example, I once pitched a zero-polling decay solution. Instead of polling every item in the game, their decay would update when the item was observed. If you drop a pile of 100 wood scrap in the woods in a corner of the map and nobody is even in local range for 2 years, the server is still polling those items up to the point where they are deleted - on every loop of every server tick. Instead, the idea would be to tag the item when it was last observed. When a player enters local and can see the item, it would then update the damage (and/or QL) of the item relative to when it was last viewed. If that was 2 years ago, then obviously the wood scraps would take so much damage to be destroyed. If instead it was a week after they were initially left on the ground, they would take a week worth of damage. This whole system would allow complete removal of polling items for decay.

 

So why not do that? Because it's really difficult. It involves adding data to every item that exists in the game and transitioning that into a brand new system. It would involve removing a ton of complicated code and re-writing it in a new way to be handled with the new system. It would involve updating the client-server communication so that the server would know when an item needs to be updated. All of these problems translate to solutions to other problems. Avoiding updating creature locations and data in un-loaded parts of the map would probably introduce new bugs. When a creature dies of old age in an unknown place on the map, it would drop a corpse and the corpse would start decaying. If the server stopped updating the age for creatures it could not see, then when it was finally viewed, it would drop a corpse with no decay introduced. Furthermore, it would be in the exact same location as when it was previously observed.

 

TLDR: Server slowdown isn't because the hardware isn't good enough or that the database checks are slow. It's because the code for the game didn't have expansion in mind when it was originally created.

  • Like 5

Share this post


Link to post
Share on other sites
12 minutes ago, Sindusk said:

Secondly, there's a ton of really poorly implemented processes in that server loop. One of the most expensive operations in Java is handling exceptions. Exceptions are used in a ton of places in the code where they could easily be handled in a faster manner. For example, whenever a function needs to find an item on a creature/player it uses an exception. This runs when you are swinging in combat (to find your weapon), when you get hit in combat (to find your armour piece), and in some extreme cases, even when some creatures are moving. A prime example of that would be when you saddle/shoe a horse, it checks all the gear it has when it's moving so that it updates how fast it goes. Each one of these operations is an expensive check that, if it fails, can be extremely expensive. If I recall correctly, this check occurs on every single creature that's either mounted or hitched to a vehicle. Anyone who is using horses without gear on a vehicle is adding quite a bit to the processing required during server polling. This problem has been seeing some work recently and exception catches are becoming less frequent. However, they're so common that it still makes up a significant portion of the server loop.

Is the problem that server is being very busy, can couse that player entering bridge with high speed, lands under a bridge, couse server does not follow his move fast enough, thus do not position player on the bridge?

 

If you have horse that is fast (all traits and high quality gear or just hellhorse) and slope below first tile of the bridge is high enough (hard to tell what is the minimum height to bug couse I have very high slopes at my deed) then when riding on the horse, you very often ride under the first tile of the bridge, and can continues to ride down, no matter what slope height tile has.

Just noticed that it hapens rarely on slow horses, and almost always on fast horses. Got this bug so many time that for me is almost impossible to move on most bridges of my deed on horse. I used to lead horses on bridges couse of many cases I landed below over the abyss and GM's needed to bring me back to the top.

My theory is that couse of extremely high slopes on my place speed of going down is naturally huge. So when entering the bridge on very fast creature, player starts to move down so fast, that server isn't able to react in time and place player on the bridge, thus players landing below it.

 

Definietly has something to do with speed of entering the bridge.

 

I loaded myself heavy, so my horse was able to run 10km/h max and tested entering the bridges. I didn't stuck so far.

 

Still cannot normally ride on my horses on my bridges, so I'am forced to go on feet each time, eventually on wagon.

Share this post


Link to post
Share on other sites
23 minutes ago, Sindusk said:

TLDR: Server slowdown isn't because the hardware isn't good enough or that the database checks are slow. It's because the code for the game didn't have expansion in mind when it was originally created.

That's my world, complete chaos ??

Share this post


Link to post
Share on other sites

Would love to see this game evolve and survive the future of hardware.

 

They should just focus on the core engine for now and stop the small updates.  Then hopefully Club Code would feel confident advertising and bring in more people to the servers.

 

It's like a client still using FoxPro or MS Access 2000 and have to rely on it and can't upgrade past Windows Server 2003 and running out of ways to work with it.

 

Share this post


Link to post
Share on other sites

New ppl are probably looking for a bit more interactive game.. more animations, better fighting, no wogic, etc.. anything you'd usually find in a 2020 mmo, wurm have some ok graphics now, but controls and animations, progress of actions is meh-ish for the most part to lure new souls into the lamp to rekindle the population.

 

Skill progression is quite slow for the average player.. 'normal' person wouldn't spend 3-6/+ months to grind 1 or 2 skills.

 

That part is the tough to handle.. wurm needs certain niche group of players.. question is how to win them, if advertising to them wasn't so hard to pull, getting them to register and login once.. is just the 1st step.. rest is.. to make them stay.. and prem up.

Than come and of course.. help you get them all affinities on chaos, sacrificing themselves for your collection ??

Share this post


Link to post
Share on other sites

Don't get me wrong, I love what wurm was before the demi god/ player god system was patched in. I keep hoping to find something close to those days and always check back to see what's changed. But all I keep seeing is "We can't do this or that" when you could, however it would take a good chunk of manpower and time. When I see that a number of the recent patches since I left in 2017/ early 2018 is causing dated players to leave, those that have spent large amounts of time playing weekly for years. I think instead of adding new things that somehow ends up breaking other parts of the game for a point of time and start revamping the games structure, fix those loose ends, rebalance, or bug fix for a while before trying to push out that next chapter of the game or quality of life that affects a number of random things it wasn't meant too.

Edited by Ronnie

Share this post


Link to post
Share on other sites

First of all, thanks for those explanations. I would love other devs commenting on as far as they have the time for. Just one question:

 

 

11 hours ago, Sindusk said:

..

First of all, pretty much the entire server runs on one thread.

 

Well, when I see all the troubles with decay or movement management, I ask myself: why? My problem is that I have no understanding of Java, I had a short glimpse at it when it came out, and decided: No. Doesn't mean that I cannot understand chunks of code, but already the whole OO "paradigm" is repulsive to me. So my perspective rather is that of a dated C and assembly language systems programmer, and I do in no way want to judge about your problems or dare to give advices, I only can ask questions. I acknowledge that the decision to use Java for a computer game was a bold and daring one, with the fascinating opportunity to be much more independent from proprietory operating system architectures. 

 

Yet the problems described beg the question: Must a Java based server process run "single threaded"? I know some game architectures (granted, just superficially) where the game engine consists of a process hierarchy where the "main loop" is sleeping almost all time, and a hierarchy of independent server processes are doing the work, interwoven by interprocess communication features such as shared memory, message queues, semaphores, sockets,  to name a few. Such server processes usually even need not to work on the same computer.

 

And the decay problem cries "database", and cries it loudly, at least as I read it in that post. Decay is damn trivial, it has, to my understanding, not many more variables than quality, decay rate, damage, and time elapsed, and the relationship among those is fairly trivial, they could be maintained and updated independently.  Yes, there are external variables adding to or substracting from damage like combat, lava, imping, or immediate destruction e.g. by saccing. But must that all be handled in the main loop of the server? Java is able to call out to RDBMs, I saw that daily in my work. And even a small database system can handle trillions of objects.

 

11 hours ago, Sindusk said:

TLDR: Server slowdown isn't because the hardware isn't good enough or that the database checks are slow. It's because the code for the game didn't have expansion in mind when it was originally created.

 

Maybe.

 

Share this post


Link to post
Share on other sites
12 hours ago, Sindusk said:

The reason the game can't handle a world as large as Xanadu cleanly is due to a couple factors.

 

First of all, pretty much the entire server runs on one thread. That means any additional load that gets added on by new features that are computation-heavy during polling will cause everything to slow down instead of using another CPU core to handle the processing. When you add an item, like was mentioned, it is polled every now and again for decay. As each item is added, the "main" server loop has to handle that. When it polls that item, it has to wait until it's done doing that poll until the next "loop" can occur for processing. When you have several thousand, it's no problem. When you get a million items that need to be processed on every loop and checked for decay, you get in a situation where it's spending more time decaying items than handling the actions for players. This is why you can run into situations where the server will stop responding and everyone will be "Refreshing..." until the server catches up.

 

Secondly, there's a ton of really poorly implemented processes in that server loop. One of the most expensive operations in Java is handling exceptions. Exceptions are used in a ton of places in the code where they could easily be handled in a faster manner. For example, whenever a function needs to find an item on a creature/player it uses an exception. This runs when you are swinging in combat (to find your weapon), when you get hit in combat (to find your armour piece), and in some extreme cases, even when some creatures are moving. A prime example of that would be when you saddle/shoe a horse, it checks all the gear it has when it's moving so that it updates how fast it goes. Each one of these operations is an expensive check that, if it fails, can be extremely expensive. If I recall correctly, this check occurs on every single creature that's either mounted or hitched to a vehicle. Anyone who is using horses without gear on a vehicle is adding quite a bit to the processing required during server polling. This problem has been seeing some work recently and exception catches are becoming less frequent. However, they're so common that it still makes up a significant portion of the server loop.

 

Third, there's systems in the game that are designed in a way that makes them very difficult to process when the volume becomes too large. Decay is currently designed in such a way that it requires collecting and polling every single item, structure, and wall in the game constantly. These are by far the most difficult problems to solve. They're not impossible, but implementing the solution is extremely difficult and would take a significant effort to mimic previous behavior. As an example, I once pitched a zero-polling decay solution. Instead of polling every item in the game, their decay would update when the item was observed. If you drop a pile of 100 wood scrap in the woods in a corner of the map and nobody is even in local range for 2 years, the server is still polling those items up to the point where they are deleted - on every loop of every server tick. Instead, the idea would be to tag the item when it was last observed. When a player enters local and can see the item, it would then update the damage (and/or QL) of the item relative to when it was last viewed. If that was 2 years ago, then obviously the wood scraps would take so much damage to be destroyed. If instead it was a week after they were initially left on the ground, they would take a week worth of damage. This whole system would allow complete removal of polling items for decay.

 

So why not do that? Because it's really difficult. It involves adding data to every item that exists in the game and transitioning that into a brand new system. It would involve removing a ton of complicated code and re-writing it in a new way to be handled with the new system. It would involve updating the client-server communication so that the server would know when an item needs to be updated. All of these problems translate to solutions to other problems. Avoiding updating creature locations and data in un-loaded parts of the map would probably introduce new bugs. When a creature dies of old age in an unknown place on the map, it would drop a corpse and the corpse would start decaying. If the server stopped updating the age for creatures it could not see, then when it was finally viewed, it would drop a corpse with no decay introduced. Furthermore, it would be in the exact same location as when it was previously observed.

 

TLDR: Server slowdown isn't because the hardware isn't good enough or that the database checks are slow. It's because the code for the game didn't have expansion in mind when it was originally created.

 

Thanks for your insight @Sindusk . How difficult would it be to make Wurm multi threaded?

Share this post


Link to post
Share on other sites
2 hours ago, solmark said:

 

Thanks for your insight @Sindusk . How difficult would it be to make Wurm multi threaded?

 

It would be possible but we are speaking of at least one year development stop.

  • Like 1

Share this post


Link to post
Share on other sites
46 minutes ago, Sklo:D said:

 

It would be possible but we are speaking of at least one year development stop.

 

Would that effort make a big difference to the performance output?

Share this post


Link to post
Share on other sites
59 minutes ago, Sklo:D said:

It would be possible but we are speaking of at least one year development stop.

 

I fail to see that, care to elaborate? In my  view it deals with a change in the game engine which would first have to be designed, meaning creating a multi threaded architecture of the same engine sourcing out some time consuming or contention causing jobs to other threads or processes (note I do not know how to do that in Java, we have done it in the past even parallelizing or distributing operating systems, so it is no unfamiliar practice in IT and CS). This would be the first step, optimizations coming in case all works later.

 

Having done that the next step would be porting the existing engine to the new architecture testing system integrity and stability. If that is generally working next would be a migration plan, and finally the change to the new architecture.

 

All that may well take a year to code and implement, sure, maybe even longer. And it would require manpower missing for other development and maintenance activities But I do not see why there should be a development stop for a year.

 

First step would be a feasability study, finding out whether it can be done at all.

Share this post


Link to post
Share on other sites

We've done plenty of optimization on the server last year and earlier this year, although it might not be ideal yet on Xan, it's improved a lot from my tests.

Certain parts of the server are already multithreaded, mostly some involving long IO operations - while redesigning the whole code to allow for reasonable multithreading of all the heavier tasks is possible, it'd take months of work for a possibly insignificant improvement, since most servers are already lag free.

The performance of the servers can still be improved in other, more effective ways and we'll be working on that more still.

 

If you're experiencing server lag and you know it's not your connection (so other people experience it at the same time too), please let us know in the Server bugs section with your server and the time it has happened - the exact time helps us find what part of the code it gets stuck on in the logs.

 

That said, I don't see the need for any big server architecture changes, unless something else requires it. What'd merging islands even help with?

  • Like 1

Share this post


Link to post
Share on other sites
37 minutes ago, Samool said:

We've done plenty of optimization on the server last year and earlier this year, although it might not be ideal yet on Xan, it's improved a lot from my tests.

..

 

Thanks for the comment. I can confirm that the lag situation on Xanadu has significantly improved compared to last year, even more in comparison to the crisis around begin of this year.

  • Like 1

Share this post


Link to post
Share on other sites
2 hours ago, Samool said:

We've done plenty of optimization on the server last year and earlier this year, although it might not be ideal yet on Xan, it's improved a lot from my tests.

Certain parts of the server are already multithreaded, mostly some involving long IO operations - while redesigning the whole code to allow for reasonable multithreading of all the heavier tasks is possible, it'd take months of work for a possibly insignificant improvement, since most servers are already lag free.

The performance of the servers can still be improved in other, more effective ways and we'll be working on that more still.

 

 

Great work on this.  Performance has been leaps and bounds better than what it was a few years ago.  Can go all day without restarting client.  No more server lag that would kill PvP or large events like it used to when everyone would lag out during battle and have to restart game.  The fix in the new client where you spin in circles when moving the mouse is awesome so no more walking off cliffs on accident.

 

Xan still is a nightmare at times with lag.  You get used to 3-5 second lag, but sometimes it's 5-10 seconds like yesterday which was annoying.  Or will take 6-12 seconds just to regen stamina at times after a few actions.  Notice a difference big time when hunting on smaller servers.  Stamina regen is 1-3 seconds and my horse keeps up with me when leading.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this