Posted July 7, 2020 Hello all! With the success of Celebration's move, we're marching forward by moving Xanadu to the new hosting provider Cloudnet. This move will take place on July 13th at 3pm server time (9am US Eastern time) and require an extended downtime of all servers of about 1.5 hours. I want to control expectations with this move in that we are not expecting that Xanadu will become "lag-free" with this move. What I do expect is performance to be the same, if not slightly improved. The new server does have newer hardware and better disk I/O speeds which may help, but the sheer size of Xanadu leaves me skeptical that it will make a huge impact. I'm most certainly looking forward to being proven wrong though! Feel free to drop your questions in replies and I'll attempt to answer them or compile them into a larger post. I'd still like to do a bigger post about Cloudnet and our roadmap for this move, but right now the main focus is getting Xanadu moved and monitoring it. Thank you and happy Wurming! (Well, except during that downtime. ) Updates 3pm: Servers down, starting transfer. 3:45pm: In the process of moving the data to the new host. All network changes have been completed. 4:30pm: Need to make a few config changes to the new host before we bring everything online, they should be completed shortly. 5pm: Servers starting up now. Watching the logs carefully! All servers online! 19 1 Share this post Link to post Share on other sites
Posted July 7, 2020 If all goes well with Xanadu, would we be expecting all of the servers move to cloudnet prior to the steam release? Also, is the plan to have the new servers launch on cloudnet? 1 Share this post Link to post Share on other sites
Posted July 7, 2020 can you migrate xanadu and then sink it at long last? 4 1 Share this post Link to post Share on other sites
Posted July 7, 2020 1 hour ago, McGarnicle said: If all goes well with Xanadu, would we be expecting all of the servers move to cloudnet prior to the steam release? Also, is the plan to have the new servers launch on cloudnet? The new servers will be launched on Cloudnet, that much is in the works already. We're not shooting to have all servers migrated before the Steam release. This is a separate time table, with the Steam release taking priority. 2 Share this post Link to post Share on other sites
Posted July 8, 2020 Keenan you get ONE forehead kiss for your efforts. 4 Share this post Link to post Share on other sites
Posted July 8, 2020 14 minutes ago, user said: Keenan you get ONE forehead kiss for your efforts. From everyone though 5 Share this post Link to post Share on other sites
Posted July 8, 2020 Thanks for the notice, good luck with your efforts, they are massively appreciated. 1 Share this post Link to post Share on other sites
Posted July 9, 2020 On 7/8/2020 at 2:30 PM, user said: Keenan you get ONE forehead kiss for your efforts. I am sure some off us here will give him two if not three forehead kisses with some maybe even giving him a sneaky kiss elsewhere Share this post Link to post Share on other sites
Posted July 10, 2020 On 7/7/2020 at 5:38 PM, Keenan said: I want to control expectations with this move in that we are not expecting that Xanadu will become "lag-free" with this move. Will build a shrine in your honor if Xanadu becomes lag free with this move. 1 Share this post Link to post Share on other sites
Posted July 12, 2020 Good luck with moving Xanadu tomorrow @Keenan, I hope your coffee machine is fully stocked and working Share this post Link to post Share on other sites
Posted July 13, 2020 I am scared someone pass be a bottle of something strong to drink .... so I can pass out until this is all done and over with LOL 1 1 Share this post Link to post Share on other sites
Posted July 13, 2020 7 minutes ago, RavenLure said: I am scared someone pass be a bottle of something strong to drink .... so I can pass out until this is all done and over with LOL We can both hit the bottle once it's over with! 4 Share this post Link to post Share on other sites
Posted July 13, 2020 Any ETA on what time servers go down today? Share this post Link to post Share on other sites
Posted July 13, 2020 3 minutes ago, Cecci said: Any ETA on what time servers go down today? They did On 7/7/2020 at 11:38 PM, Keenan said: This move will take place on July 13th at 3pm server time (9am US Eastern time) and require an extended downtime of all servers of about 1.5 hours. Share this post Link to post Share on other sites
Posted July 13, 2020 (edited) If disk I/O is an issue, maybe it would be worth to experiment with distributed file systems such as Lustre? I have some experience with such systems, maybe I could help. Edited July 13, 2020 by Platyna Share this post Link to post Share on other sites
Posted July 13, 2020 10 minutes ago, Platyna said: If disk I/O is an issue, maybe it would be worth to experiment with distributed file systems such as Lustre? I have some experience with such systems, maybe I could help. Disk I/O is a non-issue on the new host. Share this post Link to post Share on other sites
Posted July 13, 2020 (edited) 7 minutes ago, Keenan said: Disk I/O is a non-issue on the new host. Ah, okay, I got this impression from this sentence of yours, which I understood the I/O was and may continue to be a problem: On 7/7/2020 at 11:38 PM, Keenan said: The new server does have newer hardware and better disk I/O speeds Was it investigated what exactly happens on the system during the lag spikes? I will understand if you will not disclose this on public forum. It seems to be an interesting problem to solve though. Edited July 13, 2020 by Platyna Share this post Link to post Share on other sites
Posted July 13, 2020 1 minute ago, Platyna said: Ah, okay, I got this impression from this sentence of yours, which I understood the I/O was and may continue to be a problem: Was it investigated what exactly happens on the system during the lag spikes? Yes, we know what causes the spikes. It's a software issue where the main thread gets overloaded by heavily populated zones. We had a disk I/O issue on Amazon because of the EBS volumes we were using, and it wasn't financially viable to go with the higher IOPS drives as it would've quadrupled the cost of hosting per server. Share this post Link to post Share on other sites
Posted July 13, 2020 What does overloaded means here? I/O waits? Yes, EBS has issues, such as major performance penalty at bigger block sizes. Let's hope for the best then. Share this post Link to post Share on other sites
Posted July 13, 2020 5 minutes ago, Platyna said: What does overloaded means here? I/O waits? Yes, EBS has issues, such as major performance penalty at bigger block sizes. Let's hope for the best then. Big deeds for example cause lag spikes as they are polled at once (if in one zone). I wouldn't call it I/O waits in general it is I/O so reading and writing not just the I/O delay, if that is what you mean. In general the problem is that most things are single threaded blocking player interaction if something is polled and things are polled non stop otherwhise animals wouldn't move for example. 2 Share this post Link to post Share on other sites
Posted July 13, 2020 It's nice to see Xanadu getting some love. 1 1 Share this post Link to post Share on other sites
Posted July 13, 2020 As a short appraisal after a few hours: The move went fast and the server worked fine after start. Most of the in the last months worsened issues were either gone or ways less critical. After some time, maybe with recurrent population, some issues arose, such as synch troubles during sailing ("Your position .. bla .."). Still, after hours, the server feels ways better than before. To specify: I know Xanadu for 2.6 years circa, and lag has always been present. The last months, though, were among the worst. I hope the improvements are there to stay. I understand that the troubles by size and population are not addressed by the new hoster. 1 Share this post Link to post Share on other sites
Posted July 13, 2020 In game downtime message said updates would be posted here but the only thing suggesting its over is a random players feedback? 1 Share this post Link to post Share on other sites
