Posted February 8, 2019 Hi everyone, As you may have seen independence is currently experiencing some significant lag. Our investigations are pointing towards hardware issues, which mean swapping independence to different hardware. We aim to have an interim solution up tomorrow and will work towards a longer term solution over the coming days. Edit: In 7 hours we'll be starting maintenance to transition Indy to the new hardware. Downtime likely to be 1-3 hours and will apply to ALL servers. Edit (Keenan): Down time updated to reflect the data transfer speed (or lack thereof). Sorry folks! 10 Share this post Link to post Share on other sites
Posted February 8, 2019 Chaos as well, my horse, same for others is always 10 tiles behind. 1 Share this post Link to post Share on other sites
Posted February 8, 2019 (edited) thank you this has affected me greatly where do i send you my paypal info for compensation on a serious note tho, my backup horse is also always like 5 to 10 tiles behind me and to the left, so far away i usually think its killed in pvp and dont even use it Edited February 8, 2019 by platinumteef 2 Share this post Link to post Share on other sites
Posted February 8, 2019 If Inde gets brand new hardware for a bit of lag then what should Xanadu get moved to? The hadron collider?:) On a serious note: thank you for fixing it guys. We all await sleep bonus now 6 Share this post Link to post Share on other sites
Posted February 8, 2019 I am excited the lag is getting fixed! Thank you! 1 Share this post Link to post Share on other sites
Posted February 8, 2019 I hope the lag gets fixed with this new hardware. Thank you for your support and your work! Share this post Link to post Share on other sites
Posted February 8, 2019 While everything is possible, from my limited view it is not likely that hardware issues can cause something like that, especially when the web server and icmp request on the same machine are handled without any noticeable delays. 1 Share this post Link to post Share on other sites
Posted February 8, 2019 3 minutes ago, Sklo:D said: While everything is possible, from my limited view it is not likely that hardware issues can cause something like that, especially when the web server and icmp request on the same machine are handled without any noticeable delays. You have to realize they are expert investigators clearly there was something hardware wise that was failing Anyway any chance we could hear some more info on it for those curious as to what was going on as id be interested in knowing and im sure a few others are as well 1 Share this post Link to post Share on other sites
Posted February 8, 2019 I'm interested in what broke, but also what is being changed. From the sounds could be a simple hardware swap to a whole system swap. 2 Share this post Link to post Share on other sites
Posted February 8, 2019 19 minutes ago, Kelody said: I'm interested in what broke, but also what is being changed. From the sounds could be a simple hardware swap to a whole system swap. For the record I would like to know that too since Sklotopolis runs on the same machines at hetzner 2 Share this post Link to post Share on other sites
Posted February 8, 2019 (edited) tyvm for your hard work, hope it'll help Edited February 8, 2019 by ShloEmi Share this post Link to post Share on other sites
Posted February 8, 2019 Thank you, sounds like the lag torture is gonna end soon! 1 Share this post Link to post Share on other sites
Posted February 8, 2019 Drank coffee. Prepped new hardware. Got heckled by Redi. I think we're ... Redi. 7 Share this post Link to post Share on other sites
Posted February 8, 2019 Kicked off the usual 30 minute shutdown for all servers. The reason all servers need to come down for this is that we must update their references to Independence. For those who have asked, I will be posting a postmortem once we confirm that new hardware has addressed the issue. 10 Share this post Link to post Share on other sites
Posted February 8, 2019 3 minutes ago, Keenan said: Kicked off the usual 30 minute shutdown for all servers. The reason all servers need to come down for this is that we must update their references to Independence. For those who have asked, I will be posting a postmortem once we confirm that new hardware has addressed the issue. Thumbs up and GL on the fix Keenan! Share this post Link to post Share on other sites
Posted February 8, 2019 "is we all gonna die?" Jar Jar Share this post Link to post Share on other sites
Posted February 8, 2019 Thanks for the information and updates team. I know it is a real pain for people, however, looking at who amongst the management is on-line anyone can see that it is 'all hands to the pump'. They are obviously doing their best. Good luck with the medicine Share this post Link to post Share on other sites
Posted February 8, 2019 Good luck for the switch thanks for improving our game even more ? Share this post Link to post Share on other sites
Posted February 8, 2019 'Wurmageddon 2' says the little red hen. (me pessimist? never) 4 Share this post Link to post Share on other sites
Posted February 8, 2019 Hopefully the restart wont kill more animals. buddy had a tamed champ die on restart. Goodluck on the transfer and long term goals about this matter. Share this post Link to post Share on other sites
Posted February 8, 2019 Just now, ZeldaGold said: Hopefully the restart wont kill more animals. buddy had a tamed champ die on restart. Goodluck on the transfer and long term goals about this matter. i read this the first time as "Tamed Chimp" wheres my glasses! 1 Share this post Link to post Share on other sites
Posted February 8, 2019 54 minutes ago, Keenan said: Kicked off the usual 30 minute shutdown for all servers. The reason all servers need to come down for this is that we must update their references to Independence. For those who have asked, I will be posting a postmortem once we confirm that new hardware has addressed the issue. tyvm, this lag drove me crazy, barely able to play. GL!! Share this post Link to post Share on other sites
Posted February 8, 2019 UPDATE The database has been copied over to the new hardware and verified. I can give a little more explanation as it will help to understand why this is taking so long. Essentially the read performance of Independence's drives took a tank. The data integrity is fine, but it's taking an insane amount of time to actually read the data. Which makes transferring said data to new hardware take longer as well. In the interest of full transparency: we heavily suspect the hardware is the culprit, but we have not ruled out that there's a software component causing this. Diagnostics on the drives are fine, hence the data integrity being good. I always believe that a good system administrator remains a skeptic to the very end! Another note to mention is that the lag graph will not be updated to the new hardware right away. The primary focus is getting back up. Besides, I think it'll be obvious if this doesn't rectify the situation. The lag wasn't exactly difficult to notice. 11 Share this post Link to post Share on other sites
