Sign in to follow this  
Keenan

AWS Status Update - Game Server Infrastructure Complete!

Recommended Posts

Hello all!

 

I’ve made quite a bit of progress over the last week. We now have fully functioning game server instances, complete with MRTG and server stats. So the servers will have all those stats you all love. On top of that I was able to set up infrastructure for an IPSec VPN Tunnel. This VPN tunnel will allow us to uplift servers individually, rather than take all of Wurm down for a considerable amount of time to uplift all servers at once.

 

Even more exciting is that I took the test servers down to nothing and was able to restore them from backups in a short period of time. 20 minutes for Druska, which has the Jackal map at the moment! That was a pretty nice time frame for a complete server restore.

 

We are currently working out an uplift schedule, but I can tell you that the first server will be Jackal. We are starting with Jackal because it still needs to run to transfer folks off, and it will be a good test with minimal interruption. After Jackal we will likely begin the Freedom server uplift with Celebration. Again the schedule for this is forthcoming and needs to work around our patch schedule and time off.

 

It’s been a long road - much longer than I had originally anticipated. A lot of the delay was the result of failing to find time, but much of the time spent on this project was sheer trial and error. We have a unique infrastructure and trying to get it right took time.

 

Samool also helped us right-size the test servers with some memory improvements, so if you have a moment please do hop on and see how it feels. We will monitor performance of the servers we uplift and make changes to instance-size as needed. Samool also plans to monitor and make improvements to memory usage which will help with server lag overall.

 

So what’s next? I still have the build servers and the main Wurm website to do. The latter will cause some launcher downtime so I’m being careful with this. We’re getting close though!

 

That’s all for now. Happy Wurming!

  • Like 31

Share this post


Link to post
Share on other sites

Good job man for keeping the hamsters well fed and the wheels well oiled.

 

It's always good to know what is happening backstage and we can only thank you and the rest of the team for keeping the game literally (a)live on the interweb!

  • Like 3

Share this post


Link to post
Share on other sites

So test server/s and Jackal are running with aws-powered "hamsters" now?

1 hour ago, Keenan said:

...
so if you have a moment please do hop on and see how it feels. We will monitor performance of the servers we uplift and make changes to instance-size as needed. Samool also plans to monitor and make improvements to memory usage which will help with server lag overall.
...

That's an invitation to check lag/latency difference if there's any between old/new 'hosting' I'd guess?

* for these who do not know(start > run > resmon > and there find and click wurmlauncher's process' checkbox, than at network you can see your 'latency') random link explaining same thing with pictures, don't judge.. 1st google link

Share this post


Link to post
Share on other sites
11 minutes ago, Finnn said:

So test server/s and Jackal are running with aws-powered "hamsters" now?

That's an invitation to check lag/latency difference if there's any between old/new 'hosting' I'd guess?

* for these who do not know(start > run > resmon > and there find and click wurmlauncher's process' checkbox, than at network you can see your 'latency') random link explaining same thing with pictures, don't judge.. 1st google link

 

It's more an invitation to see how it feels, but any latency metrics would be handy as well.

 

10 minutes ago, Chakron said:

What AWS region(s) will the new servers be launched in?

 

They will be in Frankfurt, which is roughly close to where the servers are hosted now. We are not yet exploring the notion of having servers in different regions, but I do have VPC peering infrastructure ready if that becomes a reality. The real problem with splitting the servers into different regions is choosing which ones would go where. We have such a diverse player base that we'd be unfair to some, so we decided to keep them centrally located to where they are now.

  • Like 3

Share this post


Link to post
Share on other sites

Yeah, I sure would not create regional servers with the size of the player base as it is today.  If for some reason things went nuts with the steam release, then it could be a great option to consider.  Coming from the USA, lag is pretty minimal as it is now, unless I travel over to Xan.

Share this post


Link to post
Share on other sites

You Sir are a magician!

 

Now take a break and play some Wurm.

  • Like 1

Share this post


Link to post
Share on other sites
1 hour ago, Keenan said:

hey will be in Frankfurt, which is roughly close to where the servers are hosted now.

 

I reckon most everybody playing now is getting routed through Decix in Frankfurt anyway.

Share this post


Link to post
Share on other sites
2 hours ago, Keenan said:

They will be in Frankfurt, which is roughly close to where the servers are hosted now. We are not yet exploring the notion of having servers in different regions, but I do have VPC peering infrastructure ready if that becomes a reality. The real problem with splitting the servers into different regions is choosing which ones would go where. We have such a diverse player base that we'd be unfair to some, so we decided to keep them centrally located to where they are now.


Makes sense. I think a bunch of servers were in US Central 10+ years ago, is that right or did I imagine that?

Share this post


Link to post
Share on other sites

Indy to australia pls. I've played with australia lag for years, now the rest of the world can 

  • Like 3

Share this post


Link to post
Share on other sites
7 minutes ago, Retrograde said:

Indy to australia pls. I've played with australia lag for years, now the rest of the world can 

Gets my vote! lol

Share this post


Link to post
Share on other sites
3 hours ago, Keenan said:

They will be in Frankfurt

😍

GG EU ping in other words...

Share this post


Link to post
Share on other sites
8 minutes ago, Finnn said:

😍

GG EU ping in other words...

Hetzner servers are already in germany. 

Share this post


Link to post
Share on other sites

go keenan go keep it up ❤️ love hearing about the progress as i really want to see the fruits of this project(i mean it i really do)

also @Retrograde au internet is just horrible its sad my friend who lives in sydney and i play a game together where we connect to a server in la my ping is 165 on a bad day hers is 245-260 on a good day and its like really? there is not that much of a distance difference but the infrastructure is just so overloaded over there :(  so please dont ever move any server to au ever(i get a 80-150 ping to sydney on a average with it dropping to 60ish around midnight when what ever they got setup there stops being ######) if anything put one in nz atleast we have the infrastructure here to support it ;):P

Share this post


Link to post
Share on other sites
15 hours ago, Retrograde said:

Indy to australia pls. I've played with australia lag for years, now the rest of the world can 

 

FlippantUniqueJoey-size_restricted.gif

  • Like 5
  • Cat 1

Share this post


Link to post
Share on other sites

Well done, Keenan !  I am sure you deserve a break now after all your hard work - on stuff i have no idea what you are talking about! haha

Share this post


Link to post
Share on other sites
19 hours ago, Wargasm said:

@Keenanany update on this?  Cele lag is becoming unplayable.

 

Cele lag is server side:

http://freedom004.game.wurmonline.com/mrtg/lag.html

 

I just have to quote myself yet 500 times already...

 

On 1/13/2020 at 3:09 PM, Sklo:D said:

It is very important to stop selling AWS as the ultimate problem solver. There will just be a lot of upset people in the end. Hetzner is a very professional host and likely has the best price performance ratio in Europe. AWS will bring better uptimes because it is uses cloud instances and not dedicated hosts like the WO servers atm. It is not because Amazon is better than hetzner, there is a difference in technologies, which also comes at higher costs. On AWS Wurm will still have outages and network problems there is a lot going on on the internet and even Amazon relies on the internet.

 

AWS will not reduce server side lag. Actually when I measured AWS last time I found out that it even has slower I/O performance to disk in some setups. So after the AWS move we could even see increased server side lags depending on the chosen setup.

As long as the Celebration lag is not coming from a hardware problem it will NOT be fixed after AWS move.

 

For everyone reading this: Please stop thinking that AWS will solve all lag problems in Wurm, that is just NOT realistic, there will be a big wave of disappointment. There is a good chance that network problems will be less but also not fixed 100%, there have been similar problems monitored to the AWS network in Frankfurt as they appear with the Hetzner network. Routing in the internet is a big thing and it is often not something an endpoint like Amazon or Hetzner can improve, between some areas the internet is just not as reliable as in others, same as life is not perfect a distributed network is also not perfect.

 

  • Like 1

Share this post


Link to post
Share on other sites

Going to leave this here and direct people to go to this thread as its more recent.

Share this post


Link to post
Share on other sites
2 hours ago, Ostentatio said:

 

The issue with Celebration may be due to server hardware issues. In cases like this, the move to AWS would indeed be relevant.

 

Yeah mentioned that still... There is no excuse for not changing the hardware in constant intervals. You should never use servers longer than 3 years if you are just renting them.

 

Also the fact that there is nothing done against that is pretty sad, such a server move takes 1-2 hours and hetzner has servers available within hours if not minutes, as long as you don't rent the ones which are out because of Corona. 

Share this post


Link to post
Share on other sites
23 hours ago, Wargasm said:

@Keenanany update on this?  Cele lag is becoming unplayable.

 

I'll make an update in the post linked above as it's more recent. :)

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this