Keenan

Devblog: Server Issues Postmortem & Future

Recommended Posts

Posted (edited)

I am not sure who looks into the security aspect of things around CCAB but I hope you have someone who is confident and capable in their ability doing your security research and fortification - AWS has been a huge target in the last year for hacking - Namely a bunch of classified government-owned software got leaked from an AWS Bucket repo with poor security permissions, and now the hacking community has gone ape#### with wanting to know what else is on the service.

 

EDIT: Am I wrong in assuming that CCAB doesn't actually HAVE anyone whose main job is to oversee the security of the servers? I couldn't imagine what that person does with their time since virtually every function of the client could be hacked or modified in client-side memory up until about 3 years ago.

Edited by whereami

Share this post


Link to post
Share on other sites
1 minute ago, whereami said:

I am not sure who looks into the security aspect of things around CCAB but I hope you have someone who is confident and capable in their ability doing your security research and fortification - AWS has been a huge target in the last year for hacking - Namely a bunch of classified government-owned software got leaked from an AWS Bucket repo with poor security permissions, and now the hacking community has gone ape#### with wanting to know what else is on the service.

 

I've got a lot of experience with AWS security. I'll be ensuring that everything is locked-down as hard as possible. One thing I like about using CloudFormation is you can set up IAM policies - which means you can specify things like "only allow connections from this IP" which isn't even known until after the IP is assigned at that moment.

 

Security really isn't hard. It's when people "cheat" and think it'll be "okay" that things fall down.

 

Specifically regarding buckets - I would *never* make them public.

Share this post


Link to post
Share on other sites
1 minute ago, Keenan said:

Security really isn't hard. It's when people "cheat" and think it'll be "okay" that things fall down.

Don't try telling that to a financial firms security analyst after the spectre/meltdown crisis... Sometimes we aren't even the ones responsible for the problem. ;)

 

Good to hear that you're versed in it some, wouldn't want one of those kinds of issues to happen here.

Share this post


Link to post
Share on other sites

Hello,

 

Thank you for your hard work. If you need testers to try anything AWS server related, I'm sure you'll find many here. Something like, I don't know, I've created a plain server with 10 dragons, let's kill them together. While we have fun, you can look at all your logs and statistics to be sure everything is fine. I have no idea what I'm talking about!

 

All the information you share is incredibly interesting, even I don't understand half most of it. Keep sharing more information.

 

Regards,

 

Sarcofax

Share this post


Link to post
Share on other sites
On 3/2/2019 at 7:44 AM, Keenan said:

The idea of being able to define whole server farms in code and commit that to a repository... well, it rivals coffee.

 

Hard to beat coffee :P

Share this post


Link to post
Share on other sites

Ty for the information. Frankfurt EU yaaaaay, this is greate :D It's near me. Looking forward for the move :).

Share this post


Link to post
Share on other sites

I will be playing devil's advocate here...

I am NOT IT specialist but somehow I found odd how Sklo comments are not being quite appreciated.

If there is ANY to have a saying in this particular topic than he is the guy. Lots of facts to support his statements, the most important being the fact that he runs the best Wurm Unlimited servers/community. And this is not an overstatement.

Maybe a bit more open minded approach on his ideas would not be a bad thing... After all, he has facts to back up the statements he made.

Share this post


Link to post
Share on other sites

@ZalxisJust join the jolly train and be happy about wurm's upgrade into the cloud, enough windows hyperv stuff 😉

Share this post


Link to post
Share on other sites
1 hour ago, Zalxis said:

I will be playing devil's advocate here...

I am NOT IT specialist but somehow I found odd how Sklo comments are not being quite appreciated.

If there is ANY to have a saying in this particular topic than he is the guy. Lots of facts to support his statements, the most important being the fact that he runs the best Wurm Unlimited servers/community. And this is not an overstatement.

Maybe a bit more open minded approach on his ideas would not be a bad thing... After all, he has facts to back up the statements he made.

There are those in IT that love everything cloud and IOT and there are those who hate it and those who are in different and those in between any of those points

If done well AWS can be a huge boost to wurm or any other cloud platform for that matter
The reasons stated by sklo seem to come more from a "i read about this and these statements concern me" over a "i spend a long time with this software solution and for wurm's use case it isnt the right way to go and this is why" sklo might have the oldest and biggest WU server to date but keenan actually works in the field of IT specifically with cloud services based on what i have read from his posts in the past and in here and honestly id have to side with keenan over sklo on this one as yes they might be good concerns but if done well and properly it will be an overall benefit to wurm

As for some of the things posted by sklo it shows "enthusiast in server hosting" vs a developer in cloud services nothing against you sklo it is just how your posts come across and it has to be said before more people start to jump on out of confusion and it ends up becoming a pissing contest between 2 different ideals 

Share this post


Link to post
Share on other sites
3 hours ago, Zalxis said:

I will be playing devil's advocate here...

I am NOT IT specialist but somehow I found odd how Sklo comments are not being quite appreciated.

If there is ANY to have a saying in this particular topic than he is the guy. Lots of facts to support his statements, the most important being the fact that he runs the best Wurm Unlimited servers/community. And this is not an overstatement.

Maybe a bit more open minded approach on his ideas would not be a bad thing... After all, he has facts to back up the statements he made.

 

While we are committed to this AWS move, I'm never against being convinced otherwise. I just didn't find his arguments convincing as my own experience proves his comments wrong. I know that AWS has become one of those jargon words that people throw around to make something seem more important. It's kind of like "blockchain" and "NoSQL". However, each of those terms mentioned are tools that serve a purpose in the right context. Sklo was arguing that Wurm's server needs are not in the right context for AWS, and I stand firm in saying they are.

 

As for his facts, I believe I previously mentioned that they appear outdated. I have to agree with wipeout above that they seem to come from a place of having read some articles rather than actually using AWS. I actually recall stating the same facts three years ago when my previous company originally wanted to move into AWS. Two years later, the facts were looking different and the move was made.

Share this post


Link to post
Share on other sites
25 minutes ago, Keenan said:

I actually recall stating the same facts three years ago when my previous company originally wanted to move into AWS. Two years later, the facts were looking different and the move was made.


At my last job we had so many companies come to us with the same mindset they where introduced to it early on when everyone was hyping up cloud and aws and microsoft's cloud solutions and imb's and most of the time we had to tell them it isnt mature enough for the services you need but last year that all started to change where more and more boxes got ticked for our clients and more and more projects became focused on moving existing services onto aws and similar platforms

These last 2 years a lot has gotten better and it has turned from a gimmick that suits special cases to general server usage(web servers/databases/gaming servers and so on) being worthwhile to be put on it once you got the capital and need for it
1 thing is still true pick your plans properly and set caps as if left unchecked a server could potentially have a cpu usage or ram usage spike and your budget for that month suddenly looks like a mushroom cloud(1 client had this happen racked up a 16k bill on 1 of their servers due to them taking things easy and leaving things uncapped even after being told not to do so)

Share this post


Link to post
Share on other sites

First of all huge props to you @Keenan for this post. I hope some of the comments here don't discourage you and other devs from  doing this more often in future as some of the comments gotta make you ask why are you doing that to yourself :P. I enjoy reading this stuff and gives me some inspiritaion to go explore about things i am unfamilliar with :) Never been much into infrastructure so i can't judge who is right or wrong but i urge to people that comment to be more civil and get rid of i know everything attitude, if you have counter arguments and you think you could help with your expirience please to that in constructive manner not by harsh criticism only, please. Keenan also probably know what's he doing and have his reasons, i doubt he made this descision without much analizing. So huge suport from me and keep up with good work. I hope this will bring much benefit to Wurm :) 

Share this post


Link to post
Share on other sites

Thank you to all for a most fascinating read.  I am no longer current on the "ins and outs" of code and operating large scale server farms and nodes and such, but I am totally impressed by the Civility in the above discussion.  Very informative and professionally done by all parties.

 

Kindest regards, Hughmongus

Co-Administrator - The Albia Roads Map of Indy

Share this post


Link to post
Share on other sites
21 hours ago, Keenan said:

Not having to maintain hardware is a huge benefit.

 

I can't speak to AWS specifically, but I can attest that the above fits my experience. (In RL I'm a Linux system administrator.)

Share this post


Link to post
Share on other sites

Yesterday Indy had 300+ people and worked better than it has in weeks. Today, It's the worst it's been since moving back to the old hardware. Please investigate.

Share this post


Link to post
Share on other sites
Posted (edited)

I'm by no means an expert at anything to this level, but I am in IT, and I do deploy - https://www.nutanix.com/

 

I'm not sure how you guys are placed to house your own infrastructure, or to find a reliable data center and rack space, but if you haven't given this a look, i'd recommend it.

Edited by Dumbo

Share this post


Link to post
Share on other sites

There is nothing wrong with cloud for a case  like wurm, and done responsibly it can save money with more performance and reliability. The only issues with cost I have seen on cloud services like google and AWS is with these new scalable games, the server hardware growth as each new watchere is spun up is not linear it is exponential, which causes a fast increase in funding each time you add a player. Wurm is pretty flat in those regards so the cost aspect should not be an issue with someone familiar with it setting it up custom to handle the servers for Wurm.

Share this post


Link to post
Share on other sites

Hey, you know what? If the devs make the move and it turns out that at the end of the day, the whole game is lag free and stutter free and the move makes positive changes I can tangibly experience, I will worship Keenan like a Roman god, make indecent offers to him, and build a shrine in his name.

 

If no change can be felt but the Independence problems go away, then I'll be a happy camper and continue to play the game as I would any other time.

 

And if the change makes thing terrible and the servers start getting downtime and roll backs and the world seems to end... then I'll go play Hello Kitty Online or something.

 

That's what it all translates to in real terms to me. All the technical mumbo jumbo I leave to him because its what I pay for and he's the man with the keys to the house. I trust he'll do a great job and at the very least make things run smoothly again. He's working on it, and the matter is not forgotten. That's enough for me so I'll wait patiently.

Share this post


Link to post
Share on other sites
29 minutes ago, Angelklaine said:

Hey, you know what? If the devs make the move and it turns out that at the end of the day, the whole game is lag free and stutter free and the move makes positive changes I can tangibly experience, I will worship Keenan like a Roman god, make indecent offers to him, and build a shrine in his name.

Soon things are about to get awkward around.. but we need a better promise, something solid, this smells like a bet here.🧐

Share this post


Link to post
Share on other sites

I'm just happy to see that with all the ups and downs recently, that Keenan is having fun with this project. I too agree that AWS is a great direction to take the Wurm infrastructure, but in nearly every cloud deployment that I've seen, there are growing pains/tweaks/problems that will need to be corrected. We may be in for a little rough sailing for a bit, but I'm confident that we'll be in smooth seas in the end! :)

Share this post


Link to post
Share on other sites

That's good news, AWS will bring serenity and for sure improve the scalability and smoothness, exciting news indeed.

Share this post


Link to post
Share on other sites

Time for another update!

 

I've been working more on provisioning, which is now right down to standing up Percona MySQL for each game server paired off on an instance. There's a few hitches to work through, but soon I'll be working on getting the Wurm code deployed. The next step after that will be deploying test servers and having a go with that.

 

For the technically inclined, the way I'm handling isolating multiple instances of MySQL and Wurm is through the use of Docker. I've not worked out the logistics of Wurm's docker container yet, but the MySQL one is working perfectly.

 

I've also broken the stacks out into the EBS volumes, the network, and the instances. This makes it easier to make a change to a specific stack without bothering the rest of them.

 

That's all for now. :D

Share this post


Link to post
Share on other sites

Is this process expected to take weeks or months? From the sounds of it a test server should be fairly soon.

Share this post


Link to post
Share on other sites
2 minutes ago, Kelody said:

Is this process expected to take weeks or months? From the sounds of it a test server should be fairly soon.

 

I'm hoping to have test servers on AWS by the end of the week. I've done this so many times now, it's kind of second nature for me.

 

We'll be doing a test with live server data as well. I'm hoping we'll be giving a firm date on the move by the end of two weeks, if not before.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now