Sign in to follow this  
DevBlog

A Note About The Recent Fence Bug

Recommended Posts

On the subject of compensation, I am not really sure what to say. We do not have the people to be able to manually lead all horses back to their owners or prove ownership of horses. We do not have the ability to prove who lost how much due to the issue due to the nature of creatures being unconnected to players.

Sleep bonus is a form of compensation but maybe that is not enough? What would you guys suggest? Taking into account we will not flood the economy with free silvers, we do not have interesting items we can give out in this case that would have any relevance, we cannot provide extra/free premium to those who have lost out as I mentioned previously we have no realistic way of determining who deserves that form of compensation. Spawning horses in general is one idea but it is unlikely to compensate the people who actually lost them.

As I said in the blog post - We are sorry (I think that counts as an apology) for the issue and that people lost out but we are unable to compensate you unless someone comes up with a justifiable and realistic way of doing that. A thousand minds are better than 2 or 3.

I think a good start would be to like I suggested would be to increase the Sleep bonus cap when provided as compensation for a server crash etc. This does not mean raising the cap for normal sleep bonus (keep that as the normal 5hrs) but during periods of constant inconvience to people during major updates like 1.0 Where crashes and bugs are occurring daily only a select few would be able to take advantage of the sleep bonus thrown their way. So a new category of sleep bonus "Compensation bonus" woulld go a long way to help smooth over players such as myself that feel the effects of crashes and unscheduled server restarts but always end up with a full 5hrs SB and no time to use it before the next crash. This would be a good start I think.

As you point out Woosoo it is obviously impossible to return people their horses. Perhaps in this scenario it is impossible to truly compensate players properly which is why I think the issue of compensation should be a case by case basis.

For example I think it would make sense that for extended and multiple crashes over a period of days the downtime of the servers should be added onto a players premium time, if the servers are down for longer than say 6hrs. then you'd get 6hrs extra premium time.

Overall the issue is not one I can see being solved quickly but one that should definitely be given attention and I look forward to seeing other peoples ideas.

Ultimately it comes down to a fact that Rolf so proudly mention last night at that launch party. Everything in Wurm Online has a REAL LIFE VALUE, Not to mention any sentimental value. As such compensation should be taken as a serious matter, otherwise people are going to be left feeling cheated, angry, dissapointed and generally unhappy with the service they receive.

  • Like 4

Share this post


Link to post
Share on other sites

Sleep bonus is a form of compensation but maybe that is not enough? What would you guys suggest? Taking into account we will not flood the economy with free silvers, we do not have interesting items we can give out in this case that would have any relevance, we cannot provide extra/free premium to those who have lost out as I mentioned previously we have no realistic way of determining who deserves that form of compensation. Spawning horses in general is one idea but it is unlikely to compensate the people who actually lost them.

Sleep bonus to me is something you can use when servers have to go down for unforseen maintenance. You compensate by giving players a tool to improve themselves twice as fast in a way they see fit, thus making up for downtime. In a case where people actually lose things that were bought with silver, I'm afraid only replacement of the loss, or replacement of the silver is enough compensation. Because silver is euros, or any other real life currency. Silver is generally more likely bought by new players, and they will be more likely hit by these kind of issues.

On the subject of sleep bonus I do see a problem. The people that lost animals from the recent fence bug, were most likely the ones that couldn't be online shortly after the bug was discovered. Those people are also less likely to benefit from this small compensation (because they were already sleeping in their wurm beds). The problem is that sleep bonus is capped, to 5 hours. A player can be lucky and get 5 hours, others get no compensation at all. Would it not be possible that if the game hands out sleep bonus, for whatever reason, that the limit is the 5 hours + the compensation time?

Back to the compensation issue. As I said ideally when silver disappears, or a wurm item representing this silver, this should be the compensation. In another thread I have said that I believe that consumer laws have legal ways to get compensation for these kind of losses. Money can not just disappear. Even though it's unlikely someone will pursue legal ways, it is possible. Are silver transactions maintained in a seperate database, and can silver transactions between players be easily tracked? If it is not perhaps here lies the solution to the problem.

We all like the game very much, and only hope for things to become better.

Share this post


Link to post
Share on other sites

For example I think it would make sense that for extended and multiple crashes over a period of days the downtime of the servers should be added onto a players premium time, if the servers are down for longer than say 6hrs. then you'd get 6hrs extra premium time.

I believe that is a good suggestion.

Share this post


Link to post
Share on other sites

I have several suggestions:

1.decrease the amount of materials needed and/or decrease the mining timer for some days, so people can build some "secure" fences for themselves.

2. make an official announcement for a horse spawn(like they did for crocodiles on the impalong festival) so people can come by and just grab some horses

3. make rolf say sorry :P

4. post somewhere official, that wooden fences should be avoided for the next weeks or something until the issue is really fixed (maybe you should consider taking more time for server ups and downs AFTER them ;))

5. giving people premium time back would be my last option tbh, cause im pretty sure people would start demanding it for every little thing ;)

Share this post


Link to post
Share on other sites

The wooden fences are supposed to be 'secure' I have been told to build proper fences. I happened to like the wooden fences for animals. Just as I had a small well in one spot and a large fountain in another. It was all based on how they looked with what I had. And yes, I'd like to see an apology from Rolf at this point.

Share this post


Link to post
Share on other sites

At risk of sounding mean, I am glad I am not on Freedom. It frightens me to live on a server with so much trade going on and that losing what you paid for (twice in a row) is actually possible. I know it might be hard to think of a solution but it's hard to just forget about it.

Share this post


Link to post
Share on other sites

I completely understand games have bugs sometimes but a bug of that scale should never get through testing. I'll feel compensated if this particular bug never happens again. I think removing the cap on compensation sleep bonus is a great start too. Also, prompt notice when/if something does go wrong would be very much appreciated. The news section of the website should be the definitive source of official information (with RSS feed) so I don't have to hunt it down.

Share this post


Link to post
Share on other sites

I'd rather not see Sleep Bonus used as some sort of catchall compensation. I don't even use sleep bonus at all, to me it focuses attention on the metagaming aspects of Wurm and that's not where my enjoyment comes from. So I also prefer not having things like increasing the cap etc as it all just contributes to grinding skills and "play to win" in Wurm. I can understand other people play differently and enjoy competing aggressively for achievement and focusing their gametime on this, but please don't think all of us care all that much about Sleep Bonus. Some do, many really really don't. Same with "compensating" premium time. I think the majority of us don;t want compensation of any sort, we just want these incidents to be a lot more UNCOMMON than they currently are.

Its just such a heart break when you spend many many hours on a project only to have it all evaporate through some negligent oversight. All we want in the end, most of us, is just to know the dev staff considers these bugs to be SERIOUSLY worthy of their attention. When Indy had a 10-hour rollback, my heart sank but hey, things happen. When we next found out the fences were glitched on the servers that were still up, and many lost animals, it was confusing why they did not chose to bring the servers down to fix this, but hey whatever, I rounded up what animals I could. But then they took Indy, with its 10 hour rollback, and put it back online in the middle of a major bug, only 10 minutes before a restart that would have fixed the bug. They could not wait ten minutes to avoid having the fence glitch on top of the trrollback? They couldn't restart Indy at the "10 hour rollback" after the restart and before the fence glitch?

I just don;t understand. In my confusion it feels to me like they don't think its really that important. All the sleep bonus and comped premium time is not going to diminish that feeling.

We just want to know this is as high a priority to the staff, as it is to the players. Right now it just doesn't feel that way, and I think the number one thing you can do is to try to reverse that appearance. Sleep bonus is too "cheap" of a copout, it costs you nothing and doesn't feel like you are doing anything more than saying "ah no big deal, we will just toss them some more FREE sleep bonus and then all is ok."

Edited by Brash_Endeavors
  • Like 3

Share this post


Link to post
Share on other sites

we are always looking into ways we can improve the way we test code changes.

The very simple solution to getting the community involved in the test server is that ALL patches are released to the test server with a stated lag of a day to decide if they go to the live server. It will only work if the test server is a backup of the live server, so I can check my own deed with my own characters. If it is an old backup that gets overwritten, I certainly am not going to live and work on test, but I most certainly can run around and see if anything of mine looks OK, since I am so paranoid about patches now I am doing that anyways for myself and my neighbors on live.

Test is only being used for new features with ill equpped alts in an empty world , it needs to be used for all patches with my alts in my world. If you truly want to do something as an official response, than have Rolf come up with a plan over the holidays how he is going to implement that.

Because no amount of compensation brings back a pregnant bison that took several days in the steppe to find.

Share this post


Link to post
Share on other sites

I think the ideal situation certainly involves releasing all features on the test server, and announcing what they will be. For optimum bugfinding, announce a 1hr sleep bonus reward for the first person to find each bug in a new update.

I think it's time to leave behind the naive notion that unannounced changes are a "nice surprise" and just announce stuff with proper community testing.

  • Like 2

Share this post


Link to post
Share on other sites

We will still be surprised when we see it on test just as well on live.

The ninja release of marble and slate is a perfect example of how ninjas do not work. It was poorly implemented randomly seeding the servers without checking for exposed ores first, then a reversion fix converting most of them to iron. Now all the servers have more iron replacing more valuable ores, and marble and slate is nowhere to be found. I have more iron which means our region was 'slated' for slate (which luckily iron replacements replaced lead and zinc not my gold and silver) but now nobody can find any slate or marble.

Absolutely that would have been caught on test with a mine run thru on my deed, I would even have suicided myself over the cliff to check my gold mine since I know it is a backup copy of me. But it would not have been caught with the existing test server, an alt with no gear and no mines cannot check to see if the mines are still good. The only reason for me to go to existing test server is curiosity about new features, I would rather be surprised so I never use test.

To implement this there needs to be a process that only some features from test can move to live, with either features cut that did not work or waiting until the entire patch is solid to move it to live.

If there needs to be a new release server for major features under development with more controlled test population under NDA, then that is a seperate testing mechanism

Edited by yarnevk
  • Like 1

Share this post


Link to post
Share on other sites

Obviously we never intend for bugs like this to happen and in an ideal world they never would but as I said in the blog post this is software and bugs happen. We are improving the way we test features and patches and hopefully this will lead to less problems of this kind in the future, and we have a couple of volunteer team members dedicated to testing on the test server and we are always looking into ways we can improve the way we test code changes.

I like the idea to test on a backup of one of the live servers, that way you'll have all possible kinds of scenarios. And players are more likely to detect issues on stuff they are familiar with.

Maybe you should also introduce this policy: First 8 hours after patch are used for live testing. If everything goes well, it stays, but if something goes horribly wrong, there will be a rollback. So people can expect it and act accordingly (like not building a monument right then). Rolf would have been able to fix fences, rollback the server (maybe keep skills) to restore escaped animals and everyone would be able to accept it.

But You don't only need to improve on testing, you need to improve on update processes as well.

If Rolf had made a backup at the begin of his maintenance session he could have just gone back to that and try again, sure it might have extended downtime, but we wouldn't have lost any game time, let alone 10 hours of it.

Also there are other methods: Databases are able to keep a log of changes or do mini incremental backups so that you could restore them to any point in time.

Or you could do partial backups, Character skill data for example has no interconnection with game world data. (the change methods do, but not the data state)

If Rolf has to rollback the world he could compensate us by leaving our skills intact, that would be worth a lot more than 7 minutes of sleep bonus.

Also Rolf seems to think that we would rather play a broken game, than not play at all. This is only true if you play to spend some time, but Wurm is a game of achievements (building something, getting skill etc) So I would rather not have played, than have all my effort wasted. I was really glad that I did not go to the Impalong right away.

Share this post


Link to post
Share on other sites
Sleep bonus is a form of compensation but maybe that is not enough? What would you guys suggest?

I suggest, that you put more effort into testing. Before you upload the patch to a live server, test it! Be it on a test server, or maybe on some offline machine running tiny map (I don't how scalable is Wurm server software)?

After the patch is uploaded to live server, make it a conditional uptime of, say, half an hour. Check for any unexpected results, especially what was observed before: animals running from pens, boats getting stuck in caves etc. If any such incident is reported (and confirmed), take the server down and rollback, reverting the patch.

It is seriously better to release late than release bugged!

Edited by Hula_Girl

Share this post


Link to post
Share on other sites

I suggest, that you put more effort into testing. Before you upload the patch to a live server, test it! Be it on a test server, or maybe on some offline machine running tiny map (I don't how scalable is Wurm server software)?

After the patch is uploaded to live server, make it a conditional uptime of, say, half an hour. Check for any unexpected results, especially what was observed before: animals running from pens, boats getting stuck in caves etc. If any such incident is reported (and confirmed), take the server down and rollback, reverting the patch.

It is seriously better to release late than release bugged!

This,

And don't patch the game a few hours before a big event that you need to attend since if something big breaks down then you have a problem (as was the case now). And just don't patch the game on a friday at all if you don't work in the weekend (as we've been saying for god knows how many years now. I'm sure many of us remember all the lovely friday updates that made the game pretty much unplayable until it was fixed on the monday..)

Edited by Ecrir
  • Like 2

Share this post


Link to post
Share on other sites

Realize not everyone will have something positive to say when mistakes happen, I get Rolf is doing his best, but we all don't have to be fan boys and be understanding when things go wrong, as to the ones saying its a MMO and we should be used to it, I find it funny that programmers seem to always want a free pass when things go wrong unlike most industries,

Share this post


Link to post
Share on other sites

This,

And don't patch the game a few hours before a big event that you need to attend since if something big breaks down then you have a problem (as was the case now). And just don't patch the game on a friday at all if you don't work in the weekend (as we've been saying for god knows how many years now. I'm sure many of us remember all the lovely friday updates that made the game pretty much unplayable until it was fixed on the monday..)

Yes that is why most proffesional games have an official patch day, like Tuesday. That way the customers always know when new stuff will be added and can be excited / prepared. Aslo the staff has Monday to check over the patch one more time and the rest of the week to fix it if there are unforseen problems when it goes live.

If they cannot stick to a schedule then at least give us a few days notice on the news login page. Like patch 124.56 will be released on December 26th at 9am GMT. Then we can hope for the best and be prepared for the worst.

  • Like 1

Share this post


Link to post
Share on other sites

Exactly, since Wurm went 1.0, maybe it's time to drop the old paradigm of "it's permanent beta, players are familiar community beta testers and they have to accept any damage that happens" in favor of "we sell a product, we have customers which don't have to like us, in fact they don't even have to know about our existence".

  • Like 2

Share this post


Link to post
Share on other sites

Realize not everyone will have something positive to say when mistakes happen, I get Rolf is doing his best, but we all don't have to be fan boys and be understanding when things go wrong, as to the ones saying its a MMO and we should be used to it, I find it funny that programmers seem to always want a free pass when things go wrong unlike most industries,

It's bad when Rolf's best seems to be being stupid half the time.

You might notice that a good chunk of us complaining are actually programmers that can't understand how Rolf can mess up so badly, and repeatedly make the same mistakes.

There are some basic and easy precautions, that are common knowledge and common sense:

  • Do a backup before you do something that could possibly break stuff.
  • Have the change tested immediately and revert immediately when there is an issue.
  • Don't do something that could possible break stuff, when you know that you won't have time to fix it.

  • Also don't try to force the issue, if you can't make the change stick properly, just revert and delay the change for next week.

There is nothing worse than keeping things broken for two month and then trying to fix the accumulated damage.

And if you are trying to change something that is already broken and you can't get it right, try a simpler stopgap for the meantime.

For example: if you can't fix fences fast, stop animals randomly wandering around unless they need food.

And for the "compare to other games" argument: it seems that game programmers in general are a weaker breed of programmer. As an aspiring game designer, this saddens me.

There simply are less repercussions when a game server messes up, compared to when a business or even bank server messes up. Imagine a bank server getting a rollback of ten hours and all funds that were transferred from another in the meantime getting deleted...

Edited by Keldun
  • Like 2

Share this post


Link to post
Share on other sites

The issue is that the problems are not detectable sometimes until after many things come into play. I am sure Rolf would rather not have issues he needs to fix later. it is counterproductive. But unless the exact things happen not all bugs can be detected. The test server is not a proper representation of the normal player environment. Not enough people on not enough of the same building to represent the lag etc. Rolf i believe does the best he can. It is very easy to say what to do when it is not you who needs to do it. I have seen many a simple problem turn into a major catastrophe.

Share this post


Link to post
Share on other sites

1. giving free premium time can turn out to be bad as ppl will want it for every little thing that goes wrong.

2. If the dev's are thinking sleep bonus is not a good compensation maybe they should:

A. make the SB they give go into a compensation SB bracket that will have a limit {say 10-15 hrs max}, not just normal SB that you get from sleeping in a bed that caps at 5, not everyone can use the SB they get thus they lose out of a lot of compensation I know I have lost out on a lot of compensation over the last couple months when things went wrong.

B. maybe think about doing some double exp weekends on top of sleep bonus given out

Do patches on a set day such as monday or tuesday so you have the week to fix stuff don't do it on a friday when you wont be around on the weekend. And don't do a patch when you have a big event to attend.

And maybe do a public apology to keep your customers around. My prem ends this week, I'm going to resub as I love the game but some ppl might not. The world if full of rage quitters!!!!

Edited by Airwalkre

Share this post


Link to post
Share on other sites

Then he should do as many have suggested. No updated on Fridays or any day that he doesn't have the time to spend fixing the problems after it launches.

Share this post


Link to post
Share on other sites

This thread is actually full of some really great ideas. I think this is what frustrates me the most, it's as if the team are there with their fingers in their ears just going "nopenopenope".

Is it so hard to take advice from your customers? Isn't that the whole concept behind wurm's development?

Why use us as beta testers, if you're going to only listen when is suits you.

  • Like 2

Share this post


Link to post
Share on other sites

The issue is that the problems are not detectable sometimes until after many things come into play. I am sure Rolf would rather not have issues he needs to fix later. it is counterproductive. But unless the exact things happen not all bugs can be detected. The test server is not a proper representation of the normal player environment. Not enough people on not enough of the same building to represent the lag etc. Rolf i believe does the best he can. It is very easy to say what to do when it is not you who needs to do it. I have seen many a simple problem turn into a major catastrophe.

Rolf may be trying to do the best that he can, but he's not planning properly to allow the best that he can do actually be good enough.

This was covered fairly thoroughly by Keldun above. Procedural changes designed to prevent errors from exploding all over the face of the company would be a very good idea, especially since this is now supposed to be a released game 1.0

Barring critical failure, don't touch it unless you have a backup. When in doubt, rollback. Make changes in the first half of the week so you have the last half of the week before the weekend to hotfix or rollback if needed.

A lot of this stuff is common sense stuff, but Rolf hasn't really needed to deal with it seriously before, because Wurm was a "beta". It's not beta anymore, and people are going to expect more professional development.

Share this post


Link to post
Share on other sites

Keldun's post, with one little very practical add-on: Rolf, you should not stand up after doing an update for at least 15 minutes. Things like fencebugs and other major catastrophies are mostly found within just the first ten to fifteen minutes and reported. Be ready to flick the switch right there and then in these cases and do an *immediate* rollback (to the back-up that was of course made before any update...). So no-one gets hurt badly.

Most bad bugs wouldn't have bad consequences if just this simple rule could be followed imo.

  • Like 1

Share this post


Link to post
Share on other sites

Whilst there hasn't been a huge amount of ideas regarding compensation there has been a ton of ideas on how Rolf can improve his procedural guidelines for applying a patch to the servers or major update (If he had any kind of procedure before hand?) Certainly some of the idea mentioned here in that regard would stop or reduce any possible problems 10 fold.

Personally the thing I used to hate most was Friday updates, Rolf would seemingly update the game last thing friday, pack up his stuff and go home and be unreachable for the entire weekend. The players would be left to sweat out whatever problems would be with the client until monday. Now I entirely understand the importance of Family and RL etc, however the problem could be avoided by simply doing monday updates. Such a simple thing would make people so much happier. that said, I havn't seen a Friday update in awhile now which is comforting but i'm using this as an example of how poor implementation can alienate and hurt your player base.

I'm hoping that Rolf is reading through these responses or you'll be able to pass on some of the better ideas to him Woosoo as it would go a long way I think to helping this game.

As a note, Rolf if you do read this read I hope you understand that everyone here loves your game and appreciates the work you've done on it. If the game wasn't so great, allowing you to form attachments and place personal value on your animals and time in game. If this game wasn't so imersive and have such great potential and existing fun, people would be much less likely to care. Please take it as a compliment that some people are so upset that things mess up. By that same token, you have a resonsibilty AND the ability to provide people with an absolutely amazing gaming experience Which is what yourself and the team should be striving for. I hope that you can see past some of the negativity and frustration and take some f the suggestions for what they are. Just customers trying to tell you what would make their experience better.

  • Like 5

Share this post


Link to post
Share on other sites
Sign in to follow this