Sign in to follow this  
madnezz

Testing

Recommended Posts

Whatever happened to testing the patches before they go live?  It seems like every patch these days is broken, today's being no exception.  Why are you in such a rush?  Slow down and test your patches. 

 

1.  Design patch

2.  Code patch

3.  Run exact copy of patch on a test server and test internally for a week.  Test server should be an updated and exact copy of a live server/s.  

4.  Open test server to public and allow players to break the patch for a week.

5.  Tweak patch and re-release on test server until no more bad things happen (as long as it takes).

6.  Release patch to live servers, confident the bugs should have been found and corrected.

7.  No longer have to read all the complaints about the fact you don't test anything ever.  Avoid knee-jerk server restarts.

 

 

*Pvp changes should undergo much more rigorous testing.

Edited by madnezz
  • Like 3

Share this post


Link to post
Share on other sites

This is just as helpful as "don't break stuff".

We test all the changes, and none go live unless it works fine on test for a period of time. It's just that certain issues tend to come out only on live, most of which were unlikely or impossible to appear on test, but at least we're quick at fixing them.

  • Like 6

Share this post


Link to post
Share on other sites
2 minutes ago, Samool said:

We test all the changes, and none go live unless it works fine on test for a period of time.

 

I'm sorry, Samool, but I'm sure this is not quite as it sounds. There's a 1-week release schedule, and many things that make it into the weekly patch were not on public testing before. I even remember some being available on the test server for less than a day or two, I was trying to make time and planned to test systematically when boom, they were released. Perhaps you mean (closed) internal testing, I'm sure you do such as every developer does, but I have to agree that releasing them for public testing server/wider testing could help.

 

Perhaps the current issue was considered urgent enough, that there was no time for public or wider testing. That's part and parcel with weekly patches, I'm afraid, too. You may want to consider bi-weekly, I don't know. It's nice for people to get weekly changes, heck it's quite a feat in the industry, but it doesn't have to be so often that they don't get proper time.

 

14 minutes ago, Samool said:

It's just that certain issues tend to come out only on live, most of which were unlikely or impossible to appear on test, but at least we're quick at fixing them.

 

Definitely. And the fast work of the brave souls that are monitoring and handling things immediately is appreciated. Please don't doubt people do, even if you see frustration sometimes.

  • Like 1

Share this post


Link to post
Share on other sites

Not breaking stuff is a good start if you want to begin with that and continue the snark from your team.  I vote for less snark and more testing, however.

 

Players should be given the opportunity to test any and every change we pay you to make since there are always soooo many errors with the majority of patches.  Take your time and do it right the first time and you wont have to be quick at fixing them with multiple restarts.  Again, why are you in such a rush?

Edited by madnezz
  • Like 1

Share this post


Link to post
Share on other sites
7 minutes ago, Anarres said:

There's a 1-week release schedule, and many things that make it into the weekly patch were not on public testing before.

It's not really a 1 week release schedule, it's just recently that we've had this many patches in a row.

 

We do public testing of certain bigger features - while every single thing goes on public test servers before it goes live, I don't think it's viable to announce every single change on the test servers beforehand, when they get altered or removed very often anyway.

  • Like 1

Share this post


Link to post
Share on other sites

That is the problem, they are viable, should be announced, and vigorously tested.  More communication over poor communication and we all make the game better.  Players that enjoy playing on test servers understand a testing environment is volatile and not the end product so there will not be an expectation of perfection.   Players on production servers do have those expectations though.  It would be one thing if this was the 1980s and MMOs had never been done before.

 

I mentioned years ago to one of your colleagues this idea of creating a permanent testing environment where the players that like to test and break stuff are embraced and rewarded.  Its not too late to make it happen.

Edited by madnezz

Share this post


Link to post
Share on other sites

Actually, this is quite amusing since I actually specifically checked the skill listings before this patch went live. 

 

On test, it all worked perfectly fine, it was completely straight up and I even took pictures! 

 

We do a lot of internal testing, and all changes go onto test in the meantime, jumping on test at any point will have the dev buiild of the game.

 

The public testing is for major features and we give the full details out, but it'd be a bit difficult to constantly give out details of where the test build is at

 

TL;DR 

Things work fine on test then break on live sometimes, it sucks. We also have a permanent testing environment as the test servers 

  • Like 1

Share this post


Link to post
Share on other sites

Please explain how graphic bugs like hell horses, saddles etc work differently on test than live.  I'm genuinely curious.

Share this post


Link to post
Share on other sites
6 minutes ago, Wargasm said:

Please explain how graphic bugs like hell horses, saddles etc work differently on test than live.  I'm genuinely curious.

It's possible that the graphics things like that simply arent tested as they're byproducts of other changes. 

 

Work has been done on armour and mapping and that was the focus, geared hell horses weren't part of the criteria as it wasn't intended to affect them.

 

That's how these sorts of things happen, they're not "we worked on hell horses and they were fine" but "we worked on towers and somehow it buggered up trellises" 

Share this post


Link to post
Share on other sites

I can understand that the entire player population will find bugs better/faster than the devs, but it consistently seems like your test environment is fundamentally different somehow than production, so that the bugs simply aren't in test to be found.

  • Like 1

Share this post


Link to post
Share on other sites
4 hours ago, Roccandil said:

I can understand that the entire player population will find bugs better/faster than the devs, but it consistently seems like your test environment is fundamentally different somehow than production, so that the bugs simply aren't in test to be found.

Like any test > live environment, bugs are always going to go through. 

 

This is due to how many different use cases exist on live that we physically cannot account for on test. hundreds of people using mechanics in hundreds of ways tends to throw some out of the woodworks. 

 

We catch many as they come through, but our build literally goes dev > live 

 

The live servers are updated with the copy of the dev build that exists on the test server at the time of code freeze 

Share this post


Link to post
Share on other sites

Which is why you should test EVERY SINGLE PATCH on a test server that players have access to for a week at the very least before going live.  The bugs this week were due to a patch not going live properly, forgetting a password has the same result as forgetting a line of code, its a bug.  If you tested the patch for more than two minutes on a public test server that was an exact copy of a live server or servers you would have found the password mistake and corrected it before the update went live.   It is quite obvious that this special dev build is not the same as a copy of a live server but it sure is special.   Your build should literally go dev > public test > tweak > public test > live.

 

You still haven't answered the question, WHY ARE YOU IN SUCH A RUSH???

Edited by madnezz

Share this post


Link to post
Share on other sites
On 3/27/2019 at 2:44 PM, Retrograde said:

It's possible that the graphics things like that simply arent tested

2 hours ago, Retrograde said:

our build literally goes dev > live

 

So what you're saying is you don't do public testing

 

Gee there's something we didn't know guys

 

Now how about we do something to change that eh

Edited by whereami

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this