A Glimpse Of The Development Process (1/18/17)

Bowgentle

drayviper32 wrote:

"We do our best to cover a variety of devices and saves, but we do end up adding new scenarios to our test-case list quite regularly. (For example, Hulk (Bruce Banner) is now in the test case list!)"

Does this mean that Banner is a future buff??? Would be great if you can get the AP that Hulk matches!!!

No, it means they do now test if future updates cause his purple to crash the game. Again.

Dragon_Nexus

I'd like to ask a question with regards to R117.
While the bug that affected the AP cost of characters was, on the whole, beneficial (broken OBW was a lot of fun to use in Combined Arms, gotta say. Plus slowed down Juggernaut was a benefit in the Heroic PvE) it did also have some negative issues. All the benefitting characters didn't add up to the negative of a slowed down IM40.

With all that said, what was the decision process in releasing R117 as it was, since it was so feature-lite, and having the players endure what was a pretty significant bug for two weeks? I don't think many people here would have minded waiting a further two weeks for R117 to release if it meant we didn't have to tollerate the AP cost bug.

irwando

D3 Go! Producer, Josh Austin wrote:

i. We do our best to cover a variety of devices and saves, but we do end up adding new scenarios to our test-case list quite regularly. (For example, Hulk (Bruce Banner) is now in the test case list!)

Wait, how do you not have basic testing that tests each character's powers when you create the character? Like at least check the golden path. Start a match with the character + enough ISO to fire + any pre-requisites (X number of speciifc color, or special tiles, whatever), fire the power, make sure it at least fires?

There's now a Mystique bug where her purple doesn't fire. Surely that would have been caught by starting a game with Mystique, giving her the appropriate AP, having at least 1 purple tile on the board, and then firing the power.

Please tell me your test system at least allows you to load a match with specific AP given to any team and/or specific board configuration. If not please stop delivering features now and fix this.

The software developer in me is crying.

carrion_pigeons

I know in my own work (which isn't programming, but certainly involves development), when there are demonstrated to be hiccups in a particular part of the process, the answer is usually to insert another development step where the hiccups are.

In this case, it seems like it would make sense to maintain a build between the live build and the dev build, and just run it for a couple weeks on various devices before letting it replace the live build, to shake out any bugs. Might make your process less efficient, but ultimately efficiency isn't what players are clamoring for.

To extend the metaphor in the second post about flat tires, when you're driving a 500 mile race, it might be tempting to skip a pit stop, but you'll regret it when you blow a tire.

fmftint

Dragon_Nexus wrote:

I'd like to ask a question with regards to R117.
While the bug that affected the AP cost of characters was, on the whole, beneficial (broken OBW was a lot of fun to use in Combined Arms, gotta say. Plus slowed down Juggernaut was a benefit in the Heroic PvE) it did also have some negative issues. All the benefitting characters didn't add up to the negative of a slowed down IM40.

With all that said, what was the decision process in releasing R117 as it was, since it was so feature-lite, and having the players endure what was a pretty significant bug for two weeks? I don't think many people here would have minded waiting a further two weeks for R117 to release if it meant we didn't have to tollerate the AP cost bug.

R117 contained a new character. delaying launch 2 weeks is unacceptable (to Demiurge)

Cthulhu

Dragon_Nexus wrote:

I'd like to ask a question with regards to R117.
While the bug that affected the AP cost of characters was, on the whole, beneficial (broken OBW was a lot of fun to use in Combined Arms, gotta say. Plus slowed down Juggernaut was a benefit in the Heroic PvE) it did also have some negative issues. All the benefitting characters didn't add up to the negative of a slowed down IM40.

With all that said, what was the decision process in releasing R117 as it was, since it was so feature-lite, and having the players endure what was a pretty significant bug for two weeks? I don't think many people here would have minded waiting a further two weeks for R117 to release if it meant we didn't have to tollerate the AP cost bug.

R117 was worked on during the holidays, a lot of people on the team were off for the holidays on vacation.

Cthulhu

irwando wrote:

D3 Go! Producer, Josh Austin wrote:

i. We do our best to cover a variety of devices and saves, but we do end up adding new scenarios to our test-case list quite regularly. (For example, Hulk (Bruce Banner) is now in the test case list!)

Wait, how do you not have basic testing that tests each character's powers when you create the character? Like at least check the golden path. Start a match with the character + enough ISO to fire + any pre-requisites (X number of speciifc color, or special tiles, whatever), fire the power, make sure it at least fires?

There's now a Mystique bug where her purple doesn't fire. Surely that would have been caught by starting a game with Mystique, giving her the appropriate AP, having at least 1 purple tile on the board, and then firing the power.

Please tell me your test system at least allows you to load a match with specific AP given to any team and/or specific board configuration. If not please stop delivering features now and fix this.

The software developer in me is crying.

That is a lot of characters, with a lot of different levels and a lot of character combinations to take into consideration.

Edit: Not to mention we get builds sometimes daily as we get close to submitting a build, so checking all the characters on a build daily is a pretty difficult (if not impossible) task.

We are working on improving steps, a lot of that is taking place this month.

HaywireII

Cthulhu wrote:

That is a lot of characters, with a lot of different levels and a lot of character combinations to take into consideration.

Edit: Not to mention we get builds sometimes daily as we get close to submitting a build, so checking all the characters on a build daily is a pretty difficult (if not impossible) task.

We are working on improving steps, a lot of that is taking place this month.

Having testers and test scripts is a good and necessary part of the development process but with the number of existing characters and the addition of a new one every two weeks there needs to be unit tests. When you do a build the unit tests run and all of the assertions are tested. If a bug is found later unit tests should be added to cover the fix for the bug.

When a developer checks in code that fails the unit tests everyone gets to point at them and shame them. It's a good time for everyone.

If you tell me what development environment and tools you are using I'll even help you find a unit testing framework.

Pongie

Cthulhu wrote:

That is a lot of characters, with a lot of different levels and a lot of character combinations to take into consideration.

Edit: Not to mention we get builds sometimes daily as we get close to submitting a build, so checking all the characters on a build daily is a pretty difficult (if not impossible) task.

We are working on improving steps, a lot of that is taking place this month.

Sounds to me like the testing is done manually. Your test engineer should be working on creating new test scripts along side your dev engineer (creating new content). These test scripts should be automated and running in the background after each build. It's the only way to make sure no regressions occurs. New bugs are expected (since no software is ever bug free). But test automation should detect previously resolved issues if they are ever broken again.

Bowgentle

Cthulhu wrote:

irwando wrote:

D3 Go! Producer, Josh Austin wrote:

i. We do our best to cover a variety of devices and saves, but we do end up adding new scenarios to our test-case list quite regularly. (For example, Hulk (Bruce Banner) is now in the test case list!)

Wait, how do you not have basic testing that tests each character's powers when you create the character? Like at least check the golden path. Start a match with the character + enough ISO to fire + any pre-requisites (X number of speciifc color, or special tiles, whatever), fire the power, make sure it at least fires?

There's now a Mystique bug where her purple doesn't fire. Surely that would have been caught by starting a game with Mystique, giving her the appropriate AP, having at least 1 purple tile on the board, and then firing the power.

Please tell me your test system at least allows you to load a match with specific AP given to any team and/or specific board configuration. If not please stop delivering features now and fix this.

The software developer in me is crying.

That is a lot of characters, with a lot of different levels and a lot of character combinations to take into consideration.

Edit: Not to mention we get builds sometimes daily as we get close to submitting a build, so checking all the characters on a build daily is a pretty difficult (if not impossible) task.

We are working on improving steps, a lot of that is taking place this month.

Thank you for responding to criticism regarding your testing process.

A couple of months ago there would have been bans for posts like these.

Dragon_Nexus

Cthulhu wrote:

Dragon_Nexus wrote:

I'd like to ask a question with regards to R117.
While the bug that affected the AP cost of characters was, on the whole, beneficial (broken OBW was a lot of fun to use in Combined Arms, gotta say. Plus slowed down Juggernaut was a benefit in the Heroic PvE) it did also have some negative issues. All the benefitting characters didn't add up to the negative of a slowed down IM40.

With all that said, what was the decision process in releasing R117 as it was, since it was so feature-lite, and having the players endure what was a pretty significant bug for two weeks? I don't think many people here would have minded waiting a further two weeks for R117 to release if it meant we didn't have to tollerate the AP cost bug.

R117 was worked on during the holidays, a lot of people on the team were off for the holidays on vacation.

Forgive my ignorance but I'm not sure that answered my question at all.
I was wondering why R117 was allowed to go through knowing it had this flaw in it, rather than just holding back and publishing a combined R117 and R118 at the next update.

mpqr7

Regression testing should definitely be automated. I'm hopeful that there's an ever-increasing set of automated tests that run before every new build is released that include tests to cover:

1) Will all character powers work as expected
2) Will all character power costs be as expected
3) Will game not crash more than normal
4) Will all features work as expected
5) Will all screen transitions work as expected
6) Will all buttons work as expected

Basically you get the point.. will everything work as expected, or will there be any surprising new bugs that any of us players will discover almost immediately upon playing with the new update.

irwando

Cthulhu wrote:

R117 was worked on during the holidays, a lot of people on the team were off for the holidays on vacation.

This is exactly why major software companies and online services lock down and don't release anything during the holidays. Learn from this. The patch can wait. Do more testing.

irwando

Cthulhu wrote:

That is a lot of characters, with a lot of different levels and a lot of character combinations to take into consideration.

Edit: Not to mention we get builds sometimes daily as we get close to submitting a build, so checking all the characters on a build daily is a pretty difficult (if not impossible) task.

We are working on improving steps, a lot of that is taking place this month.

I'm glad you're improving it. I'll believe it when I see patches released without game/character breaking changes every week.

It sounds like the end to end software development process there is just broken. Schedule is prioritized above anythign else including quality, which leads to bad software. Automation is hard to set up but saves tons in the long run.

snlf25

A Glimpse Of The New Feature/New Event Implementation Process (1/18/17)

4:59:50 Okay guys, we are......wait for it.....live!
4:59:55 *high fives all around* *celebration* Yay, we did it!
4:59:59 See you guys Monday!



EDIT: Seriously though, thanks Brigby!

Roguewookie26

This is great and all but how about known issues that have existed for a long time. A good example is duplicate covers especially for 4 and 5 star characters. I have 22 champed 4* with another 4 just waiting on iso. There are 14 covers out of 135 availble 4* covers that i just cant use because they have 5 covers in that color and are not at 13 covers to champ. Somehow i have managed to pull one of these 14 unusable covers the last 8 times i have turned in 25cp. Thats ridiculous. There needs to be some type of conversion or exchange system. They do this through cs for latest legends 5* characters only currently. Why not at least all the 5*? Why not 4* as well. Im getting very frustrated that i work to save my cp which the devs rate as so valuable and thats 200 of it wasted in a row. The reward system vs time i have to spend to attain them for this game is the worst of any game i play. And then to basically just throw those rewards away time after time....i dont need more new characters (especially not the very very low tier ones like ironheart) i need a reason to continue to spend my time and/or money on this game versus the million other ones out there that actually reward a player that works hard. How about you guys work on that problem before even more players quit.

ngoni

Colorless covers have been teased for a long time but don't hold your breath waiting for them. They've also talked about being able to choose specific characters to slightly boost their pull rate but again no ETA for that feature.

Roguewookie26 wrote:

This is great and all but how about known issues that have existed for a long time. A good example is duplicate covers especially for 4 and 5 star characters. I have 22 champed 4* with another 4 just waiting on iso. There are 14 covers out of 135 availble 4* covers that i just cant use because they have 5 covers in that color and are not at 13 covers to champ. Somehow i have managed to pull one of these 14 unusable covers the last 8 times i have turned in 25cp. Thats ridiculous. There needs to be some type of conversion or exchange system. They do this through cs for latest legends 5* characters only currently. Why not at least all the 5*? Why not 4* as well. Im getting very frustrated that i work to save my cp which the devs rate as so valuable and thats 200 of it wasted in a row. The reward system vs time i have to spend to attain them for this game is the worst of any game i play. And then to basically just throw those rewards away time after time....i dont need more new characters (especially not the very very low tier ones like ironheart) i need a reason to continue to spend my time and/or money on this game versus the million other ones out there that actually reward a player that works hard. How about you guys work on that problem before even more players quit.

LordXberk

mpqr7 wrote:

Regression testing should definitely be automated. I'm hopeful that there's an ever-increasing set of automated tests that run before every new build is released that include tests to cover:

1) Will all character powers work as expected
2) Will all character power costs be as expected
3) Will game not crash more than normal
4) Will all features work as expected
5) Will all screen transitions work as expected
6) Will all buttons work as expected

Basically you get the point.. will everything work as expected, or will there be any surprising new bugs that any of us players will discover almost immediately upon playing with the new update.

Corrected:
1) Will the game open?

alphabeta

Seems appropriate to revisit this topic given we are at the release of another patch and another major bug pervades.

Seriously inability to even open the game on one of the major platforms the game is played on - how does that happen exactly?

blinktag

Cthulhu wrote:

That is a lot of characters, with a lot of different levels and a lot of character combinations to take into consideration.

Edit: Not to mention we get builds sometimes daily as we get close to submitting a build, so checking all the characters on a build daily is a pretty difficult (if not impossible) task.

We are working on improving steps, a lot of that is taking place this month.

Golly. You've just highlighted the problem.

As to the first point, yeah, that's a lot, but they should have built up over time along with the roster of available characters. And no, an automated test of each power shouldn't take long at all on a *distributed* test system. Depends on how wide you were to scale it, but 30 minutes across multiple virtual machines seems reasonable. Could be done in less.

As to the second point, builds are supposed to get *less* frequent (and more stable) as you move closer to release. If they're not, you don't have enough time for testing in your release plan. In addition, something is happening in the design or development phases to put in so many bugs that aren't caught until testing.

We're glad you believe you're improving, but the amount of naïveté I see communicated about testing policies is disheartening.