A Glimpse Of The Development Process (1/18/17)

Brigby
Brigby ADMINISTRATORS Posts: 7,757 Site Admin
edited January 2017 in MPQ General Discussion
Hey Everyone,

In the past, a lot of players have asked for us to explain the development process, and provide a behind-the-scenes look at what it really takes to address an issue. Well look no further!
I teamed up with our producer, Josh Austin, to find out what kind of steps the development team goes through in order to tackle these challenges head on.
D3 Go! Producer, Josh Austin wrote:
1. When an issue is reported, the first thing to helping the team find the issue is to try and reproduce it, and gather logs to track it down. The closest thing I can think of to this process is losing your car keys in your house and having someone move them to a location they don’t remember. You try to retrace your steps, and find out any information you can on who moved it, in order to help you find it sooner.

If you're experienced with the game code, you know the house. If not, then it’s like staying at a relative's place; it's even more difficult, but you get the general idea of how the house is laid out. Sometimes the code was changed by another programmer that was working on that section, so you may have to dig a little, and track that person down to understand the changes they did. Etc.
    i. Sometimes these issues aren’t seen in a QA environment for some reason, or they can’t be reproduced on a debug build, so the developer must review a crash log taken from the live build (or whatever information they can get to try and figure out what is causing it).
2. Depending on the issue, there would need to be a code change, which requires a full release to update. This is usually the case for the majority of issues. Sometimes a patch can be pushed to adjust something and address an issue, but typically a full build will have to be created.

There are two parts to a live mobile game like Marvel Puzzle Quest or Magic the Gathering Puzzle Quest: Data patches, and Server patches. Patches that do not require a full submission to first party can fix things like text, and sometimes art. Sometimes code level fixes can be done, but there are times when a code change could bring live build down. When this happens, it halts all work outside of the live game (new features, new content, bug fixes, etc) to a grinding halt, as everyone that is able to jumps in to help.

3. When a major issue is found, we usually look to roll the fix into the next release, because the next release is already being worked on. It’s less efficient to stop all work on it just to replicate the current live environment, fix the issue, release a fix, then finally set the next version back up for everyone to work on it again. Depending on the issue and its severity, we may be able to release the next update sooner than planned, but that's not very often.

4. Once the new build has reached gold master candidate, we then have to submit it to various first party groups for approval, either at the end or near the end of the final acceptance tests, to make sure the project is ready. Each of the first party groups' review processes usually take a few days, so we have to submit at least one and a half times the review duration. The reason we do this is to give ourselves room to adjust in case the build fails, and we need to re-submit a new one before our intended release date.
    i. We do our best to cover a variety of devices and saves, but we do end up adding new scenarios to our test-case list quite regularly. (For example, Hulk (Bruce Banner) is now in the test case list!)
5. Once the update day is upon us, we release the build for players to download. We give players a day or two to do this before we have to force the update, so players do not experience any bugs with the latency of being on an older build.
«13

Comments

  • Mr_Sinister
    Mr_Sinister Posts: 356 Mover and Shaker
    I like behind the curtain stuff. Thanks for posting this, I look forward to more in the future.
  • fanghoul
    fanghoul Posts: 311 Mover and Shaker
    D3 Go! Producer, Josh Austin wrote:
    3. When a major issue is found, we usually look to roll the fix into the next release, because the next release is already being worked on. It’s less efficient to stop all work on it just to replicate the current live environment, fix the issue, release a fix, then finally set the next version back up for everyone to work on it again. Depending on the issue and its severity, we may be able to release the next update sooner than planned, but that's not very often.
    I found this the most helpful to understanding some of the most upsetting bugs getting into the live build. As someone that's used to working on code on his own, and iterating one step at a time, it's not something I'd expect. I can see though how when you're working in a team environment, where there's maybe a number of aspects changing at the same time, that it might make less sense to try to immediately fix things. Like all four wheels on your car are important, but maybe you can't actually change a flat tire while the car's still moving.
  • STOPTHIS
    STOPTHIS Posts: 781 Critical Contributor
    Neat read. I'm hoping we could get some insight into other areas as well, like how they decide who gets a buff/rework or how much they test when it come 5* level in PvE and PvP. As always more communication is appreciated.
  • Brigby
    Brigby ADMINISTRATORS Posts: 7,757 Site Admin
    STOPTHIS wrote:
    Neat read. I'm hoping we could get some insight into other areas as well, like how they decide who gets a buff/rework or how much they test when it come 5* level in PvE and PvP. As always more communication is appreciated.
    Glad you liked it! Those are definitely some great topics to talk about, and I'll bring it up in our discussion for the next behind-the-scenes post.
  • morph3us
    morph3us Posts: 859 Critical Contributor
    I'm very much liking this Brigby chap. Can we award him a No Prize?
  • irwando
    irwando Posts: 263 Mover and Shaker
    Where are the details on the QA process? How many devices do you run on? Do you run simulated matches? Do you manually test?

    How do major issues like all powers that modify costs breaking get through when players hit right away?
  • Magere
    Magere Posts: 50 Match Maker
    morph3us wrote:
    I'm very much liking this Brigby chap. Can we award him a No Prize?

    A "no prize?" Now there's a marvel blast from the past....
  • Magere
    Magere Posts: 50 Match Maker
    While this is essentially a great overview of the process, having spent many ( so many..) years of my IT career supporting every part of this process at different times, a word to those who aren't familiar....
    Patience....Above all patience....You have entire groups working together trying to get out a product where time is always a constant pressure....Oftentimes intense pressure to provide a quality product.
    Between each of those steps are the individual group deciding on how to proceed without breaking the final product, how to implement, what's the best idea to use, and then actually doing the work.
    Then you have meetings on meetings, between individuals, groups, managers, and shops, all getting on the same plan and going forward.
    This is NOT easy. Everyone has a reason to do it their way, and it sometimes delays the process.
    So....Having seen this in action, it's easy to understand the delay in everything, but it's not ever apparent unless you live it.
    Were all here because they came out with a game we have all grown to love...And as much as I'm also chomping at the bit to see what's next and have it all go smoothe....Reality is what it is.

    So many thanks to brigby for keeping his word and making things more transparent. Were with you, man.....Even the naysayers (otherwise they wouldn't be there every day for more Deadpool)
  • DaveR4470
    DaveR4470 Posts: 931 Critical Contributor
    For Brigby:

    no-prize.jpg?auto=format&lossless=1&q=90&w=606&h=317&fit=crop
  • AionTeleos
    AionTeleos Posts: 37 Just Dropped In
    Brigby! That was a *phenomenally* interesting post. Thank you so much for lifting the curtain back. Fascinating stuff.
  • turul
    turul Posts: 1,622 Chairperson of the Boards
    Ctrl-F : beta tester
    Ctrl-F : testing

    Just as i thought.... icon_e_smile.gif
  • Alsmir
    Alsmir Posts: 508 Critical Contributor
    Personally, I' more interested how plans for the future of MPQ are discussed.

    Why are we getting new characters every 2 weeks? What's the purpose of that?
    Who decides whether the current ISO gains are fine or need changes?
    Who and how come sup with store prices (ISO especially) and hp costs for different vaults?
    What's the future of PvP & PvE? Or maybe we don't need any changes?

    And most importantly: is it even possible to give players honest answers instead of usuall "we're working on it, it will be great"?
  • Mechahamster
    Mechahamster Posts: 237 Tile Toppler
    I appreciate this Brigby. Greater understanding between players and devs creates a better game.
  • Jonny1Punch
    Jonny1Punch Posts: 440 Mover and Shaker
    As always Brigby you are appreciated and we all enjoy clear,concise communication. Thank you for all that you do sir.
  • smoq84
    smoq84 Posts: 421 Mover and Shaker
    Thank you very much for this post. But still I cannot see the most important information: what about testing the changes?
    What kind of test are you performing? Are there any functional and integration tests? Maybe some automation or regression tests?
    Because each patch brings a lot of new bugs and often new features or characters are also bugged and from the players point of view it looks like there are no tests at all :/
  • DeNappa
    DeNappa Posts: 1,398 Chairperson of the Boards
    Thanks, very interesting.

    I was wondering: how are priorities in fixing issues set? Are issues reported in the bugs section of this forum always checked? How come something like the champion selling price issue is still lingering (link to topic), despite it being reported over 6 months ago? (Asking because I was affected by it recently, obviously icon_mrgreen.gif)
  • mpqr7
    mpqr7 Posts: 2,642 Chairperson of the Boards
    This is a fantastic post. We eat this information up and love it!!! Definitely looking forward to reading more posts like this in the future. Thanks again.
  • tgibbs40
    tgibbs40 Posts: 113 Tile Toppler
    Oh so much I could say about this but I'd rather not get lectured again, or accused of attacking the developers, or whatever the uneducated with their pitchforks would do to me.

    So I'll just say this: Great insight into the process! That's a pretty standard development process. Now you just need to pinpoint the members of your team that can't handle that workload and replace them with ones that can so that we don't get broken releases every two weeks.

    And for those of you that think this would improve with fewer releases I have news for you. That is highly unlikely. You would probably just get bugged releases fewer times per year.
  • TLCstormz
    TLCstormz Posts: 1,668
    King Josh!!!!! :+D
  • drayviper32
    drayviper32 Posts: 123 Tile Toppler
    "We do our best to cover a variety of devices and saves, but we do end up adding new scenarios to our test-case list quite regularly. (For example, Hulk (Bruce Banner) is now in the test case list!)"

    Does this mean that Banner is a future buff??? Would be great if you can get the AP that Hulk matches!!!

Categories