Developers response

entrailbucket

A 90% winrate on offense is absolutely horrifically bad in this game. Just FYI, pre-nerf Gambit had a defensive winrate less than 10% and was the most powerful defensive character in the history of the game. A decent character should be winning on offense probably 98-99% of the time -- that's how big the player's advantage over the AI is.

entrailbucket

@justsing said:
This all just looks like history repeating itself where Chasm is Gambit and Riri, Hit Monkey, etc. are the "counters" (AA, Beardcap) that followed...

Yep, also the SW/Colossus damage prevention counters that saw basically zero use, the millions of 4* Polaris counters that did nothing to reduce her usage, the Okoye counters that removed TU tiles from the board...it's the same story over and over and over again. They give us these soft, narrow, active counters to incredibly powerful passives, and they never go anywhere.

DAZ0273

@DevpoolMPQ_BCS said:
@fight4thedream thanks for the ping!

I'll do my best to respond.

As for the character testing process, there are several types of testing that we run with different focuses. We have a small internal team of QA that performs high-level testing. They work pretty closely with the engineering team, identifying crashes and warnings, and looking to find odd interactions that potentially operate outside of the bounds of intention. We have a much larger external team of QA that perform a more comprehensive review that encompasses everything mentioned earlier, and more. With their broader resources, they look for optimal methods of play, strong team compositions, good and bad matchups, and a win frequency, and more.

I mentioned this elsewhere (I don't recall precisely where), but the sneak peek stats that we reveal to players ahead of time must be generated in advance, as communications like that require sign-off from numerous parties and at a certain point are handled by a completely different group.

All of this works together to make changes like the ones applied to Magik appear much more impulsive than they actually were. I realize that many players believe that Magik's sneak peek stats were strong but not overly so in theory. In truth, her win rate was north of 90%. I know players see the changes applied as a nerf, but Magik never shipped with these stat values and these might essentially be thought of as bad numbers brought on by a math error in calculating her power budget that was corrected in tandem with marketing efforts in motion.

All that said, we also look at the matter of game balance and character power as fluid. When we don't like the numbers we're seeing, we work on them. While Magik's current numbers are much more in line with the norm for 5* characters, if she needs a bump she'll get it.

Thank you for this post. I am sure the mighty @fight4thedream will respond but I wanted to just do so also.

I think the weakness in your analysis falls into the player base psyche. It might well be that Magik hit all these numerical/statistical no-no's and from that point was an issue but I think from a player perspective/psyche high damage characters are appealing for many reasons. Opposite to Chasm who hinders a player as much as he hinders an opponent, Magik just sounded...fun? Chasm is not fun. Magik sounded fun. I can't speak for them all but players seem to enjoy characters who when used help us smash opponents. It gives a bit of an adrenaline rush I guess? Like when your team wins 4 nil rather than a grinding 1 nil. It is still a win but...well anyway thank you for continuing to try and bring all these great characters to MPQ, it has certainly been a ride.🙂

KGB

@entrailbucket said:
A 90% winrate on offense is absolutely horrifically bad in this game. Just FYI, pre-nerf Gambit had a defensive winrate less than 10% and was the most powerful defensive character in the history of the game. A decent character should be winning on offense probably 98-99% of the time -- that's how big the player's advantage over the AI is.

We don't know the context of this sentence. He's not claiming it's test players vs AI win rate. I strongly suspect it's AI team vs AI team in automated testing. It's the only way they can run tens of thousands of tests with hundreds of team combos in a reasonable time frame while attempting to determine whether a power set is considered 'reasonable'.

If what I wrote is true (AI vs AI) then that is a very impressive win percentage.

KGB

Jimsta_rooney

@DevpoolMPQ_BCS said:
Hi there! Dev here!

The subject of the post is broad, but I'll do my best to span it.

EotS: this was the debut of a retired event that previously did not have the SCL tiers that are currently present. We implemented them, but players have pointed out that the event's rewards need tuning. Players also discovered other bugs, most notably an issue with Kang's interaction with wave nodes within (and elsewhere). Notes on these issues have been made, some have already been corrected for future runs, and compensation is being calculated with a plan to roll out very soon.

Kang: similarly to Chasm, Kang features interactions with the game that were laden with issues, requiring changes to the underlying engine to support. Wave nodes are a common occurrence in play and seem like obvious places to test thoroughly, but in the grand scheme of things are only a facet of all of the possible types of interactions in the game. Kang's issues have been sorted out and we're targeting a patch (likely later tonight as of this writing) to correct these issues. Compensation for these issues is being factored in with the EotS compensation.

Magik: while Magik's stats update is a big jump, her initial stats fell far outside of the power budget of a normal character. As for Magik's position as an outright Chasm counter, this isn't entirely her intention. Since the release of Chasm, numerous tools have been implemented to help players overcome challenges with revive centric teams that Chasm makes all the more powerful. Hit Monkey, Wong, Kang, and now Magik all are part of that toolkit, allowing players to strategically deal with the Chasm threat. Over time we've seen players put these tools to good use and we're seeing Chasm as less and less of a dominant player in the PVP scene. We view the game fluidly and if Magik needs a numerical bump to better serve in her role of dishing out permanent damage, we'll definitely do that.

Thanks for the post!

Tldr
Magik was a direct counter to chasm/ihulk so we have rectified that and now shes only gonna be useful boosted?

Bad

So I played my new brand champed kang on SIM.
3 battles vs inmortal bros.
Not gonna lie, without someone stealing AP it's a comfortable win, kang HM SW.
However, of course there was a tax, and I wonder why that team must be so tough.
The first stunned turn, chasm or Ihulk passive will make a dent for sure while you are on the busy process of collecting AP and then wait for the right time to fire it when his health is at reach before he revives and fully heals himself.
And that on pick 3, with SW protection and AP changing!
So on pick 2, or even pick 3, it won't change anything.
Simply they are the best defensive team and no character countering them will change that, if chasm remains unchanged.
By the way, sent away is still bugged as Ihulk revived 2 times with Shulk away, and then appeared the banner victory as if was a bugged EotS wave node.

ThaRoadWarrior

if iHulk gets a revive cap he will need a full rebalance. his self-harm is tuned for infinite revive to a degree that from full health, he can only make 15 matches assuming no incoming damage.

entrailbucket

@KGB said:

@entrailbucket said:
A 90% winrate on offense is absolutely horrifically bad in this game. Just FYI, pre-nerf Gambit had a defensive winrate less than 10% and was the most powerful defensive character in the history of the game. A decent character should be winning on offense probably 98-99% of the time -- that's how big the player's advantage over the AI is.

We don't know the context of this sentence. He's not claiming it's test players vs AI win rate. I strongly suspect it's AI team vs AI team in automated testing. It's the only way they can run tens of thousands of tests with hundreds of team combos in a reasonable time frame while attempting to determine whether a power set is considered 'reasonable'.

If what I wrote is true (AI vs AI) then that is a very impressive win percentage.

KGB

I mean, here's what he said:

"We have a much larger external team of QA that perform a more comprehensive review that encompasses everything mentioned earlier, and more. With their broader resources, they look for optimal methods of play, strong team compositions, good and bad matchups, and a win frequency, and more."

This sounds to me like they're human players playing the game and not AI vs AI. He didn't mention anything about tens of thousands of tests with hundreds of team combos.

Bowgentle

Which, assuming that the QA people are not 5* players in CL 10, PVE, sadly explains a lot.
Also makes me wonder how those MMR changes will look for us, the top 1%.

entrailbucket

What's really crazy is that Chasm, Thor, and Kang presumably went through this same process, and presumably those characters had winrates less than 90%, meaning they didn't require any fixing.

That means these testers were wiping more than one out of every ten fights using Chasm, or Thor, or Kang. How is that even possible? Can we get these players in our PvP queues?

fight4thedream

@DevpoolMPQ_BCS: I appreciate you taking the time to respond and give an insight into the testing process. Magik is primarily an offensive heavy character with a self-boost ability, match damage increase and passive damage so I don't think it's very surprising she was winning more than usual. If these are AI run tests, was the opposing team programmed to try to avoid matching her demon repeater tiles? If it's the same AI the player faces in PvP, I could see how that may factor in to her high win percentage.

My primary concern, however, is whether the testing factors in actual character usage and popular team combos for PvP. What I mean is a significant portion of the player base is encountering certain teams more frequently, most notably Chasm/iHulk teams, as opposed to a team consisting of Yelena/Heimdall. If both teams are given equal value during testing, it doesn't really reflect the general player experience of PvP and produces skewed results.

A really good example would be Chasm. When not paired with iHulk, his win percentage would presumably fall significantly since he is unable to achieve a revive loop. But a significant portion of the player base runs him almost exclusively with iHulk so it would be inaccurate for testing to evaluate all possible team compositions with him equally. More weight should be given to team compositions that have good synergy and then potential counter teams should also be given greater valuation.

Obviously this is all just conjecture on my part as I am not privy to the details of how testing is done nor do I expect you to spill the beans on the secret sauce of QA testing, but as I have pointed out she is countered pretty hard by the existing PvP meta so I am curious if she would be able to keep such a high win rate if the majority of the matches done in testing were Chasm/iHulk, Mighty Thor/Riri or Colossus/Wanda teams.

Admittedly, not all players are running meta teams and I could definitely see how she would be problematic for players who don't have access to top tier characters since she definitely had more fire power than the majority of characters in the 5* tier. But I am sure such players are already having difficulty with the current meta.

With all that said, I suppose the crux of the issue is: how do these changes help push the meta forward? I think for most of us, it's difficult to see how this power adjustment changes the status quo in any meaningful way.

As you can see, many are very eager to see a meta shift sometime soon. It doesn't look like Magik will be it, but it is my hope that she will have some part to play in the next meta.

KGB

@entrailbucket said:

@KGB said:

@entrailbucket said:
A 90% winrate on offense is absolutely horrifically bad in this game. Just FYI, pre-nerf Gambit had a defensive winrate less than 10% and was the most powerful defensive character in the history of the game. A decent character should be winning on offense probably 98-99% of the time -- that's how big the player's advantage over the AI is.

We don't know the context of this sentence. He's not claiming it's test players vs AI win rate. I strongly suspect it's AI team vs AI team in automated testing. It's the only way they can run tens of thousands of tests with hundreds of team combos in a reasonable time frame while attempting to determine whether a power set is considered 'reasonable'.

If what I wrote is true (AI vs AI) then that is a very impressive win percentage.

KGB

I mean, here's what he said:

"We have a much larger external team of QA that perform a more comprehensive review that encompasses everything mentioned earlier, and more. With their broader resources, they look for optimal methods of play, strong team compositions, good and bad matchups, and a win frequency, and more."

This sounds to me like they're human players playing the game and not AI vs AI. He didn't mention anything about tens of thousands of tests with hundreds of team combos.

I've done QA before. It involved some testing manually but most of it was scripting automated testing.

Even if their larger team is 20 or even 50 people there is no way they can do more than a few hundred manual battles given all the other testing that has to be done on bug fixes etc. You must do automated testing of some kind to get meaningful results.

KGB

Codex

Don't forget that when they tested chasm there was no Kang, hit monkey or silver surfer update and I think mThor aswell and somehow it won less than 90% of the time. Like fightthedream mentioned chasm ihulk probably won 99% of the time and chasm and x won significantly less thus skewing the overall results.

entrailbucket

@KGB said:

@entrailbucket said:

@KGB said:

@entrailbucket said:
A 90% winrate on offense is absolutely horrifically bad in this game. Just FYI, pre-nerf Gambit had a defensive winrate less than 10% and was the most powerful defensive character in the history of the game. A decent character should be winning on offense probably 98-99% of the time -- that's how big the player's advantage over the AI is.

We don't know the context of this sentence. He's not claiming it's test players vs AI win rate. I strongly suspect it's AI team vs AI team in automated testing. It's the only way they can run tens of thousands of tests with hundreds of team combos in a reasonable time frame while attempting to determine whether a power set is considered 'reasonable'.

If what I wrote is true (AI vs AI) then that is a very impressive win percentage.

KGB

I mean, here's what he said:

"We have a much larger external team of QA that perform a more comprehensive review that encompasses everything mentioned earlier, and more. With their broader resources, they look for optimal methods of play, strong team compositions, good and bad matchups, and a win frequency, and more."

This sounds to me like they're human players playing the game and not AI vs AI. He didn't mention anything about tens of thousands of tests with hundreds of team combos.

I've done QA before. It involved some testing manually but most of it was scripting automated testing.

Even if their larger team is 20 or even 50 people there is no way they can do more than a few hundred manual battles given all the other testing that has to be done on bug fixes etc. You must do automated testing of some kind to get meaningful results.

KGB

Where does it say anything, anywhere, about them doing automated testing? Yes, it'd be impossible for humans to adequately test this stuff. I'm pretty sure that's exactly why it's so uneven.

I don't know what BCS does -- I know (from talking to them, extensively) that Demiurge only did manual testing for character power level. They didn't have anything automated.

LavaManLee

I think there is a big difference between someone being paid to QA and those of us who play this game a lot. Our brain is trained to look for those imbalances, kind of like a gambler does. We quickly can determine what works and what doesn't and take advantage of every little aspect. For those who remember the 3* Gambit and making sure your build had no black covers. Or Gargs and 5Hawk are great together and pretty sad with almost everyone else.

It's great there are people being paid to QA but they don't think the same way we do. They are looking for obvious imbalances and other things that stick out but they aren't thinking about trying to win as fast as possible and look for the advantage.

entrailbucket

@revskip said:
@entrailbucket said:

What's really crazy is that Chasm, Thor, and Kang presumably went through this same process, and presumably those characters had winrates less than 90%, meaning they didn't require any fixing.

That means these testers were wiping more than one out of every ten fights using Chasm, or Thor, or Kang. How is that even possible? Can we get these players in our PvP queues?

Chasm/iHulk is very beatable it just takes a longer time to do so. I'd imagine that in AI vs AI battles that dynamic doesn't factor in. The reason that combo has a much higher float point is that it takes time to whittle down and because it eats healthpacks not because it can't be beaten. mThor and Kang are much quicker to beat but unlike the revivers a single big cascade can down them no problem. I also doubt all of their tests are using 550s since there is an absurdly small percentage of the player base running those.

Now I'm picturing an AI vs AI Chasm/Hulk mirror match -- how long do you think that'd take to resolve? 4 hours? Longer? I simply cannot believe that ever happened.

ThaRoadWarrior

I believe an AI vs AI mirror match would settle into infinite stasis a large percentage of the time. At least half would be my guess.

Bad

Actually I don't think QA AI vs AI is a wrong approach. The data you get on thousands of battles on this way is how dumb AI will do in defense.
If it wins a 90% of battles that means without AI doing smart moves it's really hard to win.
I can imagine which teams were the 10% she lost: chahulk and Mthor.
So the character can easily win without planning any tactic.
Still that doesn't mean we are wrong either: that 10% where she lost actually will be the 80% of what she will fight in pvp.

Scofie

I think the win % was over 90% because Magik was the essential character and therefore features on both teams... 🤭

revskip

@entrailbucket said:

@revskip said:
@entrailbucket said:

What's really crazy is that Chasm, Thor, and Kang presumably went through this same process, and presumably those characters had winrates less than 90%, meaning they didn't require any fixing.

That means these testers were wiping more than one out of every ten fights using Chasm, or Thor, or Kang. How is that even possible? Can we get these players in our PvP queues?

Chasm/iHulk is very beatable it just takes a longer time to do so. I'd imagine that in AI vs AI battles that dynamic doesn't factor in. The reason that combo has a much higher float point is that it takes time to whittle down and because it eats healthpacks not because it can't be beaten. mThor and Kang are much quicker to beat but unlike the revivers a single big cascade can down them no problem. I also doubt all of their tests are using 550s since there is an absurdly small percentage of the player base running those.

Now I'm picturing an AI vs AI Chasm/Hulk mirror match -- how long do you think that'd take to resolve? 4 hours? Longer? I simply cannot believe that ever happened.

Not at all sure if they are even doing AI vs AI since that was just something that was floating around earlier but if they are I'd imagine they have the ability to turn the match speed up to test in an all AI match. I know even really old chess computers can spit out hundreds of matches in under a minute. And I doubt they would be looking too hard at the length of those matches in a testing environment instead just correlating the w/l data. All hypothetical on my part since we don't know if that is even how they are testing.