VentureBeat article: MPQ's April Event Analytics

Unknown

I know I've created an older post before about the VentureBeat article series, but I think a lot more people will want to read Part 7. It talks about the numerical performances of some of April's events such as the Dark Avengers and Cap PvP events, as well as the Heroic Venom and the Hunt PvE events.

http://venturebeat.com/2014/04/29/marvel-puzzle-quests-road-to-the-mythical-1-arpdau-part-7-event-analytics/

They should probably add in another metric: off-the-wall scaling and how that affects retention...

Nemek

I think I found Davecazz in the chart (number of Hunt missions played by player):



These are always a good read.

NorthernPolarity

Copy pasted from the article's comment section, I'm really disappointed with this latest installment: This article was pretty frustrating to read. The graphs were not very well made: they could have at least added axis labels and had matching colors for the pairs of graphs. You shouldn't have to think about what the bars represent (currency spent each day I believe, which doesn't even take into account the fact that the event runs for less time on the first/last days). It shouldn't be this hard to try to understand what the graph for the Hunt is trying to represent for instance. Is it supposed to represent the number of missions completed within a single subevent, or does it track all missions completed per day throughout the course of the event? I also don't see how those metrics for the PvE event are surprising at all. The Hunt offers a new set of missions to players every 12 hours and has extra rewards for each set of missions. The Heroic mode offers a single set of missions for 3 days and does not offer any extra rewards to boot: shouldn't it be obvious that the total number of missions completed is going to be lower with the Heroic mode?

Nemek

I'd agree the article seemed a little rushed this time around. There were a handful of really silly typos and I had to really think about some of the graphs actually meant.

I wouldn't go as far as to say that they don't know why the graphs are different, though. The article doesn't present it in a "Look at how these KPIs change based on the event-type, but I have no idea why LOL!"...it's more just trying to show that different event types produce different results in their KPIs. And, theoretically, they would use their current data to better design events that prop up their KPIs.

Linkster79

I sincerely hope they do not mistake player retention for player happiness. I am starting to think that PvE events are just an exercise in how much **** players are willing to put up with. I really do enioy most of this game but some of the latest ideas really are not good.

Unknown

What really bothers me is that these aren't "apples-to-apples" comparison. Both Dark Avengers and Heroic Venom limited your roster to eight characters (again, one of them being Bagman) compared to events that let you use your stronger characters. If you don't have the right eight characters leveled up, doesn't this inherently skew the comparison?

Also, based on its performance, does this mean they never ever ever never ever run another imbalanced Heroic again? Please?

NorthernPolarity

Nemek wrote:

I'd agree the article seemed a little rushed this time around. There were a handful of really silly typos and I had to really think about some of the graphs actually meant.

I wouldn't go as far as to say that they don't know why the graphs are different, though. The article doesn't present it in a "Look at how these KPIs change based on the event-type, but I have no idea why LOL!"...it's more just trying to show that different event types produce different results in their KPIs. And, theoretically, they would use their current data to better design events that prop up their KPIs.

I think it was just this sentence that bothered me:

One interesting angle of analysis is to compare PvE events that are very different in structure to see if we can learn anything about which events appeal to which types of players.

This made it seem like their KPIs were giving them a key insight about creating events that target different types of players, when really all the graphs seem to prove is that people play when there are more rewards to do so, which seems like common sense to me. I was looking for something like "we thought that retention increased if our event structure greatly incentivized the player to play daily, and these metrics clearly show that". The particular example in the article just didn't really make their KPIs seem that valuable.

Nemek

NorthernPolarity wrote:

shouldn't it be obvious that the total number of missions completed is going to be lower with the Heroic mode?

But that's not what the graphs are trying to show or the point that Al is trying to make. The data is showing that for the Heroic, a small amount of players are playing a lot and the majority of players are not playing much in comparison to those other players, while for The Hunt, you see a very steady distribution of missions being played. The question being posed isn't what would make players play more on average, but why is there an obscene drop-off after the first group of players.

So, they have to think about what that really means...

• Is there a subsection of players that will play a lot no matter the event?
• Do you see some players 'give-up' if they feel they can't get into a particular reward zone (and thus stop playing altogether)? How does the spacing of the rewards affect that drop-off?
• Do the players in the high-amount-of-play part of the curve just so happen to have high levels of the boosted characters (or, have them at all, because of Heroic)?
• Does scaling make the curve more like the Heroic or more like The Hunt (I'd bet for sure, pre-scaling, it was obscenely more like the Heroic)

The article obviously doesn't go into answers, but it's not really trying to.

Nemek

NorthernPolarity wrote:

I think it was just this sentence that bothered me:

One interesting angle of analysis is to compare PvE events that are very different in structure to see if we can learn anything about which events appeal to which types of players.

This made it seem like their KPIs were giving them a key insight about creating events that target different types of players, when really all the graphs seem to prove is that people play when there are more rewards to do so, which seems like common sense to me. I was looking for something like "we thought that retention increased if our event structure greatly incentivized the player to play daily, and these metrics clearly show that". The particular example in the article just didn't really make their KPIs seem that valuable.

Yeah, they really never touched back on the different 'type' of players and there was certainly a lot more they could have touched on.

NorthernPolarity

Nemek wrote:

NorthernPolarity wrote:

shouldn't it be obvious that the total number of missions completed is going to be lower with the Heroic mode?

But that's not what the graphs are trying to show or the point that Al is trying to make. The data is showing that for the Heroic, a small amount of players are playing a lot and the majority of players are not playing much in comparison to those other players, while for The Hunt, you see a very steady distribution of missions being played. The question being posed isn't what would make players play more on average, but why is there an obscene drop-off after the first group of players.

So, they have to think about what that really means...

• Is there a subsection of players that will play a lot no matter the event?
• Do you see some players 'give-up' if they feel they can't get into a particular reward zone (and thus stop playing altogether)? How does the spacing of the rewards affect that drop-off?
• Do the players in the high-amount-of-play part of the curve just so happen to have high levels of the boosted characters (or, have them at all, because of Heroic)?
• Does scaling make the curve more like the Heroic or more like The Hunt (I'd bet for sure, pre-scaling, it was obscenely more like the Heroic)

The article obviously doesn't go into answers, but it's not really trying to.

Ah, I was confused about the graph itself then: I thought it was trying to show missions played as a function of time, but it's showing missions played per player. In that case, the analysis makes a lot more sense now.

LordWill

I just want to say I have suggested they use in game polls to gather data as well since their data collecting misses the point.

It's like the are looking at the metrics as if we had any choice in the matter.....(It's really a lesser of two evils)

Would you rather eat: A. Rotten eggs or B. Slime covered worms.

So they see in their magical metrics that more people selected A. than B., so they must love A. and we should give them more A.

So really its not a very fair way to gather REAL player experiences and if we are happy or not unless they directly ask us.

Here's a novel idea, if they want us to spend more money on the game, how about asking us what we would spend money on? How about conducting some in game polls of the 100,000 users and I dunno, just ask them if they like community scaling while you're at it.

They could be making a lot more money if they just asked, instead of shooting in the dark and making all these crazy changes to see how that affects their bottom line....

Pretending to know what makes your player base happiness going off of the metrics is a sure way to remain clueless.

Unknown

Yea the article made it seem like the hunt was a much more successful format than heroic.

uhh.. but the hunt had falcon as a prize while heroic was giving out 4 sets of steve rogers covers (after steve rogers was already given out a ton).

so, there was more motivation to play the hunt in order to get those covers.


let's play heroic for the first modaken covers and see how well it does.

Unknown

LordWill wrote:

Pretending to know what makes your player base happiness going off of the metrics is a sure way to remain clueless.

They don't care if you're happy. They only care if you're playing (or paying).

There's a reason why companies love metrics - it's because customer feedback is wildly unreliable. At the end of the day, you don't care what your customers think, or how they feel, you only care about what they do.

LordWill

ZenBrillig wrote:

LordWill wrote:

Pretending to know what makes your player base happiness going off of the metrics is a sure way to remain clueless.

They don't care if you're happy. They only care if you're playing (or paying).

There's a reason why companies love metrics - it's because customer feedback is wildly unreliable. At the end of the day, you don't care what your customers think, or how they feel, you only care about what they do.

Well I think that's a bad way to run a business.

Zifna

ZenBrillig wrote:

They don't care if you're happy. They only care if you're playing (or paying).

There's a reason why companies love metrics - it's because customer feedback is wildly unreliable. At the end of the day, you don't care what your customers think, or how they feel, you only care about what they do.

Don't be ridiculous. Only very terrible companies with no long-term plans don't care if their customers are happy.

When I'm having fun with this game, people around me get interested and I explain it to them, and they may give it a shot. When I'm not happy, I'm certainly not recommending the game to anyone or telling anyone what fun I'm having - quite the opposite.

What's more, it is true that you can to a certain extent "shackle" hardcore players into doing miserable content, but everyone has their "misery limit" where they say "The ratio of misery to fun in this game is too high. I'm out!"

Look at World of Warcraft. As time went by, raid instances got less and less punishing. Why? Because of the high rates of attrition among raiding players. After a while, there was even raid content you could snag a random group of players to rush at and have a reasonable hope of success. The fights are therefore much less epic than, say, C'Thun, but they still make most players "feel" challenged and then they reward them. It keeps people happy and playing.

Unknown

I'm going to leave this link for an article, as this was one of my major case studies I had to focus upon in B-school: Stop Trying to Delight Your Customers - HBR (Surprised I could find a scan of the article on the Internet)

The short answer here is that considerable business research has shown that companies should only achieve moderate levels of happiness among a community to maximize value creation. The article covers this concept well enough, from the perspective of a Customer Service role. A company just needs to have a "good enough" customer service rating in the long run, according to their metrics.

Bottom line, if the benefits a company expects to obtain from you for making you happy is less than the cost required to make you happier, then the company will not pursue an initiative that will cost them money.

reckless442

davecazz wrote:

let's play heroic for the first modaken covers and see how well it does.

Don't even suggest that in jest.

Unknown

Seriously? These guys look even worse than I estimated from the game design. The excel-sheet guys took over completely and no one there plays the f-ing game or played it even in the past.

Results example: Comparing two Versus events

Let’s look at what the numbers tell us about two PvP events that ran back-to-back:

PvP event featuring the Dark Avengers starting Friday, April 11 to Sunday, April 13.
PvP event featuring Captain America from Monday, April 14 to Wednesday April 16.

Neither of these events used brand-new characters. In my opinion, the Dark Avengers tournament should have been way more engaging — players need to compose a team of some of the less commonly used characters such as Magneto, Daken, and Venom, and that encourages new and interesting strategies. It also took place on a weekend when our engagement is higher in absolute terms. Totally wrong:

Number of Users Engaged: The Captain America event had 50 percent more players
Retention: Measuring “D1” retention for a three-day event can skew oddly. Users who join on the third day are not able to play on the final day but they are factored into our calculations. The Dark Avengers event had 41 percent D1 retention and Captain America and 45 percent D1 retention.
Monetization: Players spent 50 percent more hero points in the Captain America event. Most of that was in comic backs.

And I'm sure it has absolutely NOTHING to do with Hulk covers offered from the Cap tournament and Black sitting in the alliance... I thought anyone who plays the game actually could have made the proper prediction before the morning coffee.

More interesting strategies? DA means limited roster you either not have to start with or limits you to a few not really cool characters without synergy and real team stuff. Excludes healers. And makes you face the "favorite" enemy, Daken most of the games. So overall predictably poor experience for okay prizes -- compared to the usual routine (or better if your Cap is any evolved) for one of the most wanted cover in the game (at the tournament time, by now it's probably getting saturated).

These same guys design the PVE events, no wonder almost everyone suffers.


They probably never heard about DELAYED effects either: yes, we played the Hunt to the end cursing and crying to not lose the rewards for the pain already invested in first half -- and get that f-ing falcon. Retention -- or lack of it will show in the upcoming events as back payment. For that very experience.
Also comparing 2 sucker events is less than productive.

Unknown

Lyrian wrote:

I'm going to leave this link for an article, as this was one of my major case studies I had to focus upon in B-school: Stop Trying to Delight Your Customers - HBR (Surprised I could find a scan of the article on the Internet)

The short answer here is that considerable business research has shown that companies should only achieve moderate levels of happiness among a community to maximize value creation. The article covers this concept well enough, from the perspective of a Customer Service role. A company just needs to have a "good enough" customer service rating in the long run, according to their metrics.

Bottom line, if the benefits a company expects to obtain from you for making you happy is less than the cost required to make you happier, then the company will not pursue an initiative that will cost them money.

I think you slightly misrepresent the article, but let's not go there. What interests me how it maps to our context. Yes, we can accept it that going over "good enough" imposes diminishing returns and no gain comparable to going from "mediocre" to the "good enough" point. The article looks pretty clear that being blow the "good enough" mark is not exactly a benefit.

So where do you put MPQ on the scale? Does your dragging the article here implies they are in the green area?

IMO they are pretty low and keep going in the wrong direction more than the small steps taken for actual improvement. When you read statements like "playing Hunt is less gun than my day work" it sound like good opportunity to run the "stop and fix" workshop. Or at least pay serious attention.

Or am I lost in the past thinking the main value a game provides is FUN?

over_clocked

Surprised no one mentioned that Magneto was named as one of the DA. Do they even play their own game?

DA tourney was a flop because these characters are mostly weak, and have bad synergy, as has been noted. You just ran Daken + maxed Ares, and Rags for a pile of HP. No strategies were needed in the first place, since Moonstone and Bullseye both plain lose to Daken.

Demiurge_Will

There's no either-or between playing the game and using metrics like this. It's impossible to turn these graphs into design changes without combining them with what we're hearing from players and our own experiences playing. Responding to anecdotes alone will lead to changes that make the game worse for the silent majority of players unless you also collect and look at the data.

Re: that HBR article: one of the things that, as a designer, I love about making a free-to-play game is that the more fun your players are having, the more they value your game, and the more money you make. That's in contrast to games that you sell in a box for a flat price, where you typically get to do much less customer support or ongoing development. Developers' interests and players' are better aligned than they are when a game has a single price tag.