Results of 5*/LT Survey

Unknown

I've looked through the first set of reports from my 5*/LT survey thread (viewtopic.php?f=7&t=38063). I wanted to start a new discussion to cover the analysis, apart from future data collection on the other thread.

I have rejected the reports that were uncertain. I've also rejected the "last x number of tokens" reports, because I feel that they are too likely to be swayed by specific runs of good or bad luck. For those reports with a small range of uncertainty in total tokens drawn, I've used the average.

This leaves 50 reports, totaling 3454 LTs yielding 282 5*s, for an average hit rate of 8.1%. 24 are below the listed 10% rate and 26 are above, which signals to me that we've got a pretty decent sampling of people who are lucky and unlucky.

The luckiest person is Arondite, who is in the top 99.5 percentile of draw permutations with 6 out of 19.

The unluckiest person is Melevorn, who is in the bottom 0.23 percentile with 8 out of 220.

(Surprisingly, I am only the 13th least lucky person at 3.1 percentile with 10 out of 170).

Plugging all 3454 tokens and 282 successes into the calculator produces a p-value of around 0.00012. In fact, assuming that people have a pretty good handle on how many 5*s they've pulled, they have to have overestimated the total number of tokens pulled by more than 12% in order to exceed the common .05 significance level threshold.

Which means that, if we accept that people's estimates of their total tokens pulled are at most 12% too high, we should reject the null hypothesis that 5*s are randomly distributed at a 10% rate. The sample size is large enough, because we clearly have more than enough pulls to exceed the significance level. Sample bias is worth considering, but the entire gamut of probability is fairly well represented by the data (we have plenty of lucky, average, and unlucky reports).

The overestimation is key. 10-12%, we can be sure enough to question the system. Once overestimation of totals exceeds
15%, our p-value rises to .25. More data will allow a greater degree of leeway in terms of an estimation of total pulls.

Alternatively, we could look at just the people who have meticulously documented their pulls. That will be a smaller data set, but we can be more sure about the numbers.

Unknown

Coincidentally, the p-value for these 50 reports is the same as flipping a fair coin 13 times and getting the same result every time.

wirius

Sorry, your sample is inaccurate because you're throwing out numbers that "should" work based on the "likelihood" of inaccuracy, which you cannot verify. People lie unintentionally and intentionally, and you can't measure that. Without the hard numbers D3 has, any informal results are circumspect.

The only thing shown here is people seem obsessed with 5*'s. They're an end game legacy item, based purely on time invested, and tons of time at that. Some will get luckier, some will get unluckier. Its going to take on average 400-500 LT's to get a maxed 5*, just play the 4* game and be happily surprised when a 5* shows up.

Bowgentle

wirius wrote:

Sorry, your sample is inaccurate because you're throwing out numbers that "should" work based on the "likelihood" of inaccuracy, which you cannot verify. People lie unintentionally and intentionally, and you can't measure that. Without the hard numbers D3 has, any informal results are circumspect.

The only thing shown here is people seem obsessed with 5*'s. They're an end game legacy item, based purely on time invested, and tons of time at that. Some will get luckier, some will get unluckier. Its going to take on average 400-500 LT's to get a maxed 5*, just play the 4* game and be happily surprised when a 5* shows up.

You keep saying that, but they're not.
There are whole alliances running around with fully covered and close to maxed 5*s.
Yes, a lot of them spent a whole lot of money, but there are also people who got insanely lucky with pulls.

Like it or not, the 5* meta is here already.
And time invested means nothing - there are a lot of people who have been playing for two years+, who have most of their 4*s maxed.
Those rosters got completely invalidated by 3 months of bad token luck.
On the other side, there are people running around with 7+ covers on their 5*s who have been playing for 200 days.

Polares

Bowgentle wrote:

wirius wrote:

Sorry, your sample is inaccurate because you're throwing out numbers that "should" work based on the "likelihood" of inaccuracy, which you cannot verify. People lie unintentionally and intentionally, and you can't measure that. Without the hard numbers D3 has, any informal results are circumspect.

The only thing shown here is people seem obsessed with 5*'s. They're an end game legacy item, based purely on time invested, and tons of time at that. Some will get luckier, some will get unluckier. Its going to take on average 400-500 LT's to get a maxed 5*, just play the 4* game and be happily surprised when a 5* shows up.

You keep saying that, but they're not.
There are whole alliances running around with fully covered and close to maxed 5*s.
Yes, a lot of them spent a whole lot of money, but there are also people who got insanely lucky with pulls.

Like it or not, the 5* meta is here already.
And time invested means nothing - there are a lot of people who have been playing for two years+, who have most of their 4*s maxed.
Those rosters got completely invalidated by 3 months of bad token luck.
On the other side, there are people running around with 7+ covers on their 5*s who have been playing for 200 days.

I have to say that the sample is too small 2300 LTs is really too small to know if there is something happening, we need more than 10k, probably 100k.

But what it is obvious and nobody can say it is not true is what Bowgentle said in the end. The endgame is now just luck based, people with much less time and money invested can have more covered 5s, just beacuse they were lucky. And this is a problem, it should not be like that, I don't think it is fair for anybody .

Then another big problem are Money Alliances, some users get together, spend some big money and get 20+ 5 covers in a row. Nobody will be able to play against those users now that 5s can be championed (really big mistake allowing 5s to be championed).

Unknown

Polares wrote:

I have to say that the sample is too small 2300 LTs is really too small to know if there is something happening, we need more than 10k, probably 100k.

Well, kind of. The p-value tells us when we have a sufficient sample size. If we didn't, then we would be unable to reach a sufficient significance level to invalidate the null hypothesis. Smaller, individual sample sizes probably require a higher significance level, but the aggregate 50 report sample actually exceeds a fairly high bar (as I've suggested, 13 coin flips high).

https://en.m.wikipedia.org/wiki/P-value

My point before was that uncertainty of the total LTs drawn is potentially too high, since a modest divergence from that value sends our p-value through the roof. A larger dataset or else a smaller, but more certain dataset will help harden the numbers.

I should have been more clear. My conclusion that we should reject the null hypothesis is contingent on the reports being of high accuracy. And I know that they aren't. Still, it's an interesting enough data set to go forward with, in one of the two directions that I've suggested.

wirius

Bowgentle wrote:

wirius wrote:

Sorry, your sample is inaccurate because you're throwing out numbers that "should" work based on the "likelihood" of inaccuracy, which you cannot verify. People lie unintentionally and intentionally, and you can't measure that. Without the hard numbers D3 has, any informal results are circumspect.

The only thing shown here is people seem obsessed with 5*'s. They're an end game legacy item, based purely on time invested, and tons of time at that. Some will get luckier, some will get unluckier. Its going to take on average 400-500 LT's to get a maxed 5*, just play the 4* game and be happily surprised when a 5* shows up.

You keep saying that, but they're not.
There are whole alliances running around with fully covered and close to maxed 5*s.
Yes, a lot of them spent a whole lot of money, but there are also people who got insanely lucky with pulls.

Like it or not, the 5* meta is here already.
And time invested means nothing - there are a lot of people who have been playing for two years+, who have most of their 4*s maxed.
Those rosters got completely invalidated by 3 months of bad token luck.
On the other side, there are people running around with 7+ covers on their 5*s who have been playing for 200 days.

I keep saying it because its true. How many LT's did they open? 300-400? Then its not surprizing. Will there be a few outliers who do it in maybe 200? Certainly. Its a happy accident. You have no control. Your only effort is the time invested to gain a number of LT's, nothing else. Obsessing over this is futile. It is a carrot on the stick meant for a long term investment. A few will luck out, and will be very happy for it. A few won't luck out, and be very unhappy about it. If you think this is unfair, that's fine. But looking at the odds won't change this, the guys who make the game already know the odds, as do you. As long as you stick around in the game pursuing those intentionally long term investment 5*'s, they did their job and made a good mechanic to keep you in the game.

simonsez

Polares wrote:

I have to say that the sample is too small 2300 LTs is really too small to know if there is something happening, we need more than 10k, probably 100k.

That is complete and utter nonsense. Stop harping on your incorrect ideas about sample size, and learn about confidence intervals.