Causation, correlation, and World of Warcraft, part 3 of 3

My previous post describes a double-experience mechanism in World of Warcraft and the fact that it appears to be a deliberate product decision by Blizzard Entertainment to increase the average number of characters per active subscriber.  This decisions feels like the kind of thing a product manager or strategic manager would do based on a statistically relevant correlation between number of active characters and other preferred behaviors, such as propensity to continue subscribing to the service.  See my previous post for caveats around these guesses, but for our purposes today let's assume that all of the above is correct and indeed that the preferred behavior is increased likelihood of continued subscription.

That would make perfect sense.  A big business like this one, which appears to be in excess of one billion dollars per year, is exactly the kind of thing in which increased retention is worth a whole lot of money to the company and for which product initiatives would definitely be on order.  After all, each tenth of a percent of increased retention means more than a million dollars a year to the company.  That's worth spending a little time on analysis.

But here's where we run into the problem with causation and correlation.  Just because there exists a correlation between number of active characters and retention, that doesn't mean that gaining characters is the direct cause of increased retention.  It may be that as time progresses players tend to pick up characters and also tend to increase their overall retention rate.  It's a reasonable scenario that a veteran WOW player might tend to have more characters going and also have a much higher likelihood of continuing the service than a newby would.  But in that case both of the observed metrics owe themselves to a different cause:  Number of years as a player of the game.  In this case, there is a correlation but no causation between the two metrics.  Convincing a newby to have more characters does not necessarily increase retention.

Or maybe the causal connection goes the other way.  Perhaps the super-satisfied WOW player can be identified in part by having a large number of active characters.  Obviously these players would have a very high retention rate.  But again, artificially convincing non-super-satisfied players to add more characters doesn't necessarily address this need.  In either of these scenarios, spending product roadmap time to build out these features and changing the game environment away from a more equitable, less arbitrary state would be wasted, as it wouldn't really address the need the designers have for the game.

It's interesting to notice when these errors occur that we always attribute the characteristic we don't care about (number of characters) as a cause for the characteristic we care very deeply about (retention).  You would never look at these two factors and conclude that driving up renewals is a good method for increasing the average number of characters per player.  Because after all, shareholders don't care about number of characters per player.

Now, all we have to do is look at the stats around World of Warcraft to see that the Blizzard folks are doing a lot of things right, so even if the above speculation is 100% true I wouldn't weep too much for them.  But this basic lesson is applicable to any business with a large installed base or a large run rate or lots of traffic or any other situation that will lead to the temptation to engage in data mining.  And most businesses aren't sitting on the money-printing machine that is WOW.  So in general we would be well advised to try to make these decisions as effectively as we can.

In conclusion, am I saying not to engage in data mining?  Absolutely not.  I do it all of the time.  But I am saying to be careful about drawing conclusions that aren't supported by the data.  In particular I frequently watch people mistake correlation for causation, and that can send you down a path where you're spending your resources and giving your customers a convoluted experience without gaining the benefits you think you are.