I designed an internal system that optimises for long term outcomes. We do nothing based on whether you click “upgrade”. We look at the net change over time, including impact to engagement and calls to support months later and whether you leave 6 months after upgrading. Most of the nudges are purely for the customer’s benefit because it’ll improve lifetime value.
That's the only thing I was thinking with their A/B test. The calculator might immunize against unhappy customers later on. I think they could've looked at something like the percentage of customers who leave one or two billing cycles later.
Unfortunately, there's never enough time to run a proper experiment - we want answers now! Who cares if they're the right answers. Short-termism can't wait two months.
You could only be measuring in aggregate, no? Overall signal could be positive but one element happens to be negative while another is overly positive.
Well, adjusting nudges in aggregate but diced in various ways. Measured very much not in aggregate. We’d see positive and negative outcomes roll in over multiple years and want it per identifier (an individual). I’ve heard of companies generating a model per person but we didn’t.
A silly amount of work but honestly lots of value. Experimentation optimising for short term goals (eg upgrade) is such a bad version of this, it’s just all that is possible with most datasets.
