How confident can you be in AI predictions?
We’re often asked about the accuracy of AI – how confident we can be in its predictions.
It’s understandable. Operators are, after all, being asked to move away from an expertise-led approach to effectively hand over control to an algorithm.
It’s also true that, whether human- or machine-generated, every prediction has an error possibility.
The goal has to be as high a degree of accuracy as possible – and AI simply can’t be beaten.
For first-hand evidence, don’t miss next month’s canfitpro Business event. In a panel session entitled Everything You Need to Know About Data & AI, AI expert Ian Mullane will be joined by Troy Morgan and Brent Vaughan of Willows Health & Lifestyle Centre in Australia, where Keepme.ai is driving incredible levels of confidence in decision-making.
“Quite simply, AI is the future. We have high expectations for everything we do now,” confirms Morgan.
A short-term perspective
For now, let’s take a look at how AI compares to other forms of forecasting.
Drawing on human expertise is OK for rudimentary analysis. Say you’re looking at members coming up for renewal in the next 30 days. If you identify those who are currently active, you can fairly assume there’s a strong probability they will continue.
It’s a little simplistic. It doesn’t offer much in the way of a time window to try and change the predicted outcome for the better. But it serves a purpose and forms the basis of many a gym retention strategy.
Where this approach falls down, though, is when we want to develop an understanding that spans longer periods of time.
What if you were already able to identify in March those with the highest probability of leaving in December? Can you imagine the impact this would have on your member retention – on the interventions you could put in place?
A backwards glance
Maybe you use a Business Intelligence platform, including that stalwart of any industry dashboard: the 12-month retention chart.
Of course, it’s vital to know how your member retention breaks down by month, but what can you actually do with this information? Not a massive amount.
You can establish whether there’s an upward or downward trend. Be alerted if the previous month was considerably better or worse than the one before. But as for deliberate actions that can be driven from it and planned into your retention strategies, not much. The historical nature of the metric is limiting, and the one-dimensional presentation doesn’t provide many initiatives for action.
What you ideally want is a forward view for the coming year – a perspective that helps you understand where you need to investigate further and that gives you time to address weaknesses and/or maximize opportunities.
AI in action
AI addresses all the above issues. In fact, when it comes to prediction, the results secured from AI + machine learning will be many times better than anything any expert in your business could achieve.
Don’t want to accept that? Here’s just one (non-fitness) example: a LawGeex study that compared the effectiveness of lawyers versus AI in assessing the quality of non-disclosure agreements.
The study involved 20 lawyers, with decades of experience in companies such as Goldman Sachs, pitted against LawGeex’s proprietary AI platform. Participants evaluated the risks contained in five agreements by searching for 30 specific legal points.
The results of the study showed AI achieving an average accuracy of 94 per cent, and maximum accuracy of 100 per cent. Humans achieved 85 and 94 per cent respectively.
That might not sound wildly different, but don’t underestimate the impact of small improvements in prediction accuracy. Say you have a model that lifts your accuracy from 85 to 90 per cent, and another that lifts it from 98 to 99.9 per cent. The first improvement is impressive, as it means mistakes fall by a third. But in the second example, mistakes fall by a factor of 20.
Then, of course, there’s time efficiency. Returning to the LawGeex study, the lawyers required 92 minutes, on average, to complete the task. AI analyzed all the documents in 26 seconds.
AI in a fitness context
Let’s now look at a gym-based example from the early days of Keepme.ai.
A fitness operator passed Keepme its (somewhat patchy) data for the previous two years and held back the current year. The challenge was to identify which of the members would still be there in six months’ time, which Keepme was able to predict with 82 per cent accuracy (before tuning, may we add, which we always do before full implementation – on average, Keepme is 94–95 per cent accurate from day one).
Compare this to the operator’s previous probability of accurately predicting whether a member in month six of a 12-month contract would stay or go, which was effectively a coin-toss (aka no more than 51 per cent).
Even that initial 51 to 82 per cent jump would have provided the operator with a tremendous platform to confidently target those at risk – including members who didn’t themselves know they were at risk yet! – all in a timeframe where the outcome could still be changed.
The operator now had a real-time indicator of retention probability, which would have allowed it to craft each user journey appropriately, with personalized engagement.
Delighted with the results, we sat down with the team to hear the CFO open the meeting with: “It’s not very accurate then. It’s a long way from 100 per cent.”
Let’s remind ourselves that previous accuracy was no more than 51 per cent (this assuming anyone had the time to run these figures in a multi-thousand-member business). In contrast, they now had an opportunity to correctly predict retention more than eight out of every 10 times, with constant opportunities for further improvement (it’s all about the ‘learning’ in machine learning).
We may be biased, but to us there’s a clear winner.
Hear more about the confidence AI can bring to your decision-making.
