Image credit: technologyadvice.com
At Acorns, Jike and team were developing behavior nudges to improve people’s financial wellness through micro-investing. The team had many great nudges as behavioral recommendations for the customers. The strategic question has been: When would these recommendations be accurate enough and personalized enough for them to be successfully seen as coming from a trusted financial assistant?
What was learned was that success is more than the accuracy of the model, but also the consistency of the implementation, and the trust customers have with a data-driven solution.
How do we assess when a model is successful?
Context, Expectations, and the three “R”s
From a data science practitioner perspective, depending on the accuracy of the models we can produce, we must understand the context we intend to operate in and set realistic expectations with our customers.
As a thought leader in evaluating the success of business ventures in solving real-world problems with data science and AI, Jocelyn Goldfein, a managing director at Zetta Venture Partners, highlighted a few dimensions that must align for success in data science when she spoke at the MIT AI Conference in 2019.
We summarize the concepts she discussed here into three “R”s:
- Readiness: Is the technology accurate enough?
- Riskiness: What is the downside of being wrong?
- Reception: Will the market accept it?
Level 1: Recommendations and Ranking
At the Recommendations and Ranking level, there is a vast set of potential content to bring to the attention of the customer with limited attention. The goal is to anticipate what content could be most engaging for a customer.
In a mostly explorational context, the customer’s preference is often hard to quantify, so the expectation is that some recommendations will be relevant, but not necessarily all of them. In terms of readiness, with more deliberate user behavior aggregation, personalization algorithm can segment and target users with some accuracy. There is a huge upside for gaining engagement when we provide relevant recommendations - even a 5% engagement rate could have an out-sized business impact in many fields. At the same time, the riskiness of a recommendation being wrong is minimal. Personalization is already well accepted in our online experiences, to the extent that we may even anticipate it.
Success in recommendations is evaluated by “lift”, which maps to the incremental business impact of using a model. To illustrate “lift”, an engagement improvement from 4% to 5% would be a 25% lift (as calculated by: 5% / 4% - 1 = 25%).
Level 2: Assistant
At the Assistant level, the machine learning models are producing results close to human capability, but not quite good enough yet.
One example is model-assisted loan fraud detection, where the context calls for an algorithm to alert of suspicious loan applications, and the expectation is for human fraud investigators to assess and decide if a loan application needs to be declined. With respect to readiness, a first-pass model at detection precision of 25% would already be received as useful for the fraud investigation team. Any greater precision could come at the risk of lower recall.
Success is the significant increase in human productivity, while customers feel confident about the quality of the results as they are guaranteed by the final human decision in the loop.
Level 3: Automation
At the automation level, the machine learning models are producing results at human-parity.
Contexts including data entry (speech recognition), extracting fields out of documents, and reading MRI for a radiologist. The expectation is that while there will be errors, and results will be at least as good as what humans can do in a similar situation. These capabilities are often achieved using deep learning algorithms, readied by training on a large number of samples, at scales beyond what an average human can ever be exposed to. The risk when a model gets it wrong could be higher in healthcare-related applications, however, the benefit of providing access to important health care diagnoses can drive a warm reception from currently unreachable audiences.
Success is the adoption of models that can perform at human accuracy, such that we can significantly increase the reach to serve the majority of the population.
Level 4: Autonomous Agents
At the Autonomous Agent level, the machine learning models are producing results at superhuman capabilities. We interact with them by:
- Define the problem-solving strategy
- Allow the models to produce results
- Take time to build trust with the capabilities of the autonomous agents
One context is high-frequency trading, where trading decisions have to be made at milliseconds to nanoseconds range. No human can interpret market signals, synthesis, and respond at this time granularity. The expectation is that the models behind high-frequency trading are correct the majority of the time such that the book of trades can be profitable as a whole. The readiness of algorithms is evaluated extensively with backtesting. As being wrong carries the risk of serious financial impacts, models are usually tested extensively before being entrusted with large volumes of trades.
Success is when human operators can overcome the leap of faith to trust autonomous agents to correctly operate with the strategies we defined to produce the results we anticipate.
We summarized the four levels of confidence when we evaluate the success of a model (3 Rs) into a table. If this and the table is relevant and interests you, please comment below or contact us.
With a deeper understanding of the customers and improvements in modeling capabilities, we see solutions moving up the levels of confidence ladder, from recommendation to assistants, to automation, to autonomous agents.
At Acorns, Jike and team started with building recommendations. Through harvesting the entire company’s ingenuity in generating behavioral nudges as recommendations to improve customers’ financial wellness, the team created some nudges that generated click-through rates that was an order of magnitude higher than the typical single digit click-through rate in a regular recommendation. They also developed an infrastructure to launch and experiment with scores of new behavioral nudges each quarter, which can be used to quickly adapt to new market scenarios and provide timely assistance for the customers in the financial domain.
Time will tell if the highly relevant behavioral nudge recommendations from Acorns can be successfully perceived as help from a financial assistant from the customer’s perspective over the long term.
We have now discussed the evaluation of the success of a solution, and this concludes the Technology section of the TED-ERA fan. Other blogs on the Technology sections are understanding data characteristics of Feb 3, innovating in feature engineering of Feb 10 and clarifying modeling strategy of Feb 17. Next time we will begin discussing the rigorous application of domain insights.
Figure 8.1 Using Data to Train Models to Discover Patterns - Technology in TED-ERA fan
If you have views, questions, or examples you would like to share, feel free to comment below or contact us directly.
About the authors:
Dr. Jike Chong is a director of data science at LinkedIn, leading a team to develop products to advance people’s careers in the context of the hiring marketplace. He previously built and grew several high-performing data science teams at Acorns, Yirendai (NYSE:YRD), and Simply Hired. Dr Chong also advises several startups and teaches at Tsinghua University.
Yue Cathy Chang is an executive leader in business development, sales and marketing for data products and services. Cathy has held strategic roles serving customers using data science- and engineering-based solutions at bluechip software enterprise companies and startups. Well-versed in tech and business, she has frequently spoken at data and AI events.
No comments:
Post a Comment