The Technology in the TED-ERA Fan (part 2c): All models are wrong, but some are useful. What’s our strategy?

This article first appeared here on February 17, 2020

Image Credit: izquotes.com

In his 1987 book “Empirical Model-Building and Response Surfaces”, the then 70 years old legendary British statistician George Box wrote: “all models are wrong, but some are useful”.

If this is the case, where do we start when building any of the “wrong but useful models”? First of all, let’s differentiate “what models to build” vs “how to start building models”.

The “What” vs the “How”

Many books and blogs address “What modeling technique should I use?”. Popular models include: linear regression, logistic regression, decision tree, SVM, naive Bayes, kNN, k-means, random forest, and gradient boosted algorithms. Data science practitioners can discuss at length the choices and trade-offs in selecting particular models.

The question we are addressing today is “How to start discovering patterns in real-world scenarios?”

In earlier blogs on Feb 3 and Feb 10, we discussed the first two sets of considerations in discovering patterns in data from the technology section of the TED-ERA fan, namely understanding data characteristics and innovating in feature engineering. This time we focus on the third set of considerations, specifically clarifying modeling strategies.

Figure 7.1 Using Data to Train Models to Discover Patterns - Technology in TED-ERA fan

Modeling Strategies:

Models are effective when they match the patterns that exist in data. Domain expertise is important for forming hypotheses about the nature of the patterns we anticipate. Continuing with our example of developing anti-fraud models for return-shipping insurance, there could be four modeling strategies to consider.

Strategy 1:

One option is the momentum-based modeling strategy, also known as inertia-based modeling. It is popular in complex multi-agent environments. The model is expected to capture trends in the environments, abstracting away fundamental factors that are not expected to change in a short time window, to predict what would happen if those trends continue.

Momentum-based modeling is commonly used in high-frequency trading (capturing market microstructure and order book dynamics) and recommendation systems (such as recency, frequency, and monetization based models) where the human or algorithm decision processes work in predictable ways. For this option to be successful, three conditions must exist:

• An acute recognition that a predictable underlying process exists
• Quantifiable signals can be timely processed
• Levers exist for fast responses

In anti-fraud models for return-shipping insurance, for known fraud mechanisms that cannot be designed out of the products or services, such as offering insurance for new merchants or products, we can quantify fraud trend signals to watch, and activate levers to shut down likely fraudulent transactions before too much damage is done.

Strategy 2:

Another option is the foundational modeling strategy. This option is often used in scenarios where there are clear causal mechanisms driving the predictability of the outcome. At an extreme, it can formalize to structural modeling in economics where the structure of decision making is fully incorporated in the specification of the model.

Foundational modeling strategy can be used in application areas such as sales forecasting, infrastructure load prediction, financial account balance forecasting, where various cyclical patterns drive the outcome. The cyclical patterns could include:

• The time of the day
• The day of the week
• Monthly or semi-monthly rhythms
• Quarter or fiscal year boundaries
• Product launch dates
• Marketing campaign dates
• Regional holidays

The model then uses the training data to assess the relative weights or impact of various mechanisms to achieve good precision/recall for the prediction.

In anti-fraud models for return-shipping insurance, foundational modeling can be effective when we have a mature understanding of the common types of known fraud mechanisms, and strong latent variables assessing the risks from each type of known fraud mechanisms. For example, we may assess the fraud risk of each transaction from the following categories and assess an overall risk prediction with a foundational model:

• The customer - frequently shopping/returning products
• The merchant - with a high return rate
• The shipper - correlating with high return rate
• Merchant/customer - corroborating in fake transactions
• Previously hacked identities on an e-commerce platform
• Fake merchant/customer identities: incubated by dark web players for sale

Foundational models fail when the latent variables used for known fraud categories no longer captures the fraud mechanisms at hand.

Strategy 3:

Reflexive modeling strategy, also known as the active learning strategy, is a third option to pursue. In many scenarios, it is often challenging to define a boundary for positive and negative cases by looking at data at hand. To improve the models, we need to explore the decision boundaries in an operationally smart approach to gather additional data for training iterations.

Examples of use cases include many-class classifications such as job title taxonomies and healthcare provider taxonomies, as well as fraud detection with active feedback from fraud investigation professionals to recognize borderline cases to include in training.

When the return-shipping insurance product is at a nascent stage, a reflexive modeling strategy can be highly effective in aggregating fraud mechanisms for particular product integrations. It allows the fraud model to better clarify the separation between the good and the bad, incorporate additional features when necessary, to arrive at an operating point for the model to satisfy precision and recall requirements.

Strategy 4:

Depending on the maturity of data set and data infrastructure, the urgency of the task, and the business requirements, the situation often calls for Hybrid modeling strategies that combine a selection of the previous strategies.

Continuing with the return-shipping insurance example, early in the product development cycle, there may be some obvious fraud schemes that can be prevented with product feature design. For example, the service may not be extended to merchants who sell mostly virtual goods. We can start with some simple decision trees encoding business rules with limited distribution while collecting data on potential fraud schemes.

With limited data set collected, we can start deploying a reflexive strategy, iteratively sending borderline cases for fraud investigation professionals to review and label, to improve the model’s discrimination power.

For new merchants or products with the cold-start modeling challenge, a momentum-based strategy can help quantify fraud trend signals to watch, and activate levers to shut down likely fraudulent transactions before too much damage is done.

After a period of operations, with significant training data set collected and fraud mechanisms understood, one may move to a foundational modeling strategy with an ensemble of sub-models detecting various well-understood fraud schemes to produce latent risk scores.

A modeling strategy is shaped by deep insights into the business challenge at hand. There is no one right answer, as “all models are wrong, but some are useful”.

We have now discussed data understanding, feature engineering, and modeling strategies. This concludes the discovery of patterns in the Technology section of the TED-ERA fan. Next time we will discuss how to evaluate the success of a solution.

If you have views, questions, or examples you would like to share, feel free to comment below or contact us directly.

About the authors:

Dr. Jike Chong is a director of data science at LinkedIn, leading a team to develop products to advance people’s careers in the context of the hiring marketplace. He previously built and grew several high-performing data science teams at Acorns, Yirendai (NYSE:YRD), and Simply Hired. Dr Chong also advises several startups and teaches at Tsinghua University.

Yue Cathy Chang is an executive leader in business development, sales and marketing for data products and services. Cathy has held strategic roles serving customers using data science- and engineering-based solutions at bluechip software enterprise companies as well as startups. Well-versed in tech and business, she has frequently spoken at data and AI industry events.

The writing and opinions expressed are solely our own and are not shared, supported, or endorsed in any manner by our employers.