The Technology in the TED-ERA Fan (part 3): All models are wrong, but some are useful. How to evaluate success?

This article first appeared here on February 24, 2020

Image credit: technologyadvice.com

Success is what we aim for in building data science solutions. How do we define success in a world where “All models are wrong, but some are useful”? How accurate must a model be to be useful?

At Acorns, Jike and team were developing behavior nudges to improve people’s financial wellness through micro-investing. The team had many great nudges as behavioral recommendations for the customers. The strategic question has been: When would these recommendations be accurate enough and personalized enough for them to be successfully seen as coming from a trusted financial assistant?

What was learned was that success is more than the accuracy of the model, but also the consistency of the implementation, and the trust customers have with a data-driven solution.

How do we assess when a model is successful?

Context, Expectations, and the three “R”s

From a data science practitioner perspective, depending on the accuracy of the models we can produce, we must understand the context we intend to operate in and set realistic expectations with our customers.

As a thought leader in evaluating the success of business ventures in solving real-world problems with data science and AI, Jocelyn Goldfein, a managing director at Zetta Venture Partners, highlighted a few dimensions that must align for success in data science when she spoke at the MIT AI Conference in 2019.

We summarize the concepts she discussed here into three “R”s:

• Readiness: Is the technology accurate enough?
• Riskiness: What is the downside of being wrong?
• Reception: Will the market accept it?

Let’s look at four levels of model confidence/capabilities among these three dimensions.

Level 1: Recommendations and Ranking

At the Recommendations and Ranking level, there is a vast set of potential content to bring to the attention of the customer with limited attention. The goal is to anticipate what content could be most engaging for a customer.

In a mostly explorational context, the customer’s preference is often hard to quantify, so the expectation is that some recommendations will be relevant, but not necessarily all of them. In terms of readiness, with more deliberate user behavior aggregation, personalization algorithm can segment and target users with some accuracy. There is a huge upside for gaining engagement when we provide relevant recommendations - even a 5% engagement rate could have an out-sized business impact in many fields. At the same time, the riskiness of a recommendation being wrong is minimal. Personalization is already well accepted in our online experiences, to the extent that we may even anticipate it.

Success in recommendations is evaluated by “lift”, which maps to the incremental business impact of using a model. To illustrate “lift”, an engagement improvement from 4% to 5% would be a 25% lift (as calculated by: 5% / 4% - 1 = 25%).

Level 2: Assistant

At the Assistant level, the machine learning models are producing results close to human capability, but not quite good enough yet.

One example is model-assisted loan fraud detection, where the context calls for an algorithm to alert of suspicious loan applications, and the expectation is for human fraud investigators to assess and decide if a loan application needs to be declined. With respect to readiness, a first-pass model at detection precision of 25% would already be received as useful for the fraud investigation team. Any greater precision could come at the risk of lower recall.

Success is the significant increase in human productivity, while customers feel confident about the quality of the results as they are guaranteed by the final human decision in the loop.

Level 3: Automation

At the automation level, the machine learning models are producing results at human-parity.

Contexts including data entry (speech recognition), extracting fields out of documents, and reading MRI for a radiologist. The expectation is that while there will be errors, and results will be at least as good as what humans can do in a similar situation. These capabilities are often achieved using deep learning algorithms, readied by training on a large number of samples, at scales beyond what an average human can ever be exposed to. The risk when a model gets it wrong could be higher in healthcare-related applications, however, the benefit of providing access to important health care diagnoses can drive a warm reception from currently unreachable audiences.

Success is the adoption of models that can perform at human accuracy, such that we can significantly increase the reach to serve the majority of the population.

Level 4: Autonomous Agents

At the Autonomous Agent level, the machine learning models are producing results at superhuman capabilities. We interact with them by:

1. Define the problem-solving strategy
2. Allow the models to produce results
3. Take time to build trust with the capabilities of the autonomous agents

One context is high-frequency trading, where trading decisions have to be made at milliseconds to nanoseconds range. No human can interpret market signals, synthesis, and respond at this time granularity. The expectation is that the models behind high-frequency trading are correct the majority of the time such that the book of trades can be profitable as a whole. The readiness of algorithms is evaluated extensively with backtesting. As being wrong carries the risk of serious financial impacts, models are usually tested extensively before being entrusted with large volumes of trades.

Success is when human operators can overcome the leap of faith to trust autonomous agents to correctly operate with the strategies we defined to produce the results we anticipate.

We summarized the four levels of confidence when we evaluate the success of a model (3 Rs) into a table. If this and the table is relevant and interests you, please comment below or contact us.

With a deeper understanding of the customers and improvements in modeling capabilities, we see solutions moving up the levels of confidence ladder, from recommendation to assistants, to automation, to autonomous agents.

At Acorns, Jike and team started with building recommendations. Through harvesting the entire company’s ingenuity in generating behavioral nudges as recommendations to improve customers’ financial wellness, the team created some nudges that generated click-through rates that was an order of magnitude higher than the typical single digit click-through rate in a regular recommendation. They also developed an infrastructure to launch and experiment with scores of new behavioral nudges each quarter, which can be used to quickly adapt to new market scenarios and provide timely assistance for the customers in the financial domain.

Time will tell if the highly relevant behavioral nudge recommendations from Acorns can be successfully perceived as help from a financial assistant from the customer’s perspective over the long term.

We have now discussed the evaluation of the success of a solution, and this concludes the Technology section of the TED-ERA fan. Other blogs on the Technology sections are understanding data characteristics of Feb 3, innovating in feature engineering of Feb 10 and clarifying modeling strategy of Feb 17. Next time we will begin discussing the rigorous application of domain insights.

Figure 8.1 Using Data to Train Models to Discover Patterns - Technology in TED-ERA fan

If you have views, questions, or examples you would like to share, feel free to comment below or contact us directly.

About the authors:

Dr. Jike Chong is a director of data science at LinkedIn, leading a team to develop products to advance people’s careers in the context of the hiring marketplace. He previously built and grew several high-performing data science teams at Acorns, Yirendai (NYSE:YRD), and Simply Hired. Dr Chong also advises several startups and teaches at Tsinghua University.

Yue Cathy Chang is an executive leader in business development, sales and marketing for data products and services. Cathy has held strategic roles serving customers using data science- and engineering-based solutions at bluechip software enterprise companies and startups. Well-versed in tech and business, she has frequently spoken at data and AI events.

The Technology in the TED-ERA Fan (part 2c): All models are wrong, but some are useful. What’s our strategy?

This article first appeared here on February 17, 2020

Image Credit: izquotes.com

In his 1987 book “Empirical Model-Building and Response Surfaces”, the then 70 years old legendary British statistician George Box wrote: “all models are wrong, but some are useful”.

If this is the case, where do we start when building any of the “wrong but useful models”? First of all, let’s differentiate “what models to build” vs “how to start building models”.

The “What” vs the “How”

Many books and blogs address “What modeling technique should I use?”. Popular models include: linear regression, logistic regression, decision tree, SVM, naive Bayes, kNN, k-means, random forest, and gradient boosted algorithms. Data science practitioners can discuss at length the choices and trade-offs in selecting particular models.

The question we are addressing today is “How to start discovering patterns in real-world scenarios?”

In earlier blogs on Feb 3 and Feb 10, we discussed the first two sets of considerations in discovering patterns in data from the technology section of the TED-ERA fan, namely understanding data characteristics and innovating in feature engineering. This time we focus on the third set of considerations, specifically clarifying modeling strategies.

Figure 7.1 Using Data to Train Models to Discover Patterns - Technology in TED-ERA fan

Modeling Strategies:

Models are effective when they match the patterns that exist in data. Domain expertise is important for forming hypotheses about the nature of the patterns we anticipate. Continuing with our example of developing anti-fraud models for return-shipping insurance, there could be four modeling strategies to consider.

Strategy 1:

One option is the momentum-based modeling strategy, also known as inertia-based modeling. It is popular in complex multi-agent environments. The model is expected to capture trends in the environments, abstracting away fundamental factors that are not expected to change in a short time window, to predict what would happen if those trends continue.

Momentum-based modeling is commonly used in high-frequency trading (capturing market microstructure and order book dynamics) and recommendation systems (such as recency, frequency, and monetization based models) where the human or algorithm decision processes work in predictable ways. For this option to be successful, three conditions must exist:

• An acute recognition that a predictable underlying process exists
• Quantifiable signals can be timely processed
• Levers exist for fast responses

In anti-fraud models for return-shipping insurance, for known fraud mechanisms that cannot be designed out of the products or services, such as offering insurance for new merchants or products, we can quantify fraud trend signals to watch, and activate levers to shut down likely fraudulent transactions before too much damage is done.

Strategy 2:

Another option is the foundational modeling strategy. This option is often used in scenarios where there are clear causal mechanisms driving the predictability of the outcome. At an extreme, it can formalize to structural modeling in economics where the structure of decision making is fully incorporated in the specification of the model.

Foundational modeling strategy can be used in application areas such as sales forecasting, infrastructure load prediction, financial account balance forecasting, where various cyclical patterns drive the outcome. The cyclical patterns could include:

• The time of the day
• The day of the week
• Monthly or semi-monthly rhythms
• Quarter or fiscal year boundaries
• Product launch dates
• Marketing campaign dates
• Regional holidays

The model then uses the training data to assess the relative weights or impact of various mechanisms to achieve good precision/recall for the prediction.

In anti-fraud models for return-shipping insurance, foundational modeling can be effective when we have a mature understanding of the common types of known fraud mechanisms, and strong latent variables assessing the risks from each type of known fraud mechanisms. For example, we may assess the fraud risk of each transaction from the following categories and assess an overall risk prediction with a foundational model:

• The customer - frequently shopping/returning products
• The merchant - with a high return rate
• The shipper - correlating with high return rate
• Merchant/customer - corroborating in fake transactions
• Previously hacked identities on an e-commerce platform
• Fake merchant/customer identities: incubated by dark web players for sale

Foundational models fail when the latent variables used for known fraud categories no longer captures the fraud mechanisms at hand.

Strategy 3:

Reflexive modeling strategy, also known as the active learning strategy, is a third option to pursue. In many scenarios, it is often challenging to define a boundary for positive and negative cases by looking at data at hand. To improve the models, we need to explore the decision boundaries in an operationally smart approach to gather additional data for training iterations.

Examples of use cases include many-class classifications such as job title taxonomies and healthcare provider taxonomies, as well as fraud detection with active feedback from fraud investigation professionals to recognize borderline cases to include in training.

When the return-shipping insurance product is at a nascent stage, a reflexive modeling strategy can be highly effective in aggregating fraud mechanisms for particular product integrations. It allows the fraud model to better clarify the separation between the good and the bad, incorporate additional features when necessary, to arrive at an operating point for the model to satisfy precision and recall requirements.

Strategy 4:

Depending on the maturity of data set and data infrastructure, the urgency of the task, and the business requirements, the situation often calls for Hybrid modeling strategies that combine a selection of the previous strategies.

Continuing with the return-shipping insurance example, early in the product development cycle, there may be some obvious fraud schemes that can be prevented with product feature design. For example, the service may not be extended to merchants who sell mostly virtual goods. We can start with some simple decision trees encoding business rules with limited distribution while collecting data on potential fraud schemes.

With limited data set collected, we can start deploying a reflexive strategy, iteratively sending borderline cases for fraud investigation professionals to review and label, to improve the model’s discrimination power.

For new merchants or products with the cold-start modeling challenge, a momentum-based strategy can help quantify fraud trend signals to watch, and activate levers to shut down likely fraudulent transactions before too much damage is done.

After a period of operations, with significant training data set collected and fraud mechanisms understood, one may move to a foundational modeling strategy with an ensemble of sub-models detecting various well-understood fraud schemes to produce latent risk scores.

A modeling strategy is shaped by deep insights into the business challenge at hand. There is no one right answer, as “all models are wrong, but some are useful”.

We have now discussed data understanding, feature engineering, and modeling strategies. This concludes the discovery of patterns in the Technology section of the TED-ERA fan. Next time we will discuss how to evaluate the success of a solution.

If you have views, questions, or examples you would like to share, feel free to comment below or contact us directly.

About the authors:

Dr. Jike Chong is a director of data science at LinkedIn, leading a team to develop products to advance people’s careers in the context of the hiring marketplace. He previously built and grew several high-performing data science teams at Acorns, Yirendai (NYSE:YRD), and Simply Hired. Dr Chong also advises several startups and teaches at Tsinghua University.

Yue Cathy Chang is an executive leader in business development, sales and marketing for data products and services. Cathy has held strategic roles serving customers using data science- and engineering-based solutions at bluechip software enterprise companies as well as startups. Well-versed in tech and business, she has frequently spoken at data and AI industry events.

The writing and opinions expressed are solely our own and are not shared, supported, or endorsed in any manner by our employers.

The Technology in the TED-ERA Fan (part 2b): from Coronavirus to Ecommerce to Innovating in Feature Engineering

This article first appeared here on February 10, 2020

As the coronavirus outbreak evolves in China and around the world, we are seeing incredible robustness in the Chinese e-commerce industry (NYSE:BABA, NASDAQ:JD) so far, despite the voluntary self-quarantine measures implemented across the country. Having food, medicine, and life’s necessities delivered directly to the door-steps allows people to avoid potentially risky exposures to commutable diseases in crowded social locations.

The e-commerce industry in China may look back at the transaction data over this period as a treasure trove of customer demographic studies, as well as an assessment of the potential upside in e-commerce for serving their customers’ every need! For many e-commerce customers in China, this is likely to accelerate their shift of purchasing habits from offline to online channels.

How do we quantify purchasing habits? How could this help an e-commerce company?

Last time, we discussed the vigilance required in understanding data characteristics, using e-commerce as a data domain and mitigating return-shipping insurance fraud as a scenario.

In this blog, we discuss another set of considerations in using data to train models to discover patterns, specifically innovating in feature engineering. We will also introduce the concept of clarifying modeling strategy, which will be the focus of the next blog.

Figure 6.1 Using Data to Train Models to Discover Patterns - Technology in TED-ERA fan

Innovations in feature engineering

Let’s continue with the scenario of mitigating return-shipping insurance fraud and explore how feature engineering could help in catching the fraudsters.

Wikipedia describes feature engineering as “the process of using domain knowledge to extract features from raw data via data mining techniques. These features can be used to improve the performance of machine learning algorithms.”

In short, feature engineering allows us to summarize a vast amount of data so we can use models to meaningfully discover the patterns within. Simpler models such as linear regression rely heavily on feature engineering for summarization. More complex models such as deep neural networks may even automate the feature engineering process as part of the model training itself and use raw signals as inputs.

In many use cases, however, because of the small sample sizes, labeling difficulties, and interpretability and operability considerations, we often opt to use simpler models and engineered features to discover patterns in our scenarios.

Feature engineering can take on many levels of depth. In e-commerce, for a customer, feature engineering may include:

• Simple statistics: average price for good brought, frequency of purchase
• Complex statistics: weekday/weekend activities, impulse buy or considered purchase
• Taxonomy-based analysis: categories of purchases, diversity of needs
• Clustering analysis: the portfolio of categories, customer intent or life-stage
• Graph analysis: patterns of interaction with merchants, risks for fraud

Merchants, payment processors, shoppers can have similar stacks of feature extraction depths to summarize the diversity of information in the rich e-commerce data set.

How Do We Prioritize?

With such a diversity of feature extraction techniques, there is always limited time and resources to explore a rich data set. So how do we prioritize?

This is when domain expertise becomes crucial. In our return-shipping insurance fraud mitigation scenario, an experienced fraud investigator can often use clues from a few anomalies in customer or merchant behavior to piece together how a fraud scheme may be committed. Domain insights are valuable and necessary as starting points to focus feature extraction efforts but are not sufficient.

In one scenario, fraud investigators considered merchants with long transaction histories in physical goods more trustworthy than those merchants that sell mainly virtual goods (such as phone cards and gift cards). Historical data validated that fraudsters often look for shortcuts when building transaction history for their merchant accounts so they can be ranked higher in product search results, and virtual goods tend to be an easy way to build transaction history without the trouble of logistics operations. Understanding this domain-specific distinction can prioritize feature development and accelerate time to an accurate anti-fraud model.

In another scenario, fraud investigators discovered trends in password recovery security questions where confirmed fraudsters often use “cats” or “dogs” as answers for their favorite animals. This could be interpreted as the lack of “thoughtfulness” of fraudsters in setting up their merchant accounts. However, when assessing the trend as a fraud signal, we found that it happens that “cats” and “dogs” are most people’s favorite animals, regardless of any fraudulent intent. Using it as a signal in fraud investigations would bias fraud investigators’ assessment in an unjustified way.

These examples demonstrate that domain expertise and intuition, while crucial, must go through rigorous validation to be successfully entered into a model.

Models are effective when they match the patterns that exist in data. Domain expertise is also important for forming hypotheses about the nature of the patterns we anticipate.

We just discussed the role of innovating in feature engineering in using data to train models to discover patterns. In the next blog, we will focus on clarifying the modeling strategy where we will dive into details of different strategies and discuss how those apply to the return-shipping insurance scenario.

If you have views, questions, or examples you would like to share, feel free to comment below or contact us directly.

About the authors:

Dr. Jike Chong is a director of data science at LinkedIn, leading a team to develop products to advance people’s careers in the context of the hiring marketplace. He previously built and grew several high-performing data science teams at Acorns, Yirendai (NYSE:YRD), and Simply Hired. Dr Chong also advises several startups and teaches at Tsinghua University.

Yue Cathy Chang is an executive leader in business development, sales and marketing for data products and services. Cathy has held strategic roles serving customers using data science- and engineering-based solutions at bluechip software enterprise companies and startups. Well-versed in tech and business, she has frequently spoken at data and AI events.

The writing and opinions expressed are solely our own and are not shared, supported, or endorsed in any manner by our employers.

The Technology in the TED-ERA Fan (part 2a): from Coronavirus to Vigilance in Understanding Data Characteristics

This article first appeared here on February 3, 2020

The coronavirus outbreak is devastating much of commerce in China, sending ripples across global financial markets. One bright spot has been the e-commerce industry, where food, medicine, and pretty much anything can be ordered online and delivered to people’s doorsteps, allowing people to avoid potentially risky exposures to commutable diseases in crowded social locations.

The proliferation of online shopping habits over the past decade has enabled the build-out of some incredible logistics networks. However, the purchase friction of “what if I don’t like it?” still causes many business opportunities to be missed.

One relatively recent innovation that removes some of this friction is an insurance product called return-shipping insurance.

During an e-commerce online checkout, customers can choose to spend $1 to $2 to insure for some return-shipping credit. The insurance premium is rolled into the shopping cart so no separate payment is necessary. If a customer is not satisfied with the product purchase, he/she can just return the product within seven days, and return shipping credit will be automatically processed within 72 hours, without going through any cumbersome claim process.

This insurance product is hugely popular in e-commerce in China. ZhongAn Online P&C Insurance Co. Ltd. (6060:HK), A pioneer in online-only insurance, has issued more than 5.8 billion insurance policies to date, most of which are return-shipping insurance.

Opportunities and Caveats

As with any opportunities, there are always caveats.

Insurance fraud can be rampant in an environment where anyone can register as an online merchant, and anyone can sign up to be an e-commerce customer. When someone can pay $1-2 to get $10-20 of credit over the Internet, there is a significant incentive to find schemes to attack the insurance business model. Early trials of return-shipping insurance at one e-commerce company were reported to have claim rates as high as 90%, caused by significantly high levels of fraud, which obviously does not make a sustainable insurance business model.

Last time we discussed mapping scenarios to known optimization techniques, the first part of Technology in the TED-ERA fan, using an example of driving engagement conversion.

This time we will continue discussing Technology in TED-ERA, focusing on practices in using data to train models to discover patterns. 

How to Discover Patterns in Data to Fight Fraud?

For discovering patterns, we believe we must be vigilant in understanding data characteristics, be innovative in feature engineering, and have clarity in modeling strategy when fighting fraud using e-commerce data.

E-commerce data contains a rich range of granularity over time, entity, and interactions for catching fraud. Over the time horizon alone, as a customer is shopping for products, analysis granularities could include (but not limited to):

• the engagement with a product recommendation
• the exploratory actions within a shopping session
• a purchase decision with engagement across multiple sessions
• multiple purchases with an intent (such as Christmas, Superbowl, Valentine’s day, back-to-school)
• purchases in a life-stage, such as graduation, getting married, or the birth of a child

For each transaction, the entities involved include the customer, the merchant, the payment processor, and often the shippers for the physical goods.

There is also a rich set of interactions between these entities for each transaction, with interaction contexts and patterns across multiple transactions.

Understanding the level of granularity we choose to assess return-shipping insurance fraud is crucial for building an effective modeling strategy. However, the choice may not be so obvious.

The fraudulent intent could be coming from one or more of the following parties:

• the customers shopping for products
• the shippers
• a merchant corroborating with customers
• a hacker with access to multiple stolen identities on an e-commerce platform
• (or even) dark web players incubating merchant / customer identities for sale

Vigilance in Understanding Data Characteristics

One way to cut through the complexity is to look at the unit of decisioning. In the return-shipping insurance fraud scenario, this can be at the granularity where we can decide whether to offer the return-shipping insurance in the checkout process on the e-commerce platform. Depending on the platform’s product design choices, unit of decisioning could be at the level of an individual e-commerce transaction level, a customer level, or a merchant level. Information from the other levels of granularity can be incorporated as features in the model.

As with any evolving e-commerce business, biases in operations data can be introduced as part of business expansion, marketing campaigns, or bots/crawlers for a website. These biases can impact data characteristics such as sample size, data sparsity, and outliers. These data characteristics are crucial to assess and understand before constructing features, as common statistical metrics may often misinterpret the situation.

For anti-fraud models specifically, fraud cases are often a small portion of the overall transaction volume. Sample imbalance issues are common when training fraud models. Assess the degree of imbalance and choose more robust modeling techniques are crucial to modeling success. Common techniques include random over-sampling or under-sampling, informed under-sampling, synthetics sampling with data generation, and adaptive synthetic sampling.

The e-commerce scenario also includes a broad set of data types, including:
Individual samples such as articles, images, and videos on merchandise
Time-sequenced data such as browsing history, transaction history, customer service interaction history
Relationship-based data such as interactions between customers, merchants, payment processors, and shippers

Different data types call for different analysis infrastructures and can offer unique features and perspectives in modeling. For example, relationship-based data such as interactions between customers, merchants, payment processors, and shippers can be used to generate knowledge graphs that capture these relationships. Knowledge graph based approach has been shown to be effective in highlighting behaviors in organized financial fraud.

In summary, we discussed the first consideration of using data to train models to discover patterns: understanding data characteristics. This part includes the following practices:

• Understanding unit of decisioning
• Being aware of sample size, sparsity, and outliers
• Assessing sample distribution/imbalance
• Interpreting data types

Figure 5.1 Using Data to Train Models to Discover Patterns - Technology in TED-ERA fan

In upcoming blogs, we will be discussing other considerations in using data to train models to discover patterns, specifically innovating in feature engineering and clarifying modeling strategy.

If you have views, questions, or examples you would like to share, feel free to comment below or contact us directly.

About the authors:

Dr. Jike Chong is a director of data science at LinkedIn, leading a team to develop products to advance people’s careers in the context of the hiring marketplace. He previously built and grew several high-performing data science teams at Acorns, Yirendai (NYSE:YRD), and Simply Hired. Dr Chong also advises several startups and teaches at Tsinghua University.

Yue Cathy Chang is an executive leader in business development, sales and marketing for data products and services. Cathy has held strategic roles serving customers using data science- and engineering-based solutions at bluechip software enterprise companies and startups. Well-versed in tech and business, she has frequently spoken at data and AI events.

The writing and opinions expressed are solely our own and are not shared, supported, or endorsed in any manner by our employers.

The Technology in the TED-ERA Fan (part 1)

This article first appeared here on January 27, 2020

Image Credit: medium.com

Remember Aayana, the student with great math and statistics skills, who had encountered some vague and challenging data science interview processes? Well, in her job search, she met Brian, another UCLA alum who has been working as a Data Scientist at a social networking company for the past two years. Brian has worked on a variety of data science projects and was recently tasked with partnering with marketing and product teams to develop a set of email campaigns aimed at driving long-term user engagement.

For many data science practitioners, working with email campaigns is nothing new. We often partner with marketing and product to:

• Identify a set of users
• Develop a set of content
• Run the email campaign
• Analyze the click-through rate of emails and the engagement on the landing page

For data-driven optimizations, we may test different subject lines, and experiment with different narratives in the email content to understand what would drive the highest conversion in engagement.

But is that all?

Last time we looked at how capabilities of data scientists have evolved and shared how virtues come into the picture, expanding a familiar DS Venn diagram to a TED-ERA Fan.

This time we will begin discussing Technology in TED-ERA, starting with mapping scenarios to known optimization techniques.

Mapping Scenarios to Known Optimization Techniques in Driving Engagement Conversion

In our January 12 blog, we saw how Aayana, who had encountered a variety of data science interview processes: in one case she was asked to write down an entire data pipeline in an interview with a FinTech Startup.

In academic settings, Aayana was familiar with course project setups where students go straight to solving the most academically challenging piece of the project: discovering patterns in the data. Many of the surrounding information is either specified in the problem statement or has been provided as part of the grading infrastructure.

In real-world data science, however, the picture looks very different. How do we map a scenario to known optimization techniques? How do we evaluate the success of a solution? These questions are just as important as the specifics of building a model.

Let’s dive a bit deeper.

Levels of Practice Toward Intelligence

First of all, are we asking questions about whether we are achieving our goal of driving long-term engagement?

One practice is to 1) form a hold-out set of users as part of an A/B Test, 2) run the campaign for a number of months, and 3) assess if long-term engagement has improved. In this case, we would be analyzing what happened a few months down the road. This kind of hindsightis NOT efficient in driving improvements in operations.

Another practice is to build a model predicting long-term engagement using detectable short-term engagement characteristics such as open rate, click-through-rate (CTR), unsubscribe rate, landing page session length, and session frequency, such that we can anticipate long term effects with short-term observation. This allows us to have the foresight to anticipate campaign impact and quickly adjust our email marketing strategies from week to week.

Yet more powerful approaches can include real-time analytics on email responses to learn what segments of email recipients who tend to engage more with which marketing campaigns, and allow the insights to drive additional tweaks in the email narrative.

The most powerful approach is to prepare sequences of emails aimed at driving long-term engagement that can adapt to user responses in real-time, and help guide the user along a path of actions to realize the most value they can gain from the product. This would be intelligence.

To better understand the distinctions between hindsight, foresight, insights, and intelligence, here is a summary in an easy-to-remember 2x2 matrix.

Figure 4.1 Levels of Practice Toward Intelligence (Source: V. A. Ganesan)

Additional Considerations

Secondly, is each email recipient independent? Are there potential network effects with the campaign?

When identifying users in a social network for email campaigns, there may be intentional or unintentional network effects. Simple assumptions may treat each user as independent recipients, and select them based on the typical demographic and product usage segmentations. However, if the marketing narrative involves information about users and their networks, or promotional offers can be shared with other users, we may need to consider potential network effects. The mapping of the optimization techniques may involve simple network properties such as network graph centrality, or more advanced struct2vec deep embeddings to identify specific cohorts of users that may be more suitable for a marketing campaign.

Thirdly, in developing a set of personalized content for an email campaign, there is often a repertoire of content or the next best actions to select from and personalize with. It could be a selection of connections you can make on the social networking platform, or a set of entertainment items you can engage with. Identifying the best items to include according to relevance, coverage, diversity, novelty, or serendipity metrics to generate long-term engagement is yet another set of ranking based optimization techniques one could map the email campaign to.

In addition, when sending out the content, are there cohorts of users with identifiable habits? Many users habitually engage with certain types of content on a particular time of day or day of the week. How might we optimize the email campaign sends such that our emails are at the top of users’ mailbox when they start checking their inbox?

Technology Considerations

These are many of the ways for Mapping a seemingly straight forward email campaign scenario to optimization techniques.

Mapping scenarios to optimization techniques is the first part of Technology in the TED-ERA Fan. Two other parts of Technology are Discovering patterns in data by training models, and Evaluating the success of a solution.

In upcoming blogs, we will discuss considerations in the other two parts.

If you have views, questions, or examples you would like to share, feel free to comment below or contact us directly.

About the authors:

Dr. Jike Chong is a director of data science at LinkedIn, leading a team to develop products to advance people’s careers in the context of the hiring marketplace. He previously built and grew several high-performing data science teams at Acorns, Yirendai (NYSE:YRD), and Simply Hired. Dr Chong also advises several startups and teaches at Tsinghua University.

Yue Cathy Chang is an executive leader in business development, sales and marketing for data products and services. Cathy has held strategic roles serving customers using data science- and engineering-based solutions at bluechip software enterprise companies and startups. Well-versed in tech and business, she has frequently spoken at data and AI events.

The writing and opinions expressed are solely our own and are not shared, supported, or endorsed in any manner by our employers.

From Venn to Fan

This article first appeared here on January 17, 2020

Image Credit: Getty Images via Inc.com

Last time we shared a case of early career interview woes, discussed what fundamentally hiring managers are looking for, and gave a preview of capabilities and virtues in a successful data scientist.

This time we would like to take a look at how capabilities of data scientists have evolved and how virtues come into the picture, literally.

In the mid-2000s, after graduating from university, a young man was working in Washington, DC in the defense and intelligence communities, focusing on applying computational and statistical methods to social dynamics and organization. A few years later in 2008, right around the time DJ Patil and Jeff Hammerbacher coined the title Data Scientist, the same young man started pursuing a PhD in Political Science and went on to pioneer the art of applying computer science tools to social science. He later introduced a famous Data Science Venn Diagram.

By now, you might have figured out that we are talking about Drew Conway. The Venn diagram he introduced in 2010 clarified three pillars for success in data science: Math & Statistics Knowledge, Hacking Skills, and Substantive Expertise.

Figure 3-1: Data Science Venn Diagram 2010 (source)

What has Evolved a Decade After Inception?

While much of the 2010 original terms and ideas are still valid, there have been updates, debates and even battles on the Data Science Venn Diagram: a simple image search of these words would yield scores of variations.

To us, it is expected that the concepts have expanded greatly since inception. In 2020, a data scientist’s technology capabilities have broadened to beyond math and statistical knowledge, to include the mapping of scenario to appropriate to optimization techniques, taking care of intricate data nuances when building models to discovering patterns, and the careful evaluation of solution success.

A data scientist’s execution capabilities goes beyond hacking skills, to include architecting and mapping of complex problems into a sequence of machine learnable tasks, robustly maintain a high quality of work, and perform independently with little or no guidance, while understanding when to solicit necessary input.

Substantive expertise can be clarified into having domain knowledge of the organization’s objectives, understanding data biases and limitations in an industry, as well as the industry’s operational dynamics.

Having built data science teams in companies private and public, large and small, and after interviewing hundreds of data science candidates in the US and China, we discovered, as expected, that data scientists who are strong in all three capability dimensions are indeed quite difficult, if not impossible, to find.

A data scientist entering the field of data science with academic and industry internship background, such as Aayana from last week’s post, usually has strong capabilities in the technology dimension.

A data scientist with years of experience in industry can usually pick up the second dimension of skills in execution, which includes the ability to deploy scalable and maintainable data science solutions.

A seasoned data science practitioner who also has the third dimension of domain knowledge in a related industry would be highly valuable to any employer.

But What is Missing?

As with any teams and companies, we had our share of blind spots in building teams. While we dutifully scored candidates in their technology, execution, and domain knowledge abilities, our hiring mishaps show up as candidates get vetoed in final-round executive interviews, or worse, get hired and then have to be managed out.

Many of these failures were summarized as “not a cultural-fit”. But what does that mean? What is the “culture” for the data science discipline that we are looking to fit? How is that distinct from an organization’s culture or an industry’s culture?

To analyze these failures, we propose to expand the interviews and evaluations of a data scientist to consider not just the capabilities, but also the virtues of a data scientist in pursuing a data science career.

The Virtues of a Data Scientist

For virtues, we highlight three dimensions: Ethics, Rigor, and Attitude. We found that when data scientists maintain good practices in these three dimensions, they are more likely to produce significant impact for their organizations. On the other hand, when data scientists neglect one or more of these dimensions, it can lead them into difficult situations where they must be mentored or in some cases be managed out.

In defining the Virtues of a Data Scientist, we propose the following:

Ethics: Ethics are standards of conduct at work that enable data scientists to avoid unnecessary and self-produced breakdowns. There are many aspects of work ethics for data scientists, including data use, project execution, and teamwork.

Rigor: Rigor means producing results that are “scientific”, which are repeatable, testable, and discoverable, where the work product can be used to reliably anticipate future situations.

Attitude: Attitudes are the moods with which a data scientist approaches the situations that arise in the workplace. When left unchecked, they often reflect the instinctive reflex of the situation at hand, which may not produce the most desirable outcome.

There you have it: we have effectively transformed what was a Venn diagram from the early days of Data Science, to the TED-ERA Fan.

Figure 3-2: The TED-ERA Fan

In the near future we will introduce another Fan (the LOOP Fan) that brings the Capabilities and Virtues into action in pursuing an impactful career. For now and in the next few posts, we will focus on each part of this Capabilities-Virtues fan, with examples, interpretations, and suggested guidelines.

If you have views, questions, or examples you would like to share, feel free to comment below or contact us directly.

About the authors:

Dr. Jike Chong is a director of data science at LinkedIn, leading a team to develop products to advance people’s careers in the context of the hiring marketplace. He previously built and grew several high-performing data science teams at Acorns, Yirendai (NYSE:YRD), and Simply Hired. Dr Chong also advises several startups and teaches at Tsinghua University.

Yue Cathy Chang is an executive leader in business development, sales and marketing for data products and services. Cathy has held strategic roles serving customers using data science- and engineering-based solutions at bluechip software enterprise companies and startups. Well-versed in tech and business, she has frequently spoken at data and AI events.

The writing and opinions expressed are solely our own and are not shared, supported, or endorsed in any manner by our employers.

What’s in a Successful Data Scientist?

This article first appeared here on January 12, 2020

Image Credit: Snapwire

Last time we shared some high-level career development challenges data science practitioners may have over the span of a few decades.

This time we would like to share a real-life scenario on the early part of the career spectrum and begin discussing what makes a successful data scientist.

Interview Woes

Aayana is a graduate student studying computer science at UCLA. Her easy-going nature belies the rigor she gives to her technical work. Since arriving in the US from India 15 months ago, she already took several advanced courses in machine learning, worked with two prestigious research groups, and interned at a fast-growing, soon-to-IPO, Silicon Valley startup company.

With graduation coming up in six months, as with many ambitious young professionals, she began interviewing for a full-time position.

What she encountered in the interview process for data science positions brought her confusion and frustration. She lost her usual confidence and was desperately seeking help from mentors.

What happened?

It turns out that each data science interview Aayana encountered was pretty different.

The interview for a natural language processing (NLP) engineering position started with an informal chat about her prior projects.

Another interview in a FinTech startup started with a task to write down an entire data pipeline. Aayana was given information on the sparsity of data and the available sample size, and was asked to suggest the best algorithm to use. Then, she was asked to improve the data pipeline and better process the raw data for the final model.

These interviews looked very different from the machine learning course exams she excelled at just weeks earlier. With vastly different starting points for the data science interview processes, she was at a loss on what to expect.

Aayana was confused and frustrated about not knowing whether there is a standard process for data science interviews, or if every company and every team would have a different set of criteria they are hiring for.

What are Hiring Managers Looking For?

According to LinkedIn Talent Insight data, only 20% of data scientists work in companies with 100 data scientists or more. In large companies, there are often mature processes for interviewing, evaluating, and promoting career growth in the domain of data science. These large companies represent all but 0.25% of all companies employing data scientists in 2020.

The vast majority (67%) of data scientists work in companies with data science teams of fewer than 30 members, representing 99% of the companies employing data scientists.

The field of data science is still nascent. Interview questions for entry-level data scientists are highly dependent on the company stage, product maturity, and the skill sets hiring managers are looking for to complete their team.

Fundamentally, however, for any team to perform, what all hiring managers look for fall under two areas: ability and motivation. 

Understanding Ability and Motivation: Assessing Capabilities and Virtues

For a data science team, ability is the capability to produce accomplishments using data science skills. It is the hard technical skills that can be learned and accumulated over time.

Motivation, on the other hand, is the willingness to work together, within the team and across teams. It is the virtues or the set of soft skills we accumulate over time to harmoniously produce accomplishments together for the organizations we are a part of.

For Aayana, coming from an academic environment, the assessment of capabilities in interviews is more familiar. On the other hand, the assessment of virtues intertwined in the interviews can take conversations into many unfamiliar territories - generating confusion and frustration. However, if Aayana understood what fundamental areas interviewers are looking to assess, she would be a lot more confident walking into her interviews.

In the upcoming posts, we will discuss dimensions of Capabilities and Virtues and the pursuit of an impactful career in data science, and suggest ways we can practice capabilities and cultivate virtues over time.

As a preview, we will assess capabilities in these dimensions: Technology, Execution, and Domain knowledge, and Virtue in these dimensions: Ethics, Rigor, and Attitude.

What may be included in each of the dimensions for data science Capabilities and Virtues? For different milestones of a career? In different size companies? And in different industries?

Our upcoming posts will offer interpretations across different spectrums of a career in data science.

If you have questions or examples you would like to share, feel free to comment below or contact us directly.

About the authors:

Dr. Jike Chong is a director of data science at LinkedIn, leading a team to develop products to advance people’s careers in the context of the hiring marketplace. He previously built and grew several high-performing data science teams at Acorns, Yirendai (NYSE:YRD), and Simply Hired. Dr Chong also advises several startups and teaches at Tsinghua University.

Yue Cathy Chang is an executive leader in business development, sales and marketing for data products and services. Cathy has held strategic roles serving customers using data science- and engineering-based solutions at bluechip software enterprise companies and startups. Well-versed in tech and business, she has frequently spoken at data and AI events.

The writing and opinions expressed are solely our own and are not shared, supported, or endorsed in any manner by our employers.

Thriving as a Data Scientist in the New Roaring Twenties

This article first appeared here on January 6, 2020

Image credit: designbolts.com

Ever since the job title was coined in 2008 by D.J. Patil and Jeff Hammerbacher, then the respective leads of data and analytics efforts at LinkedIn and Facebook, 145,000+ professionals around the world have entered the field and proudly present themselves as data scientists.

It’s 2020. How is this field fairing in the new decade?

The New Gold Rush

$95,000 is the median annual base salary of an entry level data scientist.

For the first time, data scientist has surpassed software engineer (at $90,000 median annual base salary) in 2019 to become the highest paying entry-level job in the United States.

If you are a data science practitioner who has recently entered the field, congratulations!

When a field is offering a 6-figure salary with 2 years of experience out of college, what happens next?

100+ schools are now offering online and on-campus Master’s Degree in data science, and more schools are offering data science undergraduate degrees.

10,000+ data science graduates will be joining the workforce each year, and more talents are attracted to the field from other disciplines through intensive fellowship programs and online MOOC courses!

With rapid growth often comes flocking, followed by job commoditization and reduction in ability to command pay. As many new entrants enters the field, entry-level data scientist salaries are already plateauing, according to a recent PayScale survey.

Similar commoditization trends have been observed in other job functions. In the legal space, after decades of strong growth, starting in 2010, there was an over-supply of new law school graduates. For the Class of 2011, the median starting salary for law school graduates fell by nearly 17% compared to Class of 2009.

If we don’t pay attention to what differentiates between ourselves and new data science graduates, we may well get left behind. How can we ride on the popularity of data science? More importantly, how can we stand out and stay competitive?

(Career) Challenges with Nimble Teams

With the large human capital investments required for data science teams, team size at most companies have stayed relatively nimble.

95% of the companies employing data scientists have teams of fewer than ten data scientists.

Many data scientists are embedded into business product lines, or business functions such as Marketing, Operations, and Customer Service.

How does one advance a career in data science when in a small team, or when embedded in a cross-functional team? What does such a career path look like?

For many data scientists, career advancement may mean starting as a data science intern, then data scientist, and then senior data scientist. But then what?

What do vice presidents of data science or chief data scientists do? How does one make a decision of whether to work toward these roles? And if one is interested in heading that way, what would some milestones look like?

Two things are clear:

1. Hard work alone is not going to get one to senior/executive data science roles
2. Only a small fraction of data scientists will get the opportunity to take on senior leadership roles to produce outsized impact and earn significant rewards

Many other things, are not as clear.

A Different Perspective

As tens of thousands of new data scientists join the race to develop impactful careers in data science each year, the field will become increasingly competitive and one will need more than tips, tricks, and ordinary career advice to thrive and advance a career in data science.

In this blog series, we explore insights, learning and best practices from leading data science practitioners on how one can present powerful narratives and act with appropriate moods in acquiring the necessary knowledge, skills, and experience over a key career span of five to fifteen years.

We reference sage learning and career advice that withstood the test of time as well as observations and interpretations from modern analytical psychology, all have long explored the topic of human and career maturity.

These blogs are by data science practitioners for data science practitioners. We strive to comprehensively explore topics on career advancement in data science. We will reference the path highlighted in Confucius teaching “The Great Learning” and how it speaks to the process of cultivating oneself, managing a team, governing a function, and inspiring an industry, as four major career milestones. The first topic will focus on the process of cultivating oneself.

Your input on the topics is always welcome. Please share your questions/topics here.

As we embark on a new decade, we look forward to exploring the paths to Data Science career growth with you! 

About the Authors:

Dr. Jike Chong is a director of data science at LinkedIn, leading a team to develop products to advance people’s careers in the context of the hiring marketplace. He previously built and grew several high-performing data science teams at Acorns, Yirendai (NYSE:YRD), and Simply Hired. Dr Chong also advises several startups and teaches at Tsinghua University.

Yue Cathy Chang is an executive leader in business development, sales and marketing for data products and services. Cathy has held strategic roles serving customers using data science- and engineering-based solutions at bluechip software enterprise companies and startups. Well-versed in tech and business, she has frequently spoken at data and AI events.

The writing and opinions expressed are solely our own and are not shared, supported, or endorsed in any manner by our employers.