Image Credit: medium.com
Remember Aayana, the student with great math and statistics skills, who had encountered some vague and challenging data science interview processes? Well, in her job search, she met Brian, another UCLA alum who has been working as a Data Scientist at a social networking company for the past two years. Brian has worked on a variety of data science projects and was recently tasked with partnering with marketing and product teams to develop a set of email campaigns aimed at driving long-term user engagement.
For many data science practitioners, working with email campaigns is nothing new. We often partner with marketing and product to:
- Identify a set of users
- Develop a set of content
- Run the email campaign
- Analyze the click-through rate of emails and the engagement on the landing page
For data-driven optimizations, we may test different subject lines, and experiment with different narratives in the email content to understand what would drive the highest conversion in engagement.
But is that all?
Last time we looked at how capabilities of data scientists have evolved and shared how virtues come into the picture, expanding a familiar DS Venn diagram to a TED-ERA Fan.
This time we will begin discussing Technology in TED-ERA, starting with mapping scenarios to known optimization techniques.
Mapping Scenarios to Known Optimization Techniques in Driving Engagement Conversion
In our January 12 blog, we saw how Aayana, who had encountered a variety of data science interview processes: in one case she was asked to write down an entire data pipeline in an interview with a FinTech Startup.
In academic settings, Aayana was familiar with course project setups where students go straight to solving the most academically challenging piece of the project: discovering patterns in the data. Many of the surrounding information is either specified in the problem statement or has been provided as part of the grading infrastructure.
In real-world data science, however, the picture looks very different. How do we map a scenario to known optimization techniques? How do we evaluate the success of a solution? These questions are just as important as the specifics of building a model.
Let’s dive a bit deeper.
Levels of Practice Toward Intelligence
First of all, are we asking questions about whether we are achieving our goal of driving long-term engagement?
One practice is to 1) form a hold-out set of users as part of an A/B Test, 2) run the campaign for a number of months, and 3) assess if long-term engagement has improved. In this case, we would be analyzing what happened a few months down the road. This kind of hindsightis NOT efficient in driving improvements in operations.
Another practice is to build a model predicting long-term engagement using detectable short-term engagement characteristics such as open rate, click-through-rate (CTR), unsubscribe rate, landing page session length, and session frequency, such that we can anticipate long term effects with short-term observation. This allows us to have the foresight to anticipate campaign impact and quickly adjust our email marketing strategies from week to week.
Yet more powerful approaches can include real-time analytics on email responses to learn what segments of email recipients who tend to engage more with which marketing campaigns, and allow the insights to drive additional tweaks in the email narrative.
The most powerful approach is to prepare sequences of emails aimed at driving long-term engagement that can adapt to user responses in real-time, and help guide the user along a path of actions to realize the most value they can gain from the product. This would be intelligence.
To better understand the distinctions between hindsight, foresight, insights, and intelligence, here is a summary in an easy-to-remember 2x2 matrix.
Figure 4.1 Levels of Practice Toward Intelligence (Source: V. A. Ganesan)
Additional Considerations
Secondly, is each email recipient independent? Are there potential network effects with the campaign?
When identifying users in a social network for email campaigns, there may be intentional or unintentional network effects. Simple assumptions may treat each user as independent recipients, and select them based on the typical demographic and product usage segmentations. However, if the marketing narrative involves information about users and their networks, or promotional offers can be shared with other users, we may need to consider potential network effects. The mapping of the optimization techniques may involve simple network properties such as network graph centrality, or more advanced struct2vec deep embeddings to identify specific cohorts of users that may be more suitable for a marketing campaign.
Thirdly, in developing a set of personalized content for an email campaign, there is often a repertoire of content or the next best actions to select from and personalize with. It could be a selection of connections you can make on the social networking platform, or a set of entertainment items you can engage with. Identifying the best items to include according to relevance, coverage, diversity, novelty, or serendipity metrics to generate long-term engagement is yet another set of ranking based optimization techniques one could map the email campaign to.
In addition, when sending out the content, are there cohorts of users with identifiable habits? Many users habitually engage with certain types of content on a particular time of day or day of the week. How might we optimize the email campaign sends such that our emails are at the top of users’ mailbox when they start checking their inbox?
Technology Considerations
These are many of the ways for Mapping a seemingly straight forward email campaign scenario to optimization techniques.
Mapping scenarios to optimization techniques is the first part of Technology in the TED-ERA Fan. Two other parts of Technology are Discovering patterns in data by training models, and Evaluating the success of a solution.
In upcoming blogs, we will discuss considerations in the other two parts.
If you have views, questions, or examples you would like to share, feel free to comment below or contact us directly.
About the authors:
Dr. Jike Chong is a director of data science at LinkedIn, leading a team to develop products to advance people’s careers in the context of the hiring marketplace. He previously built and grew several high-performing data science teams at Acorns, Yirendai (NYSE:YRD), and Simply Hired. Dr Chong also advises several startups and teaches at Tsinghua University.
Yue Cathy Chang is an executive leader in business development, sales and marketing for data products and services. Cathy has held strategic roles serving customers using data science- and engineering-based solutions at bluechip software enterprise companies and startups. Well-versed in tech and business, she has frequently spoken at data and AI events.
The writing and opinions expressed are solely our own and are not shared, supported, or endorsed in any manner by our employers.
No comments:
Post a Comment