Personalization in Practice: Methods and Applications

Dmitri Goldenberg Kostia Kofman Javier Albert

[email protected] [email protected] [email protected]
Booking.com, Tel Aviv, Israel Booking.com, Tel Aviv, Israel Booking.com, Tel Aviv, Israel

Sarai Mizrachi Adam Horowitz Irene Teinemaa

[email protected] [email protected] [email protected]
Booking.com, Tel Aviv, Israel Booking.com, Tel Aviv, Israel Booking.com, Amsterdam,
Netherlands
ABSTRACT 1 BACKGROUND
Personalization is one of the key applications in machine learning Personalization is omnipresent in our life, with applications rang-
with widespread usage across e-commerce, entertainment, produc- ing from entertainment and commercial uses to smart devices and
tion, healthcare and many other industries. While various machine medical treatments. The integration of personalization in various
learning techniques present novel state-of-the-art advances and products turned rapidly from an unnecessary luxury to a commod-
super-human performance year-over-year, personalization and rec- ity that is expected by customers. Due to its interactive nature, and
ommender systems applications are often late-adopters of novel its strong affect on user response, the sub-field of machine learning
solutions due to problem hardness and implementation complexity. for personalization and recommendations have taken unique devel-
This tutorial presents recent advances across the personaliza- opment routes in recent years. For almost a decade, collaborative
tion industry and demonstrates their practical applications in real filtering solutions, with advanced work on matrix factorization
case-studies of world-leading online platforms. Key trends such as [30] and factorization machines [43], have been the main areas
deep learning, causality and active exploration with bandits are of progress in the field, ignited by the Netflix challenge at 2007
depicted with real examples and demonstrated alongside their busi- [7]. While other machine learning areas offer super-human perfor-
ness considerations and implementation challenges. Rising topics mance, personalization applications often involve more complexity
like explainability, fairness, natural interfaces and content gener- both in defining learning problems and in production implementa-
ation are covered, touching on aspects of both technology and tion [9].
user experience. Our tutorial relies on recent advances in the field The popular trend of deep learning earned its place among the
and on work conducted at Booking.com, where we implement per- state-of-the-art methods only in the last few years, bridging the gap
sonalization models on one of the world’s leading online travel between complex data structures and implementation challenges.
platform. Many companies use deep learning in recommendation systems -
for example, in order to predict click-through rate, Google applies
CCS CONCEPTS wide and deep learning [15], while Alibaba applies deep Interest net-
• Information systems → Personalization; Recommender sys- works [54]. Twitter uses deep learning in order to predict customer
tems. purchasing [31].
Deep learning models are also used for adding non-linearity to
KEYWORDS traditional recommender system techniques, such as Collaborative
Filtering [26, 35] and Factorization Machines [24] which are essen-
Personalization, Causality, Deep Learning, Recommender Systems,
tially linear. In the era of visual recognition tasks, deep learning
Active Exploration, Natural Experience, Content Generation, Fair-
and, particularly, convolutional neural networks (CNN) enhance
ness, Explainability, Uplift Modeling
performance [25] and push the boundaries of content-based rec-
ACM Reference Format: ommendation paradigm [39]. In recent years, use of sequential
Dmitri Goldenberg, Kostia Kofman, Javier Albert, Sarai Mizrachi, Adam models [48] has increased in the field of recommendation system
Horowitz, and Irene Teinemaa. 2021. Personalization in Practice: Methods
[10, 14, 33, 47]. Recent studies show how to use contextual data in
and Applications. In Proceedings of the Fourteenth ACM International Confer-
ence on Web Search and Data Mining (WSDM ’21), March 8–12, 2021, Virtual
a Recurrent Neural Network (RNN) based recommender system
Event, Israel. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/ [22, 38].
3437963.3441657 Another feature of personalization and recommendation systems
are their non-static setting - both in terms of items being recom-
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
mended and user preferences for items - and the resulting need
for profit or commercial advantage and that copies bear this notice and the full citation for adaptive solutions to avoid performance degradation over time.
on the first page. Copyrights for components of this work owned by others than the One approach that allows adaptiveness is utilizing Multi-Armed
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission Bandits and Contextual Bandits [46]. These sets of models and algo-
and/or a fee. Request permissions from [email protected]. rithms balance between exploration of items with low confidence
WSDM ’21, March 8–12, 2021, Virtual Event, Israel estimates and exploitation of items with high confidence, with user
© 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-8297-7/21/03. . . $15.00 feedback constantly fed into the model and updating the model
https://doi.org/10.1145/3437963.3441657 parameters [13, 23, 50]. Bandit solutions have a connection with
causal inference in the sense of learning the effect of an item from 2.2 From Theory to Applications (170 mins)
a set of possible items, with or without context. In this section we deep dive into three practical applications of
Causal inference is a key tool for personalization on online e- personalization and demonstrate the underlying theory as well as
commerce platforms. In particular, uplift modeling [12, 17] is used implementation case studies:
to estimate the causal effect of a treatment on the outcome of
individuals, such as subscribing to a service or completing a pur- 2.2.1 Sequence Modeling Recommenders (40 mins).
chase. The treatment effect estimation, conducted via various meta- • Introduction to RNNs and sequence modeling
learning techniques [17, 32], enables to select the best treatment • Sequence-aware recommendation [38]
for each user in order to maximize target business metrics. Some re- • Applications of sequences for recommendation [10, 14, 47]
cent work, aimed at developing cost-aware modeling, incorporates
value optimization within the meta-learners [53] [21], resulting on 2.2.2 Personalization via Uplift Modeling (90 mins).
personalization techniques that can be applied to marketing and • Introduction to Causality
promotional campaigns. • Uplift modeling and meta learners [17]
Other notable trends, such as explainability [36, 52], fairness • Evaluating uplift models [42]
[11, 20] and conversational recommenders [51] are adding more • Uplift modeling with cost optimization
complexity to systems simultaneously accounting for business met- • Promotions personalization application [21]
rics optimization and customer experience.
As personalization becomes more of an expected norm across all 2.2.3 Contextual Bandits in Recommender systems (40 mins).
types of online platforms and services each year, it comes with in-
• Introduction to MAB and Contextual Bandits [46]
creasing risks. Just a small list of these includes: inaccurate modeling
• Linear Bandits in Recommender Systems [34]
that effectively reduces consumer use and/or purchasing behavior
• Scalable Contextual Bandits [13, 23, 50]
[4]; inadvertently bringing privacy and data security concerns to
• Exploration and Randomness in Deep Learning [19, 44]
the forefront of consumers’ attention [1, 2]; or simple failure to
provide relevance to users’ behaviors and/or needs [28]. Companies
in various fields are newly-defining best practices and approaches 2.3 Personalization User Perception (40 mins)
for studying the role, perception and reaction of the person in order We conclude our tutorial with an investigation of how user per-
to guide modeling decision-making and to offset these risks [29]. ceptions and expectations influence the success of personalization
efforts, including discussing emerging trends in machine learning
and artificial intelligence user experience research:
2 TUTORIAL OUTLINE
(1) Variations in user perceptions
The outline of the tutorial is as follows. We begin by introducing
(2) Driving application decisions via product-market fit
the traditional personalization landscape and recommendation sys-
(3) User-centric data for modeling
tems, along with their unique challenges in implementations, and
(4) Testing and optimizing personalisation user experience
then cover key recent trends. We continue with a deep dive into
three state-of-the-art personalization solutions, covering theory
and practical implementations in the field. Lastly, we overview the 3 RELEVANCE TO THE COMMUNITY
topic from the user perspective, covering the relevance of target Personalization is a key application of the web-search and data
users’ perceptions of personalization. The total duration of the tu- mining fields. As such, it has topical importance for both research
torial is four and a half hours. The detailed format and schedule are and practical implementation. Our aim is to briefly cover the topic
as follows: of personalization by introducing recent, nontrivial trends in the
field and by demonstrating possible applications. The main focuses
of the tutorial are: (i) to survey key trends in personalization; (ii) to
2.1 Trends in Personalization (60 mins) introduce a selection of advanced techniques; (iii) to demonstrate
We begin by introducing the traditional personalization landscape practical personalization implementations.
and recommendation systems, along with their unique challenges This tutorial is suitable both as an introduction for new prac-
in implementations. We recount six trends in personalization, dis- titioners in the field and to bring experts up to date with recent
cussing their recent relevance, underlying research work, and prac- research and walk-throughs of novel applications.
tical applications:
3.1 Target Audience
(1) Deep learning applications in personalization
(2) Causality and causal inference The tutorial is designed to introduce practitioners to key concepts
(3) Active exploration techniques in personalization and expose them to recent advances and imple-
(4) Explainability and fairness mentation considerations in personalized machine learning-driven
(5) Natural interaction interfaces applications. The participants are expected to have basic famil-
(6) Personalized content generation iarity with core concepts in machine learning, deep learning and
recommender systems. However, in-depth prior knowledge is not
We conclude the first part of the tutorial with predictions of future mandatory - many advanced topics will be introduced during the
trends and examples of early-stage research and applications. tutorial.
3.2 Related Tutorials B.Sc. in Mathematics from the Ben-Gurion University, M.A. in Eco-
The general personalization and recommender systems area, along nomics from the Tel Aviv University. He is currently pursuing an
with dedicated tutorials to specific sub-topics (such as sequence- M.Sc in Statistics and Data Science in Tel-Aviv University. Kofman
modeling, exploration, causality and fairness) are a frequent com- has experience teaching undergraduate courses.
ponent of major applied data science conferences. The proposed Javier Albert is a Machine Learning Scientist at Booking.com in
tutorial was inspired by Justin Basilico’s ICML’19 AMTL workshop Tel Aviv, where he uses uplift modeling to enable the financial via-
[6], which focused on recent trends in personalization from the bility of large-scale promotional campaigns. His work was recently
Netflix perspective. published at ACM RecSys 2020. Albert has a M.Sc. in Electrical
Notable tutorials, such as by Berkovsky and Freyne at KDD’15 [8], Engineering from Tel Aviv University and two B.Sc. degrees from
cover the personalization field as a whole, presenting fundamental Technion - Israel Institute of Technology.
concepts such as collaborative filtering, paradigms of recommender Sarai Mizrachi is a Machine Learning Scientist at Booking.com.
systems and content based personalization. A tutorial by Amatriain She specializes in using deep learning models to develop recommen-
and Agarwal at RecSys’16 [3] tackles real-life implementation of dation systems in the connected trip domain. Mizrachi has an M.Sc.
advanced personalization systems, with introduction to concepts in Industrial Engineering from Ben Gurion University. She has a
such as factorization machines, deep learning and multi-armed rich academic teaching and conference presentations background,
bandits. and has published in leading peer-review journals and conferences,
More recently, at KDD’20 [37], the authors extended classical such as PLoS One and RecSys.
concepts to current trends, discussing the complexity of dynamic Adam Horowitz is a User Researcher at Booking.com’s Ma-
recommender systems in a multi-stakeholder setup and diving into chine Learning Center in Tel Aviv. Horowitz received his Ph.D.
exploratory bandits models. Another recent tutorial by Quadrana et. in Sociology from Stanford University, where he specialized in
al at RecSys and WWW [40, 41] was dedicated solely to advanced the sociology of science and technology. He was subsequently
topics in Sequence-Aware Recommender Systems. The topic of Post-Doctoral Fellow in Science, Ethics, and Democracy at the
causal inference is trending as well, with recent tutorials at WSDM, Edmond J. Safra Center for Ethics, Tel Aviv University. Horowitz
CoDS COMAD and KDD [16, 27, 45]. These works present the im- has published peer-reviewed articles in leading academic journals,
portance of causality to web applications and introduce the strong including American Sociological Review, Sociological Science, and
relationship between causality and machine learning. New Genetics and Society. At Stanford, Horowitz was instructor of
Similarly, other recent trends - including exploration with bandit- record for five courses and teaching assistant for seven courses, and
based recommenders [5, 18]), deep-learning in recommendations was the university-wide recipient of the Walter J. Gores Award for
[49], explainability [52], fairness [11, 20] and conversational rec- excellence in teaching. Horowitz was formerly a UX Researcher at
ommenders [51] - are present in recent major applied data science Facebook. At Booking.com, he works to advance best practices for
conferences (WSDM, WWW, RecSys). These tutorials introduce Machine Learning User Experience Research (ML-UXR), as part of
the main concepts of each field and guide the audience through an industry-wide endeavor.
numerous applied examples, similar to the our intended setup. Irene Teinemaa is a Machine Learning Scientist at Booking.com
in Amsterdam. She obtained her PhD on the topic of predictive and
3.3 Supporting Materials prescriptive monitoring of business processes from University of
Tartu. Her current work is focused on uplift modeling and online
Tutorial participants will be provided with supplemental materials, experimentation for personalized marketing campaigns. She is one
including tutorial slides, references to additional materials and of the trainers of an internal causal inference training and has prior
applied open-source code packages, for covered personalization experience with academic teaching. She has published her work
techniques. in top venues such as TKDD, TIST, Data Mining and Knowledge
Discovery, ECML/PKDD and ASONAM.
4 PRESENTERS’ BIOGRAPHIES
Dmitri (Dima) Goldenberg is a Machine Learning Science Team REFERENCES
Lead at Booking.com, Tel Aviv, where he leads modeling efforts [1] 2020. People AI Research. https://pair.withgoogle.com/guidebook
[2] Elizabeth Aguirre, Dominik Mahr, Dhruv Grewal, Ko de Ruyter, and Martin
in cross-product offering and promotions personalization, utiliz- Wetzels. 2015. Unraveling the personalization paradox: The effect of information
ing online learning and uplift modeling techniques. Goldenberg collection and trust-building strategies on online advertisement effectiveness.
obtained his Masters in Industrial Engineering (with honors) from Journal of retailing 91, 1 (2015), 34–49.
[3] Xavier Amatriain and Deepak Agarwal. 2016. Tutorial: Lessons learned from
Tel Aviv University, focusing on influence maximization in social building real-life recommender systems. In Proceedings of the 10th ACM Confer-
networks. He was included in the Rector’s List for excellence in ence on Recommender Systems. 433–433.
undergraduate course instruction. Goldenberg is the co-founder [4] Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi,
Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. 2019.
of Wize, an educational-social initiative that promotes informal Guidelines for human-AI interaction. In Proceedings of the 2019 chi conference on
scientific talks in a recreational setup. His work presented and pub- human factors in computing systems. 1–13.
[5] Andrea Barraza-Urbina and Dorota Glowacka. 2020. Introduction to Bandits in
lished in top journals and conferences including ESWA, RecSys, Recommender Systems. In Proceedings of the 14th ACM Conference on Recom-
IEEE TCSS, NetSci and ASONAM. mender Systems (RecSys ’20). Association for Computing Machinery.
Kostia Kofman is a Machine Learning Science Team Lead at [6] Justin Basilico. 2019. Recent Trends in Personalization: A Netflix Perspective.
ICML.
Booking.com, Tel Aviv, leading personalization and ranking model- [7] Robert M Bell and Yehuda Koren. 2007. Lessons from the Netflix prize challenge.
ing efforts. Kofman has a broad academic background, holding a Acm Sigkdd Explorations Newsletter 9, 2 (2007), 75–79.
 [8] Shlomo Berkovsky and Jill Freyne. 2015. Web personalization and recommender 309–352.
systems. In Proceedings of the 21th ACM SIGKDD International Conference on [30] Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix factorization tech-
Knowledge Discovery and Data Mining. 2307–2308. niques for recommender systems. Computer 42, 8 (2009), 30–37.
[9] Lucas Bernardi, Themistoklis Mavridis, and Pablo Estevez. 2019. 150 Successful [31] Mandy Korpusik, Shigeyuki Sakaki, Francine Chen, and Chen Yan-Ying. 2016.
Machine Learning Models: 6 Lessons Learned at Booking. com. In Proceedings of Recurrent Neural Networks for Customer Purchase Prediction on Twitter. In
the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data CBRecSys 2016.
Mining. 1743–1751. [32] Sören R Künzel, Jasjeet S Sekhon, Peter J Bickel, and Bin Yu. 2019. Metalearners for
[10] Alex Beutel, Paul Covington, Sagar Jain, Can Xu, Jia Li, Vince Gatto, and Ed H estimating heterogeneous treatment effects using machine learning. Proceedings
Chi. 2018. Latent cross: Making use of context in recurrent recommender systems. of the national academy of sciences 116, 10 (2019), 4156–4165.
In Proceedings of the Eleventh ACM International Conference on Web Search and [33] Tobias Lang and Matthias Rettenmeier. 2017. Understanding consumer behavior
Data Mining. 46–54. with recurrent neural networks. In Workshop on Machine Learning Methods for
[11] Sarah Bird, Krishnaram Kenthapadi, Emre Kiciman, and Margaret Mitchell. 2019. Recommender Systems.
Fairness-aware machine learning: Practical challenges and lessons learned. In [34] Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. 2010. A contextual-
Proceedings of the Twelfth ACM International Conference on Web Search and Data bandit approach to personalized news article recommendation. In WWW.
Mining. 834–835. [35] Dawen Liang, Rahul G. Krishnan, Matthew D. Hoffman, and Tony Jebara. 2018.
[12] Zahra Dasht Bozorgi, Irene Teinemaa, Marlon Dumas, Marcello La Rosa, and Variational Autoencoders for Collaborative Filtering. arXiv:stat.ML/1802.05814
Artem Polyvyanyy. 2020. Process Mining Meets Causal Machine Learning: [36] James McInerney, Benjamin Lacker, Samantha Hansen, Karl Higley, Hugues
Discovering Causal Rules from Event Logs. In 2020 2nd International Conference Bouchard, Alois Gruson, and Rishabh Mehrotra. 2018. Explore, exploit, and
on Process Mining (ICPM). IEEE, 129–136. explain: personalizing explainable recommendations with bandits. In Proceedings
[13] Olivier Chapelle and Lihong Li. 2011. An Empirical Evaluation of Thompson of the 12th ACM Conference on Recommender Systems. 31–39.
Sampling. In Proceedings of the 24th International Conference on Neural Information [37] Rishabh Mehrotra, Ben Carterette, Yong Li, Quanming Yao, Chen Gao, James
Processing Systems (NIPS’11). Curran Associates Inc., USA, 2249–2257. Kwok, Qiang Yang, and Isabelle Guyon. 2020. Advances in Recommender Systems:
[14] Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, and Wenwu Ou. 2019. Behavior From Multi-stakeholder Marketplaces to Automated RecSys. In Proceedings of
sequence transformer for e-commerce recommendation in alibaba. In Proceedings the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data
of the 1st International Workshop on Deep Learning Practice for High-Dimensional Mining. 3533–3534.
Sparse Data. 1–4. [38] Sarai Mizrachi and Pavel Levin. 2019. Combining Context Features in Sequence-
[15] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Aware Recommender Systems.. In RecSys (Late-Breaking Results). 11–15.
Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. [39] Michael J Pazzani and Daniel Billsus. 2007. Content-based recommendation
2016. Wide & deep learning for recommender systems. In Proceedings of the 1st systems. In The adaptive web. Springer, 325–341.
workshop on deep learning for recommender systems. 7–10. [40] Massimo Quadrana and Paolo Cremonesi. 2018. Sequence-Aware Recommen-
[16] Peng Cui, Zheyan Shen, Sheng Li, Liuyi Yao, Yaliang Li, Zhixuan Chu, and Jing dation. In Proceedings of the 12th ACM Conference on Recommender Systems
Gao. 2020. Causal Inference Meets Machine Learning. In Proceedings of the 26th (RecSys ’18). Association for Computing Machinery, New York, NY, USA, 539–540.
ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. https://doi.org/10.1145/3240323.3241621
3527–3528. [41] Massimo Quadrana, Dietmar Jannach, and Paolo Cremonesi. 2019. Tutorial:
[17] Floris Devriendt, Darie Moldovan, and Wouter Verbeke. 2018. A literature survey Sequence-Aware Recommender Systems. In Companion Proceedings of The 2019
and experimental evaluation of the state-of-the-art in uplift modeling: A stepping World Wide Web Conference. 1316–1316.
stone toward the development of prescriptive analytics. Big data 6, 1 (2018), [42] Nicholas J Radcliffe. 2007. Using control groups to target on predicted lift: Building
13–41. and assessing uplift models. Direct Marketing Analytics Journal 1 (2007), 1421.
[18] Paolo Dragone, Rishabh Mehrotra, and Mounia Lalmas. 2019. Deriving User- [43] Steffen Rendle. 2010. Factorization machines. In 2010 IEEE International Confer-
and Content-specific Rewards for Contextual Bandits. In The World Wide Web ence on Data Mining. IEEE, 995–1000.
Conference. 2680–2686. [44] Eren Sezener, Marcus Hutter, David Budden, Jianan Wang, and Joel Veness.
[19] Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian Approximation: 2020. Online Learning in Contextual Bandits using Gated Linear Networks.
Representing Model Uncertainty in Deep Learning. arXiv:stat.ML/1506.02142 arXiv:cs.LG/2002.11611
[20] Ruoyuan Gao and Chirag Shah. 2020. Counteracting Bias and Increasing Fairness [45] Amit Sharma and Emre Kiciman. 2020. Causal Inference and Counterfactual
in Search and Recommender Systems. In Proceedings of the 14th ACM Conference Reasoning. In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD. 369–370.
on Recommender Systems (RecSys ’20). Association for Computing Machinery. [46] Aleksandrs Slivkins. 2019. Introduction to Multi-Armed Bandits. CoRR
[21] Dmitri Goldenberg, Javier Albert, Lucas Bernardi, and Pablo Estevez. 2020. Free abs/1904.07272 (2019). arXiv:1904.07272 http://arxiv.org/abs/1904.07272
Lunch! Retrospective Uplift Modeling for Dynamic Promotions Recommendation [47] Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang.
within ROI Constraints.. In Proceedings of the 14th ACM Conference on Recom- 2019. BERT4Rec: Sequential recommendation with bidirectional encoder rep-
mender Systems. resentations from transformer. In Proceedings of the 28th ACM International
[22] Dmitri Goldenberg, Kostia Kofman, Pavel Levin, Sarai Mizrachi, Maayan Kafry, Conference on Information and Knowledge Management. 1441–1450.
and Guy Nadav. 2021. Booking.com WebTour 2021 Data Challenge. http: [48] Niek Tax, Irene Teinemaa, and Sebastiaan J van Zelst. 2020. An interdisciplinary
//www.bookingchallenge.com. In ACM WSDM Workshop on Web Tourism (WSDM comparison of sequence modeling methods for next-element prediction. Software
Webtour’21). and Systems Modeling 19, 6 (2020), 1345–1365.
[23] Thore Graepel, Joaquin Candela, Thomas Borchert, and Ralf Herbrich. 2010. Web- [49] Jun Xu, Xiangnan He, and Hang Li. 2018. Deep learning for matching in search
Scale Bayesian Click-Through Rate Prediction for Sponsored Search Advertising and recommendation. In The 41st International ACM SIGIR Conference on Research
in Microsoft’s Bing Search Engine. 13–20. & Development in Information Retrieval. 1365–1368.
[24] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. [50] Xiaotian Yu, Michael R. Lyu, and Irwin King. 2017. CBRAP: Contextual Bandits
DeepFM: A Factorization-Machine based Neural Network for CTR Prediction. with RAndom Projection. In AAAI.
1725–1731. https://doi.org/10.24963/ijcai.2017/239 [51] Yongfeng Zhang, Zuohui Fu, Yikun Xian, and Yi Zhang. 2020. Tutorial on Con-
[25] Benjamin Gutelman and Pavel Levin. 2020. Efficient Image Gallery Represen- versational Recommender Systems. In Proceedings of the 14th ACM Conference on
tations at Scale Through Multi-Task Learning. In Proceedings of the 43th Inter- Recommender Systems (RecSys ’20). Association for Computing Machinery.
national ACM SIGIR Conference on Research and Development in Information [52] Yongfeng Zhang, Jiaxin Mao, and Qingyao Ai. 2019. WWW’19 Tutorial on
Retrieval. Explainable Recommendation and Search. In Companion Proceedings of The 2019
[26] Balázs Hidasi, Massimo Quadrana, Alexandros Karatzoglou, and Domonkos World Wide Web Conference. 1330–1331.
Tikk. 2016. Parallel Recurrent Neural Network Architectures for Feature-Rich [53] Zhenyu Zhao and Totte Harinen. 2019. Uplift modeling for multiple treatments
Session-Based Recommendations. In Proceedings of the 10th ACM Conference on with cost optimization. In 2019 IEEE International Conference on Data Science and
Recommender Systems (RecSys ’16). Association for Computing Machinery, New Advanced Analytics (DSAA). IEEE, 422–431.
York, NY, USA, 241–248. https://doi.org/10.1145/2959100.2959167 [54] Guorui Zhou, Chengru Song, Xiaoqiang Zhu, et al. 2018. Deep Interest Network
[27] Emre Kiciman and Amit Sharma. 2019. Causal Inference and Counterfactual Rea- for Click-Through Rate Prediction. In KDD 2018.
soning (3hr Tutorial). In Proceedings of the Twelfth ACM International Conference
on Web Search and Data Mining. 828–829.
[28] Bart P Knijnenburg, Niels JM Reijmer, and Martijn C Willemsen. 2011. Each to his
own: how different users call for different interaction methods in recommender
systems. In Proceedings of the fifth ACM conference on Recommender systems.
141–148.
[29] Bart P Knijnenburg and Martijn C Willemsen. 2015. Evaluating recommender
systems with user experiments. In Recommender Systems Handbook. Springer,