70
DeepPage Framework ExperimentsMotivation
Deep Reinforcement Learning for Page-wise RecommendationsXiangyu Zhao1, Long Xia2, Liang Zhang2, Zhuoye Ding2, Dawei Yin2 and Jiliang Tang1
1: Data Science and Engineering Lab, Michigan State University 2: Data Science Lab, JD.com
7
Conclusion and Future Work
Framework Overview
Data Science and Engineering Lab
Recommender systems can mitigate the information overloadproblem by suggesting users’ personalized items
q Interactions between recommender systems and usersv Each time the system recommends a page of items to the user
v Next the user browses these items and provides real-time feedback
v Then the system recommends a new page of items…
q Two key problemsv How to efficiently capture user's dynamic preference and update
recommending strategy according to user's real-time feedback
v How to generate a page of items with proper display based on user's
preferences
We model the recommendation task as a MDP and leverage Reinforcement Learning to automatically learn the optimal recommendation strategies
1. Framework Architecture Selection
action ai
h1
h2
· · ·
(c)(b)(a) Actor Critic
state s
action a
Q(s, ai)Q(s, a2)Q(s, a1) Q(s, a)
action a
h2 h2h2
h1 h1 h1
state s state s state s
ü Actor-Critic Frameworkü Actor: current state -> a deterministic action (recommend a deterministic
page of items)
ü Critic: this state-action pair -> Q-value
✕ (a) cannot handle large and dynamic action space scenario
✕ (b) high computational cost
Q(s, a) = Es′!
r + γQ(s′, a′)|s, a"
Q∗(s, a) = Es′!
r + γmaxa′
Q∗(s′, a′)|s, a"
2. Markov Decision Process
1. Architecture of Actor Framework
1.3. Decoder for Action Generation Process
2. The Architecture of Critic Framework
· · · · · ·
· · · · · ·
eNe2e1
hNh2h1
Embedding Layer
Input Layer
sini
GRU Layer
Output Layer
ENE2E1
zt = σ(WzEt + Uzht−1)rt = σ(WrEt + Urht−1)ht = tanh[WEt + U(rt · ht−1)]ht = (1− zt)ht−1 + ztht
sini = ht
· · · · · ·
· · · · · ·
· · · · · ·
sini
Embedding Layer
Input Layer
GRU Layer
CNN Layer
scur
Attention Layer
h2h1 hT
Page Layer
Output Layer
fiei ci
Xi
XM−1
X2X1
XM· · ·
· · ·
αTα2α1
P1
p1
· · ·· · · · · ·
ei : item representationci : item
′s categoryfi : user
′s feedback
Xi = concat(Ei, Ci, Fi)
= tanh!
concat(WEei + bE,WCci + bC,WFfi + bF )"
pt = conv2d(Pt)
scur
=
T!
t=1
αtht αt =exp(Wαht + bα)!j exp(Wαhj + bα)
1.1. Encoder for Initial State Generation Process
q The Actor is designed to generate a page of recommendations according to
user's preference, which needs to tackle three challenges
v Setting an initial preference at the beginning of a new recommendation session
v Learning the real-time preference in the current session
v Jointly generating a set of recommendations and displaying them in a 2-D page
1.2. Encoder for Real-time State Generation Process
q Setting an initial preference at the beginning of a new session
v Inputs: user’s last clicked/ordered items before the current session
v Output: the representation of users’ initial preference by a vector
q Learning the real-time preference in the current session
v Inputs: item representation, item’s category (to generate complementary
recommendations) and user’s feedback (to capture user’s interests in current page)
v Output: the representation of users’ real-time preference by a vector
q Jointly generating a set of recommendations and displaying them in a page
v Inputs: users’ real-time preference
v Output: the representation of a page of recommendations (items) with proper display
q Critic aims to learn an action-value function Q(s,a), which is a judgment of
whether the action “a” matches the current state “s”v Inputs: users’ real-time preference “s” and a page of recommendations (items) “a”
v Output: Q-value, i.e., Q(s,a)
h1
h2
acur
pro
CNN
Actor Critic
User
eM
e2e1 · · ·
· · ·
eM−1
e2e1
eM
acurval Q(s, a)
ars s
· · · · · ·
· · · eM−1
· · ·
· · · · · · · · ·
· · ·
Decoderacur = deconv2d(scur) a = conv2d(acur)
1. How DeepPage performs compared to representative baselines?
2. How the components in Actor and Critic contribute to the performance?
Ø Dataset from JD.com
Ø Baselines
Ø Collaborative filtering
Ø Factorization Machines
Ø GRU for Rec
Ø DQN
Ø DDPG
training process
Ø 1: no embedding-layers
Ø 2: no category and feedback
Ø 3: no GRU for initial state
Ø 4: no CNNs
Ø 5: no attention layers
Ø 6: no GRU for real-time state
Ø 7: no DeCNN
q We make following observations:
v DeepPage-1 and DeepPage-2 validate that incorporating category/feedback information and the
embedding layer can boost the quality of recommendation
v DeepPage-3 and DeepPage-6 suggest that setting user’s initial preference, and capturing user’s
real-time preference is helpful for accurate recommendations
v DeepPage outperforms DeepPage-4 and DeepPage-7, which indicates that item display strategy
can influence the decision making process of users
q It can be observed:
v CF and FM perform worse than other baselines. These two baselines ignore the temporal
sequence of the users' browsing history
v DQN and DDPG outperform GRU. We design GRU to maximize the immediate reward for
recommendations, while DQN and DDPG are designed to achieve the trade-off between short-
term and long-term rewards
v DeepPage performs better than conventional DDPG. Compared to DDPG, DeepPage jointly
optimizes a page of items and uses GRU to learn user's real-time preference
q Conclusionv We propose a novel Actor-Critic framework, which can be applied in scenarios with
large and dynamic item space and can reduce computational cost significantly
v DeepPage framework is able to capture the real-time preference and optimize a
page of items jointly, which can boost recommendation performance
q Future Work
v We would like to handle multiple tasks such as search, bidding, advertisement and
recommendation collaboratively in one reinforcement learning framework
v We would like to validate with more agent-user interaction patterns, e.g., adding
items into shopping cart
q Acknowledgements
v This research is supported by National Science Foundation
(IIS-1714741, IIS-1715940) and Criteo Faculty Research Award
My homepage
Deconvolution Neural Network
Convolution Neural Network
