Policy-Based Reinforcement Learning
Shusen Wang
Policy Function Approximation
Policy Function 𝜋 𝑎 𝑠
• Policy function 𝜋 𝑎 𝑠 is a probability density function (PDF).
• It takes state 𝑠 as input.
• It output the probabilities for all the actions, e.g.,
𝜋 left 𝑠 = 0.2,
𝜋 right 𝑠 = 0.1,
𝜋 up 𝑠 = 0.7.
• Randomly sample action 𝑎 random drawn from the distribution.
Can we directly learn a policy function 𝜋 𝑎 𝑠 ?
• If there are only a few states and actions, then yes, we can.
• Draw a table (matrix) and learn the entries.
Action 𝑎5 Action 𝑎6 Action 𝑎7 Action 𝑎8 ⋯
State 𝑠5
State 𝑠6
State 𝑠7
⋮
Can we directly learn a policy function 𝜋 𝑎 𝑠 ?
• If there are only a few states and actions, then yes, we can.
• Draw a table (matrix) and learn the entries.
• What if there are too many (or infinite) states or actions?
Action 𝑎5 Action 𝑎6 Action 𝑎7 Action 𝑎8 ⋯
State 𝑠5
State 𝑠6
State 𝑠7
⋮
Policy Network 𝜋 𝑎 𝑠; 𝛉
Policy network: Use a neural net to approximate 𝜋 𝑎|𝑠 .
• Use policy network 𝜋 𝑎|𝑠; 𝛉 to approximate 𝜋 𝑎|𝑠 .
• 𝛉: trainable parameters of the neural net.
Policy Network 𝜋 𝑎 𝑠; 𝛉
Policy network: Use a neural net to approximate 𝜋 𝑎|𝑠 .
• Use policy network 𝜋 𝑎|𝑠; 𝛉 to approximate 𝜋 𝑎|𝑠 .
• 𝛉: trainable parameters of the neural net.
“left”, 0.2
Conv Dense Softmax
“right”, 0.1
“up”, 0.7
state 𝑠>
feature
Policy Network 𝜋 𝑎 𝑠; 𝛉
• ∑C∈𝒜 𝜋 𝑎 𝑠; 𝛉 = 1.
• Here, 𝒜 = “left”, “right”, “up” is the set all actions.
• That is why we use softmax activation.
“left”, 0.2
Conv Dense Softmax
“right”, 0.1
“up”, 0.7
state 𝑠>
feature
State-Value Function Approximation
Action-Value Function
Definition: Discounted return.
• 𝑈> = 𝑅> + 𝛾 ⋅ 𝑅>L5 + 𝛾 6 ⋅ 𝑅>L6 + 𝛾 7 ⋅ 𝑅>L7 + ⋯
• The return depends on actions 𝐴> , 𝐴>L5, 𝐴>L6, ⋯ and states 𝑆> , 𝑆>L5, 𝑆>L6, ⋯
• Actions are random: ℙ 𝐴 = 𝑎 | 𝑆 = 𝑠 = 𝜋 𝑎 𝑠 . (Policy function.)
• States are random: ℙ 𝑆 P = 𝑠 P |𝑆 = 𝑠, 𝐴 = 𝑎 = 𝑝 𝑠 P 𝑠, 𝑎 . (State transition.)
Action-Value Function
Definition: Discounted return.
• 𝑈> = 𝑅> + 𝛾 ⋅ 𝑅>L5 + 𝛾 6 ⋅ 𝑅>L6 + 𝛾 7 ⋅ 𝑅>L7 + ⋯
Definition: Action-value function.
• 𝑄S 𝑠> , 𝑎> = 𝔼 𝑈> |𝑆> = 𝑠> , 𝐴> = 𝑎> .
The expectation is taken w.r.t.
actions 𝐴>L5, 𝐴>L6, 𝐴>L7, ⋯
and states 𝑆>L5, 𝑆>L6, 𝑆>L7, ⋯
State-Value Function
Definition: Discounted return.
• 𝑈> = 𝑅> + 𝛾 ⋅ 𝑅>L5 + 𝛾 6 ⋅ 𝑅>L6 + 𝛾 7 ⋅ 𝑅>L7 + ⋯
Definition: Action-value function.
• 𝑄S 𝑠> , 𝑎> = 𝔼 𝑈> |𝑆> = 𝑠> , 𝐴> = 𝑎> .
Definition: State-value function.
• 𝑉S 𝑠> = 𝔼V 𝑄S 𝑠> , 𝐴 = ∑C 𝜋 𝑎 𝑠> ⋅ 𝑄S 𝑠> , 𝑎 .
Integrate out action 𝐴~𝜋(⋅ |𝑠> ).
State-Value Function
Definition: Discounted return.
• 𝑈> = 𝑅> + 𝛾 ⋅ 𝑅>L5 + 𝛾 6 ⋅ 𝑅>L6 + 𝛾 7 ⋅ 𝑅>L7 + ⋯
Definition: Action-value function.
• 𝑄S 𝑠> , 𝑎> = 𝔼 𝑈> |𝑆> = 𝑠> , 𝐴> = 𝑎> .
Definition: State-value function.
• 𝑉S 𝑠> = 𝔼V 𝑄S 𝑠> , 𝐴 = ∑C 𝜋 𝑎 𝑠> ⋅ 𝑄S 𝑠> , 𝑎 .
Integrate out action 𝐴~𝜋(⋅ |𝑠> ).
Policy-Based Reinforcement Learning
Definition: State-value function.
• 𝑉S 𝑠> = 𝔼V 𝑄S 𝑠> , 𝐴 = ∑C 𝜋 𝑎 𝑠> ⋅ 𝑄S 𝑠> , 𝑎 .
Policy-Based Reinforcement Learning
Definition: State-value function.
• 𝑉S 𝑠> = 𝔼V 𝑄S 𝑠> , 𝐴 = ∑C 𝜋 𝑎 𝑠> ⋅ 𝑄S 𝑠> , 𝑎 .
Approximate state-value function.
• Approximate policy function 𝜋 𝑎 𝑠> by policy network 𝜋 𝑎|𝑠> ; 𝛉 .
Policy-Based Reinforcement Learning
Definition: State-value function.
• 𝑉S 𝑠> = 𝔼V 𝑄S 𝑠> , 𝐴 = ∑C 𝜋 𝑎 𝑠> ⋅ 𝑄S 𝑠> , 𝑎 .
Approximate state-value function.
• Approximate policy function 𝜋 𝑎 𝑠> by policy network 𝜋 𝑎|𝑠> ; 𝛉 .
• Approximate value function 𝑉S 𝑠> by:
𝑉 𝑠> ; 𝛉 = ∑C 𝜋 𝑎 𝑠> ; 𝛉 ⋅ 𝑄S 𝑠> , 𝑎 .
Policy-Based Reinforcement Learning
Definition: Approximate state-value function.
• 𝑉 𝑠; 𝛉 = ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ 𝑄S 𝑠, 𝑎 .
Policy-based learning: Learn 𝛉 that maximizes 𝐽 𝛉 = 𝔼[ 𝑉 𝑆; 𝛉 .
Policy-Based Reinforcement Learning
Definition: Approximate state-value function.
• 𝑉 𝑠; 𝛉 = ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ 𝑄S 𝑠, 𝑎 .
Policy-based learning: Learn 𝛉 that maximizes 𝐽 𝛉 = 𝔼[ 𝑉 𝑆; 𝛉 .
How to improve 𝛉? Policy gradient ascent!
• Observe state 𝑠.
^ _ `;𝛉
• Update policy by: 𝛉 ← 𝛉 + 𝛽 ⋅ .
^ 𝛉
Policy gradient
Policy Gradient
Reference
• Sutton and others: Policy gradient methods for reinforcement learning with function approximation. In NIPS,
2000.
Policy Gradient
Definition: Approximate state-value function.
• 𝑉 𝑠; 𝛉 = ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ 𝑄S 𝑠, 𝑎 .
Policy gradient: Derivative of 𝑉 𝑠; 𝛉 w.r.t. 𝛉.
^ _ `;𝛉 ^S(C|`;𝛉)⋅ab `,C ^S(C|`;𝛉)
• = ∑C = ∑C ⋅ 𝑄S 𝑠, 𝑎
^ 𝛉 ^ 𝛉 ^ 𝛉
Policy Gradient
Definition: Approximate state-value function.
• 𝑉 𝑠; 𝛉 = ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ 𝑄S 𝑠, 𝑎 .
Policy gradient: Derivative of 𝑉 𝑠; 𝛉 w.r.t. 𝛉.
^ _ `;𝛉 ^ ∑c S C|`;𝛉 ⋅ab `,C
• =
^ 𝛉 ^ 𝛉
Policy Gradient
Definition: Approximate state-value function.
• 𝑉 𝑠; 𝛉 = ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ 𝑄S 𝑠, 𝑎 .
Policy gradient: Derivative of 𝑉 𝑠; 𝛉 w.r.t. 𝛉.
^ _ `;𝛉 ^ ∑c S C|`;𝛉 ⋅ab `,C
• =
^ 𝛉 ^ 𝛉
^S(C|`;𝛉)⋅ab `,C
= ∑C Push derivative inside the summation
^ 𝛉
Policy Gradient
Definition: Approximate state-value function.
• 𝑉 𝑠; 𝛉 = ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ 𝑄S 𝑠, 𝑎 .
Policy gradient: Derivative of 𝑉 𝑠; 𝛉 w.r.t. 𝛉.
^ _ `;𝛉 ^ ∑c S C|`;𝛉 ⋅ab `,C
• =
^ 𝛉 ^ 𝛉
^S(C|`;𝛉)⋅ab `,C
= ∑C
^ 𝛉
^S(C|`;𝛉) Pretend 𝑄S is independent of 𝛉.
= ∑C ⋅ 𝑄S 𝑠, 𝑎
^ 𝛉 (It may not be true.)
Policy Gradient
Definition: Approximate state-value function.
• 𝑉 𝑠; 𝛉 = ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ 𝑄S 𝑠, 𝑎 .
Policy gradient: Derivative of 𝑉 𝑠; 𝛉 w.r.t. 𝛉.
^ _ `;𝛉 ^S(C|`;𝛉)
• = ∑C ⋅ 𝑄S 𝑠, 𝑎 Policy Gradient
^ 𝛉 ^ 𝛉
Policy Gradient
Definition: Approximate state-value function.
• 𝑉 𝑠; 𝛉 = ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ 𝑄S 𝑠, 𝑎 .
Policy gradient: Derivative of 𝑉 𝑠; 𝛉 w.r.t. 𝛉.
^ _ `;𝛉 ^S(C|`;𝛉)
• = ∑C ⋅ 𝑄S 𝑠, 𝑎 Policy Gradient
^ 𝛉 ^ 𝛉
Note: This derivation is over-simplified and not rigorous.
Policy Gradient
Definition: Approximate state-value function.
• 𝑉 𝑠; 𝛉 = ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ 𝑄S 𝑠, 𝑎 .
Policy gradient: Derivative of 𝑉 𝑠; 𝛉 w.r.t. 𝛉.
^ _ `;𝛉 ^S(C|`;𝛉)
• = ∑C ⋅ 𝑄S 𝑠, 𝑎
^ 𝛉 ^ 𝛉
^ def S(C|`;𝛉)
= ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ ⋅ 𝑄S 𝑠, 𝑎
^ 𝛉
Policy Gradient
Definition: Approximate state-value function.
• 𝑉 𝑠; 𝛉 = ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ 𝑄S 𝑠, 𝑎 .
Policy gradient: Derivative of 𝑉 𝑠; 𝛉 w.r.t. 𝛉.
^ _ `;𝛉 ^S(C|`;𝛉)
• = ∑C ⋅ 𝑄S 𝑠, 𝑎
^ 𝛉 ^ 𝛉
^ def S(C|`;𝛉)
= ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ ⋅ 𝑄S 𝑠, 𝑎
^ 𝛉
^ def S 𝛉 5 ^ S 𝛉
• Chain rule: = ⋅ .
^𝛉 S 𝛉 ^ 𝛉
Policy Gradient
Definition: Approximate state-value function.
• 𝑉 𝑠; 𝛉 = ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ 𝑄S 𝑠, 𝑎 .
Policy gradient: Derivative of 𝑉 𝑠; 𝛉 w.r.t. 𝛉.
^ _ `;𝛉 ^S(C|`;𝛉)
• = ∑C ⋅ 𝑄S 𝑠, 𝑎
^ 𝛉 ^ 𝛉
^ def S(C|`;𝛉)
= ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ ⋅ 𝑄S 𝑠, 𝑎
^ 𝛉
^ def S 𝛉 5 ^ S 𝛉
• Chain rule: = ⋅ .
^𝛉 S 𝛉 ^ 𝛉
^ def S 𝛉 5 ^ S 𝛉
• è 𝜋 𝛉 ⋅ =𝜋 𝛉 ⋅ ⋅
^𝛉 S 𝛉 ^ 𝛉
Policy Gradient
Definition: Approximate state-value function.
• 𝑉 𝑠; 𝛉 = ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ 𝑄S 𝑠, 𝑎 .
Policy gradient: Derivative of 𝑉 𝑠; 𝛉 w.r.t. 𝛉.
^ _ `;𝛉 ^S(C|`;𝛉)
• = ∑C ⋅ 𝑄S 𝑠, 𝑎
^ 𝛉 ^ 𝛉
^ def S(C|`;𝛉)
= ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ ⋅ 𝑄S 𝑠, 𝑎
^ 𝛉
^ def S 𝛉 5 ^ S 𝛉
• Chain rule: = ⋅ .
^𝛉 S 𝛉 ^ 𝛉
^ def S 𝛉 5 ^ S 𝛉
• è 𝜋 𝛉 ⋅ =𝜋 𝛉 ⋅ ⋅ .
^𝛉 S 𝛉 ^ 𝛉
Policy Gradient
Definition: Approximate state-value function.
• 𝑉 𝑠; 𝛉 = ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ 𝑄S 𝑠, 𝑎 .
Policy gradient: Derivative of 𝑉 𝑠; 𝛉 w.r.t. 𝛉.
^ _ `;𝛉 ^S(C|`;𝛉)
• = ∑C ⋅ 𝑄S 𝑠, 𝑎
^ 𝛉 ^ 𝛉
^ def S(C|`;𝛉)
= ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ ⋅ 𝑄S 𝑠, 𝑎
^ 𝛉
^ def S 𝛉 5 ^ S 𝛉
• Chain rule: = ⋅ .
^𝛉 S 𝛉 ^ 𝛉
^ def S 𝛉 5 ^ S 𝛉
• è 𝜋 𝛉 ⋅ =𝜋 𝛉 ⋅ ⋅ .
^𝛉 S 𝛉 ^ 𝛉
Policy Gradient
Definition: Approximate state-value function.
• 𝑉 𝑠; 𝛉 = ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ 𝑄S 𝑠, 𝑎 .
Policy gradient: Derivative of 𝑉 𝑠; 𝛉 w.r.t. 𝛉.
^ _ `;𝛉 ^S(C|`;𝛉)
• = ∑C ⋅ 𝑄S 𝑠, 𝑎
^ 𝛉 ^ 𝛉
^ def S(C|`;𝛉)
= ∑C 𝜋 𝑎 𝑠; 𝛉 ⋅ ⋅ 𝑄S 𝑠, 𝑎
^ 𝛉
^ def S(V|`;𝛉)
= 𝔼V ⋅ 𝑄S 𝑠, 𝐴 .
^ 𝛉
The expectation is taken w.r.t. the random variable 𝐴~𝜋(⋅ |𝑠; 𝛉).
Policy Gradient
Policy gradient:
^ _ `;𝛉 ^ def S(V|`,𝛉)
= 𝔼V~S(⋅|`;𝛉) ⋅ 𝑄S 𝑠, 𝐴 .
^ 𝛉 ^ 𝛉
Calculate Policy Gradient
^ _ `;𝛉 ^ def S(V|`,𝛉)
Policy Gradient: = 𝔼V~S(⋅|`;𝛉) ⋅ 𝑄S 𝑠, 𝐴 .
^ 𝛉 ^ 𝛉
Calculate Policy Gradient
^ _ `;𝛉 ^ def S(V|`,𝛉)
Policy Gradient: = 𝔼V~S(⋅|`;𝛉) ⋅ 𝑄S 𝑠, 𝐴 .
^ 𝛉 ^ 𝛉
1. Randomly sample an action 𝑎g according to 𝜋(⋅ |𝑠; 𝛉).
Calculate Policy Gradient
^ _ `;𝛉 ^ def S(V|`,𝛉)
Policy Gradient: = 𝔼V~S(⋅|`;𝛉) ⋅ 𝑄S 𝑠, 𝐴 .
^ 𝛉 ^ 𝛉
1. Randomly sample an action 𝑎g according to 𝜋(⋅ |𝑠; 𝛉).
^ def S(Cg|`;𝛉)
2. Calculate 𝐠 𝑎g, 𝛉 = ⋅ 𝑄S 𝑠, 𝑎g .
^ 𝛉
^ _ `;𝛉
• By the definition of 𝐠, 𝔼V 𝐠 𝐴, 𝛉 = .
^ 𝛉
^ _ `;𝛉
• 𝐠 𝑎g, 𝛉 is an unbiased estimate of .
^ 𝛉
Calculate Policy Gradient
^ _ `;𝛉 ^ def S(V|`,𝛉)
Policy Gradient: = 𝔼V~S(⋅|`;𝛉) ⋅ 𝑄S 𝑠, 𝐴 .
^ 𝛉 ^ 𝛉
1. Randomly sample an action 𝑎g according to 𝜋(⋅ |𝑠; 𝛉).
^ def S(Cg|`;𝛉)
2. Calculate 𝐠 𝑎g, 𝛉 = ⋅ 𝑄S 𝑠, 𝑎g .
^ 𝛉
^ _ `;𝛉
3. Use 𝐠 𝑎g, 𝛉 as an approximation to the policy gradient .
^ 𝛉
Update policy network using policy gradient
Algorithm
1. Observe the state 𝑠> .
2. Randomly sample action 𝑎> according to 𝜋 ⋅ 𝑠> ; 𝛉> .
Algorithm
1. Observe the state 𝑠> .
2. Randomly sample action 𝑎> according to 𝜋 ⋅ 𝑠> ; 𝛉> .
3. Compute 𝑞> ≈ 𝑄S 𝑠> , 𝑎> (some estimate).
^ def S(Cm |`m ,𝛉)
4. Differentiate policy network: 𝐝l,> = ￨𝛉n𝛉m .
^ 𝛉
Algorithm
1. Observe the state 𝑠> .
2. Randomly sample action 𝑎> according to 𝜋 ⋅ 𝑠> ; 𝛉> .
3. Compute 𝑞> ≈ 𝑄S 𝑠> , 𝑎> (some estimate).
^ def S(Cm |`m ,𝛉)
4. Differentiate policy network: 𝐝l,> = ￨𝛉n𝛉m .
^ 𝛉
5. (Approximate) policy gradient: 𝐠 𝑎> , 𝛉> = 𝑞> ⋅ 𝐝l,> .
6. Update policy network: 𝛉>L5 = 𝛉> + 𝛽 ⋅ 𝐠 𝑎> , 𝛉> .
Algorithm
1. Observe the state 𝑠> .
2. Randomly sample action 𝑎> according to 𝜋 ⋅ 𝑠> ; 𝛉> .
3. Compute 𝑞> ≈ 𝑄S 𝑠> , 𝑎> (some estimate). How?
^ def S(Cm |`m ,𝛉)
4. Differentiate policy network: 𝐝l,> = ￨𝛉n𝛉m .
^ 𝛉
5. (Approximate) policy gradient: 𝐠 𝑎> , 𝛉> = 𝑞> ⋅ 𝐝l,> .
6. Update policy network: 𝛉>L5 = 𝛉> + 𝛽 ⋅ 𝐠 𝑎> , 𝛉> .
Algorithm
1. Observe the state 𝑠> .
2. Randomly sample action 𝑎> according to 𝜋 ⋅ 𝑠> ; 𝛉> .
3. Compute 𝑞> ≈ 𝑄S 𝑠> , 𝑎> (some estimate). How?
^ def S(Cm |`m ,𝛉)
4. Differentiate policy network: 𝐝l,> = ￨𝛉n𝛉m .
Option 1: REINFORCE. ^ 𝛉
5. (Stochastic) policy gradient: 𝐠o 𝛉> ≈ 𝑞> ⋅ 𝐝l,> .
• Play the game to the end and generate the trajectory:
6. Update policy network: 𝛉>L5 = 𝛉> + 𝛽 ⋅ 𝐠o 𝛉> .
𝑠5 , 𝑎5 , 𝑟5 , 𝑠6 , 𝑎6 , 𝑟6 , ⋯ , 𝑠q , 𝑎 q , 𝑟q .
• Compute the discounted return 𝑢> = ∑qsn> 𝛾 st> 𝑟s , for all 𝑡.
• Since 𝑄S 𝑠> , 𝑎> = 𝔼 𝑈> , we can use 𝑢> to approximate 𝑄S 𝑠> , 𝑎> .
• è Use 𝑞> = 𝑢> .
Algorithm
1. Observe the state 𝑠> .
2. Randomly sample action 𝑎> according to 𝜋 ⋅ 𝑠> ; 𝛉> .
3. Compute 𝑞> ≈ 𝑄S 𝑠> , 𝑎> (some estimate). How?
^ def S(Cm |`m ,𝛉)
4. Differentiate policy network: 𝐝l,> = ￨𝛉n𝛉m .
Option 2: Approximate 𝑄S using a neural^ 𝛉network.
5. (Stochastic) policy gradient: 𝐠o 𝛉> ≈ 𝑞> ⋅ 𝐝l,> .
• This leads to the actor-critic method.
6. Update policy network: 𝛉>L5 = 𝛉> + 𝛽 ⋅ 𝐠o 𝛉> .
Summary
Policy-Based Learning
• If a good policy function 𝜋 is known, the agent can be controlled
by the policy: randomly sample 𝑎> ∼ 𝜋 ⋅ 𝑠> .
• Approximate policy function 𝜋 𝑎 𝑠 by policy network 𝜋 𝑎 𝑠; 𝛉 .
• Learn the policy network by policy gradient algorithm.
• Policy gradient algorithm learn 𝛉 that maximizes 𝔼[ 𝑉 𝑆; 𝛉 .
Thank you!
