Deep Reinforcement Learning

CS 294 - 112
Course logistics
Class Information & Resources

Sergey Levine Kate Rakelly Greg Kahn Sid Reddy Michael Chang Soroush Nasiriany
Instructor Head GSI GSI GSI GSI uGSI

• Course website: http://rail.eecs.berkeley.edu/deeprlcourse

• Piazza: UC Berkeley, CS294-112
• Subreddit (for non-enrolled students): www.reddit.com/r/berkeleydeeprlcourse/
• Office hours: check course website (mine are after class on Wed in Soda 341B)
Prerequisites & Enrollment
• All enrolled students must have taken CS189, CS289, CS281A, or an
equivalent course at your home institution
• Please contact Sergey Levine if you haven’t
• Please enroll for 3 units
• Students on the wait list will be notified as slots open up
• Lectures will be recorded
• Since the class is full, please watch the lectures online if you are not enrolled
What you should know
• Assignments will require training neural networks with standard
automatic differentiation packages (TensorFlow by default)
• Review Section
• Greg Kahn will TensorFlow and neural networks on Wed next week (8/29)
• You should be able to at least do the TensorFlow MNIST tutorial (if not, make
sure to attend Greg’s lecture and ask questions!)
What we’ll cover
• Full list on course website (click “Lecture Slides”)
1. From supervised learning to decision making
2. Model-free algorithms: Q-learning, policy gradients, actor-critic
3. Advanced model learning and prediction
4. Exploration
5. Transfer and multi-task learning, meta-learning
6. Open problems, research talks, invited lectures
Assignments
1. Homework 1: Imitation learning (control via supervised learning)
2. Homework 2: Policy gradients (“REINFORCE”)
3. Homework 3: Q learning and actor-critic algorithms
4. Homework 4: Model-based reinforcement learning
5. Homework 5: Advanced model-free RL algorithms
6. Final project: Research-level project of your choice (form a group of
up to 2-3 students, you’re welcome to start early!)

Grading: 60% homework (12% each), 40% project

Your “Homework” Today
1. Sign up for Piazza (see course website)
2. Start forming your final project groups, unless you want to work
alone, which is fine
3. Check out the TensorFlow MNIST tutorial, unless you’re a
TensorFlow pro
What is reinforcement learning, and why
should we care?
How do we build intelligent machines?
Intelligent machines must be able to adapt
Deep learning helps us handle unstructured
environments
Reinforcement learning provides a formalism for
behavior
decisions (actions)

Schulman et al. ’14 & ‘15 Mnih et al. ‘13

consequences
observations
rewards
Levine*, Finn*, et al. ‘16
What is deep RL, and why should we care?
standard
features mid-level features classifier
computer
(e.g. HOG) (e.g. DPM) (e.g. SVM)
vision
Felzenszwalb ‘08

end-to-end training
deep
learning

standard
reinforcement
learning
features
? more features
? linear policy
or value func.
action

deep end-to-end training

reinforcement action
learning
What does end-to-end learning mean for
sequential decision making?
 perception

Action
(run away)

action
 sensorimotor loop

Action
(run away)
Example: robotics

robotic state
modeling & low-level
control observations estimation
prediction
planning
control
controls
pipeline (e.g. vision)
tiny, highly specialized tiny, highly specialized
“visual cortex” “motor cortex”

no direct supervision
actions have consequences
 decisions (actions)

Deep models are what allow reinforcement Actions: muscle contractions

Observations: sight, smell
Actions: motor current or torque
Observations: camera images
Rewards: task success measure (e.g.,
learning algorithms to solve complex problemsRewards: food
running speed)
consequences
endobservations
to end!
rewards

Actions: what to purchase

The reinforcement learning problem is the AI problem! Observations: inventory levels
Rewards: profit
 Complex physical tasks…

Rajeswaran, et al. 2018

 Unexpected solutions…

Mnih, et al. 2015

 Not just games and robots!

Cathy Wu
Why should we study this now?

1. Advances in deep learning

2. Advances in reinforcement learning
3. Advances in computational capability
Why should we study this now?

Tesauro, 1995

L.-J. Lin, “Reinforcement learning for robots using neural networks.” 1993
Why should we study this now?

Atari games: Real-world robots: Beating Go champions:

Q-learning: Guided policy search: Supervised learning + policy
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. S. Levine*, C. Finn*, T. Darrell, P. Abbeel. “End-to-end gradients + value functions +
Antonoglou, et al. “Playing Atari with Deep training of deep visuomotor policies”. (2015).
Reinforcement Learning”. (2013).
Monte Carlo tree search:
Q-learning: D. Silver, A. Huang, C. J. Maddison, A. Guez,
Policy gradients: D. Kalashnikov et al. “QT-Opt: Scalable Deep L. Sifre, et al. “Mastering the game of Go
J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Reinforcement Learning for Vision-Based Robotic with deep neural networks and tree
Abbeel. “Trust Region Policy Optimization”. (2015). Manipulation”. (2018). search”. Nature (2016).
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap,
et al. “Asynchronous methods for deep reinforcement
learning”. (2016).
What other problems do we need to solve to
enable real-world sequential decision making?
Beyond learning from reward

• Basic reinforcement learning deals with maximizing rewards

• This is not the only problem that matters for sequential decision
making!
• We will cover more advanced topics
• Learning reward functions from example (inverse reinforcement learning)
• Transferring knowledge between domains (transfer learning, meta-learning)
• Learning to predict and using prediction to act
Where do rewards come from?
Are there other forms of supervision?

• Learning from demonstrations

• Directly copying observed behavior
• Inferring rewards from observed behavior (inverse reinforcement learning)
• Learning from observing the world
• Learning to predict
• Unsupervised learning
• Learning from other tasks
• Transfer learning
• Meta-learning: learning to learn
 Imitation learning

Bojarski et al. 2016

More than imitation: inferring intentions

Warneken & Tomasello

 Inverse RL examples

Finn et al. 2016

Prediction
 What can we do with a perfect model?

Mordatch et al. 2015

 Prediction for real-world control

Ebert et al. 2017

How do we build intelligent machines?
How do we build intelligent machines?
• Imagine you have to build an intelligent machine, where do you start?
Learning as the basis of intelligence
• Some things we can all do (e.g. walking)
• Some things we can only learn (e.g. driving a car)
• We can learn a huge variety of things, including very difficult things
• Therefore our learning mechanism(s) are likely powerful enough to do
everything we associate with intelligence
• But it may still be very convenient to “hard-code” a few really important bits
 A single algorithm?
• An algorithm for each “module”?
• Or a single flexible algorithm?

Seeing with your tongue

Auditory
Cortex
Human echolocation (sonar)

[BrainPort; Martinez et al; Roe et al.]

adapted from A. Ng
What must that single algorithm do?
• Interpret rich sensory inputs

• Choose complex actions

Why deep reinforcement learning?
• Deep = can process complex sensory input
▪ …and also compute really complex functions
• Reinforcement learning = can choose complex actions
Some evidence in favor of deep learning
Some evidence for reinforcement learning
• Percepts that anticipate reward
become associated with similar
firing patterns as the reward
itself
• Basal ganglia appears to be
related to reward system
• Model-free RL-like adaptation is
often a good fit for experimental
data of animal adaptation
• But not always…
What can deep learning & RL do well now?
• Acquire high degree of proficiency in
domains governed by simple, known
rules
• Learn simple skills with raw sensory
inputs, given enough experience
• Learn from imitating enough human-
provided expert behavior
What has proven challenging so far?
• Humans can learn incredibly quickly
• Deep RL methods are usually slow
• Humans can reuse past knowledge
• Transfer learning in deep RL is an open problem
• Not clear what the reward function should be
• Not clear what the role of prediction should be
Instead of trying to produce a
program to simulate the adult
mind, why not rather try to
produce one which simulates the
child's? If this were then subjected general learning
to an appropriate course of algorithm

observations
education one would obtain the

actions
adult brain.
- Alan Turing
environment