Continuing Tasks

Objectives

❑ Understand the differentiate between episodic

and continuing task
❑ Using discounting for continuing tasks

Continuing Tasks 2
Continuing tasks
❑ Continuing tasks, also known as ongoing tasks or
recurring tasks, are activities that require regular
attention and maintenance over an indefinite
period. These tasks do not have a defined
endpoint and typically repeat at regular intervals
or as needed.

3
Continuing Tasks
Continuing tasks
❑ Characteristics of Continuing Tasks:
❑ Ongoing Nature: Continuing tasks persist over time and
require continuous attention and effort.
❑ Regular Intervals: They may occur at regular intervals,
such as daily, weekly, monthly, or annually, depending on
the nature of the task.
❑ No Defined Endpoint: Unlike episodic tasks, continuing
tasks do not have a clear endpoint or completion date.
They continue indefinitely or until they are no longer
necessary.

4
Continuing Tasks
Continuing tasks
❑ Characteristics of Continuing Tasks:
❑ Maintenance and Monitoring: These tasks often
involve regular maintenance, monitoring, and adjustment
to ensure they are performed effectively and efficiently.
❑ Evolution Over Time: Continuing tasks may evolve over
time to adapt to changing circumstances, requirements,
or priorities.

5
Continuing Tasks
Continuing tasks
Episodic Tasks Continuing Tasks
Definition Episodic tasks are those that Continuing tasks, also known as
have a distinct beginning and ongoing tasks or recurring tasks,
end. They are typically are those that persist over time
discrete activities or projects without a defined endpoint. They
with a defined goal or are activities that require regular
outcome attention and maintenance.
Duration These tasks are time-limited These tasks are indefinite and
and are usually completed repeat at regular intervals or as
within a relatively short needed.
timeframe.
Examples Writing a report, completing a Routine administrative tasks,
specific project, organizing an managing a website, providing
event, or studying for an customer support, and
exam are examples of maintaining equipment are
episodic tasks. examples of continuing tasks.

6
Continuing Tasks
Continuing tasks
Episodic Tasks Continuing Tasks
Characteri  They often require a  They require ongoing effort
stics focused effort until and attention.
completion.  They may not have clear
 They have clear objectives endpoints and may evolve
and deliverables. over time.
 Once finished, they do not  They often involve regular
require ongoing attention monitoring and adjustment to
or maintenance. meet changing requirements
or conditions.

7
Continuing Tasks
Continuing tasks
❑ Example: Smart thermostat which regulates the
temperature of a building

8
Continuing Tasks
Continuing tasks
❑ Example: Smart thermostat which regulates the
temperature of a building
❑ The thermostat never stops interacting with the environment.
❑ The state could be the current temperature along with details
of the situation like the time of day and the number of people
in the building.
❑ There are just two actions, turn on the heater or turn it off.
❑ The reward to be minus one every time someone has to
manually adjust the temperature and zero otherwise.
❑ To avoid negative reward, the thermostat would learn to
anticipate the user's preferences
9
Continuing Tasks
Continuing tasks
❑ Interaction goes on continually
❑ No terminal state

❑ We're summing over an infinite sequence. This

return might not be finite. What is solution ?

10
Continuing Tasks
Discounting
❑ Discount future rewards by a factor Gamma called the
discount rate.
❑ Gamma is at least zero, but less than one.
❑ The return formulation can then be modified to include
discounting.
❑ The effect of discounting on the return is simple,
immediate rewards contribute more to the some.
❑ Rewards far into the future contribute less because
they are multiplied by Gamma raised to successively
11
larger powers of k Continuing Tasks
Discounting
❑ Discount the rewards in the future by gama, where
0<gama<1

12
Continuing Tasks
Discounting
❑ We can concisely write this sum as this expression,
which is guaranteed to be finite.

13
Continuing Tasks
Discounting
❑ The effect of gama on agent behavior

❑ When Gamma = 0 the return is just the reward at the

next time step. So the agent is shortsighted and only
cares about immediate expected reward.

14
Continuing Tasks
Discounting
❑ The effect of gama on agent behavior

❑ when Gamma approaches one, the immediate and future

rewards are weighted nearly equally in the return. The
agent in this case is more farsighted.

15
Continuing Tasks
Student task
❑ Give an example of a continuing task, and analyze its
components: agent, environment, state,...

16
Continuing Tasks
Summary
❑ Understand the differentiate between episodic and
continuing tasks
❑ Using discounting for continuing tasks

Continuing Tasks 17
Q&A

Continuing Tasks 18