UNIT 5

Cognitive models
Cognitive models represent users of interactive systems.
Hierarchical models represent a user's task and goal structure.
Linguistic models represent the user-system grammar.
Physical and device models represent human motor skills.
Cognitive modeling is an area of computer science that deals with simulating human problem-
solving and mental processing in a computerized model. Such a model can be used to
simulate or predict human behavior or performance on tasks similar to the ones modeled and
improve human-computer interaction.
GOMS is a theory of the cognitive skills involved in human-computer tasks. ... For a given
task , a particular GOMS structure can be constructed and used to predict the time required to
complete the task.
In addition, the model can be used to identify and predict the effects of errors on task
performance.
GOMS is a family of predictive models of human performance that can be used to improve
the efficiency of human-machine interaction by identifying and eliminating unnecessary user
actions. GOMS stands for (Goals, Operators, Methods, and Selection).
GOMS is a theory of the cognitive skills involved in human-computer tasks. ... For a given
task , a particular GOMS structure can be constructed and used to predict the time required to
complete the task.
In addition, the model can be used to identify and predict the effects of errors on task
performance.
Cognitive models
• goal and task hierarchies
• linguistic
• physical and device
• architecturalCognitive models
• They model aspects of user:
– understanding
– knowledge
– intentions
– processing
• Common categorisation:
– Competence vs. Performance
– Computational flavour
– No clear divide
Goal and task hierarchies
• Mental processing as divide-and-conquer
• Example: sales report
produce report
gather data
. find book names
. . do keywords search of names database
. . . … further sub-goals
. . sift through names and abstracts by hand
. . . … further sub-goals
. search sales database - further sub-goals
layout tables and histograms - further sub-goals
write description - further sub-goals
goals vs. tasks
• goals – intentions
what you would like to be true
• tasks – actions
how to achieve it
• GOMS
– goals are internal
• HTA
– actions external
– tasks are abstractions
Issues for goal hierarchies
• Granularity
– Where do we start?
– Where do we stop?
• Routine learned behaviour, not problem
solving
– The unit task
• Conflict
– More than one way to achieve a goal
• ErrorTechniques
• Goals, Operators, Methods and
Selection (GOMS)
• Cognitive Complexity Theory (CCT)
• Hierarchical Task Analysis (HTA) -
GOMS
Goals
– what the user wants to achieve
Operators
– basic actions user performs
Methods
– decomposition of a goal into subgoals/operators
Selection
– means of choosing between competing methods
GOMS example
GOAL: CLOSE-WINDOW
. [select GOAL: USE-MENU-METHOD
. MOVE-MOUSE-TO-FILE-MENU
. PULL-DOWN-FILE-MENU
. CLICK-OVER-CLOSE-OPTION
GOAL: USE-CTRL-W-METHOD
. PRESS-CONTROL-W-KEYS]
For a particular user:
Rule 1: Select USE-MENU-METHOD unless another
rule applies
Rule 2: If the application is GAME,
select CTRL-W-METHOD
Cognitive Complexity Theory
• Two parallel descriptions:
– User production rules
– Device generalised transition networks
• Production rules are of the form:
– if condition then action
• Transition networks covered under
dialogue models
Example: editing with vi
• Production rules are in long-term memory
• Model working memory as attribute-value
mapping:
(GOAL perform unit task)
(TEXT task is insert space)
(TEXT task is at 5 23)
(CURSOR 8 7)
• Rules are pattern-matched to working
memory,
e.g., LOOK-TEXT task is at %LINE %COLUMN
is true, with LINE = 5 COLUMN = 23.Active rules:
SELECT-INSERT-SPACE
INSERT-SPACE-MOVE-FIRST
INSERT-SPACE-DOIT
INSERT-SPACE-DONE
Four rules to model inserting a space
New working memory
(GOAL insert space)
(NOTE executing insert space)
(LINE 5) (COLUMN 23)
SELECT-INSERT-SPACE
matches current working memory
(SELECT-INSERT-SPACE
IF (AND (TEST-GOAL perform unit task)
(TEST-TEXT task is insert space)
(NOT (TEST-GOAL insert space))
(NOT (TEST-NOTE executing insert space)))
THEN ( (ADD-GOAL insert space)
(ADD-NOTE executing insert space)
(LOOK-TEXT task is at %LINE %COLUMN)))
Notes on CCT
• Parallel model
• Proceduralisation of actions
• Novice versus expert style rules
• Error behaviour can be represented
• Measures
– depth of goal structure
– number of rules
– comparison with device description
Problems with goal hierarchies
• a post hoc technique
• expert versus novice
• How cognitive are they?
Linguistic notations
• Understanding the user's behaviour and
cognitive difficulty based on analysis of
language between user and system.
• Similar in emphasis to dialogue models
• Backus–Naur Form (BNF)
• Task–Action Grammar (TAG)
Backus-Naur Form (BNF)
• Very common notation from computer science
• A purely syntactic view of the dialogue
• Terminals
– lowest level of user behaviour
– e.g. CLICK-MOUSE, MOVE-MOUSE
• Nonterminals
– ordering of terminals
– higher level of abstraction
– e.g. select-menu, position-mouse
Example of BNF
• Basic syntax:
– nonterminal ::= expression
• An expression
– contains terminals and nonterminals
– combined in sequence (+) or as alternatives (|)
draw line ::= select line + choose points + last point
select line ::= pos mouse + CLICK MOUSE
choose points ::= choose one | choose one + choose points
choose one ::= pos mouse + CLICK MOUSE
last point ::= pos mouse + DBL CLICK MOUSE
pos mouse ::= NULL | MOVE MOUSE+ pos mouse
Measurements with BNF
• Number of rules (not so good)
• Number of + and | operators
• Complications
– same syntax for different semantics
– no reflection of user's perception
– minimal consistency checking
Task Action Grammar (TAG)
• Making consistency more explicit
• Encoding user's world knowledge
• Parameterised grammar rules
• Nonterminals are modified to include
additional semantic features
Consistency in TAG
• In BNF, three UNIX commands would be described as:
copy ::= cp + filename + filename | cp + filenames + directory
move ::= mv + filename + filename | mv + filenames + directory
link ::= ln + filename + filename | ln + filenames + directory
• No BNF measure could distinguish between this and a
less consistent grammar in which
link ::= ln + filename + filename | ln + directory + filenames
Consistency in TAG (cont'd)
• consistency of argument order made explicit
using a parameter, or semantic feature for file
operations
• Feature Possible values
Op = copy; move; link
• Rules
file-op[Op] ::= command[Op] + filename + filename
| command[Op] + filenames + directory
command[Op = copy] ::= cp
command[Op = move] ::= mv
command[Op = link] ::= ln
Other uses of TAG
• User’s existing knowledge
• Congruence between features and
commands
• These are modelled as derived rules
Physical and device models
• The Keystroke Level Model (KLM)
• Buxton's 3-state model
• Based on empirical knowledge of human
motor system
• User's task: acquisition then execution.
– these only address execution
• Complementary with goal hierarchies
Keystroke Level Model (KLM)
• lowest level of (original) GOMS
• six execution phase operators
– Physical motor:
K - keystroking
P - pointing
H - homing
D - drawing
– Mental
M - mental preparation
– System
R - response
• times are empirically determined.
Texecute = TK + TP + TH + TD + TM + TR
KLM example
GOAL: ICONISE-WINDOW
[select
GOAL: USE-CLOSE-METHOD
. MOVE-MOUSE-TO- FILE-MENU
. PULL-DOWN-FILE-MENU
. CLICK-OVER-CLOSE-OPTION
GOAL: USE-CTRL-W-METHOD
PRESS-CONTROL-W-KEY]
• compare alternatives:
• USE-CTRL-W-METHOD vs.
• USE-CLOSE-METHOD
• assume hand starts on mouse
USE-CLOSE-METHOD
P[to menu]
1.1
B[LEFT down] 0.1
M
1.35
P[to option]
1.1
B[LEFT up]
0.1
Total
3.75 s
USE-CTRL-W-METHOD
H[to kbd]
0.40
M
1.35
K[ctrlW key] 0.28
Total
2.03 s
Architectural models
• All of these cognitive models make
assumptions about the architecture of
the human mind.
• Long-term/Short-term memory
• Problem spaces
• Interacting Cognitive Subsystems
• Connectionist
• ACT
Display-based interaction
• Most cognitive models do not deal with
user observation and perception
• Some techniques have been extended
to handle system output
(e.g., BNF with sensing terminals, Display-TAG)
but problems persist
• Exploratory interaction versus planning

History of GOMS:
Developed in 1983 by Stuart Card, Thomas P. Moran and Allen Newell.
Explained in their book “The Psychology of Human Computer Interaction”
Goals - what the user intends to accomplish
Operators - are actions that are performed to reach the goal.
Methods - are sequences of operators that accomplish a goal.
Selections - there can be more than one
method available to accomplish a single goal;
used to describe when a user would select a
certain method over the others. These can be very wide reaching; from very
high-level goals (e.g. Write a book) to very
low-level goals (e.g. Type the word ‘Red’).
Higher-level goals can be divided into smaller
lower-level goals. Operators are the simple actions that are used to
accomplish your goals (e.g. ‘Left-click mouse button’
or ‘Press ALT’).
Operators cannot break down any further: they are
atomic elements (similar to those found in a
database).
Generally, it’s assumed that each operator requires a
fixed quantity of time to perform the action and this
time interval is independent of context.
For example, to double left click a mouse button takes
0.40 seconds of execution time, regardless of what
you happen to be clicking on.
Methods are procedures that describe how to accomplish goals.
A method is essentially the steps a user has to
take to complete a specified task.
For instance, one method to accomplish the goal ‘Highlight Word’ in a Windows text editor
would be to ‘move cursor’ to the beginning of the word and ‘hold shift and right arrow key’.
Another method to accomplish the same goal could involve ‘holding down the left mouse
button’ and ‘dragging to the beginning of the word’.
Selection rules specify which method is best to use when completing a goal, based on the
given context.
Since there could be several ways of achieving the same result, a selection rule utilises the
user's knowledge of the best method to achieve the required goal.
Selection rules generally take the form of a conditional statement, such as “IF the word to
highlight is less than five characters USE the arrow keys and shift ELSE the mouse dragging
method".
GOSM :
Quantitatively, GOMS offer good predictive models of performance time and learning. For
example, when choosing between two systems you can apply a GOMS model.
Application 1 has a lower start-up costs, but will be slower to perform
frequent tasks; Application 2 will be faster to perform tasks, but has a
longer learning time, etc.
With these quantitative predictions, you can examine such tradeoffs in
the light of what is important to your company.
Qualitatively, GOMS can be used to design training programs and help systems.
This approach has been shown to be an efficient way to organise help
systems, tutorials, and training programs as well as user documentation.
NGOMS:
‘Natural GOMS Language’ allows a more flexible representation of a task using ‘human’
language:
Method for goal: Deleting Icon
Step 1. Select Icon For Deletion (1.10 sec)
Step 2. Drag Icon To Trashcan
(1.10 sec)
Step 3. Update User with audio cue (0.22 sec)
This NGOMSL model predicts that it will take 2.42 seconds to delete an icon.
Problems with GOMS:
The model assumes a certain level of skill – it cannot accurately be applied to beginners.
The model doesn’t take into account time for learning the system or remembering how to
use it after a long period of disuse; for example, can you remember where all the options
are in Windows 98?
The model removes human error from the equation; even highly skilled users make the
occasional mistake!
Mental workload is not addressed in the model; it’s far more taxing when remembering a
longer process than a short one. E.g. It’s a far less stressful task to highlight text than
entering network settings manually.
Users can get tired; you’re not going to be as quick typing after three hours as you were
when you started!
Differences among users is not accounted for within the model – e.g. Those who are left
handed aren’t given special preference.
Predicting whether a system will be functional or acceptable for users is not included in the
model. E.g. Just because workers can enter data quickly into a new database system, it
doesn’t mean it’s particularly ‘user friendly’ and ‘easy to work with’.
How computer-assisted technology integrates into everyday business is not addressed in
the model. For example, entering commands using a keyboard could be the quickest
method of data capture for astronauts, however, trying to type in ‘zero gravity’ may be
difficult.

Ubiquitous computing:
Ubiquitous computing (or "ubicomp") is a concept in software engineering
and computer science where computing is made to appear anytime and everywhere.
Ubiquitous computing aims to permeate and interconnect all areas of life, and thus to enable
a ubiquitous flow of data, information
Ubiquitous computing is a paradigm in which the processing of information is linked with
each activity or object as encountered. It involves connecting electronic devices, including
embedding microprocessors to communicate information. Devices that use ubiquitous
computing have constant availability and are completely connected.
Ubiquitous computing focuses on learning by removing the complexity of computing and
increases efficiency while using computing for different daily activities.
The main focus of ubiquitous computing is the creation of smart products that are connected,
making communication and the exchange of data easier and less obtrusive.

Key features of ubiquitous computing include:

Consideration of the human factor and placing of the paradigm in a human, rather than
computing, environment
Use of inexpensive processors, thereby reducing memory and storage requirements
Capturing of real-time attributes
Totally connected and constantly available computing devices
Focus on many-to-many relationships, instead of one-to-one, many-to-one or one-to-many in
the environment, along with the idea of technology, which is constantly present
Includes local/global, social/personal, public/private and invisible/visible features and
considers knowledge creation, as well as information dissemination
Relies on converging Internet, wireless technology and advanced electronics
Increased surveillance and possible restriction and interference in user privacies, as the digital
devices are wearable and constantly connected
As technology progresses, the reliability factor of the different equipment used may be
impacted
Wood:
Ambient Wood is an innovative educational experience exploring how biological ideas can be
learnt in a real environment augmented by technology.
Augmented reality is increasingly used through smartphones, tablets and head-mounted
displays with the goal of enhancing the real world with virtual elements in an interactive
manner.
Whereas virtual reality replaces what people see and experience, augmented reality actually
adds to it. Using devices such as HTC Vive, Oculus Rift, and Google Cardboard, VR covers
and replaces users' field of vision entirely, while AR projects images in front of them in a
fixed area.
Augmented reality (AR) adds digital elements to a live view often by using the camera on a
smartphone. ... Virtual reality (VR) implies a complete immersion experience that shuts out
the physical world.
Various technologies are used in augmented reality rendering, including optical projection
systems, monitors, handheld devices, and display systems, which are worn on the human
body. A head-mounted display (HMD) is a display device worn on the forehead, such as a
harness or helmet-mounted.
Augmented reality (AR) is a technology that lets people superimpose digital content (images,
sounds, text) over a real-world environment. ... In other words, AR is on the verge of
becoming a part of everyday life.
IKEA Mobile App. ...
Nintendo's Pokémon Go App. ...
Google Pixel's Star Wars Stickers. ...
Disney Coloring Book. ...
L'Oréal Makeup App. ...
Weather Channel Studio Effects. ...
U.S. Army.
There are 3 primary categories of virtual reality simulations used today: non-immersive, semi-
immersive, and fully-immersive simulations.