Available online at www.sciencedirect.

com
Available online at www.sciencedirect.com
Available online at www.sciencedirect.com

ScienceDirect
Procedia Computer Science 00 (2019) 000–000
Procedia Computer
Procedia Science
Computer 14500
Science (2018) 730–739
(2019) 000–000 www.elsevier.com/locate/procedia
www.elsevier.com/locate/procedia

Postproceedings of the 9th Annual International Conference on Biologically Inspired Cognitive

Postproceedings of the 9thBICA
Architectures, Annual International
2018 Conference
(Ninth Annual MeetingonofBiologically Inspired Cognitive
the BICA Society)
Architectures, BICA 2018 (Ninth Annual Meeting of the BICA Society)
Metacognition
Metacognition for
for aa Common
Common Model
Model of
of Cognition
Cognition
Jerald D. Kralika,∗ , Jee Hang Leebb , Paul S. Rosenbloomcc , Philip C. Jackson, Jr.dd ,
Jerald D. Kralika,∗, Jee Hang Lee , Paul S. Rosenbloom , Philip C. Jackson, Jr. ,
Susan L. Epsteinee , Oscar J. Romeroff , Ricardo Sanzgg ,
Susan L. Epstein , Oscar J. Romero , Ricardo Sanz ,
Othalia Laruehh , Hedda R. Schmidtkeii , Sang Wan Leea,b , Keith McGreggorj
Othalia Larue , Hedda R. Schmidtke , Sang Wan Leea,b , Keith McGreggorj
a Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, 34141, South Korea
a Department of Bio and Brain Engineering, Korea Advanced Institute
bKI for Health Science and Technology, Korea Advanced Institute of of Science
Science and
and Technology
Technology (KAIST),
(KAIST), Daejeon,
Daejeon, 34141,
34141, South
South Korea
Korea
b cKI for Health Science and Technology, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, 34141, South Korea
Institute for Creative Technologies & Department of Computer Science, University of Southern California, Los Angeles, CA, USA
c Institute for Creative Technologies & Department of Computer
d TalaMind LLC, PMB #363, 55 Science,
E. Long University of Southern
Lake Rd., Troy, MI, USA California, Los Angeles, CA, USA
e Department of Computer Science, d TalaMind LLC, PMB #363, 55 E. Long Lake Rd., Troy, MI, USA
Hunter College and The Graduate Center of the City University of New York, New York, NY, USA
e Department of Computer Science, Hunter College and TheCarnegie
GraduateMellon
CenterUniversity,
of the City Pittsburgh,
University of
f Department of Machine Learning, PA,New
USAYork, New York, NY, USA
f Department of Machine
g Autonomous SystemsLearning, Carnegie
Laboratory, MellonPolitécnica
Universidad University,de Pittsburgh, PA, USA
Madrid, Spain
g Autonomous
h Department Systems Laboratory, Universidad Politécnica de Madrid, Spain
of Psychology, Wright State University, Dayton, Ohio, USA
h Department ofof
Psychology,
i Department Geography,Wright State of
University University, Dayton,OR,
Oregon, Eugene, Ohio,
USAUSA
i Department of Geography, University of Oregon, Eugene, OR, USA
j College of Computing, Georgia Institute of Technology, Atlanta, Georgia, USA
j College of Computing, Georgia Institute of Technology, Atlanta, Georgia, USA

Abstract
Abstract
This paper provides a starting point for the development of metacognition in a common model of cognition. It identifies significant
This paper provides
theoretical work on ametacognition
starting point from
for the development
multiple of metacognition
disciplines in a believe
that the authors commonworthy
modelofofconsideration.
cognition. It identifies significant
After first defining
theoreticaland
cognition work on metacognition
metacognition, from multiple
we outline disciplines
three general that of
categories themetacognition,
authors believeprovide
worthyanofinitial
consideration.
list of its After
main first defining
components,
cognition andmore
consider the metacognition, we outline
difficult problem three generaland
of consciousness, categories
present of metacognition,
examples provide
of prominent an initial
artificial list ofthat
systems its main components,
have implemented
consider the more
metacognitive difficult problem
components. Finally, of
weconsciousness, anddesign
identify pressing present examples
issues offuture.
for the prominent artificial systems that have implemented
metacognitive components. Finally, we identify pressing design issues for the future.
c 2018

© 2019 The
The Authors.
Authors. Published
Published by
by Elsevier
Elsevier B.V.
B.V.
c 2019
 The Authors. Published by Elsevier B.V.
This is an open access article under the CC BY-NC-ND license
This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/)
(https://creativecommons.org/licenses/by-nc-nd/4.0/)
This is an open access article under the CC BY-NC-ND license
Peer-review under responsibility of the scientific committee of (https://creativecommons.org/licenses/by-nc-nd/4.0/)
the 9th Annual International Conference on Biologically Inspired
Peer-review
Cognitive under responsibility of the scientific committee of the 9th Annual International Conference on Biologically Inspired
Architectures.
Cognitive Architectures.
Keywords: Metacognition; Common model; Cognition; Cognitive architecture
Keywords: Metacognition; Common model; Cognition; Cognitive architecture

∗ Corresponding author.
∗ Corresponding author.
E-mail addresses: jerald.kralik@kaist.ac.kr (Jerald D. Kralik)., jeehang@kaist.ac.kr (Jee Hang Lee)., rosenbloom@usc.edu (Paul S.
E-mail addresses:
Rosenbloom)., jerald.kralik@kaist.ac.kr(Philip
dr.phil.jackson@talamind.com (JeraldC.D.Jackson,
Kralik).,Jr.).,
jeehang@kaist.ac.kr (Jee Hang Lee).,
susan.epstein@hunter.cuny.edu ( rosenbloom@usc.edu (Paul S.
Rosenbloom).,
Susan dr.phil.jackson@talamind.com
L. Epstein)., oscarr@andrew.cmu.edu (Oscar(Philip
J. C. Jackson,ricardo.sanz@upm.es
Romero)., Jr.)., susan.epstein@hunter.cuny.edu
(Ricardo Sanz).,( othalia.larue@wright.edu (
Susan
OthaliaL.Larue).,
Epstein)., oscarr@andrew.cmu.edu (Oscar
hedda.schmidtke@gmail.com (HeddaJ. R.
Romero)., ricardo.sanz@upm.es
Schmidtke)., (Ricardo
sangwan@kaist.ac.kr Sanz).,
(Sang othalia.larue@wright.edu
Wan Lee)., (
keith.mcgreggor@gatech.edu (Keith
Othalia Larue)., hedda.schmidtke@gmail.com (Hedda R. Schmidtke)., sangwan@kaist.ac.kr (Sang Wan Lee)., keith.mcgreggor@gatech.edu (Keith
McGreggor).
McGreggor).

1877-0509
1877-0509  c 2019
© 2018 The Authors. Published
The Authors. Published byby Elsevier
Elsevier B.V.
B.V.
1877-0509
This c 2019 Thearticle
 Authors. Published by Elsevier B.V.
This isisan
anopen
openaccess under
access article the CC
under the BY-NC-ND
CC BY-NC-ND licenselicense
(https://creativecommons.org/licenses/by-nc-nd/4.0/)
(https://creativecommons.org/licenses/by-nc-nd/4.0/)
This is an open
Peer-review
Peer-review access
under
under article under
responsibility
responsibility the
ofofthe CC
the BY-NC-ND
scientific
scientific license
committee
committee (https://creativecommons.org/licenses/by-nc-nd/4.0/)
of the
of 9th 9th
the Annual International
Annual Conference
International on Biologically
Conference Inspired
on Biologically Cognitive
Inspired Archi-
Cognitive
Peer-review
Architectures.
tectures. under responsibility of the scientific committee of the 9th Annual International Conference on Biologically Inspired Cognitive Archi-
tectures.
10.1016/j.procs.2018.11.046
 Jerald D. Kralik et al. / Procedia Computer Science 145 (2018) 730–739 731
2 Author name / Procedia Computer Science 00 (2019) 000–000

1. Introduction

The goal of this paper is to begin the development of a consensus model of metacognition that spans all relevant
fields, including cognitive science, philosophy, neuroscience, and robotics. In what follows, we first define metacogni-
tion, then outline general categories of it, list its major components, discuss its relationship to consciousness, address
key design issues, and present case studies with metacognition successfully implemented computationally. Finally,
we briefly address the next steps for the project.

2. Metacognition Defined

To define metacognition, we must begin with cognition itself. Cognition is defined differently across fields and
contexts. Because an intelligent agent executes a repeating perceive-decide-act cycle, we define cognition to capture
that cycle, thus incorporating perception and action [45]. Here, “perceive” subjects the agent to a continual barrage
of signals (e.g., visual, auditory, olfactory) that describe the agent’s context. These signals are necessarily a partial
description of the environment in which the agent exists. The “decide” portion of the cycle the focus of this paper
is also treated differently across fields. Here, it is used it broadly, to capture intermediate processes that culminate in
a decision about which action(s) to execute. Necessitated by the incomplete, and possibly inconsistent, messages an
intelligent agent receives, decision incorporates, but is not restricted to, a wealth of processes. These include, attention,
reasoning, learning, planning, imagination, conscious access, and communication and understanding through natural
language [13]. Finally, “act” represents the ultimate outcome of the cognitive cycle, one that typically results in
external motor responses (e.g., via muscles, actuators).
Simply put, metacognition is cognition about cognition. Thus it includes, for example, reasoning about reasoning,
reasoning about learning, and learning about reasoning [28, 34, 43]. Broadly construed, it is any cognitive process or
structure about another cognitive process or structure (e.g., data about memory held in memory). Here we focus on
cognitive processes applied to cognitive processes, a kind of recursive processing illustrated in Fig. 1. If a particular
process is of Type-X (where X is perception, decision, or action) and receives input from another X process, it is
considered a metaprocess, and therefore metacognition. In fact, it is considered metacognitive if and only if the
process receives input from, sends output to, or both receives from and sends to the same process type.

Fig. 1. A process is metacognitive if and only if it receives input from, sends output to, or both receives from and sends to the same process type.
A process type is perception, decision, or action.

Metacognition addresses what the system knows, the importance of what is known, as well as what has been
remembered and what is worth remembering or forgetting. Its advantages include arbitration underlying competing
functions; modulation to help finetune other cognitive processes; safeguards against confusion and errors from lower
cognitive processes (especially those designed for efficiency and specialization); and data management to reduce
inefficiencies (e.g., removal of obsolete information by forgetting). The next section further clarifies this definition.

3. General Categories of Metacognition

This section delineates general categories of human-level metacognition based on their input and output. It
considers these categories in turn and provides prominent examples of each, along with empirical evidence and a de-
scription of their key properties. For clarity and brevity, the focal central process is “decide.” We align metacognition
732 Jerald D. Kralik et al. / Procedia Computer Science 145 (2018) 730–739
Author name / Procedia Computer Science 00 (2019) 000–000 3

Fig. 2. Four categories consistent with Figure 1. Category 0 is cognition itself, with primary input from perception and primary output to action
control. Categories 1, 2, and 3 comprise metacognition. Category 1: Primary input and output from and to decision processes. Category 2: Primary
input from decision processes; primary output feeds forward to action control. Category 3: Input from perception; primary output to decision
processes.

closely with cognition itself by first identifying cognition as Category 0, the mapping of perception into action (Fig. 2).

Category 1
For Category 1 metacognition, signals from other decision processes provide the main input; Category 1’s output
is primarily directed at other decision processes, with the intent to modulate or control them (Fig. 2). Examples from
cognitive neuroscience and cognitive science provide a clearer view of Category 1 metacognition.
Arbitration of Model-Free vs Model-Based Reinforcement Learning. Decision neuroscience has proposed two dis-
tinct types of reinforcement learning in the human brain: model free (MF) and model based (MB), which account for
habitual and goal-directed behavior control, respectively [12, 15]. An “arbitrator” here is a meta-control mechanism
between MB and MF systems [40]. An input to the arbitrator is the estimated reliability of each system, which is com-
puted from the average amount of state prediction errors and reward prediction errors, respectively. The arbitrator then
determines the amount of influence MB and MF should have. Neural evidence suggests that ventrolateral prefrontal
cortex (vlPFC) computes reliability, which in turn results in the model choice probability (P MB ). Given P MB , vlPFC
chooses the more reliable of MB or MF to directly control human behavior.
Self-Representation. A self-concept is the ability to situate ourselves in the world and reflect on how we act and feel.
It is presumed to arise from the lower processes that define ourselves in the first place, that is, from other processes,
including cognitive ones. Cognitive neuroscience research has identified a brain region (ventral medial prefrontal
cortex) that mediates this self-concept. It activates when we think about ourselves as opposed to others [23]. The way
a metacognitive process uses a self-representation determines its category. The next example demonstrates its use in
Category 1 metacognition.
Reflection and Self Improvement. Having a self-concept enables multiple important abilities, including that of self-
reflection, and thus of self-modification and improvement. Such behavior requires that a system assess the quality
of its own performance with respect to some standard or benchmark (including its own past behavior). The FORR
cognitive architecture, for example, can manipulate both the value it assigns to its problem-solving mechanisms and
the order in which it references them [18]. Moreover, it can develop provisional new heuristics, observe the impact
they might have had were they to participate in decisions, and gradually incorporate the most reliable of them into its
decision making [17].
 Jerald D. Kralik et al. / Procedia Computer Science 145 (2018) 730–739 733
4 Author name / Procedia Computer Science 00 (2019) 000–000

Self-control. Category 1 self-control is well illustrated by a high-profile functional imaging study in which the brain
activity of dieters was compared to that of others. The study found that dieters’ higher-level health or weight-loss self-
concept based interest modulated their lower-level valuation process in the striatum. In other words: it actually lowered
the value for junk food rather than permit a direct competition between the choice outcomes of two higher-level and
lower-level behavioral control systems [25].
Artificial Category-1 Metacognition. Further evidence for the use of this category of metacognition in artificial
systems include Clarion [9], MIDCA [10], FORR [16], Sigma [51] and Soar [52]. The key aspect here is how to
develop a representational and processing space that is connected to but separate from the base space in which direct
interaction with the world occurs. The first two of these architectures provide a separate module for metacognition,
while the latter three essentially recur on the base space to do this.

Category 2
For Category 1, input to the metacognitive processes is output from other decision processes (or other signals de-
rived during the decision process), and output is sent to decision processes (rather than, e.g., to actuators). In contrast,
Category-2 metacognitive processes do more than merely control/modulate lower-level, behavioral-control systems.
Category-2 metacognitive processes are themselves behavioral-control/problem-solving systems; they develop prob-
lem representations that lead to decision making and action selection. A key feature of Category-2 metacognition is
its use of input from other, lower-level decision processes to inform its own decision making. Social cognition is a
particularly salient example of Category 2.
Social cognition. A social setting (e.g., a multiagent system) often requires that an agent have a self-concept, a compu-
tational model of human decision making that models the agent within a multiagent environment. Such a self-concept
considers possible scenarios (i.e., decision options) with respect to potential social interactions, possibly from a game-
theoretic perspective. To the extent that the self-concept models other agents the way it models itself, higher-level
social cognition also provides an example of metacognitive elements in higher-level cognition. More specifically, ex-
tensive research in social psychology and neuroscience has established that people model each other’s beliefs, goals,
and intentions, and think about their minds much the way they think about themselves [23]. Note, however, that
all of social cognition is not necessarily meta-level or even high-level; it too shades from simple (e.g., dominance
hierarchies) to complex (e.g., theory of mind).
Social constructs may be based on larger social groups, especially social rules (from norms and conventions to
laws). Whether such rules are meta-cognitive depends on exactly how they are processed by individuals and/or mod-
eled by artificial systems. Again, this can range from simple (e.g., rules as punishment to avoid) to complex (e.g.,
moral principles) [23].
Finally, much higher-level human cognition requires sophisticated cognitive machinery to coordinate with other
cognitive systems. This is especially clear with social cognition, where almost any problem (e.g., organizing meals,
working, raising a family) must pass possible solutions through a social filter. The filter determines whether a possible
choice remains viable given the interests and dynamics of others. Such interaction across different content domains
(e.g., finding food vs. sociability) ultimately requires sophisticated coordination among a set of cognitive processes,
and therefore metacognition.
Artificial Category-2 Metacognition. Evidence for Category-2 metacognition in artificial systems includes developed
computational models of social cognition [1, 36, 37, 38, 39, 49, 50] inspired by socio-cognitive theory on human
decision making [6]. For example, regarding higher-level societal understanding, N-2APL [1] and N-Jason [38, 39]
have metacognitive components that enable decision making with social norms. Their cognitive agents can decide
whether to follow their individual goals or deontic goals (related to obligation and permission) triggered by social
norms. This allows a cognitive agent to be autonomous over social constructs, that is, it can choose the normative
goals or abandon compliance with them.

Category 3
Category-3 metacognition includes processes that receive their input primarily from feedforward representations
of environmental stimuli (e.g., from perceptual processes), but primarily project to other cognitive processes (Fig. 2).
734 Jerald D. Kralik et al. / Procedia Computer Science 145 (2018) 730–739
Author name / Procedia Computer Science 00 (2019) 000–000 5

A prominent example from cognitive neuroscience is context and abstract task relevant information. Evidence from
cognitive neuroscience shows that higher-order brain regions (e.g., regions in the prefrontal cortex) provide more
sophisticated environmental information to basic decision-making systems [23].

Summary
The categories of metacognition outlined here are meant to clarify the broad possibilities of what constitutes
metacognition. In practice, the boundaries between categories themselves can become fuzzy and gradated, especially
in sufficiently complex computational systems. Nonetheless, this categorization helps clarify the general characteris-
tics of cognitive and metacognitive processes. The next section provides a list of some of the major components of
metacognitive decision processes.

4. Components of Metacognition

This section identifies the components of metacognition for which substantial empirical evidence exists (e.g., [23]).
This list (Fig. 3) currently centers on Category-1 metacognition, with input, central focus, and output all as decision
processes (see Fig. 2).

Fig. 3. Components of Category 1 Metacognition

Monitoring. Monitoring occurs when a metacognitive process receives input from the cognitive processes it attempts
to influence. In the brain, for example, evidence implicates particular brain regions (e.g., medial prefrontal cortex, and
in particular, anterior cingulate) involved in monitoring [23].
Evaluation. Once activity from the monitored cognitive systems is received, the metacognitive system must then
evaluate it. A particularly strong example of this is evidence that a region of the prefrontal cortex in the human brain
arbitrates among candidate behavioral-control systems via an evaluation process that compares their relative likelihood
of success (see Category 1 Section above) [7, 12, 33, 40].
Planning. Because evaluation by higher-level metacognitive control systems is relatively sophisticated, evaluation
should include an assessment of future success as well as identification of the best action policies to achieve it. Plan-
ning systems can be quite complex. For example, they may have goal hierarchies that require dynamic management
to use and update them during task completion [23].
Mental Simulation. Similar to and often in conjunction with planning, mental simulation provides the ability to play
out imagined possible scenarios before a given action is chosen. Such simulations require relatively rich mental models
of the problem environment. Consciousness, described in Section 5, also appears to play an important role in forming
mental models; it integrates aspects of the present, the past, and the future as part of a correlated scene [27].
Control. Category-1 metacognitive processes are dedicated to coordinating (or orchestrating) activities of lower-level
behavioral-control systems. They include arbitrating among systems (i.e., choosing among mutually exclusive ones)
and multitasking, including such sub-processes as scheduling and task switching. Control by Category-1 processes is
normally expected to either modulate or bias the behavioral-control systems it addresses [23, 42].
This list of components is merely a starting point. Other functionality expected to be added includes those related to
self-reflection and self-improvement (e.g., understanding, awareness, generating, organizing, maintaining, modifying,
debugging, healing, configuring, adapting) [26, 27, 32, 36, 48, 53]. We turn next to perhaps the most quintessentially
‘meta’ cognition: conscious processing and consciousness.
 Jerald D. Kralik et al. / Procedia Computer Science 145 (2018) 730–739 735
6 Author name / Procedia Computer Science 00 (2019) 000–000

5. Consciousness

Consciousness involves perceiving, thinking about, and experiencing elements derived from other decision pro-
cesses (e.g., our concept of ‘self’). Thus, consciousness is also a form of metacognition. The ‘Hard Problem’ [8] is
explaining the first-person, subjective experience of human consciousness that goes from self-concept to the interpre-
tation of our experiences as sentient. How and why, for example, people are able to experience things like love, the
color red, self-doubt [11, 14, 22, 23, 24, 47, 58, 59]. To date, there is no philosophical or scientific consensus on this,
but there are notable, important developments, which we outline here.
Tononi [58] and Tononi and Koch [59] described and refined integrated information theory (IIT) as a theoretical
framework to describe and measure consciousness. IIT argues that a theory of consciousness must begin from a set of
axioms based on the phenomena to be explained, and then derive a set of postulates from those axioms. Central to IIT
is the notion that a proper theory of consciousness must first consider the essential properties of the phenomenon that
the conscious being has had, that is, its own experience of the phenomenon. McGreggor [41] established a theoretical
framework that allows such experiences to be considered as proper knowledge representations, a crucial connection
between the various theories of consciousness and the analytical techniques of cognitive science and AI.
Jackson [28] discussed how computers could potentially obtain enough self-awareness to achieve human-level
AI by adapting the ‘axioms of being conscious’ proposed by Aleksander and Morton [2] for research on artificial
consciousness. For a system to approach artificial consciousness, there are a set of metacognitive “observations” it
must achieve:
Observation of an external environment.
Observation of itself in relation to the external environment.
Observation of internal thoughts.
Observation of time: the present, the past, and potential futures.
Observation of hypothetical or imaginative thoughts.
Reflective observation: Observation of observations.
To attain these observational abilities, an AI system would need to create and process data structures that rep-
resent them. Indeed, there appears to be nothing inherently impossible about creating such data structures. Jackson
([28], p.245) discussed how the potential to support artificial consciousness is illustrated by the TalaMind prototype
demonstration system.
Dehaene [13] described consciousness as “the mind’s virtual reality simulator.” The functions of consciousness,
he argued, are the stable retention of information (as opposed to the fleeting signals of perception), the compression
of information to facilitate routing and further processing, and the ability to broadcast information through language.
These capabilities should also be considered in a cognitive model of consciousness.
An interpreter and inner speech are also critical features of human consciousness. The former is a unified cognitive
process that mediates the sense of “I” or “me”, and the control we believe we have over our decisions [22]. Function-
ally, this interpreter is an overarching cognitive system that organizes the “findings” of the multiple other lower-level
processes to produce one coherent story (and sense of self). This interpreter appears to seek a consistent narrative
that makes sense of the world causally, with this story considered as a set of higher-level beliefs. The interpreter can
then use these beliefs to manipulate and affect the agent’s goal-directed behavior. Consciousness thus appears to have
aspects drawn from Types 1, 2 and 3 metacognition to orchestrate other systems, and to follow its own muse and
decision policies. Representation of the interpreter and inner speech in metacognition would support several of the
axioms of consciousness: observation of internal thoughts, observation of hypothetical or imaginative thoughts, and
reflective observation.
Johnson-Laird ([30], pp. 448-477) discussed how a computational system could approach artificial consciousness.
He reasoned that such a system must process in parallel, and that a form of self-awareness could result if the system
could recursively represent mental models within mental models and have a higher-level model of its own operating
system.
Beyond the emulation of human consciousness, Sanz [54] offered an alternative perspective to Aleksander and
Morton’s axioms. Gamez [21] offered an approach to neutralize philosophical conundrums around consciousness to
ground a scientific, measurable theory that can be used in the analysis of consciousness in metacognition for both
humans and machines.
736 Jerald D. Kralik et al. / Procedia Computer Science 145 (2018) 730–739
Author name / Procedia Computer Science 00 (2019) 000–000 7

Finally, in neuroscience, affective processing has been identified as playing a key role in conscious experience
[11, 23, 46]. In short, the sentience that humans feel derives at least in part from a highly integrated (and likely
“resonant loop”) signal of both deeply bottom-up body-state signals with those from the highest top-down conceptual
understanding. Progression towards a greater understanding will therefore require focus on the processes involved in
the large-scale integration of cognitive processing. Metacognition should play a prominent role, since it focuses on
the mechanisms of system-wide integration.

6. Design Issues and Case Studies

To help provide a roadmap for constructing models of metacognition, this section presents general design issues,
followed by case studies where metacognitive components have been successfully added to working computational
models of cognition.

6.1. Design Issues for Metacognition

The homunculus fallacy. A persistent issue for metacognition concerns whether to consider metacognition as a
central module for executive control. This can give rise to the homunculus fallacy, where a little person inside one’s
brain reasons like an intelligent being to deal with the situation it observes. Such reasoning leads to an infinite regress:
to explain how the homunculus functions, one must assume that it has a mind, which itself implies another homunculus
inside it, which must contain yet another homunculus, and so on. One possible solution to this problem is to provide
a priori constraints on what the highest-level executive system should entail, both in what it can achieve and how it
is constructed. From a functional and architectural perspective, it may be best to conceive of metacognition not as a
collection of parts but holistically, as a whole distributed over many components. Ultimate understanding is not located
in any one of the components, but in the network of their relations or interconnections, and in their inter-processing.
Internal languages of thought. In principle, we can generate arbitrarily long recursive metacognitive processes and
their output (e.g., long, embedded sentences), In practice, however, probably only a few combinations require execu-
tion in a metacognitive system. One general approach would be to build additional components for the model once
sufficient evidence supports them. Another design possibility, however, would make the system itself able to add a
metacognitive process when it is needed. This could, for example, be prompted by a decision point relative to an
ongoing process. For example, “I don’t know how to do X, so I can try to learn how to do X.” Systems could also
be designed to halt metacognitive processes when they are no longer deemed worthwhile. This might use mecha-
nisms corresponding to an ‘economy of mind’ [60]. In addition, recursively nested mental models [30], and a ‘natural
language of thought’ [29] can be tools for representation and implementation of metacognition.
A multi-language approach to an internal language of thought proposed a cognitive hierarchy of logical languages.
Other logical language families (e.g., AI’s description logics or classical mathematical logic with its propositional
logic, first-order and higher-order predicate logics) emphasize cognitive adequacy as measured by time complexity,
following Newell [45]. In particular, the continuous domains of perception require high-complexity description logics
built on taxonomic reasoning. Decision processes, however, must handle perceptual input much faster than taxonomic
inference. Accordingly, a logical hierarchy was developed that is based on a minimalistic language for reasoning about
continuous domains equivalent to a fragment of propositional logic, that is, fast enough for real-time processing at the
time scale higher end of perception and the lower end of reasoning (see [55]). A recent result has shown that, through
a strictly logical, ontology-free, and particularly simple reasoning mechanism, this primitive language’s formulae give
rise to graphical representations analogous to the content expressed in the formulae [55]. This result opens new ways
to connect the lower end of decision with the higher end of perception. It is also fundamental for higher cognition
because it provides a simple mechanism for the construction of mental images from logical representations, which
can be employed for the construction or reconstruction of remembered, inferred, or communicated contents. That is,
the hierarchy can provide a primitive step that facilitates metacognitive tasks.
Limitations of human cognition. Finally, an important design consideration is the extent to which any model of
cognition including metacognition should mirror human abilities. On the one hand, human cognition provides an
important existence proof of some of the highest cognitive and metacognitive abilities known to exist, yet it often
 Jerald D. Kralik et al. / Procedia Computer Science 145 (2018) 730–739 737
8 Author name / Procedia Computer Science 00 (2019) 000–000

makes simple logical errors [30, 31]. Thus, there may be a downside to building cognitive systems that fall into the
same logical traps as people. Perhaps we should not be guided solely by human models.

6.2. Metacognition in Cognitive Architectures

Computational frameworks for cognition have tackled metacognition from different perspectives depending on the
underlying neuropsychological and psychological theories used for their construction. Next, we briefly present three
case studies for metacognition: the ACT-R [3], CLARION [57], and LIDA [20] cognitive architectures.
While a metacognitive module is not included in the core ACT-R architecture, recent work has developed a
metacognitive module [4] that consciously assesses what one knows and how to extend it to solve a problem. This ac-
tivity is associated with the rostrolateral prefrontal cortex (rlPFC), which has been linked to reflective functions (e.g.,
prospective memory, reasoning about analogies). The metacognitive module implements this activity by reflecting on
declarative representations of cognitive procedures. Use of the metacognitive module was illustrated with Exception
and Regular (simpler) mathematical problem solving. Solution of Exception problems required the modification or re-
placement of elements in procedures for solving Regular problems. The metacognitive module builds a representation
of the required elements. The working memory holds the problem representation, while the metacognitive module
holds declarative representations of procedures to be modified and “rehearses” the modified procedure.
CLARION’s hybrid architecture comprises two representation levels; symbolic and subsymbolic. It is based on
Flavell’s notion of metacognition as the active monitoring and consequent regulation and orchestration of cognitive
processes in relation to the cognitive objects and data of which they bear) [19]. CLARION uses multiple metacog-
nitive criteria to decide when and how to use symbolic or sub-symbolic processing; a particular learning method
(e.g., reinforcement, supervised, unsupervised) or combination of them; and a specific reasoning mechanism (e.g.,
rule-based, similarity-based). CLARION also provides cross-layer learning mechanisms to synchronize (accommo-
date and assimilate knowledge) in both symbolic and subsymbolic layers, to enable both top-down and bottom-up
learning. Moreover, CLARION includes a variety of metacognitive processes to set parameters (e.g., learning rates,
thresholds, temperature in stochastic decision making, action costs); set dynamic goals driven by competition; and set
reinforcement functions to measure the agent’s degree of satisfaction. CLARION’s metacognition thereby depends
heavily upon interaction with a motivational subsystem concerned with drives and their interactions.
Unlike CLARION, LIDA does not define a specific module or subsystem for metacognition; instead, metacog-
nition emerges from the interaction of cascading sequences of cognitive cycles corresponding to action-perception
loops. Metacognition in LIDA is based on Sloman’s classification of levels of control [56]. These include reactive (for
agents requiring little flexibility in their action selection), deliberative (higher-level cognitive processes as planning,
scheduling and problem solving), and metacognitive levels (monitoring deliberative processes, allocating cognitive
resources, and regulating cognitive strategies). Metacognition in LIDA is implemented by a collection of appropriate
behavior streams, each with its own metacognitive task. Metacognitive control adds yet another level of flexibility to
an agent’s decision making, allowing it to function effectively in an even more complex and dynamically changing en-
vironmental niche. Additionally, LIDA defines an artificial consciousness mechanism based on the Global Workspace
theory, a neuropsychological theory of consciousness and cognition [5]. Attention ‘codelets’ are little processes that
bring items of interest to consciousness, gather current information from the workspace, and compete to see which
can bring its information to consciousness. The winner’s information is broadcast widely throughout the cognitive ap-
paratus. The purpose of the conscious broadcast is to recruit appropriate resources with which to deal with the current
situation. Though various types of resources can, theoretically, be recruited, the conscious broadcast is mostly aimed
at procedural memory, where it can directly bring to bear the information in the contents of consciousness so as to
affect the next action to be chosen.

7. Discussion and Conclusions

The first steps towards an architecture for metacognition are to develop a common language, and to outline the main
concepts and research across the relevant fields, which the current paper has begun. The next steps should elaborate
on every section, and begin piecing them together to construct a consensus model of metacognition. In 1973, Allen
Newell challenged scientists to achieve “a science of [humans] adequate in power and commensurate with [their]
 738 Jerald D. Kralik et al. / Procedia Computer Science 145 (2018) 730–739
Author name / Procedia Computer Science 00 (2019) 000–000 9

complexity” [44]. The endeavor to include metacognition in a Common Model of Cognition is one way to accept his
challenge [35].

Acknowledgements

Contributions by JDK, JHL and SWL were supported by the ICT R&D program of MSIP/IITP [2016-0-00563,
Research on Adaptive Machine Learning Technology Development for Intelligent Autonomous Digital Companion],
Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government
(MSIT) (No.2018-0-00677, Development of Robot Hand Manipulation Intelligence to Learn Methods and Proce-
dures for Handling Various Objects with Tactile Robot Hands), and Samsung Research Funding Center of Sam-
sung Electronics under Project Number SRFC-TC1603-06. Contributions by OJR were sponsored by the U.S. Army,
Verizon-CMU InMind project. Contributions by PSR were supported by the U.S. Army under contract W911NF-14-
D-0005. Statements and opinions expressed do not necessarily reflect the position or the policy of the United States
Government, and no official endorsement should be inferred.

References

[1] Alechina, N., Dastani, M., Logan, B., 2012. Programming norm-aware agents, in: Proceedings of the 11th International Conference on
Autonomous Agents and Multiagent Systems-Volume 2, International Foundation for Autonomous Agents and Multiagent Systems. pp. 1057–
1064.
[2] Aleksander, I., Morton, H., 2007. Depictive architectures for synthetic phenomenology. Artificial consciousness , 67–81.
[3] Anderson, J.R., 2009. How can the human mind occur in the physical universe? Oxford University Press.
[4] Anderson, J.R., Fincham, J.M., 2014. Extending problem-solving procedures through reflection. Cognitive psychology 74, 1–34.
[5] Baars, B.J., 2007. The global workspace theory of consciousness, in: The Blackwell companion to consciousness, pp. 236–246.
[6] Bandura, A., 2001. Social cognitive theory: An agentic perspective. Annual review of psychology 52, 1–26.
[7] Boureau, Y.L., Sokol-Hessner, P., Daw, N.D., 2015. Deciding how to decide: self-control and meta-decision making. Trends in cognitive
sciences 19, 700–710.
[8] Chalmers, D.J., 1995. Facing up to the problem of consciousness. Journal of consciousness studies 2, 200–219.
[9] Coward, L.A., Sun, R., 2004. Criteria for an effective theory of consciousness and some preliminary attempts. Consciousness and Cognition
13, 268–301.
[10] Cox, M.T., Alavi, Z., Dannenhauer, D., Eyorokon, V., Munoz-Avila, H., Perlis, D., 2016. Midca: A metacognitive, integrated dual-cycle
architecture for self-regulated autonomy., in: AAAI, pp. 3712–3718.
[11] Damasio, A., 2011. Thinking about brain and consciousness, in: Characterizing Consciousness: From Cognition to the Clinic?. Springer, pp.
47–54.
[12] Daw, N.D., Niv, Y., Dayan, P., 2005. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control.
Nature neuroscience 8, 1704–1711.
[13] Dehaene, S., 2014. Consciousness and the brain: Deciphering how the brain codes our thoughts. Penguin.
[14] Dennett, D., 1991. Consciousness explained. New York: Little Brown & Co .
[15] Doya, K., 1999. What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural networks 12, 961–974.
[16] Epstein, S.L., 1994. For the right reasons: The FORR architecture for learning in a skill domain. Cognitive science 18, 479–511.
[17] Epstein, S.L., Freuder, E.C., Wallace, R.J., 2005. Learning to support constraint programmers. Computational Intelligence 21, 336–371.
[18] Epstein, S.L., Petrovic, S., 2011. Learning a mixture of search heuristics, in: Autonomous Search. Springer, pp. 97–127.
[19] Flavell, J.H., 1976. Metacognitive aspects of problem solving. The nature of intelligence , 231–235.
[20] Franklin, S., Madl, T., Strain, S., Faghihi, U., Dong, D., Kugele, S., Snaider, J., Agrawal, P., Chen, S., 2016. A lida cognitive model tutorial.
Biologically Inspired Cognitive Architectures 16, 105–130.
[21] Gamez, D., 2018. Human and machine consciousness. Open Book Publishers.
[22] Gazzaniga, M., 2011. Who’s in Charge? NY:Harper Collins.
[23] Gazzaniga, M., Ivry, R., Mangun, G., 2013. Cognitive Neuroscience: The Biology of the Mind (Fourth Edition). W. W. Norton. URL:
https://books.google.co.kr/books?id=MBdBmwEACAAJ.
[24] Gazzaniga, M.S., 2018. The Consciousness Instinct: Unraveling the Mystery of How the Brain Makes the Mind. Farrar Straus and Giroux.
[25] Hare, T.A., Camerer, C.F., Rangel, A., 2009. Self-control in decision-making involves modulation of the vmpfc valuation system. Science
324, 646–648.
[26] Hernández, C., Bermejo-Alonso, J., Sanz, R., 2018. A self-adaptation framework based on functional knowledge for augmented autonomy in
robots. Integrated Computer-Aided Engineering , 1–16.
[27] Holyoak, K.J., Morrison, R.G., 2012. The Oxford handbook of thinking and reasoning. Oxford University Press.
[28] Jackson, P.C., 2014. Toward human-level artificial intelligence: Representation and computation of meaning in natural language .
[29] Jackson, P.C., 2018. Natural language in the common model of cognition, in: The 9th Annual International Conference on Biologically Inspired
Cognitive Architectures. (To appear).
 Jerald D. Kralik et al. / Procedia Computer Science 145 (2018) 730–739 739
10 Author name / Procedia Computer Science 00 (2019) 000–000

[30] Johnson-Laird, P.N., 1983. Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness. Harvard University
Press, Cambridge, MA, USA.
[31] Kahneman, D., Egan, P., 2011. Thinking, fast and slow. volume 1. Farrar, Straus and Giroux New York.
[32] Korpan, R., Epstein, S.L., Aroor, A., Dekel, G., 2017. Why: Natural explanations from a robot navigator. arXiv preprint arXiv:1709.09741 .
[33] Kowaguchi, M., Patel, N.P., Bunnell, M.E., Kralik, J.D., 2016. Competitive control of cognition in rhesus monkeys. Cognition 157, 146–155.
[34] Kralik, J., 2017. Architectural design of mind & brain from an evolutionary perspective, in: Proceedings of the AAAI Fall Symposium A
Standard Model of the Mind.
[35] Laird, J.E., Lebiere, C., Rosenbloom, P.S., 2017. A standard model of the mind: Toward a common computational framework across artificial
intelligence, cognitive science, neuroscience, and robotics. AI Magazine 38.
[36] Lee, J., Kralik, J., Jeong, J., 2018a. A general architecture for social intelligence in the human mind and brain, in: AAAI Fall Symposium:
Common Model of Cognition. (To appear).
[37] Lee, J., Kralik, J., Jeong, J., 2018b. A sociocognitive-neuroeconomic model of social information communication: To speak directly or to
gossip, in: The 40th Annual Meeting of the Cognitive Science Society.
[38] Lee, J., Padget, J., Logan, B., Dybalova, D., Alechina, N., 2014a. N-jason: Run-time norm compliance in agentspeak (l), in: Engineering
Multi-Agent Systems, Springer. pp. 367–387.
[39] Lee, J., Padget, J., Logan, B., Dybalova, D., Alechina, N., 2014b. Run-time norm compliance in bdi agents, in: Proceedings of the 2014
international conference on Autonomous agents and multi-agent systems, pp. 1581–1582.
[40] Lee, S.W., Shimojo, S., ODoherty, J.P., 2014c. Neural computations underlying arbitration between model-based and model-free learning.
Neuron 81, 687–699.
[41] McGreggor, K., 2017. An experience is a knowledge representation, in: AAAI Fall Symposium Series Technical Reports.
[42] Miller, E.K., Cohen, J.D., 2001. An integrative theory of prefrontal cortex function. Annual review of neuroscience 24, 167–202.
[43] Nelson, T.O., 1992. Metacognition: Core readings. Allyn & Bacon.
[44] Newell, A., 1973. You can’t play 20 questions with nature and win: Projective comments on the papers of this symposium. Visual Information
Processing , 283–310.
[45] Newell, A., 1990. Unified theories of cognition. Harvard University Press.
[46] Ortony, A., Norman, D.A., Revelle, W., 2005. Affect and proto-affect in effective functioning. Who needs emotions? , 173–202.
[47] Pinker, S., 1999. How the mind works. Annals of the New York Academy of Sciences 882, 119–127.
[48] Project CogX, . http://cogx.eu/. Accessed 20180930.
[49] Pynadath, D.V., Marsella, S.C., 2005. Psychsim: Modeling theory of mind with decision-theoretic agents, in: IJCAI, pp. 1181–1186.
[50] Pynadath, D.V., Rosenbloom, P.S., Marsella, S.C., 2014. Reinforcement learning for adaptive theory of mind in the sigma cognitive architecture,
in: International Conference on Artificial General Intelligence, Springer. pp. 143–154.
[51] Rosenbloom, P.S., Demski, A., Ustun, V., 2016. The sigma cognitive architecture and system: Towards functionally elegant grand unification.
Journal of Artificial General Intelligence 7, 1–103.
[52] Rosenbloom, P.S., Laird, J.E., Newell, A., 1988. Meta-levels in soar, in: Rosenbloom, P.S., Laird, J.E., Newell, A. (Eds.), Meta-Level Archi-
tectures and Reflection. Amsterdam, NL: North Holland, pp. 227–240.
[53] Sampson, W.W., Khan, S.A., Nisenbaum, E.J., Kralik, J.D., 2018. Abstraction promotes creative problem-solving in rhesus monkeys. Cognition
176, 53–64.
[54] Sanz, R., López, I., Rodrı́guez, M., Hernández, C., 2007. Principles for consciousness in integrated cognitive control. Neural Networks 20,
938–946.
[55] Schmidtke, H.R., 2018. Logical lateration–a cognitive systems experiment towards a new approach to the grounding problem. Cognitive
Systems Research .
[56] Sloman, A., 1999. What sort of architecture is required for a human-like agent?, in: Foundations of rational agency. Springer, pp. 35–52.
[57] Sun, R., 2007. The motivational and metacognitive control in clarion. Modeling integrated cognitive systems , 63–75.
[58] Tononi, G., 2008. Consciousness as integrated information: a provisional manifesto. The Biological Bulletin 215, 216–242.
[59] Tononi, G., Koch, C., 2015. Consciousness: here, there and everywhere? Phil. Trans. R. Soc. B 370, 20140167.
[60] Wright, I., 2000. The society of mind requires an economy of mind, in: Proceedings AISB00 Symposium Starting from Society - the Applica-
tion of Social Analogies to Computational Systems, AISB, Birmingham, UK. pp. 113–124.