Accepted Manuscript

Measuring digital literacies: Junior high-school students' perceived competencies

versus actual performance

Erez Porat, Ina Blau, Azy Barak

PII: S0360-1315(18)30173-8
DOI: 10.1016/j.compedu.2018.06.030
Reference: CAE 3394

To appear in: Computers & Education

Received Date: 28 May 2017

Revised Date: 27 June 2018
Accepted Date: 30 June 2018

Please cite this article as: Porat E., Blau I. & Barak A., Measuring digital literacies: Junior high-school
students' perceived competencies versus actual performance, Computers & Education (2018), doi:
10.1016/j.compedu.2018.06.030.

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to
our customers we are providing this early version of the manuscript. The manuscript will undergo
copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please
note that during the production process errors may be discovered which could affect the content, and all
legal disclaimers that apply to the journal pertain.
 ACCEPTED MANUSCRIPT
Measuring Digital Literacies: Perceived Versus Actual
Competencies of Junior High-School Students

Title page

Erez Porat, M.A., PhD candidate

PT
jezporat@gmail.com
Department of Learning, Teaching and Teacher Education; Faculty of Education
University of Haifa, Israel

RI
SC
Ina Blau, Ph.D. (corresponding author)
Associate Professor

U
Website: https://www.openu.ac.il/en/personalsites/ProfInaBlau.aspx
Office: 972-9-778-1325
AN
Fax: 972-9-778-0632
inabl@openu.ac.il; ina.blau@gmail.com
M

Department of Education and Psychology

The Open University of Israel
1 University Road, P.O.B. 808
D

Ra'anana 43107, Israel

TE

Azy Barak, Ph.D.

EP

Professor Emeritus

Website: http://azy.edu.haifa.ac.il/
C

azy@edu.haifa.ac.il
AC

Department of Counseling and Human Development; Faculty of Education

University of Haifa, Israel

Conflict of Interest: The authors declare that they have no conflict of interest.
 ACCEPTED MANUSCRIPT

Measuring Digital Literacies: Junior High-School Students’

Perceived Competencies versus Actual Performance

Abstract

PT
The widespread belief is that youth, "digital natives", who live their entire
lives in media-rich digital environments and are ubiquitously connected

RI
through social networks, naturally develop digital competencies. This study
investigated digital literacies among 280 junior-high-school students with the
aim of comparing participants’ perceived digital literacy competencies and

SC
their actual performance in relevant digital tasks. The findings showed that
only a few of participants' perceived skills were related to their actual

U
performance. Generally, participants displayed high confidence in their
digital literacies and significantly over-estimated their actual competencies.
AN
This gap was most evident in social-emotional skills, which were, on
average, perceived by students as their strongest skills, while their actual
level of performance was very low. Positive strong correlations were found
M

between participants' self-reported evaluations of different digital skills,

indicating their perception as a single factor, while actual performance tests
D

revealed low- to medium-size correlations between different literacies. For

educational decision-makers, the findings highlight the importance of
TE

designing training programs aimed to develop students’ digital literacies,

with a special emphasis on social-emotional competencies. Such training may
enhance important competencies needed, reduce unfounded self-perceptions,
EP

and thus, develop efficient digital functioning in contemporary society.

C

Keywords: digital literacy skills; self-perceptions of digital literacy

AC

competencies; measurement of actual digital literacy competencies; digital

literacy perception-performance gap; technology-enhanced learning.

1
 ACCEPTED MANUSCRIPT

Introduction
The ever-changing variation and rapid growth of information and communication
technologies (ICT) and technology-enhanced work environments, have enriched and
improved professional activities, as well as the private lives of individuals and the way
they work, learn and communicate (Ala-Mutka, 2011). In parallel, these rapid changes

PT
have led to increasing information overload and challenging situations which require a
growing set of digital competencies (Eshet-Alakali, 2012). Developing new literacies is
a crucial determinant of effective functioning in the digital age, enabling individuals to

RI
experience more fulfilling and productive personal and professional lives (Leu et al.,
2013).

SC
Becoming a literate person in the age of knowledge poses considerable challenges
for the 21st-century learner (Ferrari, 2012; Pellegrino & Hillton, 2012). The nature of

U
reading, writing, and interpersonal communication has been, and continues to be,
AN
fundamentally transformed by digital environments. Successful navigation in hypertext
and hypermedia in non-linear environments requires psychosocial, socio-contextual
M

knowledge and skills (Alexander, 2012). These cognitive challenges involve finding
and gathering relevant information, building comprehension from multiple texts, and
D

integrating textual, graphic, and multimedia information to construct representations of

inter-connected sources (Barzilai & Eshet-Alkalai, 2015; Barzilai, Tzadok, & Eshet-
TE

Alkalai, 2015).

Schools and education systems have been called to integrate digitally-rich learning
EP

platforms and tools for teaching and learning, in acknowledgement of the potential
benefits of emerging new technologies and digital learning environments which have
C

become an integral part of youth's everyday lives (e.g., Blau & Shamir-Inbal, 2016).
AC

School ICT accessibility has been found to be positively related to students’ self-
reported digital skills (Zhong, 2011). In contrast, Zhong found a negative association
between the rate of ICT integration in different countries and adolescents’ digital skills.
This finding implies that increased ICT penetration rate per se, without the integration
of school programs which teach digital skills explicitly, does not guarantee that
adolescents learn how to use ICT effectively in an educational setting. Moreover, since
technology is dynamic and constantly changing, students are required to update their
knowledge and competencies on an ongoing basis (OECD, 2015). Indeed, high-

2
 ACCEPTED MANUSCRIPT

performance workplaces rapidly integrate emerging technologies to increase

productivity. This trend has led to fundamental changes in both the practices and nature
of the literacies needed to function productively within organizations (Barzilai & Eshet-
Alkalai, 2015; Leu et al., 2013).

This study investigated the digital literacy competencies of junior high school

PT
students and compared their self-evaluations of their own literacy level to their actual
performance on digital tasks. The purpose of this comparison was to examine the

RI
relationships between subjective perceptions and objective achievements, to reach a
better understanding of youth’s behavior in digital environments. The next sections first

SC
define the concept of digital literacy and discuss various frameworks of digital
competencies. Following that, we focus on the younger generation’s level of digital
competencies and ways of developing digital literacies in an educational setting.

Digital Literacy: Definition and Frameworks

U
AN
Mastering digital literacies is "crucial for survival" (Ala-Mutka, 2011; Eshet-
Alkalai, 2012) in the knowledge society. It comprises more than just arming oneself
M

with the technical abilities to operate digital devices, tools, and surfing the Internet, but
rather comprises a set of knowledge, skills, and attitudes needed to become functional in
D

a digital environment (Ferrari, 2012). Digital literacy is described as complex thinking

TE

competencies, involving cognitive, motor, emotional and social skills (Eshet-Alkalai,

2012), that enable users to operate intuitively and effectively in digital environments for
work, learning, and daily functioning (ECDL Foundation, 2012, 2016; Hague & Payton,
EP

2010; Mohammadyari, & Singh, 2015). Digital literacy includes the ability to find
relevant information and evaluate its credibility, communicate successfully with
C

invisible others (mostly through written text) and create original content to express
AC

oneself in a manner consistent with one's personal or/and professional goals (Iordache,
Mariën, & Baelden, 2017).

Researchers who study digital literacies use different terms, a variety of meanings,
and competing definitions of these competencies (Ala-Mutka, 2011; Ferrari, 2012;
Iordache et al., 2017). One possible explanation for this might be different
understandings of the competencies by researchers coming from different academic
disciplines – sociology, psychology, media and communication, computer sciences,

3
 ACCEPTED MANUSCRIPT

education, or information sciences (Bawden, 2001; 2008). Furthermore, regardless of

the field, some scholars perceive digital literacy as primarily associated with technical-
operational skills (Cihak et al., 2015; Murray, & Pérez, 2014; Oblinger, & Oblinger,
2005), whereas others view it as focused on cognitive and social-emotional aspects
(Eshet-Alkalai, 2012; Gilster, 1997; Greene et al., 2014; Meyers et al., 2013; Pangrazio,
2016; Rouet, 2006; Tapscott, 1998; Wiley et al., 2009).

PT
Several conceptual (e.g., Ala-Mutka, 2011; Ng, 2012) and empirically-tested

RI
(Helsper, & Eynon, 2013; van Deursen, & van Dijk, 2008; 2010; van Deursen, Helsper,
& Eynon, 2015) frameworks of digital literacy have been proposed, emphasizing

SC
instrumental-operational and selected cognitive aspects. For example, a report published
by the European Commission Joint Research Centre (Ala-Mutka, 2011) suggested a
conceptual model of digital competence considering (1) instrumental knowledge and

U
skills for using digital tools and media; (2) advanced skills and knowledge required for
AN
communication and collaboration, information management, learning and problem-
solving, and meaningful participation; (3) attitudes related to strategic use of these skills
M

in intercultural, critical, creative, responsible and autonomous ways. According to Ala-

Mutka, instrumental knowledge and skills comprise a basis for developing or using
more advanced skills. Similarly, Ng’s (2012) digital literacy model consists of three
D

intersecting dimensions (i.e., technical, cognitive, social-emotional) of digital literacy.

TE

Regarding the empirically-tested models for measuring digital literacy, the vast
majority of studies (e.g., Helsper, & Eynon, 2013; van Deursen, & van Dijk, 2008;
EP

2010; van Deursen et al., 2015) have examined participants' subjective evaluations of
their own competencies, rather than their actual digital literacy performance. In one
C

such study, Helsper and Eynon (2013) measured four skill categories, which are both
operational and strategic (i.e., technical, social, critical, creative skills). However, the
AC

measures used were problematic. For example, in order to measure creativity, they used
items such as "uploading photos" and "downloading music", which express the
exchange of digital content and are not commonly used for measuring creativity.
Moreover, the same items were included in more than one index, namely, "participating
in discussions online" was included in both the technical and the social indexes, instead
of the social index alone; "learning to use a new technology" was a part of both
technical and creative skills, instead of technical skills alone.

4
 ACCEPTED MANUSCRIPT

Another empirically-tested digital literacy framework, based on participants' self-

report, was suggested by van Deursen, and van Dijk (2008). This model consists of four
types of skills with operational definitions, which can be used to measure digital skills
(i.e., operational, formal, information, and strategic skills). Two additional skills were
later added to this framework (i.e., communication and content creation skills (van Dijk,
& van Deursen, 2014). More recently, these authors proposed an updated framework of

PT
digital literacy (van Deursen et al., 2015) which was cross-culturally examined and
consisted of five different types of internet skills: operational, information navigation,

RI
social, creative, and mobile. However, two out of five indexes – operational and mobile
- measure basic technical skills rather than cognitive or social-emotional literacies.

SC
In contrast with most of the contemporary frameworks, the original definition of
digital literacy, as the ability to make informed judgments concerning what one found

U
online (Gilster, 1997), emphasized thinking processes to a greater extent than technical
AN
aspects of the concept. Therefore, although Gilster's definition of digital literacy was
introduced decades ago, it still remains relevant (Ala-Mutka, 2011). A holistic and
M

comprehensive framework of digital literacy (Eshet-Alkalai, 2004, 2012), which is

consistent with Gilster’s original definition, covers most of the cognitive competencies
that users or learners employ while working in digital environments. Moreover, it
D

"brings additional elements into discussion that are crucial to ensure a full take-up and
TE

capital-enhancing usage of digital media contents and online environments" (Iordache,

Mariën, & Baelden, 2017, p.12). This framework conceptualizes digital literacies
EP

primarily as cognitive competencies, beyond technical skills, as well as thinking

strategies and mindsets. Methodologically, in contrast with previous self-report studies,
the framework of digital literacy is based on the analysis of users' actual performance in
C

digital environments, with tasks that require the application of a variety of competencies
AC

(Barzilai & Eshet-Alkalai, 2015; Eshet-Alkalai & Amichai-Hamburger, 2004; Eshet-

Alkalai & Chajut, 2009, 2010; Eshet-Alkalai & Geri, 2010).

The six cognitive and social-emotional skills that comprise the conceptual model of
digital literacy by Eshet-Alkalai (2012) include: 1) photo-visual thinking: the ability to
intuitively understand messages and instructions displayed in visuals and in graphic
interfaces. 2) Re-production thinking: the ability to create new meanings and original
interpretations from pre-existing independent pieces of digital information or content. 3)

5
 ACCEPTED MANUSCRIPT

Informational thinking: critical evaluation of the quality of online information and the
credibility of digital sources. 4) Branched thinking: the ability to construct knowledge
and reach understanding through non-linear navigation in hypertextual environments,
and maintain focus without "getting lost" in cyberspace. 5) Social-emotional thinking:
the ability to benefit from digital communication, information sharing, exchange of
insights, expression of emotions, and the projection of one's online persona, while

PT
avoiding online risks. 6) Real-time thinking: the ability to effectively process
large amounts of stimuli and information simultaneously at a high-speed in digital

RI
environments, such as learning simulations, digital games or synchronous online
teaching and learning. Thus, Eshet-Alkalai's framework seems to include three digital

SC
literacies which are not covered by Helsper and Eynon's (2013) model: photo-visual
competencies; social-emotional skills, and real-time thinking, as well as two digital

U
literacies which are not covered by van Deursen and van Dijk’s (2014) original model
AN
or the updated framework by van Deursen et al. (2015): photo-visual skills and real-
time thinking.
M

The topic of computer and information literacy (CIL), which is similar to digital
literacy in some ways, was explored in The International Computer and Information
Literacy Study (ICILS; Fraillon, Ainley, Schulz, Friedman, & Gebhardt, 2014). This
D

study was first administrated in 2013 in 21 countries, and is planned to be repeated in

TE

2018 in additional countries, including the US. It is important to note that the
framework of CIL is essentially different from digital literacy and consists of two
EP

strands that frame the following competencies (Fraillon et al., 2014): Strand 1 -
collecting and managing information - incorporates aspects such as knowing about and
understanding computer use, accessing and evaluating information, and managing
C

information. Strand 2 of the CIL construct - producing and exchanging information -

AC

includes aspects such as transforming information, creating information, sharing

information, and using information safely and securely. As can be seen, some of these
competencies are technical and related to the operation of technology (i.e., computer
literacy), while others, such as accessing and evaluating information, are cognitive in
nature and correspond to information literacy in the DL framework (Eshet-Alkalai,
2012). Additional aspects of CIL show only partial overlapping with the DL framework:
creation of information is a part of reproduction thinking – cognitive DL skills, while

6
 ACCEPTED MANUSCRIPT

online safety and security, ethical use of electronic information and sharing information,
are part of socio-emotional digital literacy. Thus, in comparison with CIL, the scope of
the DL framework (1) excludes technical-operational skills of knowing about and
understanding computer use and (2) includes cognitive literacies such as photo-visual,
branching, and real-time thinking, which are not in the focus of the CIL model, and (3)
addresses reproduction thinking and social-emotional literacies in a broader manner

PT
than CIL. Moreover, the operationalization of students' ICT self-efficacy in the ICILS
questionnaire is focused exclusively on the technical-procedural elements of using

RI
computers. In sum, although some recent studies have reported on performance and
students’ self-report of their perceived competencies (e.g., Fraillon et al., 2014, Rohatgi,

SC
Scherer, & Hatlevik, 2016), they explored the concept of CIL that is essentially different
from DL, which is the focus of the current study.

Developing Youth’s Digital Literacies

U
AN
Young people are presented by popular media and addressed in the public debate as
"digital natives" (Palfrey, & Gasser, 2008; Prensky, 2001). "Generation Z" is described
M

as "fluent users of technology" who feel emotionally attached to digital environments

(Ng, 2012; Turner, 2015). This perspective—digital native versus digital immigrants—
D

was lately criticized by many researchers (cf. Kirschner, & van Merriënboer, 2013),
TE

including Prensky (2009) himself. However, in educational practice, the young

generation of learners is perceived as having "natural abilities" to use digital
technologies, sophisticated technological expertise, and even new cognitive
EP

abilities (Ahn & Jung, 2016; Dede, 2005).

With regard to the interplay between the perceptions and actual digital literacy
C

competencies of the young generation, research shows that youth evaluate themselves as
AC

experts and rate their competency level as high or very high (Smith & Caruso, 2010). In
practice, however, they are far from being proficient and productive users of digital
environments. There are significant gaps between what students think they know about
the technology and their actual performance (Hargittai, 2010; Porat, Blau, & Barak,
2017). Moreover, there is a gap between the use of technologies for social purposes and
"digital wisdom" (Blau, Peled & Nusan, 2014; Shamir-Inbal & Blau, 2016), which
refers to students’ productive use of technologies, by students for effective learning and

7
 ACCEPTED MANUSCRIPT

by teachers as added-value to their instruction. Therefore, some researchers refer to

"digital savviness" as an urban legend (Kirschner, & van Merriënboer, 2013).
According to this argument, youngsters’ daily engagement with digital media is
primarily for entertainment and social practices. Thus, when the requirements are
beyond the technical use of digital devices but rather involve critical thinking, and
cognitive, social-emotional, and ethical dimensions related to the effective use of digital

PT
technologies for learning, the level of students’ performance tends to be mediocre to
poor (ACARA, 2015; Calvani et al., 2012; ECDL, 2016; Gui, & Argentin, 2011).

RI
Regarding the development of digital literacy in education settings, previous

SC
research has shown that digital natives can and should be taught digital literacy (Ng,
2012). However, Ng’s study was conducted in a higher education setting, was
exclusively based on self-reports, and showed improvement in students’ technical rather

U
than cognitive or social-emotional competencies. In relation to the instructional design
AN
which is recommended for developing students’ digital literacies, some authors (e.g.,
Blau et al., 2014; Hicks, & Hawley Turner, 2013; McLoughin & lee, 2008; Peled, Blau,
M

& Grinberg, 2015) have criticized teaching practices associated with developing digital
literacy competencies. They have argued that the use of classroom technology mostly
serves to preserve existing teaching practices, rather than as an opportunity to develop
D

students’ digital literacy competencies. Other scholars have criticized traditional teacher
TE

professional development programs and training workshops for being ineffective in

providing future and in-service teachers with the kinds of experiences, knowledge, and
EP

competencies needed to prepare them for effective integration of technology in the

classroom (Blau, 2011b; Carlson, & Tidiane Gadio, 2002; Peled et al., 2015).
C

Research Aims and Questions

AC

As the literature reviewed above shows, most previous studies have explored digital
literacies based on self-report alone, and have therefore suffered from issues of external
validity. To the best of our knowledge, no previous research in the field of education
has examined calibration by comparing self-evaluation of digital literacies with actual
performance, and only one study (van Deursen, 2010) examined such calibration in the
field of communication. The current study explored the perceived digital literacies of
junior high-school students, their actual competencies revealed in performance tasks

8
 ACCEPTED MANUSCRIPT

and the association between the two. Based on the framework of digital literacy
competencies defined by Eshet-Alkalai (2012), this study explored the following
research questions: (1) What is the nature of the associations between students’
performance level on different digital literacy competencies? (i.e., photo-visual,
reproduction, branching, information, social-emotional, and real-time thinking tasks) (2)
What is the nature of the associations between students’ self-reported evaluations of

PT
their own competence in relation to different digital literacies? (3) Is there an
association between students’ perceived digital literacy competencies and their actual

RI
performance?

SC
Method
Participants

U
Participants included 280 Israeli junior-high school students in the seventh grade
AN
(approximately 13 years old). The students attended a variety of geographically
dispersed Hebrew-speaking schools in the state education system. The participants
included 142 (50.7%) boys and 138 (49.3%) girls. These young learners are considered
M

"digital natives" (Prensky, 2001) and thus, on average, are expected to have relatively
high proficiency in the use digital technologies.
D

None of the participants studied in one-to-one computing classrooms, but rather,

TE

they all studied according to the mainstream technological model currently integrated in
Israeli schools (Blau & Shamir-Inbal, 2016), based on whole-class technologies, such as
EP

a class computer connected to an overhead projector or an interactive whiteboard. In

this model only teachers have access to a computer which is connected to the internet
C

and can be used to project digital contents and digital textbooks on a screen. Students
who participated in the study only had occasional access to a computer lab and mobile
AC

laptop cart and did not use technology for learning purposes in school on a daily basis.

Procedure

The study received the approval of the institutional ethics committee. Data was
collected in the spring of 2016. The participants were allocated 90 minutes to perform a
battery of six digital literacy tasks and to evaluate their perceived competencies on those
tasks (see below). A few students requested extra time to complete the tasks and were

9
 ACCEPTED MANUSCRIPT

given extensions. Data regarding performance on digital tasks was collected through an
online environment developed for the study using a Moodle platform. Self-estimations
of digital competencies were reported using a Google Forms platform. The data were
analyzed using SPSS (v. 22).

Instruments

PT
Performance on Digital Literacy Tasks

The actual digital literacies of students were tested using a series of authentic tasks

RI
relevant to learning settings, based on previous studies which showed that practical
performance tests can reliably check actual levels of digital skills (Eshet, & Amichai-

SC
Hamburger, 2004; Gui, & Argentin, 2009, 2011). Three researchers in the field of
educational technology assessed the face validity of each performance task. Each of the

U
digital tasks required effective use of a particular type of digital literacy skill. The tasks
AN
reflect typical assignments designed to apply digital knowledge and skills in real-world
learning challenges.
M

Photo-visual literacy was examined similarly to previous research (Eshet, &

Amichai-Hamburger, 2004) by using an online photo design and editing application
D

(Pixlr Express; see https://pixlr.com/express) to create a greeting card based on a given

digital image combined with a given text. This application has a sophisticated user
TE

interface, which includes toolbars and navigation menus that consist of icons, symbols,
and other visuals. Interacting intuitively with the environment through graphical icons
EP

and understanding the usage cues provided through graphical representation of symbols
and icons, require photo-visual literacy. All the participants reported that they were not
familiar with the Pixlr Express application. They received no instruction on how to use
C

it and needed to decipher the graphic interface in order to use it effectively and perform
AC

the task.

Reproduction literacy: The original performance task, as developed previously by

Eshet and Amichai-Hamburger (2004), tested only textual reproduction literacy. In that
task, participants were asked to convey a new meaning using a short text by re-
arranging the words and sentences of the given text. In the current study, two tasks were
designed to test reproduction literacy: a task that examined visual reproduction skills,
and a task that measured textual reproduction skills. In the visual assignment, students

10
 ACCEPTED MANUSCRIPT

were asked to create a digital poster for a given topic ("friendship") using preexisting
digital materials found on the Internet. Students could select, cut, paste, and "digitally
recycle" existing content in order to create an original artifact. In the textual task, as in
Eshet and Amichai-Hamburger's study, students were given an 140-word text divided
into four paragraphs. They were asked to create a new original text by combining and
rearranging the information in the most appropriate way.

PT
The original evaluation task for Information literacy (Eshet, & Amichai-

RI
Hamburger, 2004) focused solely on the critical evaluation of the quality and reliability
of biased information from seven different news resources representing different

SC
political agendas for the same news item. In the current study, we used the evaluation
task, which was developed by Gui and Argentin (2009, 2011), inspired by van Deursen
and van Dijk’s (2008) information skill task, aiming to test information evaluation

U
practices. The original task, developed by Gui and Argentin, presented several listings
AN
that were retrieved through a search engine for the query "wind energy". Since wind
energy is rarely used in Israeli as a source of green energy, in order to adapt the search
M

query to the local context the participants received five listings for the term "solar
energy". In contrast to screenshots presented in a previous study, we provided active
links to websites students could visit and critically examine. In the first sub-task
D

("identifying the source type"), participants were asked to choose and sort the nature of
TE

each website and the information sources behind them. In the second sub-task ("critical
evaluation of information"), participants were asked to evaluate whether the information
EP

on each of the five websites was reliable, presented a neutral and objective point of
view, and was appropriate for a research assignment. The answers for both sub-tasks
were multiple-choice questions. Each participant’s score was determined according to
C

the number of correct answers given for the five sub-tasks.

AC

Branching literacy was examined using a task from a previous study (Eshet, &
Amichai-Hamurger, 2004). Students were directed to a tourist website and were asked
to plan a long weekend trip to a European city they had never visited. They were asked
to include a map, a daily schedule, and information about each tourist attraction.
Performing the task effectively required the utilization of branching literacy in order to
construct coherent knowledge from independent sources of information, while choosing
the right navigation paths in a non-linear environment.

11
 ACCEPTED MANUSCRIPT

Social-Emotional literacy was originally tested by exposing participants to a

chatroom discussion on a current controversial political topic (Eshet & Amichai-
Hamburger, 2004). However, synchronous discussions via textual chat are normally
characterized by short messages that do not promote in-depth discussions, thus might be
inferior to asynchronous discussions (Blau, & Barak, 2012; Johnson, 2006). Therefore,
learning-related online discussions are mostly conducted in education systems through

PT
discussion forums. Thus, examining this literacy was conducted in an asynchronous
discussion group, which allowed reflection and careful drafting of responses to others’

RI
posts (Branon & Essex, 2001; Meyer, 2003; Shin Yi, & Overbaugh, 2007; Tu & Corry,
2003). The subject of the discussion was "online safety", an issue that is relevant to

SC
participants' age and interests. This topic is an integral part of learning activities
conducted during the "Internet safety week", which takes place once a year in Israeli

U
elementary and middle schools. During this week students take part in discussions,
AN
workshops, lectures and special projects to raise awareness of the topic [the data
collection was not related to these activities]. Participants were asked to think about
positive and negative aspects of the topic, provide examples, and express their opinions
M

in the discussion group. The instructions emphasized the importance of relating to

others’ opinions and responding to their posts. This task required the ability to project
D

"social presence" in digital communication (Rourke, Anderson, Garrison, & Archer

TE

2001), express both knowledge and personal perspectives, as well as sharing feelings
and personal stories. Two raters coded the posts using a quantitative content analysis
technique based on the scheme for assessing social presence in written discussions in
EP

the community of inquiry (Garrison, Anderson, & Archer, 2000, 2010). The score
representing the level of social-emotional literacy for each participant was the average
C

raters' scores in three categories of the social presence concept according to the model:
AC

emotional expression, open communication, and group cohesion.

Real-time thinking literacy has not been tested empirically in previous studies. In
the current study this was measured through a challenging digital game (Multitask
game; see: http://multitaskgames.com/multitask-game.html). In this game, users were
exposed to multiple stimuli that move on the screen at a very high speed and "bombard"
the cognition in real-time in random temporal and spatial distribution. The score that
represented successful performance on the real-time literacy task was determined based

12
 ACCEPTED MANUSCRIPT

on the length of time (in seconds) that users managed to "survive" in the challenging
game, thus demonstrating their ability to effectively process simultaneous stimuli. All
the participants reported that they had no previous experience of playing this game.
Each participant was given three attempts to familiarize themselves with the game and
then to execute five real trials. For each trial, the result was recorded by the system as
the time (in seconds) until the participant was disqualified. The level of real-time

PT
literacy was calculated as the highest score reached by the student in one of the five
trials.

RI
Apart from real-time literacy, the other digital literacy skills were evaluated

SC
independently by two raters after the initial training provided by the researchers based
on the criteria presented in Appendix. The raters used scales ranging from 0 (i.e.,
demonstrated no skill) to 5 (i.e., demonstrated very high skill). The inter-rater reliability

U
was high - Cohen's κ = .82-87 for different literacies. Table 1 presents descriptive
AN
statistics for digital literacies. As Table 1 shows, apart from social-emotional literacy
(for which the score was relatively low), the performance measures were
M

normally distributed.

Table 1. Descriptive Statistics of Performance of Digital Literacies (n = 280)

D

Photo- Visual Textual Information Information Social- Real-

Branching
Visual Reproduction Reproduction -Source -Critical Emotional Time
literacy
TE

Literacy Literacy Literacy Identification Evaluation Literacy Literacy

Mean 2.35 2.41 2.69 2.09 2.99 2.13 0.89 38.98
Median 2.50 2.50 3.00 2.00 3.00 2.00 0.50 47.00
Std.
EP

1.43 1.31 1.29 1.31 1.16 1.86 0.98 27.72

Deviation
Skewness 0.24 -0.09 -0.31 0.56 -0.51 0.21 1.15 0.16
Minimum 0 0 0 0 0 0 0 0
C

Maximum 5 5 5 5 5 5 5 151
AC

Self-Perceived Evaluations of Digital Literacy Competencies

Participants evaluated and reported their digital literacy competencies by

responding to a questionnaire (see the items in the left column of Table 2) that was
developed by Blau and Shamir-Inbal (2014) based on Eshet-Alkalai's (2012) conceptual
DL model, similarly to the performance tasks described above. The questionnaire
consisted of 20 items (three to four items for each literacy skill). An example of the
visual literacy item is: "Understanding information presented in an illustration",

13
 ACCEPTED MANUSCRIPT

reproduction item: "Connecting between a number of different online sources when

writing a new text of my own"; branching skill: "Constructing meaning from
information on a website with many web pages"; information literacy: "Comparing
information from different websites to check whether the information I found is
reliable"; social-emotional literacy: "Respectfully relating to the opinions of others
when responding through email, forum, SMS, WhatsApp, Facebook, etc.", real-time

PT
thinking: "Responding and reacting quickly when I’m playing a digital game or
simulation". Participants evaluated to what extent they were competent in performing

RI
each of the different digital literacy tasks. Responses were rated on a Likert scale
ranging from 1 (i.e., "with great difficulty") to 6 (i.e., "with great ease"). Table 2

SC
compares the measurements of digital literacies employed in this study – self-report
questionnaire and performance tasks.

U
Table 2: Comparisons of DL measurements - self-evaluation and performance
AN
Digital Competencies Digital tasks measuring actual DL
Questionnaire measuring performance
self-evaluation of DL
M

Photo-visual 1 Understanding information Participants asked to create a greeting card

literacy presented in an illustration based on a given image and text when using
2 Understanding information an unfamiliar application with a graphic user
presented in a map interface. Operating the built-in graphic
D

3 Understanding meanings editing wizard requiring participants to

represented by the icons of an decode the user-interface and interact with
TE

app visual representations such as

toolbars, icons, symbols and menus.
4 Addressing things that other In the textual reproduction skills task
Reproduction people wrote online, when participants were provided with a 140- word
EP

literacy writing a new text of my own digital text divided into four paragraphs. They
5 Connecting between a were asked to create a new original text of
number of different online their own by combining, rearranging,
sources when writing a new rewriting and connecting the information in
C

text of my own the most appropriate way.

6 Using others’ illustrations to
AC

create a new illustration/ In the visual reproduction skills task,

collage of my own participants were asked to create a digital
7 Using others’ videos to create poster for a given topic ("friendship") using
a new video of my own preexisting digital materials they found on the
internet and "digitally recycling" content in
order to create an original artifact.
Branching 8 Navigating my way through a Participants were directed to an online travel
literacy complex website with many guide and were asked to plan a trip to a
web pages European city they had never visited by
9 Constructing meaning from navigating the site. They were asked to
information on a website with include a map, a daily schedule, and

14
 ACCEPTED MANUSCRIPT

Digital Competencies Digital tasks measuring actual DL

Questionnaire measuring performance
self-evaluation of DL
many web pages information about each tourist attraction.
10 Not “getting lost” on a
website with many web pages

Information 11 Finding the information I’m The first sub-task "identifying the source
literacy looking for on the internet type" asked participants to correctly identify

PT
12 Identifying incorrect or the nature of five given websites and the
inaccurate information in a list information sources presented to them.
of internet search results

RI
13 Comparing information The second sub-task "critical evaluation of
from different websites to information" asked participants to evaluate
check whether the information whether the information on the websites was

SC
I found is reliable reliable, presented a neutral and objective
point of view, and was appropriate to the task
Social- 14 Being careful not to post Participants were exposed to a discussion
Emotional personal information about group in which the subject of discussion was

U
literacy myself when I send a message an "online safety" issue relevant to teens.
through email, forum, SMS, Participants were asked to reflect about
AN
WhatsApp, Facebook, etc. positive and negative aspects of online
15 Being careful not to post communication, provide examples, and
personal information about my express their opinions and feelings in the
friends when I send a message discussion group, emphasizing the importance
M

through email, forum, SMS, of relating to the opinions of others.

WhatsApp, Facebook, etc.
16 Staying aware of the
D

possibility that a message that I

wrote in an email, forum,
SMS, WhatsApp, Facebook,
TE

etc., could reach other people,

such as parents or teachers
17 Respectfully relating to the
opinions of others when
EP

responding through email,

forum, SMS, WhatsApp,
Facebook, etc.
C

Real-Time 18 Ignoring ads that pop up Exposing participants to a challenging

thinking while looking for information multitasking digital game with simultaneous
AC

literacy for an assignment cognitive stimuli.

19 Ignoring messages that pop
up (for example, friends’
statuses) while looking for
information for an assignment
20 Responding and reacting
quickly when I’m playing a
digital game or simulation

The questionnaire has been tested among a sample of Israeli school students having
similar background with the current study participants and showed good parameters of

15
 ACCEPTED MANUSCRIPT

internal consistency and validity (Rozmarin, Shamir-Inbal, & Blau, 2017). Table 3
presents confirmatory factor analysis with Varimax rotation method. A discriminant
validity analysis suggests that digital literacy components were relatively independent
of one another (all r's < .12).

Table 3: Digital Competencies Questionnaire – Confirmatory factor analysis

PT
Components
Visual Reproduction Branching Information Social R-Time
DL1_Visual .610

RI
DL2_Visual .655
DL3_Visual .569

SC
DL4 Reproduction .655
DL5 Reproduction .619
DL6 Reproduction .741

U
DL7 Reproduction .732
DL8 Branching .665
AN
DL9_Branching .657
DL10_Branching .623
DL11_Information .587
M

DL12_Information .601
DL13_Information .664
D

DL14 Social-Emotional .778

DL15 Social-Emotional .727
TE

DL16 Social-Emotional .752

DL17 Social-Emotional .733
DL18 Real-Time .780
EP

DL19 Real-Time .699

DL20 Real-Time .683
C

Rotation Method: Varimax with Kaiser Normalization

Table 4 presents descriptive statistics of the participants' estimations of their digital
AC

literacy competencies.
Table 4. Descriptive Statistics for Self-Estimations of Digital Literacies (n = 280)
Literacies Photo-Visual Reproduction Information Branching Social-Emotional Real-Time
Mean 4.61 4.23 4.68 4.58 5.22 5.07
Median 4.75 4.25 4.67 4.67 5.67 5.33
Std. Deviation 0.85 1.02 0.89 1.13 0.95 0.92
Skewness -0.48 -.0.48 -0.44 -0.57 -1.46 -1.28
Minimum 1 1 2 1 1 1
Maximum 6 6 6 6 6 6

16
 ACCEPTED MANUSCRIPT

Cronbach's
.78 .73 .69 .79 .83 .64
Alpha

As can be seen in Table 4, on average, participants estimated their digital literacy

competencies as high or very high. Apart from the estimations of social-emotional and
real-time literacies, the scales were normally distributed and, taking into consideration

PT
the small number of items per scale, they all had high levels of internal consistency.

Results

RI
Actual Performance on Digital Literacy Tasks

SC
The first research question examined the relationships between the participants’
performance level on digital literacy tasks representing different competencies (i.e.,
photo-visual, reproduction, branching, information, social-emotional, and real-time

U
thinking). Table 5 shows the Pearson correlation coefficients between different digital
AN
literacies as measured by the performance tests.

Table 5. Pearson Correlation Coefficient Among Actual Digital Literacies (n = 280)

M

Information
Information
Photo- Visual Textual -Identifying Social-
- Critical Branching
Literacies visual Reproduction Reproduction the Type of Emotional
D

Evaluation
Source
TE

Photo-Visual

Visual
.236**
Reproduction
EP

Textual
.128* .198**
reproduction
Information
C

Identifying
.009 .159** .128*
the Type of
AC

Source
Information –
Critical .135* .178** .209*** .325***
Evaluation
Branching -.007 .212*** .245*** .280*** .247***

Social-
-.013 -.049 .002 .023 .030 -.041
Emotional

Real-Time -.001 .113+ .140* .210*** .195*** .203** .067

*** p < .001, ** p < .01, * p < .05, + p<.06

17
 ACCEPTED MANUSCRIPT

As shown in Table 5, the correlations among the six digital literacy skills were of
moderate strength, at most; many of them did not reach statistical significance.

Self-Appraisal of Digital Literacy Competencies

The second question examined the nature of the associations between the perceived

PT
levels of different digital literacy competencies as reported by the participants. Table 6
shows the Pearson correlations between the different digital literacy competencies based

RI
on the participants’ self-evaluations.

SC
Table 6. Pearson Correlations between Perceived Digital Literacies (n = 280)

Literacies Photo-visual Reproduction Information Branching Social-emotional

Photo-visual

Reproduction .572***
U
AN
Information .599** .469**
M

Branching .626** .518** .607**

Social-emotional .341*** .256*** .438*** 438***.

D

Real time .333*** .369*** .482*** .398*** .531***

TE

***p < .001

As demonstrated in Table 6, all the correlation coefficients between the different
perceived competencies were positive and statistically significant, and most of them
EP

were relatively high. These findings show interconnections between various perceived
competencies needed for effective performance of complex digital tasks and suggest the
C

generalized view that students hold regarding their competencies.

AC

Digital Literacy: Actual Performance versus Self-Appraisal

Finally, we tested the association between participants’ self-perceptions of their

digital literacy competencies and their actual performance. The results of this analysis
are presented in Table 7, which shows Pearson correlations between self-appraisals of
digital skills and the actual literacies as measured by performance tasks.

18
 ACCEPTED MANUSCRIPT

Table 7. Correlations Between Actual and Perceived Digital Literacies (n = 280)

Information
Information
Photo- -Identifying Social- Real-
Literacies Reproduction - Critical Branching
Visual the Type of Emotional Time
Evaluation
Source

PT
r -.040 .081 .025 .108 .253*** .070 .134*

***p<.001, *p<.05, +p=.07

RI
The results presented in Table 7 show that most of the associations between the

SC
perceptions of digital literacies and the actual competencies demonstrated by the
participants were low to non-existent. A moderate correlation was found for branching
skills, a statistically significant but weak correlation for real-time thinking, and a weak

U
marginally significant correlation for information-critical evaluation literacies. Non-
AN
significant correlations were found between perceived and actual performance level of
other skills. Although findings in Table 2 showed the high confidence of students in
M

their digital literacies, Table 5 reveals that this digital confidence is only very partially
associated with the actual performance of digital tasks in educational context.
D

Discussion
TE

This study addressed the digital literacies of junior high-school students.

Specifically, the research explored students' actual performance on digital tasks, their
EP

self-perceptions regarding their digital literacy skills, as well as the association between
the two. The study extended some aspects of the digital literacy model and measured
not only textual reproduction thinking but also visual reproduction literacy, and
C

explored information thinking separately for identification of the information source

AC

and for critical evaluation of information.

The first research question regarding the nature of associations between students’
performance on different digital literacy tasks (i.e., photo-visual, reproduction,
branching, information, social-emotional, and real-time thinking) revealed that the
correlations were at the most medium, and the majority were not statistically significant.
The weak associations between level of performance on different competency tasks,
which comprise the digital literacy framework, indicate that each of them employs a

19
 ACCEPTED MANUSCRIPT

unique literacy skill. In other words, mastering one competency does not guarantee
effective performance on other digital literacy tasks. Some researchers (e.g., Lankshear,
& Knobel, 2006; Knobel, & Lankshear, 2008) have argued that digital literacy is a
unidimensional concept. Our results do not support this claim and, as far as actual
performance literacies are concerned, are consistent with previous studies which have
argued that digital literacy is multi-dimensional in nature (Eshet-Alkalai, 2012; Helsper,

PT
& Eynon, 2013; van Deursen, & van Dijk, 2008; 2010). According to the latter view,
supported by our findings, the concept of digital literacy includes a complex variety of

RI
skills: cognitive (Barzilai, & Zohar, 2006; Barzilai, & Eshet-Alkalai, 2015), motor,
social, and emotional (Eshet-Alkalai & Amichai-Hamburger, 2004). These

SC
competencies are essential for completing tasks and solving problems in digital
environments, for successful adaptation to the rapid changes in ICT (Eshet-Alkalai,

U
2012). Note that the framework adopted in this study focuses exclusively on cognitive
AN
and social-emotional literacies. It excludes important technical aspects of the concept,
such as computer literacy addressed by the CIL framework (Fraillon et al., 2014). For
instance, a recent qualitative study (Shamir-Inbal, Blau, & Avdiel, 2018), which was
M

conducted among graduate students on digital literacies, suggested a pyramid according

to the literacies' complexity. Bottom-up coding revealed cognitive, social-emotional
D

AND technical literacies, as well as suggested addressing self-regulation as one of the

TE

new literacies essential for technology-enhanced learning and work. More qualitative
studies are needed in order to advance theoretical discussion on the topic of digital
literacies and reach a deeper understanding of whether different frameworks are
EP

competing, or should rather be perceived as complementary and, perhaps, still need to

be addressed as preliminary.
C

Moreover, medium correlations between the "branching", "information -

AC

identifying the type source" and "information - critical evaluation" literacies suggest
difficulties in constructing knowledge from associative non-linear navigation in
hypermedia environments and in evaluating the reliability and credibility of those
resources. This discrepancy may pose problems in individuals’ capacity to further apply
new knowledge, meanings and interpretations constructed from the information located
in those digital environments. This assertion is consistent with the results of PISA tests
(OECD, 2015). According to PISA results, and similar to the association between

20
 ACCEPTED MANUSCRIPT

branching and information literacies found in our study, students who navigate
effectively in digital environments are successful in finding relevant digital information
and evaluating its credibility, as well as constructing knowledge based on information
from multiple sources.

The finding regarding the very low performance on social-emotional literacy tasks,

PT
which was the only literacy skill that was not significantly correlated with any other
skill, calls for special attention. These findings point to the unique nature of mastering

RI
interpersonal e-communication and e-collaboration literacies which include multiple
social (Ala-Mutka, 2011; Litt, 2012), emotional (Amichai-Hamburger, & Hayat, 2011;

SC
Lebduska, 2014) and psychological aspects (Barak, & Suler, 2008; Palfrey, & Gasser,
2015; Potter, & Banaji, 2012). Social-emotional skills constantly evolve and become
more complex as cyberspace develops and rapidly changes. Although other digital

U
literacies also need modifications as cyberspace evolves, the changes that are needed for
AN
navigating more complex online environments, building understanding based on more
information sources, or adapting to more demanding learning games, are incompatible
M

with dramatic changes that have occurred in the last decade in social-emotional skills.
For example, social networks radically changed the nature of online interactions – from
mostly anonymous communication, a significant part of which was conducted among
D

strangers (Barak & Suler, 2008), to identified communication, mostly with friends and
TE

acquaintances (Blau, 2011a, 2014).

Regarding the second research question, which referred to the associations between
EP

self-perceived competency in different digital literacy domains, our findings showed

that the correlations between the scales were all positive, significant and relatively high.
C

This finding is consistent with Rozmarin, Shamir-Inbal, and Blau’s (2017) study
findings of high correlations between self-appraised digital literacy competencies
AC

among a sample of Israeli elementary school students studying in one-to-one computing

classrooms. The meaning of this finding in both studies is essential, as it seems that
students do not differentiate between various digital literacies and, actually, generally
refer to them as a single “g” factor. That is, in general, students perceive themselves to
be on a low—high continuum in regard to their digital competencies, regardless of the
very different nature of the components which comprise digital skills. This finding,
therefore, has important implications in relation to literacy awareness and education.

21
 ACCEPTED MANUSCRIPT

The third research question explored in this study related to the association
between students’ self-perceived digital literacy competencies and their actual
performance. To the best of our knowledge, this study is a pioneer in investigating the
relationship between actual performance of students' literacies on digital tasks and their
self-appraisals of competence levels. Although recent studies reported direct and
indirect relationships between students’ performance and self-reported competencies

PT
(e.g., Fraillon et al., 2014, Rohatgi et al., 2016), as discussed in detail in the literature
review, they explored the concept of CIL that is essentially different from DL. The

RI
associations between performance and self-report found in these studies primarily
relates to the technical skills of using computers, the self-evaluation of which is

SC
relatively straightforward. Our findings suggest that self-evaluation of digital literacies
is significantly more complex and less accurate. The weak association between DL

U
performance and self-evaluation found in our study questions the validity of the
AN
methodology of investigating digital literacy competencies based on self-report.

Our findings clearly point at the gap between the self-perceived and actual level of
M

digital skills, which implies a "digital overconfidence" effect (Ackerman & Goldsmith,
2011; Gross, & Latham, 2012; Kruger & Dunning, 1999; Lauterman & Ackerman,
2014; Sidi, Ophir, & Ackerman, 2016). This effect refers to over-estimation
D

of participants' actual expertise and knowledge in a digital setting. It is explained by the

TE

lack of appropriate and sufficient learning and training in digital environments, that
could lead to better internalization of individual capabilities and higher self-awareness.
EP

That is, the amount of time spent in digital environments obviously increases students’
self-efficacy perceptions in regard to technical skills and, may consequently contribute
to biased over-estimation of more complex and advanced digital literacies, perhaps
C

caused by limited tutoring and personal feedback. Indeed, studies have shown that the
AC

amount of time spent online is only relevant in promoting technical-operational skills

(Ng, 2012; van Deursen, & van Dijk, 2008; van Deursen et al., 2011; van Deursen et al.,
2014), but not for promoting complex skills requiring activation of cognitive and
strategic mechanisms (e.g., effective navigation in hypertext and hypermedia; the ability
to find, process and evaluate information in order make data-driven decisions and
achieve personal and professional goals). Similarly, Livingstone (2015) argued that
youth do not naturally acquire digital literacies through basic online activities. Rather,

22
 ACCEPTED MANUSCRIPT

the acquisition of digital literacies requires awareness, effort, and time invested in
meaningful learning and practice supported by properly-designed pedagogy that
transforms students from novices to experts in cognitive and social-emotional digital
competencies.

Thus, of special importance is the finding regarding the clear gap between

PT
perceived social-emotional competence and actual performance in the task that
evaluates this competence. In fact, it is interesting to note that the measured perception

RI
of this social skill was the highest in comparison to other perceived skills (Mean = 5.2
on a 1 to 6 scale), while the raters’ evaluation of actual performance in this task was the

SC
lowest (Mean = 0.8 on a 0 to 5 scale). The level of students' appraisals of their social-
emotional literacy was similar to findings of a previous study (Rozmarin et al., 2017).
However, Rozmarin and colleagues' study focused on self-perceptions of digital literacy

U
and did not explore actual student performance on social literacy tasks. In the current
AN
study, however, the social task examined the ability to share information, express
personal opinion, and effectively relate them to others’ information and opinions while
M

participating in a discussion group. This enabled monitoring and evaluation of

participants’ performance in actual, relevant tasks. Thus, in contrast to self-report, we
focused on testing the actual expression of "social presence" in digital environments
D

(Garrison et al., 2000, 2010; Rourke et al., 2001). Previous research has reported that
TE

social presence projected by students’ discussions in academic courses was not only
important but also positively associated with the quality of online learning (Caspi, &
EP

Blau, 2008).

Cyberspace is an important arena for adolescents’ social lives, through which they
C

have the opportunity to express social presence and the space to comfortably interact
with others and share their personal life (Barak, & Suler, 2008; Turner, 2015). This
AC

includes, among other aspects, social and learning-related online interactions with peers
(Blau, 2011a, 2014). In practice, the very low level of performance on the social task,
based on online interactions with peers which was found in our study, suggests that the
basic digital and social communication practices that young students engage in online
on a daily basis do not promote the development and mastery of this complex skill in an
educational context. For instance, Rozmarin and colleagues (2017) showed that, in
contrast with common arguments related to digital literacies, the average level of

23
 ACCEPTED MANUSCRIPT

students’ perceived social-emotional competencies remained the same at the beginning

and at the end of the school year. This was found despite the fact that participants
studied in one-to-one computing classroom with network laptops (Chromebooks), a
technology which affords easy e-communication and e-collaboration with peers. Thus,
the cumulative evidence highlights the need for carefully-designed educational training
programs aimed to enhance specific social literacies. This training should focus on

PT
promoting effective social-emotional competencies among students in educational
contexts, such as peer dialogue in online learning communities and collaboration in

RI
virtual teams. It should be noted that this competence is important not only for effective
technology-enhanced school learning, but can be further transferred to online

SC
communication and collaboration in the workplace, professional training, or life-long
learning settings.

Conclusion and Implications

U
AN
This study investigated the digital literacy competencies of junior high school
students and compared their perceived literacy level to their actual performance on
M

digital tasks. The findings revealed students’ high self-evaluated competence, and clear
over-confidence compared to the actual performance levels exhibited, which were
D

mediocre-to-low. This gap was especially noticeable with regard to social-emotional

TE

skills, which were self-estimated as the highest among the rated skills, while actual
performance on social tasks was found to be the poorest. These findings call for
educational decision-makers to take action and encourage training that aims to develop
EP

the digital literacies of school students. It is important for schools in general, and for
teachers in particular, to take responsibility for nurturing the digital literacies of
C

students, and to design learning and evaluation activities that develop these
AC

competencies. Effective education for digital literacy could equip youth with
appropriate knowledge and competences for lifelong learning, for effective coping with
cognitive and social-emotional challenges introduced by the knowledge society, for
successful functioning in future workplaces, and active involvement in economic and
social life.
In contrast to previous studies that tended to focus on limited types of digital
competencies, this study explored the entire set of digital literacies as based on the
comprehensive theoretical framework suggested by Eshet-Alkalai (2012). Moreover, the

24
 ACCEPTED MANUSCRIPT

current study extended some aspects of the digital literacy model and measured both
textual and visual reproduction literacy, as well as exploring information thinking
separately for identification of the information source and for critical evaluation of
information. In addition, real-time thinking, which was introduced by Eshet-Alkalai in
his updated theoretical framework, was empirically tested for the first time in this study.
Although our study was conducted in a large and geographically diverse sample of

PT
junior high school students, all of the participants learned in the same whole-class
technology model. Future studies may continue exploring the contribution of other

RI
technological models, such as one-to-one computing initiatives, on the development of
students’ digital literacies. In addition, the current study measured students’ digital

SC
competencies at a single time-point and in a particular educational context. In future
studies, it could be of importance to examine changes in digital literacies over time and

U
in the context of different educational contents.
AN
References
ACARA (2015). National Assessment Program – ICT Literacy Year 6 and 10 Report 2014. Technical -
=Report. Sydney: ACARA. Retrieved April 9th, 2017 from
M

http://nap.edu.au/verve/_resources/D15_10858__NAP
ICT_Technical_Report_2014_Final_051015.pdf
Ackerman, R., & Goldsmith, M. (2011). Metacognitive regulation of text learning: On screen versus on
D

paper. Journal of Experimental Psychology: Applied, 17, 18- 32.

Ahn, J., & Jung, Y. (2016). The common sense of dependence on smartphone: A comparison between
TE

digital natives and digital immigrants. New Media & Society, 18(7), 1236-1256.
Ala-Mutka, K. (2011). Mapping digital competence: Towards a conceptual understanding. Luxembourg:
Publications Office of the European Union: JRC-IPTS. Retrieved April 9th 2017 from
EP

http://ipts.jrc.ec.europa.eu/publications/pub.cfm?id=4699
Alexander P. A., & the Disciplined Reading and Learning Research Laboratory. (2012). Reading into the
future: Competence for the 21st century. Educational Psychology, 47, 259-280.
C

Amichai-Hamburger, Y. & Hayat, Z. (2011). The impact of the Internet on the social lives of users: A
representative sample from 13 countries. Computers in Human Behavior, 27, 585-589.
AC

Barak, A., & Suler, J. (2008). Reflections on the psychology and social science of cyberspace. In A.
Barak (Ed.), Psychological aspects of cyberspace: Theory, research, applications. (pp. 1-12).
Cambridge, UK: Cambridge University Press.
Barzilai, S., & Zohar, A. (2006). How does information technology shape thinking? Thinking Skills and
Creativity, 1, 130-145.
Barzilai, S., & Eshet-Alkalai, Y. (2015). The role of epistemic perspectives in comprehension of multiple
author viewpoints. Learning and Instruction, 36, 86-103.
Barzilai, S., Tzadok, E., & Eshet-Alkalai, Y. (2015). Sourcing while reading divergent expert accounts:
Pathways from views of knowing to written argumentation. Instructional Science, 43(6), 737-766.

25
 ACCEPTED MANUSCRIPT

Blau, I. (2011a). Application use, online relationship types, self-disclosure, and Internet abuse among
children and youth: Implications for education and Internet safety programs. Journal of Educational
Computing Research, 45, 95-116.
Blau, I. (2011b). Teachers for "Smart classrooms": The extent of implementing of an Interactive
Whiteboard-based professional development program on elementary teachers' instructional
practices. Interdisciplinary Journal of E-Learning and Learning Objects, 7, 275-289.
Blau, I., & Barak, A. (2012). How do personality, synchronous media, and discussion topic affect
participation? Educational Technology & Society, 15, 12-24.

PT
Blau, I. (2014). Comparing online opportunities and risks among Israeli children and youth Hebrew and
Arabic speakers. New Review of Hypermedia and Multimedia, 20, 281-299.
Blau, I., Peled, Y., & Nusan, A (2014). Technological, pedagogical and content knowledge in one-to-one
classroom: Teachers developing "digital wisdom". Interactive Learning Environments, 24(6), 1215-

RI
1230.
Blau, I. & Shamir-Inbal, T. (2016). Digital competences and long-term ICT integration in school culture:
The perspective of elementary school leaders. Education and Information Technologies, 1-19. DOI:

SC
10.1007/s10639-015-9456-7
Bawden, D. (2001). Information and digital literacies; a review of concepts. Journal of Documentation,
57, 218-59.

U
Bawden, D. (2008). Origins and concepts of digital literacy. In C. Lankshear & M. Knobel (Eds), Digital
literacies: concepts, policies and practices (pp. 17-32). New York: Peter Lang Publishing.
AN
Branon, R. F., & Essex, C. (2001). Synchronous and asynchronous communication tools in distance
education: a survey of instructors. TechTrends, 45, 36–42.
Calvani, A., Fini, A., Ranieri, M., & Picci, P. (2012). Are young generations in secondary schools
M

digitally competent? A study on Italian teenagers. Computer & Education, 58, 797-807.
Carlson, S. & Tidiane Gadio, C. (2002). Teacher professional development in the use of technology. In
Haddad, W. D. & Draxler, A. (Eds). Technologies for Education (pp. 118-132). Paris: UNESCO and
D

the Academy for Educational Development.

Caspi, A., & Blau, I. (2008). Social presence in online discussion groups: testing three conceptions and
TE

their relations to perceived learning. Social Psychology of Education, 11, 323-346.

Cihak, D. F., Wright, R., McMahon, D., Smith, C. C., & Kraiss, K. (2015). Incorporating functional
digital literacy skills as part of the curriculum for high school students with intellectual disability.
Education and Training in Autism and Developmental Disabilities, 50, 155-171.
EP

Dede, C. (2005). Planning for neo millennial learning styles. EDUCAUSE Quarterly, 28, 7-12.
ECDL Foundation (2012). Delivering the Digital Agenda for Europe. Retrieved April 9th, 2017 from
http://www.ecdl.org/media/DeliveringtheDAEReport_ECDLF1.pdf.pdf
C

ECDL Foundation (2016). Perception and reality. Retrieved April 9th, 2017 from
http://ecdl.org/media/perceptionandreality-measuringdigitalskillsineurope-
AC

ecdlfoundationpositionpaper1.pdf
Eshet-Alkalai, Y. (2004). Digital literacy: A conceptual framework for survival skills in the digital era.
Journal of Educational Multimedia and Hypermedia, 13, 93-106.
Eshet, Y., & Amichai-Hamburger, Y. (2004). Experiments in digital literacy. Cyberpsychology and
Behavior, 7, 425-434.
Eshet-Alkalai, Y. (2012). Thinking in the digital era: A revised model for digital literacy. Issues in
Informing Science and Information Technology, 9, 267-276.
Eshet-Alkalai, Y., & Chajut, E. (2009). Changes over time in digital literacy. Cyberpsychology &
Behavior, 12, 713-715.

26
 ACCEPTED MANUSCRIPT

Eshet-Alkalai, Y., & Chajut, E. (2010). You can teach old dogs new tricks: The factors that affect changes
over time in digital literacy. Journal of Information Technology Education, 9, 173-181.
Eshet-Alkalai, Y., & Geri, N. (2010). Does the medium affect the message? The effect of congruent
versus incongruent display on critical reading. Human Systems Management, 29, 243-251.
Ferrari, A. (2012). Digital Competence in Practice: An Analysis of Frameworks. Luxembourg:
Publications Office of the European Union: JRC IPTS. (DOI: 10.2791/82116)
Fraillon, J., Ainley, J., Schulz, W., Friedman, T., & Gebhardt, E. (2014). Preparing for life in a digital
age: The IEA International Computer and Information Literacy Study international report. Available

PT
at https://link.springer.com/book/10.1007/978-3-319-14222-7#authorsandaffiliationsbook
Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment:
Computer conferencing in higher education. The Internet and Higher Education, 2, 87-105.

RI
Garrison, D. R., Anderson, T., & Archer, W. (2010). The first decade of the community of inquiry
framework: A retrospective. The Internet and Higher Education, 13, 5-9.
Gilster, P. (1997). Digital literacy. N.Y: Wiley Computer Publishing.

SC
Gross, M., & Latham, D. (2012). What's skill got to do with it? Information literacy skills and self-views
of ability among first-year college students. Journal of The American Society for Information
Science & Technology, 63, 574-583.

U
Gui, M., & Argentin, G. (2009). How deep are different forms of digital skills divide among
Youngpeople? Results from an extensive survey of 1000 northern-Italian high school
students.Media@LSE electronic working papers, 15, London School of Economics and
AN
PoliticalScience, London, UK.
Gui, M., & Argentin G. (2011). The digital skills of Internet-natives. The role of ascriptive differences in
the possession of different forms of digital literacy in a random sample of northern Italian high
M

school students. New Media and Society, 13, 963-980.

Hague, C., & Payton, S. (2010). Digital literacy across the curriculum. UK: Futurelab.
Hargittai, E. (2010). Digital Na(t)ives? Variation in Internet Skills and Uses among Members of the "Net
D

Generation". Sociological Inquiry. 80, 92-113.

Helsper, E. J., & Eynon, R. (2013). Distinct skill pathways to digital engagement. European Journal of
TE

Communication, 28, 696-713.

Hicks, T., & Turner, K. H. (2013). No longer a luxury: Digital literacy can't wait. English Journal, 102,
58-65.
EP

Johnson, G. (2006). Synchronous and asynchronous text-based CMC in educational contexts: A review of
recent research. Techtrends: Linking Research & Practice To Improve Learning, 50, 46-53.
Iordache, E., Mariën, I., & Baelden, D. (2017). Developing Digital Skills and Competences: A QuickScan
Analysis of 13 Digital Literacy Models. Italian Journal of Sociology of Education, 9, 6-30.
C

Kirschner, P. A., & van Merriënboer, J. J. (2013). Do learners really know best? Urban legends in
education. Educational Psychologist, 48, 169-183.
AC

Knobel, M., & Lankshear, C. (2008). Digital literacy and participation in online social networking
spaces. In In C. Lankshear and M.Knobel (Eds.), Digital Literacies: Concepts, Policies and Practices
(pp.249-278). New York: Peter Lang Publishing.
Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one's own
incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77,
1121-1134.
Lankshear, C. & Knobel, M. (2006). Digital literacy and digital literacies: Policy, pedagogy and
research considerations for education. Nordic Journal of Digital Literacy. 1, 12-24.
Lauterman, T., & Ackerman, R. (2014). Overcoming screen inferiority in learning and
calibration. Computers in Human Behavior, 35, 455-463.

27
 ACCEPTED MANUSCRIPT

Lebduska, L. (2014). Emoji, Emoji, what for art Thou? Harlot: A Revealing Look at the Arts of
Persuasion, 1. Retrieved April 9th, 2017 from
http://www.harlotofthearts.org/index.php/harlot/article/view/186/157
Leu, D. J., Kinzer, C. K., Coiro, J., Castek, J., Henry, L. A. (2013). New literacies: A dual level theory of
the changing nature of literacy, instruction, and assessment. In Alvermann, D.E., Unrau, N.J., &
Ruddell, R.B. (Eds.). (2013). Theoretical models and processes of reading (6th ed.). (pp. 1150-
1181). Newark, DE: International Reading Association.
Litt, E. (2012) Knock, knock. Who’s there? The imagined audience. Journal of Broadcasting &
Electronic Media, 56, 330-345.

PT
Livingstone, S. (2015). Digital skills for European citizens and consumers. Research Report. Retrieved
April 9th, 2017 from http://blogs.lse.ac.uk/mediapolicyproject/2015/06/17/digital-skills-for-
european-citizens-and-consumers/

RI
McLoughlin, C., & Lee, M. (2008). Mapping the digital terrain: New media and social software as
catalysts for pedagogical change. Ascilite Melbourne.
Meyer, K. A. (2003). Face-to-face versus threaded discussions: the role of time and higher-order thinking.

SC
Journal of Asynchronous Learning Networks, 7, 55–65.
Meyers, E. M., Erickson, I., & Small, R. V. (2013). Digital literacy and informal learning environments:
an introduction. Learning, media and technology, 38, 355-367.

U
Mohammadyari, S., & Singh, H. (2015). Understanding the effect of e-learning on individual
performance: The role of digital literacy. Computers & Education 82, 11-25.
AN
Murray, M. C., & Pérez, J. (2014). Unraveling the digital literacy paradox: How higher education fails at
the fourth literacy. Issues in Informing Science and Information Technology, 11, 85-100.
Ng, W. (2012). Can we teach digital natives digital literacy? Computers & Education, 59, 1065-1078.
M

Oblinger, D. G., & Oblinger, J.(2005). Educating the Net Generation. Educase Online book. Retrieved
April 9th, 2017 from http://www.educause.edu/ir/library/pdf/pub7101.pdf
OECD (2015). Students, Computers and Learning: Making the Connection. PISA, OECD Publishing.
D

Retrieved April 9th, 2017 from http://tinyurl.com/os77645

Palfrey, J., & Gasser, U. (2008). Born digital: Understanding the first generation of digital natives. New
TE

York, NY: Basic Books.

Pangrazio, L. (2016). Reconceptualising critical digital literacy. Discourse: Studies in the Cultural
Politics of Education, 37, 163-174.
EP

Pellegrino J.W., & Hilton, M. L. (2012). Education for life and work: Developing transferable knowledge
and skills in the 21st century. Committee on Defining Deeper Learning and 21st Century Skills.
Division on Behavioral and Social Sciences and Education. National Research Council.
Washington, DC: National Research Council.
C

Peled, Y., Blau, I., & Grinberg, R. (2015). Does 1:1 computing in a junior high-school change the
pedagogical perspectives of teachers and their educational discourse? Interdisciplinary Journal of e-
AC

Skills and Lifelong Learning - IJELL, 11, 257-271.

Potter, J., & Banaji, S. (2012). Social media and self-curatorship : Reflections on identity and pedagogy
through blogging on a masters module. Comunicar, 19, 83-91.
Porat, E., Blau, I., & Barak, A. (2017). Digital literacy of school students – self-report versus actual
performance. In Y. Eshet-Alkalai, I. Blau, A. Caspi, N. Geri, Y. Kalman, & V. Silber-Varod,
(Eds.), Learning in the Technological Era (pp.160-173). Ra'anana, Israel: The Open University of
Israel. [in Hebrew].
Prensky, M. (2001). Digital natives, digital immigrants: Part 1. On the Horizon, 9, 1 - 6. Do:
10.1108/10748120110424816. Retrieved April 9th, 2017 from
http://dx.doi.org/10.1108/10748120110424816

28
 ACCEPTED MANUSCRIPT

Prensky, M. (2009). H. Sapiens Digital: From Digital Immigrants and Digital Natives to Digital Wisdom.
Journal of Online Education, 5. Retrieved April 9th, 2017 from
http://nsuworks.nova.edu/innovate/vol5/iss3/1
Rohatgi, A., Scherer, R., & Hatlevik, O. E. (2016). The role of ICT self-efficacy for students' ICT use and
their achievement in a computer and information literacy test. Computers & Education, 102, 103-
116.
Rouet, J. F. (2006). The skills of document use: From text comprehension to web-based learning.
Mahwah, NJ: Lawrence Erlbaum Associates.

PT
Rourke, L., Anderson, T., Garrison, D.R., & Archer, W. (2001). Assessing social presence in
asynchronous text-based computer conferencing. Journal of Distance Education, 14, 1-18.
Rozmarin, T., Shamir-Inbal, T., & Blau, I. (2017). What are the implication of integration one-to-one
computing with network laptops (Chromebooks) on students' digital literacy skills? In Y. Eshet-

RI
Alkalai, I. Blau, A. Caspi, N. Geri, Y. Kalman, & V. Silber-Varod, (Eds.), Learning in the
Technological Era (pp.201-210). Ra'anana, Israel: The Open University of Israel. [in Hebrew].
Shamir-Inbal, T. & Blau, I. (2014). Development of digital literacy and challenge of collaboration: Can

SC
they go together? In Y. Yair & E. Shmueli (Eds.), Proceedings of the 12th annual MEITAL national
conference 2014 (pp. 269-274). Levinsky College of Education, Israel.
Shamir-Inbal, T. & Blau, I. (2016). Developing digital wisdom by students and teachers: the impact of

U
integrating tablet computers on learning and pedagogy in an elementary school. Journal of
Educational Computing Research, 54(7), 967-996.
AN
Shamir-Inbal, T., Blau, I., & Avdiel, O. (2018). Developing digital literacies in academia through
collaborative learning activities and interactions in a learning community. Paper presented at the
AERA (American Educational Research Association) 2018 Annual Meeting: The dreams,
possibilities, and necessity of public education. New York, US.
M

Shin Yi, L., & Overbaugh, R. C. (2007). The Effect of Student Choice of Online Discussion Format on
Tiered Achievement and Student Satisfaction. Journal Of Research On Technology In Education
(International Society For Technology In Education, 39, 399-415.
D

Sidi, Y., Ophir, Y., & Ackerman, R. (2016). Generalizing screen inferiority-does the medium, screen
versus paper, affect performance even with brief tasks? Metacognition and Learning, 11, 15-33.
TE

Smith, S.D., & Caruso, J. B. (2010). The ECAR Study of Undergraduate Students and Information
Technology, 2010. Boulder, CO: EDUCAUSE Center for Applied Research, 2010. Retrieved April
9th, 2017 from https://net.educause.edu/ir/library/pdf/ERS1006/RS/ERS1006W.pdf
Tapscott, D. (1998). Growing up digital: The rise of the Net generation. New York: McGraw-Hill.
EP

Tu, C. H., & Corry, M. (2003). Designs, management tactics, and strategies in asynchronous learning
discussions. The Quarterly Review of Distance Education, 4, 303-315.
Turner, A. (2015). Generation Z: Technology and Social Interest. The Journal of Individual Psychology,
C

71, 103-113.
van Deursen, A. J. A. M. (2010). Internet Skills: vital assets in an information society. Enschede, the
AC

Netherlands: University of Twente.

van Deursen, A. & van Dijk, J. (2008). Measuring digital skills. Performance tests of operational, formal,
information and strategic internet skills among the Dutch population. Retrieved April 9th, 2017 from
http://www.utwente.nl/gw/vandijk/news/measuring_digital_skills/mds.pdf
van Deursen, A. J. A. M., & Van Dijk, J. A. G. M. (2010). Measuring Internet skills. International
Journal of Human-Computer Interaction, 26, 891–916.
van Deursen, A. J. A. M., Van Dijk, J. A. G. M., & Peters, O. (2011). Rethinking Internet skills. The
contribution of gender, age, education, Internet experience, and hours online to medium- and
content-related Internet skills. Poetics, 39, 125–144.
van Deursen, A.J.A.M., Helsper, E.J., & Eynon, R. (2015). Development and validation of the Internet
Skills Scale (ISS). Information, Communication & Society, 19, 804-823.

29
 ACCEPTED MANUSCRIPT

van Deursen, A. J.A.M., Görzig, A., Van Delzen, M., Perik, H.T.M., & Stegeman, A.G. (2014). Primary
school children’s internet skills: a report on performance tests of operational, formal, information,
and strategic internet skills. International Journal of Communication, 8, 1343-1365.
van Dijk, J. A. G. M., & Van Deursen, A. J. A. M. (2014). Digital skills, unlocking the information
society. New York, NY: Palgrave Macmillan
Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. K., & Hemmerich, J. A. (2009).
Source evaluation, comprehension, and learning in internet science inquiry tasks. American
Educational Research Journal, 46, 1060-1106.

PT
Zhong, Z. J. (2011). From access to usage: The divide of self-reported digital skills among
adolescents. Computers & Education, 56, 736-746.

RI
Appendix

SC
Guidelines for raters:

Each performance task has a specific outcome that examines the level of a specific

U
digital literacy. The following document contains criteria for evaluating each digital
literacy task. Apart from the socio-emotional literacy task, please examine and rate each
AN
outcome in accordance with the criteria listed below on a 5 point Likert scale: 1 = very
low, 2 = low, 3 = moderately, 4 = high, 5 = very high.
To evaluate the socio-emotional literacy task, read the discussion transcripts in the
M

forum for each participant to assess social presence. If the participant's text corresponds
with one of the indicators for social presence listed below, copy and paste the relevant
text from the forum and place it in the appropriate cell in the rating spreadsheet. Finally,
D

calculate the number of times a particular indicator appears in the writing of each
participant, sum up his/her total for each category (emotional responses, open
TE

communication, social cohesion), and calculate the average score on the three social-
emotional literacy categories.
EP

Digital literacy performance tasks: criteria for assessment on a 5-point Likert scale
Photo-Visual thinking: decoding the user interface of an unknown application
C

• Decoding the patch and adjustment interface

• Decoding the effects interface
AC

• Decoding the layers interface

• Decoding the frame interface
• Decoding the label interface
• Decoding the writing interface

Reproduction Thinking –Visual: creating new meaning by rearranging and manipulating

existing images
• The quality and sophistication of the artifact
• The complexity of the artifact

30
 ACCEPTED MANUSCRIPT

• The creativity and originality of the artifact

Reproduction Thinking-Textual: rearranging and rewriting text to convey an original

claim or new meaning
• The quality and sophistication of the rearranged and rewritten text
• The complexity of the rearranged and rewritten text
• The creativity and originality of the statement or meaning in the rearranged and

PT
rewritten text

Information thinking- Identifying the source type: critical assessment of information

RI
sources
• The number of items (out of 5) correctly associated with the description of the

SC
information source nature
Information thinking - Critical evaluation of information: credibility, objectivity, and
appropriateness of information sources

U
• The number of items (out of 5) correctly identified as trustworthy, objective, and
appropriate for the task
AN
Branching Thinking - Planning a trip to an unknown destination by building knowledge
through navigating in an information domain
M

Completion of the assignment:

• All days of the tour were planned
D

Coherence:
• Logical, consistent planning
TE

Depth of the planning:

• Number of sites in the tour
•
EP

Diversity of the sites

• Practical details about the sites
• The presence of a map
C

Social-emotional thinking – successful social-emotional group discussion in the forum

AC

The number of social-emotional indicators in the content analysis of participation in a

group discussion:
• Emotional responses
• Open communication indicators
• Group cohesion indicators

Real-Time thinking – successful functioning in a digital game with multiple cognitive

stimuli

31
 ACCEPTED MANUSCRIPT

• The longest time (in seconds) among three attempts that a user "survived"
functioning in the unknown complex game with multiple cognitive stimuli, after
a short initial training.

PT
RI
U SC
AN
M
D
TE
C EP
AC

32
 ACCEPTED MANUSCRIPT

Measuring Digital Literacies: Perceived Versus Actual

Competencies of Junior High-School Students

Highlights
• Previous studies mostly explored students’ self-report digital literacies

PT
• This study compares six perceived digital literacies with their actual performance
• Students (n=280, age~13) significantly over-estimated their digital competencies
• Perceived literacies were high to very-high, while performance was average-low

RI
• This perception-performance gap was most evident in social-emotional literacies

U SC
AN
M
D
TE
C EP
AC