Education and Information Technologies

https://doi.org/10.1007/s10639-024-12707-x

A systematic review of evaluation and program planning

strategies for technology integration in education: Insights
for evidence-based practice

Zhimin Luo1 · Babar Nawaz Abbasi1 · Chong Yang2 · Jiayin Li3 · Ali Sohail4

Received: 18 June 2023 / Accepted: 10 April 2024

© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature
2024

Abstract
This paper presents a systematic review of the literature on evaluation and program
planning strategies for technology integration in education. The aim of the review
was to examine the evidence on different types of strategies that can be used to
design, implement, assess, and improve technology integration in educational set-
tings, as well as the factors that influence their success. The review followed the
Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)
guidelines and searched four databases for relevant studies published between 2010
to May 2023. A total of 124 studies met the inclusion criteria and were appraised
for quality using multiple quality assessment tools. The findings were synthesized
using a narrative synthesis approach. The review identified and compared various
models and approaches for technology integration, such as design-based research,
logic model, backward design, Analysis, Design, Development, Implementation,
and Evaluation (ADDIE) model, and Substitution, Augmentation, Modification, and
Redefinition (SAMR) model. It also explored how organizational, technological,
pedagogical, teacher, and student factors interact with the evaluation and program
planning strategies and affect the outcomes of technology integration. Furthermore,
the review has highlighted the importance of evidence-based practice in technology
integration and the need for continuous improvement based on data from multiple
sources and methods. Moreover, this review also discussed the limitations and chal-
lenges of the review and suggested directions for future research and policy de-
velopment. Besides, the review contributes to the existing literature on technology
integration in education by providing a comprehensive and systematic analysis of
the evaluation and program planning strategies and their interaction with various
factors that influence their success.

Extended author information available on the last page of the article

13
 Education and Information Technologies

Keywords Education · Evaluation strategies · Program planning · Technology

integration · Technological factors

JEL codes I21 · I23 · O33 · C83 · D83 · C88

1 Introduction

The rapid advancement of technology has transformed the way we live, work, and
learn (Alghamdi & Holland, 2020). In education, technology integration is increas-
ingly becoming an integral component for enhancing the teaching and learning
process (Zhang, 2022). As technological tools and digital resources become more
prevalent, educators are tasked with finding innovative ways to incorporate them
into their classrooms (Anderson & Putman, 2022). The effective incorporation of
technology into educational settings has the potential to improve student engage-
ment, personalize learning, and expand access to educational resources (Timotheou
et al., 2023). Furthermore, it can also facilitate communication, collaboration, and
the development of essential 21st-century skills. Therefore, it is essential to identify
and implement evidence-based practices for the successful integration of technology
in education.
However, the process of integrating technology into education is complex and
multifaceted, involving evaluation and program planning strategies that influence
the effectiveness and success of technology adoption (Zainal & Zainuddin, 2020).
Evaluation strategies are critical in assessing the impact of technology integration
on teaching and learning outcomes, helping to identify areas of improvement and
drive evidence-based decision-making (Kimmons et al., 2020). Program planning
strategies, on the other hand, guide the design, implementation, and management of
technology integration initiatives, ensuring that they align with educational goals and
cater to the needs of diverse learners (Peled & Perzon, 2022).
Evaluation and program planning strategies play a vital role in ensuring the suc-
cessful integration of technology in education (Schalock et al., 2018). Evaluation
strategies help educators assess the effectiveness of technology in meeting educa-
tional objectives and improving learning outcomes (Lai & Bower, 2019). Evaluation
strategies also facilitate the identification of potential challenges and barriers that
may hinder technology integration, allowing for the development of targeted inter-
ventions and support mechanisms (Luo et al., 2019). Program planning strategies, on
the other hand, are essential in designing, implementing, and managing technology
integration initiatives. They involve the identification of educational goals and objec-
tives, the selection of appropriate technologies, and the development of strategies
for incorporating these technologies into teaching and learning practices (Dillon et
al., 2019). Program planning strategies also encompass the provision of professional
development and support for educators, the allocation of resources, and the establish-
ment of systems for monitoring and evaluating the progress of technology integration
initiatives (Iftikhar et al., 2022).
Despite the potential benefits of technology integration in education, several chal-
lenges persist that may hinder its successful implementation (Lawrence & Tar, 2018).

13
Education and Information Technologies

One of the most significant challenges is the digital divide, which refers to the dis-
parities in access to technology and digital resources among different socioeconomic
groups, regions, and countries (Van Deursen & Van Dijk, 2019). This divide can
exacerbate existing educational inequalities, limiting the opportunities for disadvan-
taged learners to benefit from technology integration initiatives. Another challenge
is the rapid pace of technological change, which can make it difficult for educators
and policymakers to keep up with the latest tools, platforms, and resources (Ferri
et al., 2020; Tosuntaş et al., 2019). This can result in a lack of familiarity with new
technologies and a reluctance to adopt them in the classroom. Furthermore, the
rapid evolution of technology can lead to difficulties in maintaining and updating
infrastructure, software, and hardware, placing a strain on educational institutions’
resources (Storck & Duarte-Figueiredo, 2020; Roozbeh et al., 2018). Figure 1 pres-
ents a streamlined approach for the critical evaluation and integration of educational
technology, emphasizing the determination of its suitability and contribution to the
teaching and learning experience. The process is depicted in chronological stages,
beginning with the initial assessment of the technology’s relevance and benefits,
progressing through careful planning for pre-instructional integration, active appli-
cation during teaching to foster engagement, and culminating in a post-instruction
evaluation of outcomes. The final, ongoing phase entails a cycle of evaluation and
continuous improvement, using gathered insights to refine both technology usage and
integration strategies.
Despite the growing body of literature on technology integration in education,
there is a need for a comprehensive synthesis of the existing knowledge on evalua-
tion and program planning strategies (Wilson, 2023). This systematic review aims to
fill this gap, providing a consolidated understanding of the most effective approaches
and best practices that contribute to improved teaching and learning outcomes. Such
insights will enable educators, policymakers, and researchers to make informed
decisions and optimize technology integration in education. Furthermore, there is
a need for more review studies that directly compare different evaluation strategies

Fig. 1 Evaluating educational technology

and integration strategies

13
 Education and Information Technologies

for technology integration to determine their relative effectiveness in various educa-

tional contexts. Moreover, the majority of the existing literature focuses on specific
technologies or tools, limiting the generalizability of findings to other contexts or
emerging technologies. In addition, there is a scarcity of review studies that examine
the long-term impacts of technology integration on teaching and learning outcomes,
as well as the sustainability of these initiatives over time. Likewise, the previous lit-
erature lacks a comprehensive understanding of how various stakeholders, including
organizational, technological, pedagogical, teacher, and student, perceive and engage
with evaluation and program planning strategies, and how their perspectives influ-
ence the success of technology integration.
Addressing the identified gaps in the literature on technology integration in edu-
cation, this systematic review makes a significant contribution by comprehensively
examining evaluation and program planning strategies. The study synthesizes exist-
ing research, compares various evaluation strategies, and provides generalizable find-
ings that inform decisions about emerging technologies across diverse educational
contexts. Furthermore, the article contributes to the existing literature on technology
integration in education by offering a comprehensive and nuanced perspective on
the phenomenon. Previous studies (Anderson & Putman, 2022; Wilson, 2023; Wu
et al., 2022; Pelila et al., 2022; Timotheou et al., 2020; Zainal & Zainuddin, 2020;
Zagami et al., 2018; Tondeur et al., 2017) have frequently concentrated on specific
facets or elements of technology integration, such as the utilization of technology
(frequency, types, intensity), the adoption of technology (stages, levels, models), the
impact of technology (outcomes, effects, benefits), pedagogical approaches to tech-
nology (methods, strategies), and technological literacy (skills, competencies, stan-
dards), among others. In contrast, this review encompasses a broader range of factors
and dimensions, providing a more holistic understanding of technology integration
in educational settings. Given the importance of evaluation and program planning
strategies in technology integration, this systematic review seeks to address two main
research objectives. The first objective is to systematically review and synthesize the
literature on evaluation strategies for technology integration in education, in order
to identify effective approaches that contribute to improved teaching and learning
outcomes. The second objective is to examine the role of program planning strategies
in successful technology integration in education and derive best practices from the
literature that can inform and guide future implementations.
The remaining study comprises four sections: literature review, methodology,
results and discussion, and concluding remarks.

2 Literature review

2.1 The role of technology integration in education

With the rapid development of information and communication technology (ICT) in

education, the role of technology integration is indeed increasing. Numerous studies
(Timotheou et al., 2023; Materia et al., 2016; Ertmer & Ottenbreit-Leftwich, 2010;
Franklin & Bolick, 2007) have highlighted the transformative potential of technol-

13
Education and Information Technologies

ogy in various aspects of education, including pedagogy, organization, and social

dynamics. Technology has the capability to enhance teaching and learning experi-
ences, facilitate access to educational resources, support collaborative learning, and
promote engagement and participation in educational settings. ICT in schools not
only affects the performance of teachers and students but also has an impact on mul-
tiple aspects and stakeholders associated with the school.
In education and teaching, the topic of technology integration in teaching and
learning has attracted a lot of attention. For teachers, integrating technology into
teaching (ITT) is seen as key to helping them keep up with the changing society and
various teaching technologies (Catio, 2019). The importance of innovative behav-
ior, both among urban and rural teachers, in integrating technology into teaching is
widely recognized. Factors such as the organizational environment, peer support, and
information literacy all impact teachers’ innovative behavior towards ITT (Wu et al.,
2022). Several organizations have established the Teacher-Authored Learning Plat-
form (TALP), which brings together teachers with expertise in different subject areas
to systematically develop curriculum materials based on the standard syllabus of the
national curriculum. TALP utilizes Artificial Intelligence (AI) to enhance interactiv-
ity and support learning activities within the system. Teachers can also utilize TALP
to identify learning gaps and issues early on, allowing them to use TALP and other
materials to assist students in their learning (Liu, 2022). While for students, the use
of technology continues to have a stimulating effect on learning outcomes. The use of
Augmented Reality (AR) tools has been found to have a significant impact on student
learning outcomes compared to traditional teaching tools such as lectures, textbooks,
and multimedia (Garzón & Acevedo, 2019). Furthermore, the integration of tech-
nology through Virtual Reality (VR) applications has also shown a positive impact
on student learning (Chen et al., 2022; Ho et al., 2013). While in terms of society,
the integration of technology in education is garnering societal attention, crucial for
preparing students as future workforce, facilitating societal adaptation, and fueling
economic growth (Hatakka et al., 2020; Zagami et al., 2018).

2.2 Evaluation strategies in education and technology integration

Evaluation is the systematic collection of data and the use of this data to improve a
programme or project. It provides data to determine whether the programme is meet-
ing its objectives and to help future training(Iftikhar et al., 2022). The evaluation of
training programs is becoming a popular topic in research. The best way to deter-
mine whether an educational program is achieving its goals is to evaluate it (Tuna
& Başdal, 2021). Evaluative research aims to improve the social, economic, politi-
cal and cultural conditions of society using rigorous empirical work, where policy
makers and program planners are the primary audience for evaluation research. If
evaluation is done correctly, everyone benefits and if it is not done correctly, it will
be a waste of resources. It can be in the form of formative assessment or summative
assessment.
Assessment of learning outcomes serves as a crucial tool for educational qual-
ity assurance, aiding educators in evaluating and refining educational programs
promptly, ensuring educational objectives are met (Chin et al., 2009). There are two

13
 Education and Information Technologies

main types of assessment internationally recognized: formative and summative. For-

mative assessment occurs during the learning process, allowing for real-time progress
monitoring by teachers and students, typically within classroom settings. Summative
assessment, conversely, is a comprehensive evaluation of learning over a set period,
aiming to assign grades or qualifications based on overall student performance, often
culminating in a final exam. Gagne’s (1985) learning theory emphasizes the necessity
of both, advocating for initial formative assessment to identify and address learning
issues, followed by summative assessment to ascertain the achievement of learning
outcomes, emphasizing that omitting either hampers the educational process.
Formative assessment enables teachers to gather information through vari-
ous activities and student self-assessment, facilitating adjustments in teaching and
learning methods. A study by Ozan and Kıncal (2018) found that students engaged
in formative assessment demonstrated superior performance and higher learning
achievement compared to a control group. Digital technology is enhancing forma-
tive assessment’s efficiency, allowing for immediate evaluations, tracking progress,
and enabling dynamic assessments. Educational technology further supports forma-
tive assessment by boosting engagement, pinpointing knowledge gaps, and fostering
deeper learning (Majid, 2020). Studies have shown that online formative assessments
through digital signals can more effectively assess educational outcomes (Petrović
et al., 2017), and that science experiments conducted via computer simulations in
classrooms provide authentic experiences, with model-building identified as a crucial
formative assessment task for enhancing cognitive skills, knowledge performance,
and understanding learners’ grasp of the material (Peffer et al., 2015).

2.3 Program planning strategies in education and technology integration

Effective teaching and learning begin with effective instructional planning (Jones &
Davis, 2008). In the modern era, the rapid pace of educational technology develop-
ment is a key driver influencing the design and development of educational instruc-
tion (Adnan et al., 2022). This development not only impacts the technological
capacity of educational institutions to integrate hardware and software solutions
into existing learning infrastructures but also influences educational teaching and
learning practices, stakeholder acceptance of new technologies, and the administra-
tive structures of higher education institutions. Chugh et al. (2023) found that the
dynamic interplay of educational technology features, stakeholder perceptions of the
effectiveness of technology integration decisions, theoretical frameworks and models
related to technology integration in teaching and learning practices, and indicators
of post-implementation success are key dimensions in establishing effective path-
ways for educational technology implementation. Amidst this complex backdrop, the
implementation of a needs assessment emerges as a critical step in laying the ground-
work for successful technology integration programs. A needs assessment is vital
for initiating technology integration programs, requiring a thorough and balanced
approach. It should focus on how technology affects student learning and instruc-
tion, be conducted impartially by state or third-party agencies, include both inter-
nal (school data, instructional methods, processes), and external (educational trends,
technologies, legislation, unions, labor) reviews, and utilize a mix of quantitative

13
Education and Information Technologies

and qualitative data for a well-rounded and accurate understanding of the technology
integration needs and challenges.

2.4 Theoretical frameworks guiding technology integration evaluation and

planning

Despite the rapid growth of technology integration in education and the resources,
it requires in terms of time, money, and infrastructure, there is often inadequate
evidence of the effectiveness of educational technology in teaching and learning.
Furthermore, there is a lack of clarity regarding what constitutes ‘effectiveness’ and
how it can be achieved (King, 2014). To use technology effectively in education, it
is crucial to have a proper conceptual understanding that can guide the integration
process. A range of planning/evaluation frameworks have been developed and used
as a reference point in research and practice. Among these frameworks are the Diffu-
sion of Innovation Theory (DIT), the Technological Pedagogical Content Knowledge
(TPACK) framework, and the Substitution, Augmentation, Modification, and Redefi-
nition (SAMR) model, as well as the Analysis, Design, Development, Implementa-
tion, and Evaluation (ADDIE) model. All these models are widely recognized as
robust theoretical foundations for assessing and planning technology integration (see
Fig. 2 for a visual representation).
The Technological Pedagogical Content Knowledge (TPACK) framework is
widely acknowledged as a foundational theory for teaching with technology, pro-
viding teachers with a critical tool for successful technology integration into their
instructional practices. TPACK framework primarily focuses on analyzing the inter-
play of teachers’ technology knowledge, pedagogy, and content knowledge. The
TPACK framework offers a systematic approach for both pre-service and in-service
teachers to consider teaching methods and promote student-centered learning. A
study conducted by Deng and Zhang (2023) confirmed that pre-service teachers’ abil-
ity to integrate technology into their teaching improved after completing the TPACK
training program. This improvement was beneficial for both general and special edu-
cation educators. Similar findings were reported in the studies by Koh et al. (2017)
and Hsu (2016), where participants concluded that the TPACK framework facilitated
teachers’ use of technology in language classes. This finding is crucial in encourag-
ing teachers to embrace technology in their teaching practices and has a significant
impact on shaping teachers’ attitudes towards technology integration.
The ADDIE model is a systematic instructional design framework that consists
of five interconnected phases: Analysis, Design, Development, Implementation, and
Evaluation. It guides the process of creating effective learning experiences by first
analyzing the learning context and learners’ needs, then designing a detailed plan
including objectives and instructional strategies, followed by developing the actual
learning materials and activities. The implementation phase involves delivering the
instruction, and finally, the evaluation phase assesses the effectiveness of the learn-
ing experience. The iterative nature of the model allows for continuous improvement
based on feedback, ensuring that learning outcomes are achieved in a structured and
adaptable manner (Branch, 2009; Larson, 2014; Molenda, 2015; Piskurich, 2015).

13
 Education and Information Technologies

Fig. 2 Theoretical framework

The Substitution, Enhancement, Modification, Redefinition (SAMR) model helps

to reflect on the current use of the platform, demonstrating the integration of technol-
ogy along the levels of substitution, enhancement and modification (Drugova et al.,
2021). Harmandaoğlu et al. (2018) revealed that pre-service teachers exhibited posi-
tive attitudes towards SAMR model training. The training increased their awareness
of potential and innovative ways to integrate ICT into their teaching practice. The
participants expressed a desire to utilize the SAMR model in future lessons if they
had the necessary resources and opportunities. However, it is important to note that
the reality of insufficient technological facilities remains a challenge (Puentedura,
2014). The SAMR model, similar to the TPACK framework, is context-specific and
provides a framework for teachers to understand the levels of technology integra-
tion in their instructional practices. It emphasizes moving beyond mere substitution
to transforming and redefining learning experiences through the effective use of
technology.

13
Education and Information Technologies

Each application of technology is categorized into one of four tasks: replacement,

enhancement (both termed ‘enhancements’), and modification, redefinition (both
termed ‘transformations’). Despite its visual representation as a ladder, the SAMR
model isn’t strictly hierarchical. Yet, a sequential progression through its levels can
aid teachers in enhancing their (TPACK) (Geer et al., 2017).
The TPACK and SAMR models can be utilized to analyze the process of integrat-
ing educational technology (Mishra & Koehler’s, 2006; Puentedura’s, 2014). Taken
together, the TPACK model and the SAMR model complement each other to pro-
vide valuable insights into student learning when selecting technology. The TPACK
model helps contextualize the teacher knowledge required for technology integra-
tion, while the SAMR model assists in analyzing the specific levels of enhancement
and transformation. This combination allows for a comprehensive examination of
technology integration from both teacher and student perspectives, helping educators
understand the challenges and opportunities that arise in student learning.
Rogers (2003) is a well-established empirical framework (Miller, 2015), and Dif-
fusion of Innovations Theory (DIT) has proven successful in many areas, particu-
larly the adoption of educational technology in education (Jwaifell & Gasaymeh,
2013). Recently, DIT has been applied to examine the adoption of online supervised
examinations during the pandemic, and experiential learning through live labora-
tories (Raman et al., 2020). In addition, it is continually providing answers to how
society responds emotionally and behaviorally to innovation (Hains & Hains, 2020).

2.5 Challenges and barriers to technology integration in education

Technology integration makes class effective and shifting the focus from the teacher-
centred methodology to the student-centred methodology. In addition, technology
integration continues to have a positive impact on teaching and learning activities
as well as the teaching and learning environment. However, technology integra-
tion always does not get the privilege of selection from the instructors and learners
because of its weaknesses and challenges. Technology integration has some loop-
holes that are worth researching (Adhikari, 2023). There are also some issues, chal-
lenges and opportunities for technology integration that deserve to be discussed.
Moreover, although the national contexts are different in developed and developing
countries, developed countries have similar concerns in technology integration like
teacher apprehension and motivation, and lack of appropriate educational software
and technical support. Numerous studies have identified current problems in technol-
ogy integration. (Iftikhar et al., 2022), conclude that pre-service teachers showed
more anxiety than in-service teachers were less technology competent than the prac-
ticing teachers.
Teachers and administrators feel confident using ICT in schools but lack clear
integration guidelines, while students believe ICT is underutilized (Gülbahar, 2007).
Educational institutions often suffer from insufficient technical and financial sup-
port. Although some state-controlled institutions receive training and equipment, the
absence of follow-up and qualified operators leads to underuse, rendering the equip-
ment eventually unusable. Ali and Azhar (2018) found lack of support for technology
integration, while Singh and Meena (2019) emphasized the necessity of ongoing pro-

13
 Education and Information Technologies

fessional training for teachers. However, schools continue to face barriers in technol-
ogy integration, ranging from infrastructural issues to a lack of expert support and
resolution of basic technological challenges in the classroom. Akram et al. (2022)
also highlighted additional hurdles, including resource scarcity, insufficient leader-
ship, limited ICT access, time constraints, vague policies, and a lack of professional
development, technical support, and suitable instructional models.
Technology integration in schools has its challenges. Many tech tools and teaching
methods are still in the early stages of development, and might not work well in every
type of classroom. Teachers’ own tech skills are key for making technology work in
education. However, many teachers do not have the training they need, leading to
less effective use of tech tools in teaching (Singh, 2019). Challenges also come from
low digital literacy, not enough access to tech resources, a lack of support, and old
negative beliefs from teachers, schools, and students (Kandel, 2022). Furthermore,
there’s a mismatch in how school leaders, teachers, and students view technology’s
role in education, and this can make tech integration harder (Pagunsan-Sumbise &
Montecalvo, 2021). In developing countries, the lack of access to tech tools creates a
digital divide, widening the gap in tech skills among teachers, students, and schools
(Kandel, 2022).
The success of any program depends on the collaboration of all stakeholders
involved. Teachers, students, educators, and policymakers play vital roles in integrat-
ing technology into classrooms and educational programs. However, many school
managements are not supportive enough, operating in a top-down manner, but their
support can significantly encourage the integration of technology in education (Vatan-
artiran & Karadeniz, 2015). Funding issues pose a significant barrier, hindering edu-
cational technology initiatives, research, and teacher training (Ali & Azhar, 2018).
Moreover, the lack of leadership support and unclear technology policies contribute
to these challenges (Akram et al., 2022). For technology integration to be effective, it
is crucial to have clear and supportive ICT policies (Kandel, 2022).

3 Methodology

3.1 Search strategy and databases

To conduct a systematic review of the literature on evaluation and program planning

strategies for technology integration in education, the guidelines of the Preferred
Reporting Items for Systematic Reviews and Meta-Analyses were followed (Moher
et al., 2009). Peer-reviewed journal articles published in English from 2010 to May
2023 were searched using four electronic databases: Web of Science, Scopus, ERIC,
and ScienceDirect. These databases were selected because they cover a wide range
of disciplines and topics related to education, technology, evaluation, and program
planning. A combination of keywords and Boolean operators was employed to con-
struct the search queries. The keywords were derived from the main concepts of the
research question: evaluation, program planning, technology integration, and edu-
cation. Synonyms, related terms, and truncations were also utilized to expand the

13
Education and Information Technologies

search results. The following Table 1 shows an example of the search query for Web
of Science.
The final search query was (evaluat* OR assess* OR measure* OR indicat*) AND
(program* OR plan* OR design* OR strateg* OR framework* OR model*) AND
(technolog* OR ICT OR digital* OR online OR e-learning OR blended OR hybrid)
AND (educat* OR teach* OR learn* OR school* OR college* OR universit*).
The same logic was applied to construct similar queries for the other databases,
adjusting the syntax and filters as needed. The search was limited to articles contain-
ing the keywords in their titles, abstracts, or keyword fields. The initial search yielded
a total of 6,543 articles across the four databases. These results were exported to a
reference management software (EndNote), and duplicates were removed. Following
this process, 4,217 articles remained.

3.2 Inclusion and exclusion criteria

The remaining 4,217 articles were screened by their titles and abstracts using the fol-
lowing inclusion and exclusion criteria. Covidence software was used to manage the
screening process of the titles and abstracts of the articles. Covidence is a web-based
platform that facilitates the conduct of systematic reviews by providing features such
as deduplication, screening, data extraction, and quality assessment. Two reviewers
were assigned to independently screen the articles using the inclusion and exclusion
criteria. Discrepancies between reviewers were resolved through discussion or, if
necessary, by consulting a third reviewer. This approach ensured that the selection
procedure was consistent, transparent, and reliable. Previous studies have recom-
mended the use of Covidence software and the importance of resolving discrepancies
in systematic reviews (Beller et al., 2018; McKeown & Mir, 2021).

3.2.1 Inclusion criteria

The article reports on an empirical study that involves evaluation or program plan-
ning of technology integration in education. The article provides sufficient details on
the methods, results and implications of the evaluation or program planning process.
The article focuses on formal education settings (e.g., K-12 schools, higher education
institutions) or informal education settings (such as museums and libraries) that are
relevant to our research question.

Table 1 illustrates an example Concept Keywords

of the search query
Evaluation Evaluat* OR assess* OR measure* OR
indicat*
Program planning Program* OR plan* OR design* OR
strateg* OR framework* OR model*
Technology Technolog* OR ICT OR digital* OR on-
integration line OR e-learning OR blended OR hybrid
Source: Researcher’s Education Educat* OR teach* OR learn* OR
Computation school* OR college* OR universit*

13
 Education and Information Technologies

3.2.2 Exclusion criteria

The article is not written in English or is not peer-reviewed. The article does not
address evaluation or program planning of technology integration in education as its
main topic or objective. The article is a review, meta-analysis, editorial, commentary,
or opinion piece that does not present original data or findings. After applying the
criteria, 372 articles met the inclusion criteria and proceeded to full-text assessment.
The full texts of these articles were retrieved from the databases or other sources
(e.g., Google Scholar, ResearchGate) and assessed them for eligibility using the same
criteria as above. The reference lists of these articles were also examined to identify
additional relevant studies that may have been missed in the initial database search.
Through this process, 248 articles were excluded for not meeting the criteria or being
inaccessible, resulting in the inclusion of 124 articles in the systematic review. Fig-
ure 3 presents a PRISMA flow diagram summarizing the search and selection process.

3.3 Data extraction and synthesis

To extract data from the included studies, a standardized data extraction form was
developed and pilot-tested on a sample of 10 studies. The data extraction form
included the following variables of interest.

Fig. 3 A PRISMA flow diagram

13
Education and Information Technologies

3.3.1 Study characteristics

Author(s), year, title, journal, country, aim/purpose, research question(s), and fund-
ing source(s).

3.3.2 Evaluation strategies

Type, purpose, scope, criteria, methods, tools, and outcomes of evaluation of technol-
ogy integration in education.

3.3.3 Program planning strategies

Type, purpose, scope, criteria, methods, tools, and outcomes of program planning for
technology integration in education.

3.3.4 Outcomes

Type, measure, effect size, significance level, and direction of the effects or impacts
of evaluation or program planning strategies for technology integration in education
on various outcomes, such as teacher knowledge, skills, attitudes, beliefs, behaviors,
or student learning outcomes.
Covidence software was employed to manage the data extraction process. Two
reviewers were assigned to independently extract data from each study using the data
extraction form. Discrepancies between reviewers were resolved through discussion
or by consulting a third reviewer if necessary. The same two reviewers who per-
formed data extraction also conducted the quality assessment. These reviewers, Dr. X
and Dr. Y, are both researchers at Z University with expertise in evaluation, program
planning, technology integration, and education. Authors of the original studies were
contacted to request additional information or clarification when necessary.
To synthesize the findings from the extracted data, a narrative synthesis approach
was employed (Popay et al., 2006). Narrative synthesis is a method of summarizing
and explaining the findings from multiple studies using words and text. It involves
four main elements: (a) developing a theory of how the intervention works, why and
for whom; (b) developing a preliminary synthesis of the findings of the included
studies; (c) exploring relationships within and between studies; and (d) assessing
the robustness of the synthesis. The following steps were undertaken to address the
research questions and objectives, and to offer insights for evidence-based practice.

3.4 Quality assessment

To assess the quality of the included studies, established tools that are appropriate for
different study designs were employed. For Randomized Controlled Trials (RCTs),
the Cochrane Collaboration’s risk of bias tool (Higgins et al., 2011) was utilized. This
tool evaluates six domains of bias: selection bias, performance bias, detection bias,
attrition bias, reporting bias, and other bias. Each domain was rated as having low,
high, or unclear risk of bias, with justifications provided. For Non-Randomized Stud-

13
 Education and Information Technologies

ies (NRS), the Critical Appraisal Skills Programme (CASP) tool was used. This tool
consists of 12 questions that cover three broad areas: validity, results, and applicabil-
ity. For each question in the CASP tool, answers were categorized as yes, no, or can-
not tell, accompanied by a brief explanation. For Mixed-Methods Studies (MMS),
the Mixed-Methods Appraisal Tool (MMAT) (Hong et al., 2018) was employed. This
tool evaluates the quality of both quantitative and qualitative aspects of a study, as
well as their integration. The MMAT features five criteria for each component and
one for integration. Each criterion was rated as yes, no, or cannot tell, with a rationale
provided for each rating.
Covidence software was employed to manage the quality assessment process. Two
reviewers were assigned to independently assess the quality of each study using the
relevant tool. Discrepancies between reviewers were resolved through discussion or
by consulting a third reviewer, if needed. Authors of the original studies were con-
tacted for additional information or clarification when needed. The quality assessment
results informed the synthesis and interpretation of the findings. These results were
reported in a table and summarized narratively. Sensitivity analyses were conducted
to examine the impact of study quality on overall findings. Studies with high risk
of bias or low quality were excluded from the synthesis, and results were compared
both with and without these studies. Confidence in the evidence was assessed using
the GRADE approach (Guyatt et al., 2008), considering risk of bias, inconsistency,
indirectness, and imprecision. Confidence levels were rated as high, moderate, low,
or very low for each outcome.

3.5 Reporting of results

The results of the systematic review were reported in accordance with the PRISMA
guidelines (Moher et al., 2009), which offer a 27-item checklist aimed at enhancing
the transparency, completeness, and quality of reporting for systematic reviews and
meta-analyses.

3.6 Overview of included studies

A total of 124 studies meeting the inclusion criteria and reported on evaluation or
program planning strategies for technology integration in education were included
in this review. The Table 2 provides a summary of the main characteristics of the
included studies, such as author(s), year, title, journal, country, aim/purpose, study
design, sample size, setting, intervention, outcome(s), and quality assessment result.
Figure 4 illustrates a map displaying the countries where the included studies were
conducted. The circles on the map correspond to the number of studies conducted in
each respective country, with larger circles indicating a greater number of studies.
Figure 5 depicts a bar chart representing the number of studies published by year.
The figure clearly demonstrates the number of studies published each year shows an
increasing trend, indicating a growing interest and investment in research.
Figure 6 presents a pie chart showcasing the distribution of study designs among
the included studies. The chart reveals that the majority of studies utilized quantita-

13
Education and Information Technologies

Table 2 An example of a sum- Author(s) Example: Ho et al. (2013)

mary of the main characteristics
Year 2013
of the included studies
Title Using theoretical models to examine the ac-
ceptance behavior of mobile phone messag-
ing to enhance parent–teacher interactions
Journal Computers & Education
Country Taiwan
Aim/Purpose To investigate teacher behavior of adopting
mobile phone messaging as a parent-teach-
er communication medium by applying and
comparing the TAM, C-TAM-TPB, and
UTAUT models.
Study Design Survey research using questionnaires with
7-point Likert scales.
Sample Size 315 usable responses from primary school
teachers.
Setting Primary schools in Shin Chu area, Taiwan.
Outcome(s) Perceived usefulness, perceived ease of use,
attitude, subjective norms, and perceived
behavioral control were found to influence
teachers’ intention to use mobile messaging
for parent communication.
Quality Assessment The measurement scales showed adequate
Source: Researcher’s Result reliability and validity based on the analysis
Computation provided.

Fig. 4 Geographical distribution of included studies

tive methods, accounting for 54% of the total. Following this, mixed-methods studies
constituted 28%, while qualitative methods were employed in 18% of the studies.
Figure 7a-b displaying the frequency of evaluation and program planning strate-
gies reported by the included studies. According to the Fig. 7a, the most common
evaluation strategy reported was framework-based evaluation, accounting for 32% of
the strategies. This was followed by goal-based evaluation at 24%, and process-based

13
 Education and Information Technologies

Fig. 5 The number of studies published by year (2010 to May 2023)

Fig. 6 The distribution of study designs among the included studies

evaluation at 16%. Regarding Fig. 7b, program planning strategies, the most preva-
lent approach reported was design-based research, representing 28% of the strategies.
The second most common program planning strategy was the use of a logic model,
accounting for 20% of the reported strategies. Lastly, backward design was reported
as the third most common program planning strategy, making up 16% of the total
strategies reported.

3.7 Evaluation strategies for technology integration

One of the main objectives of this systematic review was to identify and synthesize
the evidence on evaluation strategies for technology integration in education. Evalu-

13
Education and Information Technologies

Fig. 7 The frequency of the most common evaluation strategy. The frequency of the most common
program planning strategy

ation is defined as “the systematic collection and analysis of information to make

judgments, usually about the effectiveness, efficiency and/or appropriateness of an
educational intervention” (Kirkpatrick & Kirkpatrick, 2006, p. 21). Evaluation can
serve different purposes, such as providing feedback, improving practice, inform-
ing decision-making, or demonstrating accountability (Patton, 2012). The review
found that the included studies reported on various types of evaluation strategies for
technology integration, such as framework-based evaluation, goal-based evaluation,
process-based evaluation, outcome-based evaluation, and mixed-methods evalua-
tion. These strategies were systematically categorized based on their primary focus
and approach, as presented in Table 3.
The findings from the included studies were synthesized using a narrative synthe-
sis approach (Popay et al., 2006). The review found that framework-based evaluation
could help teachers and evaluators to understand the current level and quality of tech-
nology integration; to identify strengths and weaknesses; to set goals and standards;
to plan improvement actions; and to communicate and share best practices (Garzón &
Acevedo, 2019; Davies & West, 2014; Ledford, 2016). However, framework-based
evaluation can also pose challenges such as selecting or adapting an appropriate
framework; aligning the framework with the curriculum; training teachers and evalu-
ators to use the framework; ensuring reliability and validity of the framework; and
addressing contextual factors that may affect technology integration (Jones & Davis,
2008).
Goal-based evaluation can help teachers and evaluators to clarify the purpose and
expectations of technology integration; to align technology use with learning objec-
tives; to design and deliver technology-enhanced learning activities; to monitor and
measure progress and achievement; and to provide feedback and recognition (Wu et
al., 2022). However, goal-based evaluation can also pose challenges such as defining
realistic and measurable goals; selecting or developing appropriate assessment tools;
collecting and analyzing data; interpreting and reporting results; and addressing ethi-
cal issues such as fairness, equity, and privacy.
Process-based evaluation can help teachers and evaluators to understand the
change process of technology integration; to identify the needs and concerns of the
stakeholders; to provide support and guidance; to facilitate collaboration and com-
munication; to document and reflect on the implementation process; and to adjust
and improve the implementation strategies (Timotheou et al., 2023; Summak et al.,

13
 Table 3 Evaluation strategy classification
Evaluation Definition Example Reference

13
Strategy
Framework- An evaluation that uses an existing or The Technology Integration Matrix (TIM) is a framework that describes five levels (Florida Center for
based adapted framework or model to guide the of technology integration (entry, adoption, adaptation, infusion, and transformation) Instructional Technol-
evaluation evaluation design, data collection, analysis, and five characteristics of meaningful learning environments (active, collaborative, ogy, 2011)
and reporting. constructive, authentic, and goal-directed). The TIM provides a tool for teachers
and evaluators to assess and improve technology integration practices.
Goal-based An evaluation that focuses on the extent to The SAMR model is a framework that describes four levels of technology integra- (Puentedura, 2013)
evaluation which the intended goals or objectives of a tion (substitution, augmentation, modification, and redefinition) based on the degree
technology integration program or interven- of transformation of the learning task. The SAMR model provides a tool for teach-
tion are achieved. ers and evaluators to set and measure goals for technology integration.
Process- An evaluation that focuses on the The Concerns-Based Adoption Model (CBAM) is a framework that describes the (Hall & Hord, 2015)
based implementation process of a technology stages of concern and levels of use of an innovation among adopters. The CBAM
evaluation integration program or intervention, such as provides a tool for teachers and evaluators to monitor and support the change pro-
the inputs, activities, outputs, and feedback cess of technology integration.
mechanisms.
Outcome- An evaluation that focuses on the outcomes The Technology Integration Assessment Instrument (TIAI) is a survey instrument (Abbitt & Klett,
based or impacts of a technology integration pro- that measures teachers’ perceptions of their knowledge and skills related to technol- 2007)
evaluation gram or intervention on various indicators, ogy integration; their beliefs about the value and role of technology in education;
such as teacher knowledge, skills, attitudes, their self-efficacy and anxiety toward using technology; and their frequency and
beliefs, behaviors, or student learning type of technology use in teaching. The TIAI provides a tool for teachers and evalu-
outcomes. ators to assess the outcomes of technology integration.
Mixed- An evaluation that combines quantita- The Technology Integration Evaluation Model (TIEM) is a mixed-methods model (Sun et al., 2017)
methods tive and qualitative methods to collect and that evaluates technology integration from four dimensions: context (the envi-
evaluation analyze data from multiple sources and ronmental factors that influence technology integration), input (the resources and
perspectives. support for technology integration), process (the implementation strategies and
activities for technology integration), and product (the outcomes and impacts of
technology integration). The TIEM provides a tool for teachers and evaluators to
conduct a comprehensive and holistic evaluation of technology integration.
Source: Researcher’s Computation
Education and Information Technologies
Education and Information Technologies

2010). However, process-based evaluation can also pose challenges such as defining
the scope and duration of the evaluation; selecting or developing appropriate data
collection methods; managing and analyzing data; ensuring trustworthiness and cred-
ibility of the data; and dealing with resistance and barriers to change (Stufflebeam &
Coryn, 2014; Summak et al., 2010.
Outcome-based evaluation can help teachers and evaluators to determine the
effects or impacts of technology integration on various indicators; to compare the
results with benchmarks or standards; to demonstrate accountability and effective-
ness; to justify investment and resource allocation; and to inform decision-making
and policy-making (Yıldırım & Şen, 2021; Mikasa et al., 2013). However, outcome-
based evaluation can also pose challenges such as establishing causal relationships
between technology integration and outcomes; controlling for confounding vari-
ables; attributing outcomes to technology integration; generalizing results to other
contexts or populations; and addressing unintended or negative outcomes (Japee &
Oza, 2021).
Mixed-methods evaluation can help teachers and evaluators to obtain a compre-
hensive and holistic picture of technology integration; to triangulate data from mul-
tiple sources and perspectives; to complement or supplement quantitative data with
qualitative data or vice versa; to explore complex phenomena or issues in depth; and
to enhance the validity and reliability of the evaluation (Sun et al., 2017; Wyant et al.,
2015). However, mixed-methods evaluation can also pose challenges such as select-
ing or designing an appropriate mixed-methods design; integrating quantitative and
qualitative data effectively; balancing the strengths and weaknesses of each method;
managing the complexity and workload of the evaluation; and resolving conflicting
or contradictory findings (Tuna & Başdal, 2021).

3.8 Program planning strategies for technology integration

Another objective of this systematic review was to identify and synthesize the evi-
dence on program planning strategies for technology integration in education. Pro-
gram planning is defined as “the process of designing, developing, implementing, and
evaluating educational programs to achieve specific educational goals” (Kaufman et
al., 2003, p. 4). Program planning can help educators to align technology use with
curriculum standards, learning objectives, instructional methods, and assessment
strategies; to allocate resources and support for technology integration; to monitor
and improve the implementation process; and to ensure the sustainability and scal-
ability of technology integration (Chugh et al., 2023; Adnan et al., 2022).
The review found that the included studies reported on various types of program
planning strategies for technology integration, such as design-based research, logic
model, backward design, ADDIE model, and SAMR model. These strategies were
categorized based on their main components and steps, as shown in the following
Table 4.
The synthesis of findings utilized a narrative approach (Popay et al., 2006) to eval-
uate the effectiveness of program planning strategies for technology integration. The
present study found that design-based research could help educators to develop and
implement innovative and effective solutions or interventions for technology integra-

13
 Table 4 Program planning strategy classification
Program Plan- Definition Components/Steps
ning Strategy

13
Design-based A research approach that involves (1) Identify a practical problem or challenge in education. (2) Review relevant literature and theory to develop a
research designing, implementing, testing, and tentative solution or innovation. (3) Implement the solution or innovation in a naturalistic setting with multiple
refining educational interventions in cycles of testing and refinement. (4) Collect and analyze multiple sources of data to evaluate the effectiveness
real-world settings. and impact of the solution or innovation. (5) Generate design principles and theoretical insights that can inform
future practice and research.
Logic model A graphical representation that shows (1) Identify the problem or need that the program or intervention aims to address. (2) Specify the inputs or
the logical relationships between the resources that are required for the program or intervention. (3) Describe the activities or processes that will be
inputs, activities, outputs, outcomes, carried out by the program or intervention. (4) Define the outputs or products that will result from the program
and impacts of an educational program or intervention activities. (5) Determine the outcomes or changes that will occur among the participants or ben-
or intervention. eficiaries of the program or intervention in the short, medium, and long term. (6) Identify the impacts or effects
that will occur at the broader level or system as a result of the program or intervention outcomes.
Backward A curriculum design approach that (1) Identify the desired results or learning goals based on curriculum standards and student needs. (2) Determine
design starts with identifying the desired re- the acceptable evidence or assessment criteria that will demonstrate student achievement of the learning goals.
sults and then works backward to plan (3) Plan the learning experiences and instruction that will enable students to acquire the knowledge and skills
the learning experiences and assess- necessary to achieve the learning goals and meet the assessment criteria.
ments that will lead to those results.
ADDIE model A generic instructional design model (1) Analysis: Analyze the needs, goals, objectives, learners, context, and constraints of the instructional problem
that consists of five phases: analysis, or situation. (2) Design: Design a blueprint or plan for the instructional solution, including learning outcomes,
design, development, implementation, assessment strategies, content outline, instructional methods, media selection, and delivery mode. (3) Develop-
and evaluation. ment: Develop or produce the instructional materials and resources based on the design plan using appropriate
tools and technologies. (4) Implementation: Implement or deliver the instructional solution to the intended
learners using effective facilitation and management strategies. (5) Evaluation: Evaluate the effectiveness and
efficiency of the instructional solution using formative and summative methods and data sources; revise and
improve the instructional solution based on feedback and findings.
SAMR model A framework that describes four levels (1) Substitution: Technology acts as a direct tool substitute with no functional change (e.g., using a word pro-
of technology integration (substitu- cessor instead of a typewriter). (2) Augmentation: Technology acts as a direct tool substitute with functional im-
tion, augmentation, modification, and provement (e.g., using a word processor with spell check instead of a typewriter). (3) Modification: Technology
redefinition) based on the degree of allows for significant task redesign (e.g., using a word processor with collaborative features instead of a type-
transformation of the learning task. writer). (4) Redefinition: Technology allows for creation of new tasks that were previously inconceivable (e.g.,
using a word processor with multimedia features to create an interactive digital story instead of a typewriter).
Source: Researcher’s Computation
Education and Information Technologies
Education and Information Technologies

tion in authentic settings; to engage in collaborative and iterative cycles of inquiry

and improvement; to generate practical and theoretical knowledge that can inform
future practice and research; and to address complex and ill-structured problems or
challenges in education (Harrison & West, 2014; Xu et al., 2022; Wang, 2020; Wang
et al., 2014). However, design-based research can also pose challenges such as bal-
ancing rigor and relevance; managing multiple stakeholders and perspectives; ensur-
ing ethical and quality standards; documenting and disseminating the design process
and outcomes; and scaling up and sustaining the solutions or interventions.
Logic model can help educators to clarify the purpose and logic of a program or
intervention for technology integration; to align the inputs, activities, outputs, out-
comes, and impacts with the program or intervention goals; to communicate and
share the program or intervention plan with stakeholders; to monitor and evaluate
the program or intervention progress and performance; and to identify and address
gaps or issues in the program or intervention implementation (McLaughlin & Jordan,
2015). However, logic model can also pose challenges such as capturing the com-
plexity and dynamics of a program or intervention; accounting for contextual factors
and external influences; ensuring validity and reliability of the data and assumptions;
revising and updating the logic model as needed; and avoiding oversimplification or
overgeneralization of the logic model.
Backward design can help educators to design curriculum that is aligned with cur-
riculum standards and student needs; to focus on the desired results or learning goals
rather than the activities or content; to plan assessment strategies that are valid and
reliable measures of student learning; to plan learning experiences and instruction
that are engaging and effective; and to ensure coherence and consistency across the
curriculum (Lamberton & Raschke, 2023; Fox & Doherty, 2012). However, back-
ward design can also pose challenges such as defining clear and measurable learning
goals; selecting appropriate assessment tools and criteria; differentiating instruction
for diverse learners; integrating cross-curricular connections and skills; and adapting
to changing circumstances or feedback.
ADDIE model can help educators to follow a systematic and structured process for
instructional design; to apply a variety of methods, tools, and technologies for each
phase of the process; to involve stakeholders in each phase of the process; to evaluate
each phase of the process for quality assurance; and to revise and improve each phase
of the process based on feedback (Stapa & Mohammad, 2019; Nichols Hess & Greer,
2016; Yeh & Tseng, 2019). However, ADDIE model can also pose challenges such
as being too linear or rigid for complex or dynamic situations; requiring a lot of time
and resources for each phase of the process; coordinating multiple tasks and roles for
each phase of the process; integrating formative evaluation throughout the process;
and coping with uncertainty or ambiguity in the process.
SAMR model can help educators to understand the different levels of technology
integration; to evaluate the current level of technology integration in their practice;
to set goals for moving from lower to higher levels of technology integration; to
design learning tasks that leverage technology to transform learning; and to foster
student creativity, collaboration, communication, and critical thinking (Wahyuni et
al., 2020; Caukin & Trail, 2019). However, SAMR model can also pose challenges
such as defining the criteria for each level of technology integration; distinguishing

13
 Education and Information Technologies

between augmentation and modification levels of technology integration; selecting or

creating appropriate technologies for each level of technology integration; assessing
student learning outcomes at each level of technology integration; and addressing
equity issues related to technology access and use (Hamilton et al., 2016).

3.9 Factors influencing the success of technology integration

The present systematic review not only aggregates and analyzes evidence concerning
evaluation and program planning strategies for technology integration in educational
settings but also investigates the factors affecting the success of such integration.
Success is defined as the degree to which the technology integration meets predefined
objectives, encompassing improvements in student learning, teacher efficacy, and
systemic educational transformations. A socio-technical framework was employed to
acknowledge that successful technology integration is a product of both technologi-
cal and social factors, which are intrinsically interconnected and interactive with the
applied evaluation and planning strategies.
The present study found that the included studies reported on various types of
factors that influence the success of technology integration in education, such as
organizational, technological, pedagogical, teacher, and student factors. The specific
attributes of these domains are detailed in Table 5. This way of categorizing helps
us better understand the various factors that contribute to the success of integrating
technology in education.
This review synthesized the findings from the included studies on the interac-
tion between these factors and the evaluation and program planning strategies for
technology integration using a narrative synthesis approach (Popay et al., 2006). The
present study found that organizational factors can influence the selection, implemen-
tation, and evaluation of evaluation and program planning strategies for technology
integration by providing or limiting the vision, direction, support, resources, lead-
ership, culture, and policy for technology integration. For instance, having a well-
defined and shared vision for technology integration can ensure that evaluation and
program planning strategies are in line with the organization’s goals and objectives
(Mahmoud, 2022; Franklin & Bolick, 2007). A supportive and innovative culture can
encourage collaboration and feedback among stakeholders involved in the evaluation
and program planning process (Leyva, 2015). Moreover, strong and effective leader-
ship is essential in facilitating and sustaining the implementation and improvement
of evaluation and program planning strategies (Adnan et al., 2022). These key fac-
tors contribute to the successful integration of technology and maximize the benefits
derived from evaluation and program planning efforts.
Technological factors can influence the design, delivery, assessment, and evalu-
ation of evaluation and program planning strategies for technology integration by
enabling or inhibiting certain functions, features, affordances, and constraints of the
technology tools or systems that are used for technology integration (Materia et al.,
2016). For instance, the availability of accessible and user-friendly technology tools
or systems can significantly enhance the participation and engagement of stakehold-
ers in the evaluation and program planning process (Nelson et al., 2019; Hamutoglu,
2021). Similarly, reliable and secure technology tools or systems play a crucial role in

13
 Table 5 Various types of factors that influence the success of technology integration in education
Factor Definition Example
Organi- Factors related to the (1) Vision and mission: The clarity and alignment of the organizational vision and mission for technology integration with
zational structure, culture, policy, the curriculum standards, learning objectives, and stakeholder needs. (2) Strategy and planning: The comprehensiveness and
factors leadership, and management effectiveness of the organizational strategy and planning for technology integration, including goal setting, resource allocation,
of the educational organiza- implementation process, evaluation plan, and improvement actions. (3) Support and infrastructure: The availability and quality
tion or system that supports of the organizational support and infrastructure for technology integration, such as technical support, professional develop-
or hinders technology ment, funding, equipment, network, software, etc. (4) Leadership and management: The role and influence of the organization-
integration. al leaders and managers in facilitating and sustaining technology integration, such as providing direction, guidance, feedback,
recognition, communication, collaboration, etc. (5) Culture and climate: The norms and values of the organizational culture
and climate that affect technology integration, such as innovation, collaboration, trust, openness, flexibility, etc.
Techno- Factors related to the fea- (1) Accessibility and usability: The ease of access and use of the technology tools or systems for technology integration by dif-
Education and Information Technologies

logical tures, functions, affordanc- ferent users in different contexts. (2) Reliability and security: The dependability and safety of the technology tools or systems
factors es, and constraints of the for technology integration in terms of performance, functionality, stability, privacy, etc. (3) Compatibility and interoperability:
technology tools or systems The degree of fit and integration of the technology tools or systems with other technologies or systems that are used for tech-
that are used for technology nology integration. (4) Functionality and flexibility: The range and diversity of functions and features that the technology tools
integration. or systems offer for technology integration to meet different needs and purposes. (5) Affordance and constraint: The potential
and limitation of the technology tools or systems for enabling or inhibiting certain actions or behaviors related to technology
integration.
Peda- Factors related to the (1) Pedagogical approach: The underlying philosophy or theory that guides the design and delivery of teaching and learning
gogical design, delivery, assess- with technology. (2) Pedagogical method: The specific technique or strategy that is used to implement teaching and learning
factors ment, and evaluation of with technology. (3) Pedagogical content: The subject matter or topic that is taught or learned with technology. (4) Pedagogi-
teaching and learning with cal outcome: The intended or achieved result or impact of teaching or learning with technology. (5) Pedagogical evaluation:
technology. The process or method of collecting and analyzing data to measure the effectiveness or quality of teaching or learning with
technology.

13
 Table 5 (continued)
Factor Definition Example

13
Teacher Factors related to the (1) Knowledge and skills: The cognitive and practical abilities of the teachers to use technology for teaching and learning, such
factors knowledge, skills, attitudes, as technological, pedagogical, content, and TPACK knowledge; digital literacy; problem-solving; critical thinking; etc. (2) At-
beliefs, behaviors, and titudes and beliefs: The affective and cognitive dispositions of the teachers toward using technology for teaching and learning,
characteristic of the teachers such as motivation; interest; confidence; self-efficacy; anxiety; perceived usefulness; perceived ease of use; etc. (3) Behaviors
who use technology for and characteristics: The observable actions and traits of the teachers when using technology for teaching and learning, such as
teaching and learning. frequency; type; intensity; duration; style; preference; etc.
Student Factors related to the (1) Knowledge and skills: The cognitive and practical abilities of the students to use technology for learning and assessment,
factors knowledge, skills, attitudes, such as technological, pedagogical, content, and TPACK knowledge; digital literacy; problem-solving; critical thinking; etc.
beliefs, behaviors, and char- (2) Attitudes and beliefs: The affective and cognitive dispositions of the students toward using technology for learning and
acteristics of the students assessment, such as motivation; interest; confidence; self-efficacy; anxiety; perceived usefulness; perceived ease of use; etc. (3)
who use technology for Behaviors and characteristics: The observable actions and traits of the students when using technology for learning and assess-
learning and assessment. ment, such as frequency; type; intensity; duration; style; preference; etc.
Source: Researcher’s Computation
Education and Information Technologies
Education and Information Technologies

ensuring the quality and validity of the data and information collected and analyzed
during the evaluation and program planning activities (Chen et al., 2019). Further-
more, the compatibility and interoperability of technology tools or systems enable
the seamless integration of different technologies or systems utilized for technology
integration purposes Ertmer and Ottenbreit-Leftwich (2010). These factors contrib-
ute to the effectiveness and efficiency of the evaluation and program planning process
in leveraging technology for educational purposes.
Pedagogical factors can influence the effectiveness, efficiency, appropriateness,
and impact of evaluation and program planning strategies for technology integration
by determining the pedagogical approach, method, content, outcome, and evalua-
tion that are used for teaching and learning with technology (Nelson et al., 2019).
For example, adopting a constructivist pedagogical approach can greatly support the
design and implementation of evaluation and program planning strategies that pri-
oritize learner-centered, inquiry-based, and authentic learning experiences (Kim et
al., 2013). Incorporating a formative pedagogical evaluation component ensures con-
tinuous and timely feedback, allowing for iterative improvements to the evaluation
and program planning strategies (Catio, 2019). Furthermore, employing a student-
centered pedagogical outcome assessment enables the measurement and evaluation
of the impact of these strategies on student learning outcomes, encompassing aspects
such as knowledge acquisition, skill development, attitude formation, and behavioral
changes (Majid, 2020). These pedagogical considerations contribute to the effective-
ness and relevance of evaluation and program planning efforts in enhancing educa-
tional outcomes.
Teacher factors can influence the adoption, integration, and evaluation of evalu-
ation and program planning strategies for technology integration by affecting the
knowledge, skills, attitudes, beliefs, behaviors, and characteristics of the teachers
who use technology for teaching and learning (Nelson et al., 2019). For example,
a strong foundation in technological, pedagogical, content, and TPACK knowledge
and skills empowers teachers to effectively and efficiently employ evaluation and
program planning strategies. Positive attitudes and beliefs towards technology use
in teaching and learning serve as motivational factors that encourage teachers to
willingly and confidently utilize these strategies (Kim et al., 2013). Furthermore,
teachers’ diverse and frequent engagement with technology, as well as the various
behaviors and characteristics exhibited during its integration, reflect their level of
proficiency and the quality of technology integration in their instructional practices
(Ifinedo et al., 2020). These factors collectively contribute to the successful imple-
mentation of evaluation and program planning strategies in educational settings.
Student factors can influence the engagement, learning, and assessment of evalu-
ation and program planning strategies for technology integration by affecting the
knowledge, skills, attitudes, beliefs, behaviors, and characteristics of the students
who use technology for learning and assessment (Deng & Zhang, 2023). For instance,
when students possess advanced technological, pedagogical, content, and TPACK
knowledge and skills, they are able to effectively and efficiently utilize evaluation and
program planning strategies (Alhababi, 2017). Positive attitudes and beliefs towards
technology in the context of learning and assessment can serve as motivating factors
for students, encouraging them to willingly and confidently employ evaluation and

13
 Education and Information Technologies

program planning strategies (Sheffield et al., 2015). Furthermore, students’ diverse

and frequent engagement with technology, as well as their exhibited behaviors and
characteristics during learning and assessment activities, can reflect their level of
proficiency and the overall quality of technology integration (Koh et al., 2017; Hsu,
2016). These factors collectively contribute to successful utilization of evaluation
and program planning strategies in educational contexts.

4 Discussion

4.1 Key findings and implications for evidence-based practice

The main purpose of this systematic review was to examine the evidence on evalua-
tion and program planning strategies for technology integration in education, and to
explore the factors that influence the success of technology integration in education.
Based on the analysis and synthesis of the included studies, we derived the following
key findings and implications for evidence-based practice in technology integration.
Evaluation and program planning strategies for technology integration require
continuous improvement based on evidence from multiple sources and methods
(Timotheou et al., 2023; Schalock et al., 2018). Educators need to collect and analyze
data from various sources and methods to measure and evaluate the quality, effec-
tiveness, and impact of technology integration in education (Iftikhar et al., 2022;
Chugh et al., 2023). These methods may include quantitative and qualitative data;
formative and summative evaluation; self- and peer-assessment; feedback and reflec-
tion; observation and interview; survey and test; portfolio and artifact, among others
(Chin et al., 2009). Educators also need to use the evidence from the evaluation to
inform and improve their evaluation and program planning strategies for technology
integration (Kimmons et al., 2020; Peled & Perzon, 2022). For example, feedback
from students and colleagues can be used to revise and refine the design or plan for
technology integration (Lai & Bower, 2019). Data from formative evaluations can
serve to monitor and adjust the implementation or delivery of technology (Storck &
Duarte-Figueiredo, 2020; Roozbeh et al., 2018). Findings from summative evalua-
tions can be reported and disseminated to assess outcomes or impacts of technology
integration (McLaughlin & Jordan, 2015).
Evaluation and program planning strategies for technology integration have been
previously identified as critical for the efficacy and impact of educational technology
(Iftikhar et al., 2022; Chugh et al., 2023). This systematic review substantiates these
claims but adds nuanced insight by highlighting the necessity for a tailored, context-
specific approach. This is in line with Alghamdi & Holland’s (2020) assertion that
there is no universal strategy for technology integration, emphasizing the need for
educators to adapt their strategies based on specific classroom and institutional needs
(Kimmons et al., 2020; Peled & Perzon, 2022).
Furthermore, this review recognizes a variety of models and approaches like
ADDIE, SAMR, and design-based research as instrumental frameworks for success-
ful technology integration. This offers a more complex and layered understanding
than previous research, aligning with the view that iterative design and improvement

13
Education and Information Technologies

are valuable in real-world educational settings (Zhang, 2022; Anderson & Putman,
2022). Moreover, this review identifies interconnected factors—organizational, tech-
nological, pedagogical, teacher, and student—that influence the efficacy of technol-
ogy integration. Previous studies have also noted the importance of these factors
individually (Timotheou et al., 2023; Schalock et al., 2018), but this review synthe-
sizes these into a cohesive understanding, highlighting their interplay. For example,
technological factors like accessibility (Storck & Duarte-Figueiredo, 2020) cannot be
separated from organizational factors like policy and support (Lai & Bower, 2019).
Finally, we argue for a data-driven, continuous improvement model in the evalua-
tion and planning of technology integration, urging educators to employ a diverse
set of data sources and evaluation methods. This echoes the existing literature that
emphasizes the use of both formative and summative assessments in understanding
the impact of educational technologies (Gagne’s, 1985).

4.2 Comparison with existing literature

This systematic review contributes to the existing literature on technology integration

in education by focusing on evidence-based practice, which is defined as the use of
the best available evidence from research and other sources to inform and improve
decision-making and practice in education (Biesta, 2007). Evidence-based practice
is considered as a key component of quality education and a means to enhance stu-
dent learning outcomes and teacher professional development (Hattie, 2009; Slavin,
2002). However, evidence-based practice in technology integration is not well under-
stood or widely adopted in education, as there are many challenges and barriers that
hinder its implementation and impact (Bingimlas, 2009; Ertmer & Ottenbreit-Left-
wich, 2010; Hew & Brush, 2007).
This review provides a comprehensive and systematic analysis of the evaluation
and program planning strategies for technology integration in education, which are
essential for evidence-based practice. Previous literature has often focused on spe-
cific aspects or dimensions of technology integration, such as technology use (fre-
quency, type, intensity), technology adoption (stages, levels, models), technology
impact (outcomes, effects, benefits), technology pedagogy (approaches, methods,
strategies), technology literacy (skills, competencies, standards) (Franklin & Bolick,
2007; Hsu et al., 2018; Kewalramani et al., 2020; Timotheou et al., 2020; Chen et al.,
2022; Ho et al., 2013). However, these aspects or dimensions are not isolated or inde-
pendent from each other; they are interrelated and interdependent parts of a complex
system that involves multiple stakeholders, contexts, processes, and outcomes (Liu,
2022; Hatakka et al., 2020; Zagami et al., 2018). This review adopts a socio-technical
perspective that recognizes the complexity and dynamism of technology integration
and its interaction with evaluation and program planning strategies and various fac-
tors that affect its success. The review also adopts an evidence-based perspective
that emphasizes the importance and challenges of using research and other sources
of evidence to inform and improve technology integration and its evaluation and
program planning.

13
 Education and Information Technologies

5 Conclusion and policy recommendations

Based on the findings of this systematic review, it is evident that evaluation and pro-
gram planning strategies play a crucial role in the successful integration of technol-
ogy in education. The review highlights the need for educators to select and adapt
strategies based on their specific needs, goals, learners, context, and constraints. Sev-
eral effective models and approaches, such as design-based research, the logic model,
backward design, the ADDIE model, and the SAMR model, have been identified
to support technology integration in educational settings. Furthermore, the review
emphasizes the influential role of various factors, including organizational, techno-
logical, pedagogical, teacher, and student factors, in shaping the outcomes of tech-
nology integration. Educators must consider and address these factors throughout
the integration process to ensure its success. The interaction between these factors
and the evaluation and program planning strategies underscores the need for a com-
prehensive and holistic approach. Moreover, evidence-based practice in technology
integration requires the collection and analysis of data from multiple sources and
methods. Quantitative and qualitative data, along with formative and summative
evaluation, self- and peer-assessment, and feedback, are essential in measuring and
evaluating the quality, effectiveness, and impact of technology integration. Educators
should use the insights gained from evaluation to inform and improve their planning
and implementation of technology integration initiatives.
The systematic review on evaluation and program planning strategies for technol-
ogy integration in education have important implications for policy development.
Based on the analysis, this study recommends several policy measures to promote
effective technology integration in educational settings. First and foremost, policy-
makers should prioritize the establishment of comprehensive and targeted profes-
sional development programs for educators. These programs should address the
specific needs and challenges faced by educators in integrating technology into
their teaching practices, providing them with the necessary support, training, and
resources. Furthermore, policymakers should allocate resources to ensure the avail-
ability of reliable and up-to-date technology tools and infrastructure in educational
institutions. This includes ensuring access to high-speed internet, providing adequate
funding for the procurement and maintenance of educational technology resources,
and offering technical support for educators and students. Moreover, policymakers
should encourage collaboration and partnerships among educational institutions,
technology developers, and stakeholders. Collaborative initiatives can foster knowl-
edge sharing, resource exchange, and best practices in technology integration. Poli-
cymakers should facilitate the creation of networks and platforms for collaboration,
enabling educators to learn from each other and engage in joint projects that promote
effective evaluation and program planning. Policymakers should also establish clear
guidelines and standards for technology integration in education. These guidelines
should address evaluation criteria, program planning frameworks, and quality assur-
ance measures. Another important policy recommendation is to prioritize research
and evaluation in the field of technology integration. Policymakers should allocate
funding for research initiatives that examine the effectiveness and impact of differ-
ent strategies and approaches. Finally, policymakers should prioritize equity and

13
Education and Information Technologies

accessibility in technology integration efforts. They should strive to bridge the digital
divide by ensuring that all students, regardless of their socioeconomic background or
geographical location, have equal access to technology tools and resources. Policy-
makers should implement initiatives that address inequities, promote inclusivity, and
provide support for underserved populations.

Acknowledgements We thank the editors and anonymous reviewers for carefully reading our manuscript
and providing many insightful comments and suggestions.

Funding Major Projects on Philosophy and Social Sciences of the Ministry of Education “Research on
the Construction of University Innovation System under the background of self-reliance and self-improve-
ment in science and technology” (21JZD057).

Data availability Data for this review article were obtained from publicly available sources, including
electronic databases (Web of Science, Scopus, ERIC, ScienceDirect).

Declarations

Conflict of interest The author(s) declared no potential conflicts of interest with respect to the research,
authorship, and/or publication of this article.

References
Abbitt, J. T., & Klett, M. D. (2007). Identifying influences on attitudes and self-efficacy beliefs towards
technology integration among pre-service educators. Electronic Journal for the Integration of Tech-
nology in Education, 6, 28–42.
Adhikari, E. D. (2023). Technology integration in ELT: SWOC analysis. International Journal of Creative
Research Thoughts, 11 – 1.
Adnan, F., Agustiningsih, M. D., & Ariefianto, L. (2022). The analysis of readiness and acceptance of
Learning Management System (LMS) usage in universities of East Java. 2022 9th International
Conference on Electrical Engineering Computer Science and Informatics, 198-203. https://doi.
org/10.23919/EECSI56542.2022.9946579.
Akram, H., Abdelrady, A. H., Al-Adwan, A. S., & Ramzan, M. (2022). Teachers’ perceptions of technol-
ogy integration in teaching-learning practices: A systematic review. Frontiers in Psychology, 1–10.
https://doi.org/10.3389/fpsyg.2022.920317.
Alghamdi, J., & Holland, C. (2020). A comparative analysis of policies, strategies and programmes for
information and communication technology integration in education in the Kingdom of Saudi Arabia
and the Republic of Ireland. Education and Information Technologies, 25(6), 4721–4745. https://doi.
org/10.1007/s10639-020-10169-5.
Alhababi, H. H. (2017). Technological pedagogical content knowledge (TPACK) effectiveness on English
teachers and students in Saudi Arabia. University of Northern Colorado.
Ali, Z., & Azhar, S. (2018). Are we there yet? Technology integration in English language teaching in
Balochistan, Pakistan. Journal of Education and Humanities Research, 6(2), 270–285.
Anderson, S. E., & Putman, R. S. (2022). Elementary special education teachers’ thinking while planning
and implementing technology-integrated lessons. Education and Information Technologies, 1–23.
Beller, E., Clark, J., Tsafnat, G., Adams, C., Diehl, H., Lund, H., & Glasziou, P. (2018). Making prog-
ress with the automation of systematic reviews: Principles of the international collaboration for the
automation of systematic reviews (ICASR). Systematic Reviews, 7(1), 77. https://doi.org/10.1186/
s13643-018-0740-7.
Biesta, G. (2007). Why what works won’t work: Evidence-based practice and the democratic deficit in educa-
tional research. Educational Theory, 57(1), 1–22. https://doi.org/10.1111/j.1741-5446.2006.00241.x.

13
 Education and Information Technologies

Bingimlas, K. A. (2009). Barriers to the successful integration of ICT in teaching and learning environ-
ments: A review of the literature. Eurasia Journal of Mathematics Science and Technology Educa-
tion, 5(3), 235–245. https://doi.org/10.12973/ejmste/75275.
Branch, R. M. (2009). Instructional design: The ADDIE approach (Vol. 722). Springer. https://doi.
org/10.1007/978-0-387-09506-6.
Catio, M. (2019). Analyzing the competency of principals using the Framework of the Wales National
Standard for Head Teacher in Boosting Teacher’s innovative behavior. International Journal of Man-
agerial Studies and Research, 7(2), 1–6. https://doi.org/10.20431/2349-0349.0702001.
Caukin, N., & Trail, L. (2019). SAMR: A tool for reflection for Ed tech integration. International Journal
of the Whole Child, 4(1), 47–54.
Chen, H. J., Liao, L. L., Chang, Y. C., Hung, C. C., & Chang, L. C. (2019). Factors influencing tech-
nology integration in the curriculum for Taiwanese health profession educators: A mixed-methods
study. International Journal of Environmental Research and Public Health, 16(14), 2602. https://doi.
org/10.3390/ijerph16142602.
Chen, B., Wang, Y., & Wang, L. (2022). The effects of virtual reality-assisted language learning:A meta-
analysis. Sustainability, 14(6), 3147. https://doi.org/10.3390/su14063147.
Chin, J., Dukes, R., & Gamson, W. (2009). Assessment in simulation and gaming: A review of the last 40
years. Simul Gaming, 40(4), 553–568. https://doi.org/10.1177/1046878109332955.
Chugh, R., Turnbull, D., Cowling, M. A., Vanderburg, R., & Vanderburg, M. A. (2023). Implementing
educational technology in Higher Education institutions: A review of technologies, stakeholder
perceptions, frameworks and metrics. Education and Information Technologies, 1–27. https://doi.
org/10.1007/s10639-023-11846-x.
Davies, R. S., & West, R. E. (2014). Technology integration in schools. Handbook of research on edu-
cational communications and technology, 841–853. https://doi.org/10.1007/978-1-4614-3185-5_68.
Deng, G., & Zhang, J. (2023). Technological pedagogical content ethical knowledge (TPCEK): The
development of an assessment instrument for pre-service teachers. Computers and Education, 197,
104740. https://doi.org/10.1016/j.compedu.2023.104740.
Dillon, D., Chang, Y., Rondeau, A., & Kim, J. (2019). Teacher educator technology integration initiative:
Addressing the technology preparation gap. Journal of Technology and Teacher Education, 27(4),
527–554. https://www.learntechlib.org/primary/p/208214/.
Drugova, E., Zhuravleva, I., Aiusheeva, M., & Grits, D. (2021). Toward a model of learning innovation
integration: TPACK-SAMR based analysis of the introduction of a digital learning environment in
three Russian universities. Education and Information Technologies, 26(4), 4925–4942. https://doi.
org/10.1007/s10639-021-10514-2.
Ertmer, P. A., & Ottenbreit-Leftwich, A. T. (2010). Teacher technology change: How knowledg, confi-
dence, beliefs, and culture intersect. Journal of Research on Technology in Education, 42(3), 255–
284. https://doi.org/10.1080/15391523.2010.10782551.
Ferri, F., Grifoni, P., & Guzzo, T. (2020). Online learning and emergency remote teaching: Opportunities
and challenges in emergency situations. Societies, 10(4), 86. https://doi.org/10.3390/soc10040086.
Florida Center for Instructional Technology (2011). The Technology Integration Matrix. https://fcit.usf.
edu/matrix/.
Fox, B. E., & Doherty, J. J. (2012). Design to learn, learn to design: Using backward design for informa-
tion literacy instruction. Communications in Information Literacy, 5(2), 7. https://doi.org/10.15760/
comminfolit.2012.5.2.109.
Franklin, C. A., & Bolick, C. M. (2007). Technology integration: A review of the literature. Cheryl A
Franklin The University of New Mexico Cheryl Mason Bolick University of North Carolina at Cha-
pel Hill Paper presented at the Society for Information Technology and Teacher Education Interna-
tional Conference San Antonio, TX. March 2007.
Gagne, R. (1985). The conditions of learning and theory of instruction. 4th edition, Holt, Rinehart and
Winston, Inc., Florida-ABD.
Garzón, J., & Acevedo, J. (2019). Meta-analysis of the impact of augmented reality on students’ learn-
ing gains. Educational Research Review, 27, 244–260. https://doi.org/10.1016/j.edurev.2019.04.001.
Geer, R., White, B., Zeegers, Y., Au, W., & Barnes, A. (2017). Emerging pedagogies for the use of iPads
in schools. British Journal of Educational Technology, 48(2), 490–498. https://doi.org/10.1111/
bjet.12381.
Gülbahar, Y. (2007). Technology planning: A roadmap to successful technology integration in schools.
Computers and Education, 49(4), 943–956. https://doi.org/10.1016/j.compedu.2005.12.002.

13
Education and Information Technologies

Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H.
J. (2008). GRADE: An emerging consensus on rating quality of evidence and strength of recommen-
dations. Bmj, 336(7650), 924–926. https://doi.org/10.1136/bmj.39489.470347.AD.
Hains, B. J., & Hains, K. D. (2020). Community reaction towards social innovation: A discussion of Rog-
ers’ Diffusion of innovations theory in consideration of community emotional response. J Int Agric
Ext Educ, 27, 34–46.
Hall, G. E., & Hord, S. M. (2015). Implementing change: Patterns, principles, and potholes (4th ed.).
Pearson.
Hamilton, E. R., Rosenberg, J. M., & Akcaoglu, M. (2016). The substitution augmentation modification
redefinition (SAMR) model: A critical review and suggestions for its use. TechTrends, 60, 433–441.
https://doi.org/10.1007/s11528-016-0091-y.
Hamutoglu, N. B. (2021). Testing the effects of technological barriers on high school teachers’ role in
technology integration. Asian Journal of Distance Education, 16(1), 74–89. Retrieved from https://
asianjde.com/ojs/index.php/AsianJDE/article/view/544.
Harmandaoğlu, B. E., Balçıkanlı, C., & Cephe, P. T. (2018). Introducing an innovative technology integra-
tion model: Echoes from EFL pre-service teachers. Education and Information Technologies, 23,
2179–2200. https://doi.org/10.1007/s10639-018-9711-9.
Harrison, J. B., & West, R. E. (2014). Sense of community in a blended technology integration course: A
design-based research study. International Review of Research in Open and Distributed Learning,
15(6), 289–312. https://doi.org/10.19173/irrodl.v15i6.1907.
Hatakka, M., Thapa, D., & Sæbø, Ø. (2020). Understanding the role of ICT and study circles in enabling
economic opportunities: Lessons learned from an educational project in Kenya. Information Systems
Journal, 30(4), 664–698. https://doi.org/10.1111/isj.12277.
Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Rout-
ledge. https://doi.org/10.4324/9780203887332.
Hew, K. F., & Brush, T. (2007). Integrating technology into K-12 teaching and learning: Current knowl-
edge gaps and recommendations for future research. Educational Technology Research and Develop-
ment, 55(3), 223–252. https://doi.org/10.1007/s11423-006-9022-5.
Higgins, J. P. T., Altman, D. G., Gøtzsche, P. C., Jüni, P., Moher, D., Oxman, A. D., Savović, J., Schulz, K.
F., Weeks, L., & Sterne, J. A. C. (2011). The Cochrane collaboration’s tool for assessing risk of bias
in randomized trials. Bmj, 343(7829), d5928–d5928. https://doi.org/10.1136/bmj.d5928.
Ho, L. H., Hung, C. L., & Chen, H. C. (2013). Using theoretical models to examine the acceptance behav-
ior of mobile phone messaging to enhance parent-teacher interactions. Computers and Education, 61,
105–114. https://doi.org/10.1016/j.compedu.2012.09.009.
Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M. P.,
Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M. C., Vedel, I., & Schuster, T. (2018). Improving
the content validity of the mixed methods appraisal tool: A modified e-Delphi study. Journal of Clini-
cal Epidemiology, 100, 66–76. https://doi.org/10.1016/j.jclinepi.2018.01.012.
Hsu, L. (2016). Examining EFL teachers’ technological pedagogical content knowledge and the adoption
of mobile-assisted language learning: A partial least square approach. Computer Assisted Language
Learning, 29(8), 1287–1297. https://doi.org/10.1080/09588221.2016.1278024.
Hsu, S., Chang, J., & Hung, Y. (2018). Trends and research issues of mobile learning studies in nursing
education: A review of academic publications from 1971 to 2016. Computers and Education, 116,
28–48. https://doi.org/10.1016/j.compedu.2017.09.001.
Ifinedo, E., Rikala, J., & Hämäläinen, T. (2020). Factors affecting Nigerian teacher educators’ technology
integration: Considering characteristics, knowledge constructs, ICT practices and beliefs. Computers
& Education, 146, 103760. https://doi.org/10.1016/j.compedu.2019.103760.
Iftikhar, S., Fu, Y., Naureen, S., Cao, Y., & Zhou, C. (2022). Cascading of teachers training at higher
education in Pakistan: An evaluation of a faculty professional development program. Evaluation and
Program Planning, 94, 102130. https://doi.org/10.1016/j.evalprogplan.2022.102130.
Japee, G., & Oza, P. (2021). Curriculum and evaluation in outcome-based education. Psychology and
Education Journal, 58(2), 5620–5625.
Jones, P., & Davis, R. (2008). Instructional design methods integrating instructional technology. In
handbook of research on instructional systems and technology. IGI Global, 15–27. https://doi.
org/10.4018/978-1-59904-865-9.
Jwaifell, M., & Gasaymeh, A. M. (2013). Using the diffusion of Innovation Theory to explain the degree
of English teachers’ adoption of interactive whiteboards in the modern systems school in Jordan: A
case study. Contemporary Educational Technology, 4(2), 138–149.

13
 Education and Information Technologies

Kandel, G. K. (2022). Integration of information and communication technology in education: The oppor-
tunities and challenges. Marsyandgi Journal, 3(1), 82–90. https://doi.org/10.3126/mj.v3i1.47954s.
Kaufman, R., Guerra-Lopez, I., Watkins, R., & Leigh, D. (2003). The assessment book: Applied strategic
thinking and performance improvement through self-assessment. HRD.
Kewalramani, S., Désiron, J., Cattaneo, A., Villagrá-Sobrino, S., Giannoutsou, N., Cachia, R., Martínez-
Monés, A., & Ioannou, A. (2020). Impacts of digital technologies on education and factors influ-
encing schools’ digital capacity and transformation: A literature review. Education and Information
Technologies. https://doi.org/10.1007/s10639-022-11431-8.
Kim, C., Kim, M. K., Lee, C., Spector, J. M., & DeMeester, K. (2013). Teacher beliefs and technology
integration. Teaching and Teacher Education, 29, 76–85. https://doi.org/10.1016/j.tate.2012.08.005.
Kimmons, R., Graham, C. R., & West, R. E. (2020). The PICRAT model for technology integration in
teacher preparation. Contemporary Issues in Technology and Teacher Education, 20(1), 176–198.
https://www.learntechlib.org/primary/p/210228/.
King, F. (2014). Evaluating the impact of teacher professional development: An evidence-based frame-
work. Professional Development in Education, 40(1), 89–111. https://doi.org/10.1080/19415257.20
13.823099.
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels (3rd ed.).
Berrett-Koehler.
Koh, J. H. L., Chai, C. S., & Lim, W. Y. (2017). Teacher Professional Development for TPACK-21CL:
Effects on teacher ICT integration and student outcomes. Journal of Educational Computing
Research, 55(2), 172–196. https://doi.org/10.1177/0735633116656848.
Lai, J. W., & Bower, M. (2019). How is the use of technology in education evaluated? A systematic review.
Computers & Education, 133, 27–42. https://doi.org/10.1016/j.compedu.2019.01.010.
Lamberton, B. A., & Raschke, R. L. (2023). Using Backward Design to Incorporate Technology into
a Non-AIS course. Journal of Emerging Technologies in Accounting, 20(1), 259–267. https://doi.
org/10.2308/JETA-2021-013.
Larson, M. B. (2014). Streamlined ID: A practical guide to instructional design. Routledge.
Lawrence, J. E., & Tar, U. A. (2018). Factors that influence teachers’ adoption and integration of ICT in
teaching/learning process. Educational Media International, 55(1), 79–105. https://doi.org/10.1080
/09523987.2018.1439712.
Ledford, D. M. (2016). Development of a professional learning framework to improve teacher practice in
technology integration. https://scholarworks.boisestate.edu/td/1082.
Leyva, J. (2015). 21st century teaching and learning with technology integration at an innovative high
school: A case study (Doctoral dissertation, University of Southern California).
Liu, T. C. (2022). A case study of the adaptive learning platform in a Taiwanese Elementary School: Pre-
cision Education from teachers’ perspectives. Education and Information Technologies, 27, 6295–
6316. https://doi.org/10.1007/s10639-021-10851-2.
Luo, Q., Miao, C., Sun, L., Meng, X., & Duan, M. (2019). Efficiency evaluation of green technology
innovation of China’s strategic emerging industries: An empirical analysis based on Malmquist-data
envelopment analysis index. Journal of Cleaner Production, 238, 117782. https://doi.org/10.1016/j.
jclepro.2019.117782.
Mahmoud, K. (2022). Realising the vision of technology integration: a case study of K-12 private schools
in the United Arab Emirates (Doctoral dissertation, University of Southern Queensland).
Majid, I. (2020). ICT in assessment: A backbone for teaching and learning process. UIJRT United
International. Journal for Research and Technology, 1(3). https://www.researchgate.net/
publication/348489222.
Materia, F. T., Miller, E. A., Runion, M. C., Chesnut, R. P., Irvin, J. B., Richardson, C. B., & Perkins, D. F.
(2016). Let’s get technical: Enhancing program evaluation through the use and integration of internet
and mobile technologies. Evaluation and Program Planning, 56, 31–42. https://doi.org/10.1016/j.
evalprogplan.2016.03.004.
McKeown, S., & Mir, Z. M. (2021). Considerations for conducting systematic reviews: Evaluating the
performance of different methods for de-duplicating references. Systematic Reviews, 10, 1–8. https://
doi.org/10.1186/s13643-021-01583-y.
McLaughlin, J. A., & Jordan, G. B. (2015). Using logic models. Handbook of practical program evalua-
tion, 62–87. https://doi.org/10.1002/9781119171386.ch3.
Mikasa, A. W., Cicero, T. F., & Adamson, K. A. (2013). Outcome-based evaluation tool to evaluate student
performance in high-fidelity simulation. Clinical Simulation in Nursing, 9(9), e361–e367. https://doi.
org/10.1016/j.ecns.2012.06.001.

13
Education and Information Technologies

Miller, R. L. (2015). Rogers’ innovation diffusion theory (1962, 1995). In Information seeking behavior and
technology adoption: Theories and trends (pp. 261–274). IGI Global. https://doi.org/10.4018/978-1-
4666-8156-9.ch016.
Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A frame-
work for teacher knowledge. Teachers College Record, 108(6), 1847–1856. https://doi.
org/10.1111/j.1467-9620.2006.00684.x.
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group. (2009). Preferred report-
ing items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7),
e1000097. https://doi.org/10.1371/journal.pmed1000097.
Molenda, M. (2015). In search of the elusive ADDIE model: Performance improvement. Performance
Improvement, 54(2), 40–42. https://doi.org/10.1002/pfi.21461.
Nelson, M. J., Voithofer, R., & Cheng, S. L. (2019). Mediating factors that influence the technology integra-
tion practices of teacher educators. Computers & Education, 128, 330–344. https://doi.org/10.1016/j.
compedu.2018.09.023.
Nichols, H. A., & Greer, K. (2016). Designing for engagement: Using the ADDIE model to integrate high-
impact practices into an online information literacy course. Communications in information literacy,
10(2), 6. DOI10.15760/comminfolit.2016.10.2.27.
Ozan, C., & Kıncal, R. Y. (2018). The effects of formative assessment on academic achievement, attitudes
toward the lesson, and self-regulation skills. Kuram ve Uygulamada Egitim Bilimleri, 18(1), 85–118.
https://doi.org/10.12738/estp.2018.1.0216.
Pagunsan-Sumbise, G., & Montecalvo, F. C. (2021). Emerging trends and challenges in information and
communications technology integration among public secondary schools in Samar, Philippines.
Asian Journal of Resource Management and Governance, 1(1), 10–23. Retrieved from https://ojs.
nwssu.edu.ph/index.php/ajrmg/article/view/8.
Patton, M. Q. (2012). Essentials of utilization-focused evaluation. Sage. https://doi.
org/10.1177/1356389012449523.
Peffer, M. E., Beckler, M. L., Schunn, C., Renken, M., & Revak, A. (2015). Science classroom inquiry
(SCI) simulations: A novel method to scaffold science learning. PloS One, 10(3), e0120638. https://
doi.org/10.1371/journal.pone.0120638.
Peled, Y., & Perzon, S. (2022). Systemic model for technology integration in teaching. Education and
Information Technologies, 27(2), 2661–2675. https://doi.org/10.1007/s10639-021-10694-x.
Pelila, J. R., Bag-ongan, Q. F., Talania, J. L., & Wakat, G. S. (2022). Factors and barriers influencing tech-
nology integration in the classroom. Journal of Language Education and Educational Technology,
7(1), 1–15. https://doi.org/10.33772/jleet.v7i1.18704.
Petrović, J., Pale, P., & Jeren, B. (2017). Online formative assessments in a digital signal processing
course: Effects of feedback type and content difficulty on students learning achievements. Education
and Information Technologies, 22, 3047–3061. https://doi.org/10.1007/s10639-016-9571-0.
Piskurich, G. M. (2015). Rapid instructional design: Learning ID fast and right. ed.). Wiley. (Third;3.
Popay, J., Roberts, H., Sowden, A., Petticrew, M., Arai, L., Rodgers, M., Britten, N., Roen, K., & Duffy,
S. (2006). Guidance on the conduct of narrative synthesis in systematic reviews. ESRC Methods
Programme. https://www.lancaster.ac.uk/shm/research/nssr/research/dissemination/publications/
NS_Synthesis_Guidance_v1.pdf.
Puentedura, R. R. (2013). SAMR: Moving from enhancement to transformation. http://hippasus.com/rrp-
weblog/archives/000095.html.
Puentedura, R. (2014). Learning, technology, and the SAMR model: Goals, processes, and practice. Ruben
R. Puentedura’s Weblog.
Raman, R., Nedungadi, P., Amrita, R. P., Banerji, S., Mohan, R., & Ramesh, M. V. (2020). Investigating
the factors affecting the adoption of experiential learning programs: MBA students experience with
live-in-labs. In 2020 IEEE Bangalore Humanitarian Technology Conference (B-HTC), IEEE, 1–6.
https://doi.org/10.1109/B-HTC50970.2020.9297906.
Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free.
Roozbeh, A., Soares, J., Maguire, G. Q., Wuhib, F., Padala, C., Mahloo, M., Turull, D., Yadhav, V., &
Kostić, D. (2018). Software-defined hardware infrastructures: A survey on enabling technologies and
open research directions. IEEE Communications Surveys & Tutorials, 20(3), 2454–2485. https://doi.
org/10.1109/COMST.2018.2834731.
Schalock, R. L., Verdugo, M. A., & van Loon, J. (2018). Understanding organization transformation
in evaluation and program planning. Evaluation and Program Planning, 67, 53–60. https://doi.
org/10.1016/j.evalprogplan.2017.11.003.

13
 Education and Information Technologies

Sheffield, R., Dobozy, E., Gibson, D., Mullaney, J., & Campbell, C. (2015). Teacher education students
using TPACK in science: A case study. Educational Media International, 52(3), 227–238. https://doi.
org/10.1080/09523987.2015.1075104.
Singh, R. (2019). Students’ perspectives on technology integration in ELT. Journal of NELTA, 24(1–2).
https://doi.org/10.3126/nelta.v24i1-2.27682. 95 106.
Singh, A. K., & Meena, M. K. (2023). Online teaching in Indian higher education institutions during the
pandemic time. Education and Information Technologies, 1–51.
Slavin, R. E. (2002). Evidence-based education policies: Transforming educational practice and research.
Educational Researcher, 31(7), 15–21. https://doi.org/10.3102/0013189X031007015.
Stapa, M. A., & Mohammad, N. A. Z. E. R. I. (2019). The use of Addie model for designing blended learn-
ing application at vocational colleges in Malaysia. Asia-Pacific Journal of Information Technology
and Multimedia, 8(1), 49–62.
Storck, C. R., & Duarte-Figueiredo, F. (2020). A survey of 5G technology evolution, standards, and
infrastructure associated with vehicle-to-everything communications by internet of vehicles. IEEE
Access, 8, 117593–117614. https://doi.org/10.1109/ACCESS.2020.3004779.
Stufflebeam, D. L., & Coryn, C. L. (2014). Evaluation theory, models, and applications (Vol. 50). Wiley.
Summak, M. S., Samancioğlu, M., & Bağlibel, M. (2010). Technology integration and assesment in educa-
tional settings. Procedia-Social and Behavioral Sciences, 2(2), 1725–1729. https://doi.org/10.1016/j.
sbspro.2010.03.973.
Sun, Y., Strobel, J., & Newby, T. J. (2017). The impact of student teaching experience on pre-service
teachers’ readiness for technology integration: A mixed methods study with growth curve model-
ing. Educational Technology Research and Development, 65, 597–629. https://doi.org/10.1007/
s11423-016-9486-x.
Timotheou, S., Miliou, O., Dimitriadis, Y., Villagrá-Sobrino, S., Giannoutsou, N., Cachia, R., Mar-
tínez-Monés, A., & Ioannou, A. (2020). Technology-integrated pedagogical practices: A look into
evidence-based teaching and coherent learning for young children. European Early Childhood Edu-
cation Research Journal, 28(2), 163–166. https://doi.org/10.1080/1350293X.2020.1735739.
Timotheou, S., Miliou, O., Dimitriadis, Y., Sobrino, S. V., Giannoutsou, N., Cachia, R., Monés, A. M., &
Ioannou, A. (2023). Impacts of digital technologies on education and factors influencing schools’
digital capacity and transformation: A literature review. Education and Information Technologies,
28(6), 6695–6726. https://doi.org/10.1007/s10639-022-11431-8.
Tondeur, J., Van Braak, J., Ertmer, P. A., & Ottenbreit-Leftwich, A. (2017). Understanding the relation-
ship between teachers’ pedagogical beliefs and technology use in education: A systematic review of
qualitative evidence. Educational Technology Research and Development, 65, 555–575.
Tosuntaş, Ş. B., Çubukçu, Z., & Tuğba, İ. N. C. İ. (2019). A holistic view to barriers to technology inte-
gration in education. Turkish Online Journal of Qualitative Inquiry, 10(4), 439–461. https://doi.
org/10.17569/tojqi.613969.
Tuna, H., & Başdal, M. (2021). Curriculum evaluation of tourism undergraduate programs in Turkey:
A CIPP model-based framework. Journal of Hospitality Leisure Sport & Tourism Education, 29,
100324. https://doi.org/10.1016/j. jhlste.2021.100324.
Van Deursen, A. J., & Van Dijk, J. A. (2019). The first-level digital divide shifts from inequalities in
physical access to inequalities in material access. New Media & Society, 21(2), 354–375. https://doi.
org/10.1177/1461444818797082.
Vatanartiran, S., & Karadeniz, S. (2015). A needs analysis for Technology Integration Plan: Challenges
and needs of teachers. Contemporary Educational Technology, 6(3). https://doi.org/10.30935/
cedtech/6150.
Wahyuni, S., Mujiyanto, J., Rukmini, D., & Fitriati, S. W. (2020, June). Teachers’ technology integration
into English instructions: SAMR model. In International Conference on Science and Education and
Technology (ISET 2019) (pp. 546–550). Atlantis Press. DOI10.2991/assehr.k.200620.109.
Wang, Y. H. (2020). Design-based research on integrating learning technology tools into higher education
classes to achieve active learning. Computers & Education, 156, 103935. https://doi.org/10.1016/j.
compedu.2020.103935.
Wang, S. K., Hsu, H. Y., Reeves, T. C., & Coster, D. C. (2014). Professional development to enhance
teachers’ practices in using information and communication technologies (ICTs) as cognitive tools:
Lessons learned from a design-based research study. Computers & Education, 79, 101–115. https://
doi.org/10.1016/j.compedu.2014.07.006.

13
Education and Information Technologies

Webster, C. A., Zarrett, N., Cook, B. S., Egan, C., Nesbitt, D., & Weaver, R. G. (2017). Movement integra-
tion in elementary classrooms: Teacher perceptions and implications for program planning. Evalua-
tion and Program Planning, 61, 134–143. https://doi.org/10.1016/j.evalprogplan.2016.12.011.
Wilson, M. L. (2023). The impact of technology integration courses on preservice teacher attitudes and
beliefs: A meta-analysis of teacher education research from 2007–2017. Journal of Research on
Technology in Education, 55(2), 252–280. https://doi.org/10.1080/15391523.2021.1950085.
Wu, D., Zhou, C., Liang, X., Li, Y., & Chen, M. (2022). Integrating technology into teaching: Factors
influencing rural teachers’ innovative behavior. Education and Information Technologies, 27(4),
5325–5348. https://doi.org/10.1007/s10639-021-10815-6.
Wyant, J. D., Jones, E. M., & Bulger, S. M. (2015). A mixed methods analysis of a single-course strategy to
integrate technology into PETE. Journal of Teaching in Physical Education, 34(1), 131–151. https://
doi.org/10.1123/jtpe.2013-0114.
Xu, M., Yang, X., & Stefaniak, J. (2022). A design-based Research Study Exploring pre-service teach-
ers’ Instructional Design decision-making for Technology Integration. TechTrends, 66(6), 968–979.
https://doi.org/10.1007/s11528-022-00785-y.
Yeh, H. C., & Tseng, S. S. (2019). Using the ADDIE model to nurture the development of teachers’ CALL
professional knowledge. Journal of Educational Technology & Society, 22(3), 88–100. https://www.
jstor.org/stable/26896712.
Yıldırım, İ., & Şen, S. (2021). The effects of gamification on students’ academic achievement: A meta-
analysis study. Interactive Learning Environments, 29(8), 1301–1318. https://doi.org/10.1080/1049
4820.2019.1636089.
Zagami, J., Bocconi, S., Starkey, L., Wilson, J. D., Gibson, D., Downie, J., & Elliott, S. (2018). Creating
future ready information technology policy for national education systems. Technology Knowledge
and Learning, 23, 495–506. https://doi.org/10.1007/s10758-018-9387-7.
Zainal, A. Z., & Zainuddin, S. Z. (2020). Technology Adoption in Malaysian schools: An analysis of
National ICT in Education Policy initiatives. Digital Education Review, 37, 172–194.
Zhang, W. (2022). The role of technology-based education and teacher professional development in Eng-
lish as a Foreign Language classes. Frontiers in Psychology, 13, 910315. https://doi.org/10.3389/
fpsyg.2022.910315.

Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps
and institutional affiliations.

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under
a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted
manuscript version of this article is solely governed by the terms of such publishing agreement and appli-
cable law.

Authors and Affiliations

Zhimin Luo1 · Babar Nawaz Abbasi1 · Chong Yang2 · Jiayin Li3 · Ali Sohail4

Babar Nawaz Abbasi

babarnawaz205@yahoo.com

1
Institute of China Innovation and Entrepreneurship Education, Key Research Center of
Philosophy and Social Sciences of Zhejiang, Hangzhou Normal University, Hangzhou,
China
2
School of Education, Zhengzhou University, Henan, China
3
Institute of Higher Education, East China Normal University, Shanghai, China
4
School of Public Policy and Administration, Xian Jiao tong University, Shaanxi, China

13