Do PISA data justify PISA-based education policy?

Luisa Araujo, Andrea Saltelli, Sylke V. Schnepf

(Article Analysis)

Rida Evilina Situmorang - 24012388

Introduction:

This writing is intended to analyse the article under the title ‘Do PISA data justify PISA-
based education policy?’ written by Araujo, Luisa, Andrea Saltelli, and Sylke V Schnepf and
published in 2017, by the Journal of International Journal of Comparative Education and
Development. The article talked about how some scholars questioned the result of PISA and
how PISA maintained their survey with their methodology, is it reliable and valid data despite
the different circumstances of each member that took the survey? And whether the data can
be used as a resource to make policies related to education since the PISA survey is to
measure the readiness of students to face real-life challenges despite the technical or cultural
differences among all countries. The writers use survey methodologies and identify
conflicting arguments and elaborate on the problem clearly and give a solution to the
conflicting arguments. The article is understandable when discussing the topic and uses
formal and technical language for academic purposes.

Summary:

Programme for International Student Assessment, known as PISA is organized by OECD as

one of the international assessments to measure the student’s ability, usually at the age of 15,
in preparing them to face the challenges that they will face outside the school world. This test
is used as a comparative study of student’s performance in the area of mathematics, reading,
and science literacies. The PISA survey is carried out on a sample of
students aged 15 years in each member of OECD countries and other participating countries.
The survey was taken in a three-year cycle to measure students’ performance and some other
factors related to how the students are ready to face the outside world from the school context
(Firman, harry, 2016). The writers concluded that the controversy over PISA data is based on
two dimensions; methodological and ideological.
The main arguments presented are; that the methodological flaws undermine PISA’s
reliability for policy decisions, the writers confronted that at some point the survey neglected
cultural and development differences among all the countries that took the survey and then
made the result as based for the policy decision, which they presume that it did not represent
as the sampling and the data transparency is also questioned. Second, PISA promotes a
narrow, neoliberal vision of education, where the goal of the test is to get a higher position
which will lead to more testing time for students. Since the orientation of the teaching is ‘the
test’ of course it will lead the education in the wrong direction. The third argument the writers
mention is that the overreliance on the PISA result leads to a dangerous education
reformation, because in some countries, after getting the result of the survey they quickly
make some changes to the educational system which may bring effects to the students and
teachers. All these arguments highlighted the critiques from the expert’s concern about PISA
methodology, ideological bias and policy implications.

The writers use qualitative methodology in explaining their arguments by interpreting the
conflicts about the PISA and supported with examples.

The Analysis:

The writers give a critical perspective on the PISA methodology and validity including the
result of the survey, the writers confronted that at some point the survey neglected cultural
and development differences among all the countries that took the survey and then made the
result as the based for policy decision, which they presume that it did not represent as the
sampling and the data transparency is also questioned. The lack of clarity in what specific
dimensions are being measured, and the challenge of achieving cultural neutrality in a global
survey also supported by Gillis, S., Polesel, J. & Wu, M (2016) where stated that some
countries overuse and lead to misleading about the PISA data, ignoring technical issues and
the contextual factors related to some cultural barrier and socio-economic that can influence
the results.

The writers provide a detailed critique of the methodological limitations of PISA, pointing
out issues with cross-sectional data, sample representativeness, and the use of statistical
models to summarize complex phenomena like student abilities. The critique focuses on the
cultural biases in test design and the lack of transparency in PISA’s methodology which lead
to the important questions about the validity of the conclusions drawn from the data. Another
strength that the writer gives is that not only discusses the lack of the methodology of PISA
but the writers propose an actionable solution for improvement of the PISA survey which is
adopting a Post-Normal Science (PNS) approach, that involves broader stakeholder
participants, then a transparency in methodological choices and also more comprehensive of
PISA data, which will make the survey more balanced. The limitation that may arise from
this article is when the writers mention its methodological limitations that could be improved
or compensated by giving examples such as; longitudinal design, but at the same time, the
writers did not give a clear explanation and detail about how this might be implemented in
the global assessment.

When we come to the arguments presented, the writers can elaborate on them clearly and
support them with some data and other supporting ideas. For example, when the writers argue
that PISA promotes a narrow, neoliberal vision of education, PISA brings effect to the global
governance and power dynamics of the members. The writers give an additional explanation
that PISA has brought significant changes in governance education worldwide, where PISA
focused on the data, measurement tools, and global ranking of the members to improve the
qualities of the students’ outcomes in the specific areas where PISA test on, but at some point,
the survey neglected on non-cognitive skills that are needed in real life, for example,
emotional resilience, creativity and also collaboration. As Volante also mentioned (2015)
PISA’s emphasis on standardized and measurable academic outcomes only on three subjects
has marginalized broader educational objectives, where there are also other subjects, for
example; moral, civic, artistic, and physical development which result in reduced the vision
of the purpose of education itself.

On the other side, the writers only give slight evidence about how PISA’s influence or effect
to detrimental education reform in various countries. It will be excellent if the writers give
concrete case studies, ideologically the article can seem more abstract and less convicting to
those who are not familiar with these issues. We can take the example stated by Volante
(2015) that educational comparison measurement has caused a shift to short-term fixed in
education policy, countries race to improve their rank, for example; Japan. Following a PISA
performance between 2000 and 2006, Japan did some reformation in promoting reading
skills, introducing a national achievement test and then revising the curriculum. The policy is
not intended to fix the curriculum but rather to achieve quick improvement and raise in the
rank.
This article offers valuable contributions to the ongoing debate about the use of standardized
testing in education policy. Its significance lies in its critical perspective on the
methodological and ideological foundations of PISA, and it has important implications for
education systems worldwide. The article contributes by providing a clear criticism and clear
description of the debate of PISA data but at the same time, the writers also provide with
solution of the PNS framework and hope that the discussion of the article would change the
politicians how to react toward the PISA result related to education policy. When delivering
the critique, the writers use a critical tone but at the same time keep the critical issue
balanced, furthermore, the writers also coherently keep the arguments in a way so the readers
can keep on track of the article. At the end of the article, the writers hope that the discussion
about PISA data can a give clear perspective of the debates of the issue.

Conclusion:

The article gives a strong critique of the PISA result and how it uses the methodology, as well
as strong arguments about how PISA data give changes in educational policy and narrowed
the educational system. It proposed that PISA should be more transparent in the processes
and urges caution in using PISA data as a basis for educational policy. Gillis, S., Polesel, J. &
Wu, M (2016) support the idea that PISA rankings are frequently misinterpreted as an
absolute indicator of education quality where the government needs to make some changes
while ignoring other technical issues and contextual factors that can influence the result of the
survey. It suggested that for a broader and more nuanced view of education to consider the
diverse and complex purposes of education. Not to mention also as the media’s role in
shaping public opinion and perception in creating narratives of educational success or failure
based on international rankings. (Pron. X. 2017)

The article helped me to understand more about how PISA results influence some countries
and educational policy, and now I can link it with my course about the critical issues in
education. How to think critically about what happened in the education field and see in a
broad and wide way of thinking. Since I gained more understanding about the topic, I intend
to conduct another research about how the PISA score influences the national curriculum in
my country, in the future.
References;

Firman, Harry, (2016) What can PISA 2012 data tell us; Performance and Challenges in Five
Participating Southeast Asian Countries. pp.63-64. Available from:
<https://doi.org/10.1163/9789463004688_006> [accessed 19 October 2024].

Gillis, S., Polesel, J. & Wu, M., (2016) PISA data: Raising concerns with its use in policy
settings. Australian Educational Researcher [online], 43(2), pp.131–146. Available from:
<https://doi.org/10.1007/s13384-015-0183-2> [accessed 19 October 2024].

Pons, X., (2017) Fifteen years of research on PISA effects on education governance: A
critical review. European Journal of Education [online], 52(2), pp.131-144. Available from
One Search [accessed 19 October 2024].

Volante, L., (2015) The impact of PISA on education governance: Some insights from highly
reactive policy contexts. International Studies in Educational Administration [online], 43(2),
pp.103-117. Available from: <https://search.ebscohost.com/login.aspx?
direct=true&db=a9h&AN=114122444&site=ehost-live> [accessed 19 October 2024].