PISA – THE OECD PROGRAMME FOR INTERNATIONAL STUDENT ASSESSMENT

OECD countries PISA partner countries

Australia Hungary Norway Albania Hong Kong-China Peru
Austria Iceland Poland Argentina Indonesia Qatar
Belgium Ireland Portugal Azerbaijan Israel Republic of Montenegro
Canada Italy Slovak Republic Brazil Jordan Republic of Serbia
Czech Republic Japan Spain Bulgaria Kyrgyz Republic Romania
Denmark Korea Sweden Chile Latvia Russian Federation
Finland Luxembourg Switzerland China (Shanghai) Liechtenstein Singapore
France Mexico Turkey Chinese Taipei Lithuania Slovenia
Germany Netherlands United Kingdom Colombia Macao-China Thailand
Greece New Zealand United States Croatia Macedonia Tunisia
Estonia Panama Uruguay
“ Quality education is the most valuable asset
for present and future generations. Achieving
it requires a strong commitment from everyone,
including governments, teachers, parents and
students themselves. The OECD is contributing
to this goal through PISA, which monitors
results in education within an agreed framework, allowing
for valid international comparisons. By showing that some
countries succeed in providing both high quality and equitable
learning outcomes, PISA sets ambitious goals for others. ”
Angel Gurría
OECD Secretary-General
4

Building on PISA’S Knowledge Base

eveloped jointly by OECD member countries through the PISA surveys each focused on a particular subject area: reading

D OECD’s Directorate for Education, the Programme for

International Student Assessment (PISA) aims to measure
how far students approaching the end of compulsory education
(in 2000), mathematics (in 2003) and science (in 2006). The
programme will now conduct a second set of surveys in 2009,
2012 and 2015.
have acquired some of the knowledge and skills essential for
In addition to monitoring student performance in the three
full participation in the knowledge society. PISA is an important
main subject areas, PISA will seek to deepen policy insights by:
part of the work of the Directorate for Education, which collects
data and provides comparative indicators of education systems > Developing better ways of tracking student progress, including
in OECD member and partner countries. exploring the possibility of comparing student progress
between primary education and the age of 15 years
In 2006, PISA completed its first set of surveys of the knowledge
and skills of 15-year-olds in OECD member countries and partner > Allowing closer comparisons between performance and
countries, covering 90% of the world economy. These PISA surveys instruction, by extending the option of examining student
examined the performance of students in key subject areas and performance within a single grade and gathering more data
also looked at a wider range of educational outcomes that about classroom experiences
include students’ motivation to learn, their beliefs about
> Making use of computer-based assessments, not only to
themselves and their learning strategies. They examined how
measure Information and Communication Technology (ICT)
performance varies between the genders and between socio-
literacy skills but also to allow for a wider range of dynamic
economic groups. They also provided insights into some of the
and interactive tasks and to explore more efficient ways of
factors that influence the development of knowledge and skills
carrying out the main tests of student knowledge and skills
at home and at school, how these factors interact and what the
in reading, mathematics and science
implications are for policy development. Most importantly, PISA
has shed light on those countries that succeed in achieving high These innovations will initially be explored by countries as a
performance standards while at the same time providing an supplementary, optional component of PISA, but where appropriate
equitable distribution of learning opportunities. The first three may be integrated into the survey’s core.
 Building on PISA’S Knowledge Base 5

Assessment year 2000 2003 2006 2009 2012 2015

Subjects Reading Reading Reading Reading Reading Reading

assessed Mathematics Mathematics Mathematics Mathematics Mathematics Mathematics
Science Science Science Science Science Science
Problem solving

Students’ Approaches to Approaches to Approaches to To be defined To be defined To be defined

self- learning, learning, learning,
assessment engagement attitudes to attitudes to
with reading mathematics science + One developmental assessment per survey
6

The Story So Far: PISA 2000-2006

ISA seeks to measure how well young adults, at age 15

P and therefore approaching the end of compulsory schooling,

are prepared to meet the challenges of today's knowledge
societies – what PISA refers to as “literacy”. The assessment is
forward looking, focusing on young people's ability to use their
knowledge and skills to meet real-life challenges, rather than
merely on the extent to which they have mastered a specific
school curriculum. This orientation reflects a change in the
goals and objectives of curricula themselves, which increasingly
address what students can do with what they learn at school
and not merely whether they can reproduce what they have
learned.

Well over one million students have so far been assessed. In

addition to paper-and-pencil tests measuring reading, mathematical
and scientific literacy, students filled in questionnaires about
themselves, while their principals completed questionnaires
about their schools. This has produced an unprecedented
comparative knowledge base of school systems and their outcomes,
and allows these outcomes to be monitored over time.

DID YOU KNOW … that more than 60 OECD member

countries and partner countries have participated in PISA so far
and that they cover roughly 90% of the world economy?
 The Story So Far: PISA 2000-2006 7

PISA’s key features

> Its policy orientation, with design and reporting methods
determined by the need of governments to draw policy lessons

> Its innovative “literacy” concept, which is concerned with the

capacity of students to apply knowledge and skills in key
subject areas and to analyse, reason and communicate
effectively as they pose, solve and interpret problems in a
variety of situations

> Its relevance to lifelong learning, which does not limit PISA
to assessing students’ curricular and cross-curricular competencies
but also asks them to report on their own motivation to learn,
their beliefs about themselves and their learning strategies
> Its regularity, which will enable countries to monitor their
progress in meeting key learning objectives

> Its contextualisation within the system of OECD education

indicators, which examine the quality of learning outcomes,
the policy levers and contextual factors that shape these
outcomes, and the broader private and social returns to
investments in education

> Its breadth of geographical coverage and collaborative nature,

with more than 60 countries (covering roughly nine-tenths
of the world economy) having participated in PISA assessments
to date, including all 30 OECD countries
8

Development of PISA
In its first three surveys, PISA has been characterised by

> The establishment of comprehensive > Exploration of student abilities and characteristics
frameworks for assessing reading, beyond school subjects
mathematics and science
> From its outset, PISA has assessed student performance beyond
> In each three-yearly PISA survey, one subject was chosen as the confines of reading, mathematics and science, as cross-
a focus while two other subject areas have been assessed curricular competencies like ICT skills, communication and
more briefly. This allowed, for each subject area, a detailed problem-solving will be important to students’ futures. The
profile of what a country’s students can do every nine years, surveys asked students about their characteristics as learners,
and an update of their performance every three years. The including their learning habits, their motivation levels, their
assessment framework for each subject was finalised in the attitudes to learning in general and to reading, science and
year that it featured as a focus. The development of these mathematics in particular, and their habits and competence
frameworks has been central to PISA, which takes an innovative in using ICT. This has allowed analysis of student approaches
approach to assessing knowledge, skills and attitudes, by to learning and of self-assessed cross-curricular competencies
identifying the key concepts and processes that underpin such as computer proficiency. In addition, the direct assessment
each subject, rather than merely examining curriculum content. in 2003 of student performance in solving problems, outside
the context of a single curriculum area, was a first step in
describing such competencies through external testing rather
than self-reporting.

DID YOU KNOW …that each PISA cycle has produced

more than 10 OECD publications and numerous national
reports in the participating countries?
 The Story So Far: PISA 2000-2006 9

> Analysis of students, schools and school systems’

characteristics, set alongside performance data
> PISA’s database provides an unprecedented array of information,
based on questionnaires, from which to analyse potential
influences on student performance across countries. Although
it does not track individual students and cannot therefore
establish causal links, this analysis can compare the degree
of association with educational outcomes of various factors
in different countries. At the individual level, such factors
include socio-economic background, immigration status and
cultural possessions in the home. At the school level, they
include student perceptions of instructional practices,
disciplinary environment and, importantly, the collective
socio-economic background of students at each school. At
the school system level, the extent of school autonomy and
the structural organisation of students in secondary education
can be compared to the overall performance and distribution
of the performance of students aged 15.
10

Taking Forward a Global

Collaboration to Monitor
Education Systems
ISA is a collaborative effort, bringing together scientific

P expertise from the participating countries, steered jointly

by their governments on the basis of shared, policy-driven
interests.The project’s Governing Board, on which all participating
countries are represented, decided in 2005 to take forward this
partnership into a new phase lasting from 2009 to 2015. As in
the first phase, PISA will aim to better assist countries in
understanding the processes that shape quality and equity in
learning outcomes within the educational, social and cultural
contexts in which education systems operate.
 Taking Forward a Global Collaboration to Monitor Education Systems 11

Maintaining the knowledge base:

PISA 2009-2015
The central part of the PISA survey will continue to be a two- > Include in each survey one developmental assessment area
hour assessment of 15-year-olds’ knowledge and skills, focused to be selected by participating countries. This could be a
on reading, mathematics and science, combined with questionnaires component involving test performance, like the assessment
filled out by students and school principals. However, this core of problem-solving skills in 2003, or it could rely on student
activity will continue to be strengthened and developed, with self-assessments, as with self-reported attitudes to science
the potential for adding new elements, either permanently or in 2006.
as one-off components. Specifically, the core of PISA will:
> Continue to contextualise performance by administering
> Remain focused on assessing the cumulative yield of education questionnaires to students and principals, and extend the
at the age of 15 years, the highest age at which participation analytical power of PISA by adding an optional parent
in formal education in OECD countries is still largely universal. questionnaire.
This will allow the impact of educational change to be tracked
> Provide the opportunity for countries to further describe
and compared across countries for at least the first 15 years
students’ instructional settings by linking PISA to other surveys,
of this century.
e.g. teacher surveys.
> Continue to assess the extent to which students have acquired
key competencies and dispositions needed for further learning
and for adult life, and therefore focus on proficiency in applying
knowledge acquired at school to a wider context, rather than
in just reproducing it.

> Build on existing assessment frameworks in order to improve

their efficiency in assessing desired outcomes, but without
compromising comparability of results over time.
12

Extending our understanding: three new areas of enquiry

ith the core programme of work, PISA places the While the core of PISA allows countries to maintain a common

W emphasis on cross-national comparisons of the

cumulative yield of education towards the end of
compulsory education. It assesses to what extent education
and consistent approach to assessing student knowledge and
skills over time, PISA seeks to remain open to methodological
and substantive innovation. To this end, PISA will explore the
systems succeed in ensuring that young adults acquire the key development of optional modules which seek to extend policy
competencies and dispositions to learning believed to contribute insights into:
to the foundations for further learning and a successful transition
> The measurement of learning progress over time and the
into adult life; as well as what individual, systemic and contextual
comparison of progress across countries
factors contribute to this success. Because of the ongoing nature
of PISA, the core programme will also provide trend indicators > The relationship between aspects of instruction and learning
that will allow countries to measure improvements in educational outcomes
outcomes.
> The assessment of ICT competencies, as well as the use of
technology as a means to capture a broader range of assessment
tasks

DID YOU KNOW …that the PISA survey is typically

administered to between 4 500 and 50 000 students in each
country? And that these students are selected from a random
sample of both public and private schools and are chosen
according to age (from 15 years and 3 months to 16 years
and 2 months at the beginning of the assessment)
as opposed to which grade they belong?
 Taking Forward a Global Collaboration to Monitor Education Systems 13

> Assessment of student progress: How can we best measure the progress of students in acquiring
knowledge and skills in the course of their education?

The objective of this optional module would be to investigate The integration of the assessment of a younger age cohort would,
the possibility of measuring learning progress within education in particular:
systems, by linking assessments of student knowledge and
> Give an indicator of outcomes of the earlier years of education,
skills at different stages of schooling.
and allow analysis of whether they are sustained through
PISA has so far given a snapshot of student performance at a secondary school
single point of time, near the end of compulsory education.
> Help link PISA outcomes more closely to what happens within
While this gives an indication of the cumulative yield of school
schools and classrooms: a connection that is easier to measure
systems, it does not show the extent to which students progress
in the more uniform organisation of primary schools, where
over time. This requires information on performance at different
each student generally has one main teacher
stages of education.
> Be able to look at how factors other than cognitive performance
Although some countries have programmes of tracking the
develop over time – for example, whether students’ levels of
individual students assessed in PISA, it is not at present feasible
motivation change and how their learning styles evolve
for all the countries to do this. However, measuring the overall
performance of students at different ages in different education > Allow a common framework to be developed for monitoring
systems does allow comparison across countries of the progress the development of certain basic skills, which could be
of whole student populations. Ideally, periodic surveys at different translated into estimates of the average yearly progress made
ages would be designed to ensure that a given cohort’s performance by students in subjects such as reading
is measured at different ages in successive surveys.
This option is a longer-term strategic objective for PISA but
there are no plans for its implementation in the context of the
PISA 2009 assessment.
14

> Instruction and outcomes: How can we improve our capacity to describe those aspects of
instruction that influence student outcomes?
The second optional module would facilitate analyses of This module would comprise three separate components:
instructional aspects that could lead to enhanced learning
> A supplementary assessment of students in a single grade or
outcomes.
class (in addition to the usual survey of a sample of 15-year-
PISA is currently able to show the extent to which positive olds), allowing the characteristics of students’ instructional
learning outcomes are associated with particular features of experience to be described more precisely
schooling. So far, the strongest effects have been noted at a
> A more comprehensive collection of information about students,
fairly general level. For example, students in schools with a
schools and school systems, based on extra questions posed
strong disciplinary climate do better on average. More precise
to students and principals, and a collection of more data about
links with the educational experiences of individuals have been
school systems as a whole
limited by the scope of questionnaires administered and by the
fact that 15-years-olds are not all in one grade or one class, > The exploration of links with the OECD’s Teaching and Learning
making it hard to draw precise links with their experiences in International Survey (TALIS), which will first take place in
the classroom. 2007 and which will enable extensive analysis of differences
in national approaches to teaching and learning
This optional module seeks to enable countries to trial several
extra components in the 2009 survey in order to permit better
measurement of the links between students’ instructional
experiences and their performance. These will all be developed
further in subsequent surveys.
 Taking Forward a Global Collaboration to Monitor Education Systems 15

> Computers and PISA: How can ICT strengthen the assessment of students’ knowledge and skills?
The third optional module would seek to develop computer- in themselves are also becoming a highly relevant subject for
delivered assessments to facilitate better coverage of those assessment.
aspects of the PISA frameworks that are very difficult to capture
In its first phase, PISA made a start in both areas. In 2003, it
with static paper-and-pencil documents. In this way, the
used a student questionnaire to survey the extent to which
assessment process will become more efficient (e.g. via adaptive
students used computers and felt comfortable using them. In
testing), with reductions both in operational costs and in student
2005, 13 countries undertook a pilot computer-based assessment
response time. In the longer term, these developments should
of science. The second phase will take this work further, and
improve the assessment’s targeting across and within countries.
potentially lay the groundwork for introducing a computer-
ICT forms an essential part of life in the modern world. Besides based assessment in the PISA core.
the computer’s relevance for student assessment, ICT competencies

> Anticipated timeline for the computer-delivered assessment and the assessment of ICT literacy

Assessment year 2009 2012 2015

Development Test whether computerised Develop “adaptive” assessment

Activity assessment is compatible (see below)
with pencil-and-paper tests;
develop framework for
assessing ICT literacy.

Assessment Assessment of reading electronic Implement supplementary computer Continue implementation of all parts
(see comments texts assessment in focus areas
on previous Assess ICT skills
diagram) Pilot “adaptive” technology enabling
students of different abilities to
respond to different sets of questions
16

p to and including the 2009 survey, this work will focus Over the longer term, there is potential for computer-based

U on assessing the reading of electronic texts. It will only

be possible to explore fuller assessments in each of the
three subject areas and of students’ ICT skills for the 2012 survey
assessments to play a part in improving the delivery of the PISA
core, bringing several advantages. In particular:

> It may be possible to assess certain types of thinking in

and onwards. This reflects the importance of developing
computer-based assessments that are difficult to assess in
instruments that are fully compatible with the rest of PISA,
pencil-and-paper tests. A pilot in 2005 showed, for example,
especially when using computers to assess reading, mathematics
that it is possible to simulate the sequential steps taken in
and science.
scientific experiments by this method.

> A computer-delivered approach could, in the longer term,

ensure that tests align better with the performance levels of
individual students, giving them more challenging or more
manageable tasks according to how they perform in early
questions. This permits in particular a more fine-grained
analysis of the most and least able students. It also provides
a more useful profile of countries whose students are clustered
in particularly high or low areas of performance.

> Finally, computer delivery brings practical advantages, including

lower data entry costs.
 PISA’S Future: Monitoring Change, Deepening Understanding 17

PISA’S Future: Monitoring Change,

Deepening Understanding
n 2009, the students who will be assessed in PISA will have Above all, this evolution is guided by the priorities of educational

I entered primary school at about the time of the first PISA

survey in 2000. As a result, countries will be able to gauge to
what extent any changes in their education systems may have
policy makers, who want to ensure that further changes in
education systems are firmly rooted in good evidence. In the
coming years, PISA will offer countries more elaborate instruments
produced improved outcomes compared with the benchmarks for making linkages between students’ experiences at school
set by the original 2000 survey. A central objective of PISA – and their knowledge and skills near the end of secondary
providing a stable point of reference against which to monitor education. By taking the developmental approach to innovations
the evolution of education systems – will therefore have been in student assessment described above, PISA will be able to test
achieved. which methods work well enough to eventually be incorporated
into its core.
At the same time, PISA continues to seek a deepened understanding
of strong student performances, to get better at measuring them
and to examine links between outcomes and educational
processes. In the first set of surveys from 2000 to 2006, preliminary
steps were taken to measure a wider set of competencies than
had been tested in international surveys, and to collect contextual
data that would show which factors are linked to student
performance. The new elements of PISA described here will take
this process further, without compromising PISA’s integrity as
a stable instrument for monitoring student knowledge and skills.
18

DID YOU KNOW …that you can try sample interactive questions
from PISA and consult all publicly released test questions on the
PISA website?

…that the micro-level PISA data, as well as

their accompanying manuals, are available for further analysis
on the PISA website?

www.pisa.oecd.org
www.oecd.org/media
www.oecd.org/publications
www.oecd.org/statistics
ORGANISATION FOR ECONOMIC
CO-OPERATION AND DEVELOPMENT (OECD) DIRECTORATE FOR EDUCATION
The OECD groups 30 member countries sharing a commitment The OECD views education in a “cradle-to-grave” lifelong learning
to democratic government and the market economy and provides framework, involving formal, informal and non-formal settings.
a unique forum for them to discuss, develop and refine economic Internationally comparable statistics and indicators underpin
and social policies. The OECD’s mission is to promote policies the work but it has a strong qualitative dimension as well. The
designed to: ultimate outputs are policy recommendations designed to
increase both the quality and equity of education systems. The
> Achieve sustainable economic growth and employment and
OECD’s work on education is relevant, not only for government
rising standards of living in member countries while maintaining
and local authorities, but also for civil society, researchers,
financial stability, so contributing to the development of the
professional practitioners and an informed lay audience. The
world economy
extent of the audience is evident in the public discussion and
> Assist sound economic expansion in member countries and debate that OECD’s publications on education often generate.
other countries in the process of economic development

> Contribute to growth in world trade on a multilateral, non-

discriminatory basis

The OECD also has active relationships with some 70 other

countries, non-government organisations and civil society, and
these help give its activities a global reach.
PISA – Programme for International Student Assessment
www.pisa.oecd.org

ORGANISATION FOR ECONOMIC

www.oecd.org CO-OPERATION AND DEVELOPMENT