W H I T E PA P E R

The GlobalEnglish The purpose of this paper is

to describe the GlobalEnglish
Standard Test for English

Standard Test Professionals Plus (STEP+), an

online, computer adaptive test
used by global professionals

for English and international organizations

to assess Business English

Professionals Plus
proficiency.

First, the paper details the design,

(STEP+) development and administration

of the test, including its four
sections (grammar, reading,
Development and Validation listening and speaking), the model
of communicative competence on
which it is based, and how scores
Lindsay Oishi, Ph.D. are reported and interpreted.
Second, the paper presents
Associate Product Owner Progress and Measurement
information on test reliability and
GlobalEnglish Corporation
validity, including an empirical
validation study, inter-rater
reliability, test-retest reliability, and
Silvie Liao, Ph.D. Item Response Theory analyses.
ESL/EFL Learning Specialist
GlobalEnglish Corporation

April 2013

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus

 Contents

3 Introduction

3 Test Description
Design
Administration
Number of Items
Format
Test Construct

9 Content Design and Development

Item Development and Vocabulary Selection
Item Recording

10 Score Reporting
Scores and Weights
Score Use
Score Interpretation

12 Validation
Study Design
Reliability
Validity

16 Conclusion

16 About the Authors

17 About GlobalEnglish

18 References

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus ( S T E P+ ) 2

Introduction adaptive testing is that test-takers receive questions that are
appropriate for their level.
GlobalEnglish STEP+ is an online test designed to evaluate
an individuals ability to understand and use English in global There are a total of eleven item types, as summarized in
business contexts. It is intended for adults and takes about the table below. For every item in the Grammar, Listening
an hour to complete. The test can be taken at any time, at any and Reading sections, and for Question-Response items in
location, and requires a computer with a standard Internet the Speaking section, there are four possible answers and
connection, audio output, and a microphone. A combination one correct answer. The other Speaking items (reading a
of computer and human scoring generates reliable, paragraph out loud, and open-ended questions) are scored
meaningful results that offer a valid snapshot of an individuals on a range of 0 to 5.
proficiency in Business English. Table 1. STEP+ Sections and Item Types

Business English proficiency is how well a person can

comprehend and use English for common business Section Item Types
purposes, and includes many aspects of general English
Error Detection, Fill in the Blank,
language ability. GlobaEnglish STEP+ tests a persons Grammar
Question-Response
capacity to understand spoken and written English on general
and business topics, as well as his or her ability to produce Listening
Describe Picture, Question-Response, Short
Conversation, and Long Conversation
effective spoken English. Corporations and other institutions
can use test scores to inform employment, training, and General Comprehension, Vocabulary, Pronoun
Reading
advancement decisions, as well as measure the outcomes of Reference

English language programs. Read a Paragraph, Question-Response

Speaking (multiple choice with spoken answer),
Question-Response (open-ended)

Test Description GlobalEnglish STEP+ questions are oriented to the

Design context of adults working in global business, and include
functional language on standard topics such as meeting
GlobalEnglish STEP+ is a computer-based adaptive test that
with colleagues, business socializing, telephoning, giving
selects a total of 66 questions for each individual test-taker
presentations and travel. GlobalEnglish STEP+ does not,
out of an item bank that includes hundreds of questions.
however, test knowledge of specific industries, such as
Individuals answer multiple-choice questions across four
manufacturing or finance. Further, it is not limited to a
sections: Grammar, Listening, Reading and Speaking (the
particular geographic variant of English such as British or
Speaking section also includes other question types, as
American English. Audio recordings by men and women
shown in Table 1). In all sections except for the Speaking
feature native English speakers from multiple regions,
section, there are five difficulty levels of questions, and
providing a range of common accents and speaking styles.
test-takers receive questions in pairs. All test-takers receive
the first pair of questions at Level 3 (Intermediate). If they Scoring for the first three sections of GlobalEnglish STEP+
answer both questions correctly, the next pair of questions is (Grammar, Listening and Reading) is by computer, while
at Level 4. If they answer one question correctly, the next pair trained experts in English language and teaching evaluate
of questions is at Level 3. If they miss both questions, the the Speaking section. Scores are available on a secure
next pair of questions is at Level 2. This process of receiving website within 48 hours. The GlobalEnglish STEP+ report
easier or more difficult questions based on previous answers includes a raw score from 0 to 1,581, and an assignment
continues until the final Speaking section. The advantage of to an English proficiency level from Beginner to Advanced

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus ( S T E P+ ) 3

(Levels 0 to 10). These assignments also correspond to Format
levels of study in the GlobalEnglish Edge online curriculum.
This section provides a description of each test section as
Administration well as examples and explanations of the types of questions
found in each. In all sections, questions are presented one
Administration of GlobalEnglish STEP+ is via an Internet-
at a time, with an answer required before the individual can
capable computer and generally takes about an hour.
move on to the next question. There are equal numbers
Test-takers are allowed a maximum of 52 minutes; however,
of each type of item within a section; however, they can
it is not required to use all the time allotted. There are also
be presented at any time (i.e., items of one type are not
untimed parts of the test, at the beginning and between each
grouped together).
section, during which the test-taker can read instructions,
calibrate computer settings, and prepare for the next section.
Part 1: Grammar
Each section is timed separately and test-takers cannot use
leftover time in one section in any other section. It is also
The Grammar section includes two types of items: Error
not possible to go back and change any answer or review
Detection and Fill in the Blank.
any previous part of the test. While it is recommended that
people take the test in a single session, it is possible to take In an Error Detection question, the test-taker reads a
the test in multiple sittings, either through pausing the test or sentence that has four underlined components. Each
by exiting and returning later. This does not affect total time underlined component is a potential answer choice, and the
allowed, and the test resumes at the last point saved. task is to select which answer is grammatically incorrect.
For Fill in the Blank questions, the test-taker must choose
Number of Items the grammatically correct response out of the four potential
The GlobalEnglish STEP+ computerized assessment system answers, which are typically similar and represent common
presents a total of 66 items in four separate sections to each errors.
test-taker. Items are drawn from a large item pool following an
Example: Error Detection
adaptive algorithm.

Table 2. STEP+ Sections, Items and Time Allowed Instructions:

Read the question. Find the mistake.
Click A, B, C or D to choose the answer.
Section Items Time Allowed
Presented (minutes) After he received his second promotion, John was the
most happy employee in the company.
Grammar 26 15

(A) After
Listening 16 15
(B) received
(C) most happy
Reading 16 15 (D) in

Speaking 8 7

Total 66 52

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus ( S T E P+ ) 4

Example: Fill in the Blank In Question-Response items, test-takers listen to a spoken
question and select the written answer that is the most
Instructions:
appropriate response. The question is not displayed as
Read the question. Click A, B, C or D to choose the answer
text and the test-taker can only listen to it twice. There is a
that best completes the sentence.
photo along with the question to provide context, but it is
A: Where was Elena this morning? She was supposed to not required to answer the question correctly.
meet us in the conference room at 10:00AM. Example: Question-Response

B: She ___________ an important call regarding the Heyns Instructions:

account. Listen to the question and select the most appropriate
response to that question.
(A) must take
PLAY
(B) had to take
(C) would take Question (not displayed): How
(D) should take often do you play golf?
(A) Yes, lets play next
week.
Part 2: Listening (B) I play about twice a
week.
There are four item types in the Listening section: Picture,
(C) Ill probably golf today.
Question-Response, Short Conversation, and Long (D) I play golf with my dad.
Conversation.
In Short and Long Conversation items, test-takers listen
In the Picture questions, the test-taker sees a photograph
to a conversation between two people and then answer
of a common social or business situation, and listens to four
a written question about that conversation. The answer
answer choices being read aloud. The answer choices are
choices are also presented as text. Short conversations
not displayed as text, and the test-taker can only listen to
typically involve one sentence given by each person in
them twice.
the dialogue. Long conversations typically involve more
than one sentence by each person in the dialogue. The
Example: Picture
conversations are not presented as text and test-takers can
Instructions:
only listen to each item twice.
Look at the picture. Click PLAY and listen (You can listen
only twice). Click on A, B, C or D to select the best answer. Example: Short Conversation

Instructions:
PLAY Listen to the conversation between the two people. Then,
(A) answer the question about the conversation you heard.
(B) What does the man offer to
(C) do?
(D)
(A) Ask the woman to call Ms.
Jones immediately
Answer choices (not displayed): (B) Call Ms. Jones again
the next day
(A) The man is drinking coffee. (B) The women are shaking
(C) Request that Ms. Jones
hands. (C) The man and woman are meeting at a PLAY
return the call later
restaurant. (D) The men are drinking wine.
(D) Invite the woman to call
Ms. Jones in the morning

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus ( S T E P+ ) 5

 Example: General Comprehension
Part 3: Reading
Instructions:
The Reading section is comprised of General Read the document below. Then read the question. Click A,
Comprehension, Vocabulary, and Pronoun Reference items B, C or D to select the best answer.
in approximately equal numbers. General Comprehension Which of the following is NOT a reason for Dymo
questions test a readers ability to recognize explicit Industries increased sales?
information, make reasonable inferences from a text, (A) A new sales team
and use this knowledge to select a written answer with (B) New markets
appropriate semantic content in response to a written (C) Sales training
question. Vocabulary items require the test-taker to identify (D) Performance in international divisions
a synonymous word or phrase for a selected word from the
reading passage, as it is used in that context. Finally, Pronoun
Example: Vocabulary
Reference items ask the test-taker to identify the noun that
a pronoun such as it, he or they refers to. In easier Instructions:
questions, this noun is clearly stated in the passage; in harder Look at the word that is highlighted in yellow in the reading
questions, the noun may have to be inferred from the text. As passage. Choose the word or words that have the same
meaning as the highlighted word.
the reading passages are short but complete texts, typically
a few hundred words long, there are two questions for each Look at the word divisions in the article. Which words or
passage. phrases mean the same thing as divisions in this article?
(A) Disagreements within the company
Example: Reading Passage
(B) Groups or teams within an organization
(C) Sales methods
Annual Sales Report
(D) Individual employees
Dear Dymo Employees,

We had an excellent year at Dymo Industries. Total sales

were US$550 million. This is an impressive growth of 10% Example: Pronoun Reference
compared to last year.

There are a few important reasons for the large increase in Instructions:
sales. All the international divisions performed very well. The Instructions: Look at the pronoun that is highlighted in
new sales training helped many of us promote our new line of yellow in the reading passage. Choose the word or words
toys. In addition, the creation of new products opened more that have the same meaning as the highlighted pronoun.
markets for us. The word or words will come before the highlighted
pronoun. Some may even be in the previous sentence.
We continued last year to provide our customers with
improved products. We all know that product enhancement Look at the word this in the article. What does the word
can lead to larger market shares. Therefore, we hope to this refer to in the previous sentence?
continue increasing our sales by creating better products in (A) Total sales
the coming year.
(B) Dymo Industries
Congratulations on your great teamwork! (C) Last year
(D) Division performance
Jon Ngyuen

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus ( S T E P+ ) 6

 In Part B, test-takers must listen to audio cues, choose
Part 4: Speaking a written response that is appropriate in the context, and
speak it. They are evaluated on their ability to choose
In the speaking section, examinees read a written paragraph
the correct answer and to say it clearly. Part B has
out loud, answer five short questions (where responses are
five questions and is designed to test communicative
provided in text), and respond to two open-ended questions.
competence. Communicative competence refers to
Test-takers are instructed to speak naturally, using correct
a speakers ability to apply linguistic, discourse, and
pronunciation, intonation, stress and rhythm. Test-takers can
cultural knowledge to produce appropriate language in
only record their answers once, and while recording, they
a particular context.
cannot pause the timer. There is an exception to this rule
Example: Select & Speak Response
if the audio recorded is too poor to produce a score-able
response; in that case, the test-taker is allowed to re-record
Instructions:
his or her response. If test-takers produce two recordings
Click PLAY and listen. You can only listen twice. Click
without intelligible audio, the test takes them back to the
RECORD and then read the correct response aloud. Click
microphone calibration tool.
STOP when you are finished.
In Part A, test-takers read a short paragraph aloud. Once
Audio Cue (not displayed): We dont
they begin recording, they have 60 seconds total to finish PLAY
have time to discuss everything on the
reading the paragraph aloud; however, they may stop the agenda at todays meeting.
recording at any point if they finish before 60 seconds RECORD
are used. Part A has one question and is designed to test (A) The meeting started an hour ago.
pronunciation, including the ability to say all of the sounds of STOP (B) Whats the next point on the
English, stress the correct syllables, pause appropriately, and agenda?
use the correct intonation. This allows scorers to evaluate the (C) Thats okay, we can continue
candidates ability to produce intelligible and comprehensible tomorrow.
(D) Well, what in particular concerns
speech. Intelligibility refers to the production of sound
you?
patterns that are recognizable as English. Comprehensibility
refers to the ability to speak so that people can understand
Part C has two open-ended questions and is designed to
the meaning. Test-takers read paragraph aloud that has all of
test overall linguistic performance. Open-ended questions
the phonemes of English and a variety of sentence types.
allow evaluation of how well test-takers construct a
Example: Read Paragraph response to show evidence of their acquired knowledge.
The first question requires only a beginner to low
Instructions:
Click RECORD and read the paragraph aloud. Click STOP intermediate level of English to answer adequately, while the
when you are finished. second question requires greater topic development and
Our CEO announced a decision today linguistic complexity, and demonstrates high intermediate to
RECORD at the monthly company meeting. The advanced ability when answered well. Test-takers are given
executives think that we should expand
75 seconds for the low-level response and 105 seconds for
STOP our product line. We just need to add
a few luxury items to some electronic the high-level response, including preparation time.
goods. That should help us increase our Table 3. Examples of Open-ended Questions
market share in Europe and Asia. Future
profits depend on our dealing with this
Level Question
issue now. We hope to increase sales by Low What is your favorite hobby or activity? Why?
10% in the next year.
How does the Internet affect your life? What are the
High
advantages and disadvantages of the Internet?

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus ( S T E P+ ) 7

 Test Construct
Figure 2. Language Skills Assessed by STEP+
Canale and Swains (1980) model of communicative
competence is one of the most influential frameworks
in language testing, and argues for the need to assess
grammatical, sociolinguistic, strategic and discourse Language skills
competence. In this model, simple declarative knowledge
of the language (e.g., being able to state the past tense of
a verb) is not sufficient evidence of ability to actually use Grammar Listening Reading Speaking
the language for real communicative purposes. Figure 1
Morphosyntactic Comprehension Main idea Topic
presents the theoretical framework of this model. Tests
Syntactic Inferences Detail development
of communicative competence not only reflect the need Functional Inferences Grammar control

for language to be used interactively with other speakers Vocabulary Fluency

Pronunciation
of that language, but also incorporate authentic contexts,
Communicative
situations, and topics. In this way, tests based on this model effectiveness
aim to evaluate and individuals overall ability to communicate
successfully in real-life situations.

Figure 1. STEP+ Test Construct

Figure 3. Communicative Competence and
Language Skills Tested in STEP+
Communicative Competence

Communicative Competence
Grammatical Sociolinguistic Strategic Discourse

Vocabulary Sociocultural Communication Cohesion and

Syntax rules of use (e.g. strategies (e.g. coherence (e.g. Grammatical Sociolinguistic Strategic Discourse
Morphology appropriateness strategies used knowledge to
Phonology of language use for unfamiliar understand/
Sentence- in context) topics or grammar produce texts in
grammar forms) different genres Grammar Listening Reading Speaking
semantics

Morphosyntactic Comprehension Main idea Topic

As evident in the figure above, communicative competence Syntactic Inferences Detail development
is a complex construct that involves more than one language Functional Inferences Grammar control
Vocabulary Fluency
skill (such as reading or speaking per se). To provide a
Pronunciation
strong evaluation of a candidates actual communicative Communicative

ability, a test must therefore address multiple language skills. effectiveness

Figure 2 shows the language skills assessed in GlobalEnglish

STEP+, and Figure 3 illustrates how each section within the
test addresses multiple facets of communicative competence.
As a comprehensive test, GlobalEnglish STEP+ is designed
to measure more than English language knowledge or
general linguistic skill. Instead, it specifically targets the ability
to effectively use English in global business contexts.

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus ( S T E P+ ) 8

Content Design and Development Test items in the speaking section underwent a different
procedure to create the three distinct sub-sections. In Part
The GlobalEnglish curriculum and STEP+ were developed
A (reading paragraphs), test writers were advised to write
by a team of experts with extensive training and experience
paragraphs that met the following criteria:
teaching English as a foreign language, creating curricula,
and writing coursework. GlobalEnglish Corporations world- Length of 65 to 75 words.
class Academic Advisory Board, including four former Score between 75 and 85 on the Flesch Reading
presidents of International TESOL, aided this team. Dr. David Ease scale.
Nunan, the senior academic advisor to GlobalEnglish, was
Paragraph subjects related to business.
director and chair of applied linguistics at the University of
Hong Kong, and the president of International TESOL in
Contain at least two instances of every phoneme in
English.
1999-2000. He has published over 100 books and articles
in the areas of curriculum and materials development, A variety of consonant clusters (e.g. the lpf in the
classroom- based research, and discourse analysis. For more word helpful).
information, visit www.globalenglish.com/whoweare/aboutus/ Contain words with the phonemes /s/ and /z/ at the
academic_advisors. end (e.g. boss, was).

Item Development and Vocabulary Selection Some variation in sentence intonation (e.g., questions
and statements).
As GlobalEnglish STEP+ was designed to align with
GlobalEnglish levels, test items were created with reference Items in Part B and Part C were adapted from questions in
to content and learning objectives from the levels, which are the GlobalEnglish curriculum. After those speaking items
available by request or through the GlobalEnglish website. were drafted, 30 non-native speakers of English were
The vocabulary used in GlobalEnglish STEP+ was also based recruited to take a pilot test. Based on their responses,
on the GlobalEnglish courses, and test-writers were given a items that seemed inappropriate, confusing or dependent
comprehensive list of 3,150 vocabulary words, as well as the on factors such as test-takers working experiences,
syllabus for each level. regions or familiarity with certain English dialects were
either eliminated or revised.
For the grammar section, test writers were given the test
framework, which listed grammatical features across different Overall, the GlobalEnglish STEP+ item bank was designed
English proficiency levels, and were advised to provide so that any two randomly selected test forms would have
a balance across the three types of grammar features: similar test content and difficulty. The item bank has also
morpho-syntactic (e.g. articles a/an/the), syntactic (e.g. undergone two thorough reviews. Dr. Kathleen Bailey1, a
use of a preposition), and functional (e.g. tense marking in recognized expert in English language testing, analyzed
conditionals). For the listening section, test writers were the item pool, checking item difficulty, and examining
given a list of common categories of expressions in business each item for its correspondence to the test construct.
communication (e.g. asking for and giving opinions). For the Dr. Jean Turner2 of the Monterey Institute, a recognized
reading section, test writers worked from a list of genres expert in English language testing, then conducted a
common in the workplace, such as memos, forms, charts and statistical analysis of each test items discrimination
announcements. (capacity to separate low-level examinees from high-level

1.
Dr. Kathleen Bailey is a professor of applied linguistics at the Monterey Institute of International Studies in Monterey, California. Director of the TESOL M.A. program
there for six years, she was also the Director of the Intensive English as a Second Language Program for three years. Dr. Bailey was the 1998-1999 president of
TESOL. She has worked with language teachers in many countries and is the author or coauthor of numerous scholarly articles and books. Her two most recent
books are Pursuing Professional Development: The Self as Source (co-authored with Andy Curtis and David Nunan, Heinle & Heinle, 2001) and Practical English
Language Teaching: Speaking (McGraw-Hill, 2004). Dr. Bailey earned her Ph.D. in applied linguistics from the University of California, Los Angeles.
2.
Dr. Jean Turner is a professor of applied linguistics at the Monterey Institute of International Studies. She is a recognized expert in the field of English language
testing. She received her Ph.D. from the Graduate School of Language and Educational Linguistics, Monterey Institute of International Studies.

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus ( S T E P+ ) 9

examinees, based on their overall pattern of responses). The raw score for each section is the sum of points
Items with appropriate item discrimination were retained awarded for correct answers. Easier questions have
in the GlobalEnglish STEP+ item bank, while others were lower point values, and harder questions have higher
revised under Dr. Baileys guidance. This effort reduced point values, with a range of 2.5 (easiest) to 27.5 (hardest)
measurement error, increased reliability, and enhanced points awarded for a single correct answer. Total score
construct validity. for the grammar section is divided by 1.625 in order to
provide an equal weighting of grammar with listening and
Item Recording reading sections (the grammar section has 26 questions,
Sixteen professional voice talents (7 males and 9 females) while the listening and reading sections each have 16). It
were hired to record listening items and speaking prompts. is possible for a user to be awarded more points than the
All voice actors were native speakers with a range of maximum section score. In this case, the section score is
common American and British English accents. Recordings set to the maximum.
were made in a professional recording studio in South San
The Speaking section is evaluated by trained, expert
Francisco, California. An ESL expert was on site to ensure
human scorers with extensive experience in teaching and
that the speed and pronunciation were clear, natural and
assessing English as a foreign language. There are three
appropriate to the items at different levels of difficulty.
parts to the Speaking section, with a different question type
in each.

Score Reporting Scoring for Part A of the Speaking section (reading a

Scoring for the first three sections of GlobalEnglish STEP+ paragraph aloud) is holistic and depends on two key
(Grammar, Listening and Reading) is computerized, and factors: intelligibility and comprehensibility. Scorers are
when the audio files from the Speaking section have been instructed to think about the extent to which pronunciation
scored (within 48 hours and often sooner), the test-taker and fluency problems reduce understanding of what
receives an email with a link to a secure results web page. the test-taker said, and to gauge how much any accent
interferes with basic intelligibility. Each question is given a
The Score Report includes: score of 0.0 to 5.0, with half-points allowed.
A raw score from 0 to 1,581.
Part B of the Speaking section contains multiple choice
The test-takers recommended English level at which questions in increasing order of difficulty, and all questions
to begin studying (i.e., Level 3: High Beginner), and a are scored as correct or incorrect.
description of that level.
Scoring for Part C of the Speaking section (two open-
Performance (raw score) in each of the four sections:
ended questions) is holistic, but the rubric includes
Grammar, Listening, Reading, and Speaking.
consideration for grammar and topical development (how
A description of strengths and weaknesses by section. completely and well the response answers the question),
as well as pronunciation and fluency. Each of these four
Scores and Weights
sub-scores is given a score of 0 to 5, with no half-points
GlobalEnglish STEP+ has a score range of 0 to 1,581, allowed.
calculated by summing the scores for each of the four
sections. The maximum possible scores vary by section: Score Use
403 in Grammar, 392 in Listening, 395 in Reading and 391 GlobalEnglish STEP+ is a diagnostic tool that can be
in Speaking. used in multiple ways. First, it is a valid indicator of the
most appropriate level for an individual to begin studying

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus ( S T E P+ ) 10

Business English. Second, organizations can use the test Score Interpretation
to estimate the overall Business English proficiency of a
GlobalEnglish STEP+ raw scores are reported along with
potential or current employee, and use this information to
a GlobalEnglish Business English Proficiency Level from
inform hiring, training and advancement decisions. Third,
0 (Low Beginner) to 10 (Advanced). Table 4 shows how
GlobalEnglish STEP+ can be used to ascertain progress
GlobalEnglish levels align with GlobalEnglish STEP+ scores.
in Business English after a significant and sustained
period of study. These scores can also be interpreted using the Common
European Framework of Reference (CEFR, Council of
Scores on GlobalEnglish STEP+ are valid and reliable
Europe, 2001), a widely used standard that describes six
indicators of Business English proficiency when the test
broad levels of proficiency in any language. The CEFR was
is taken in compliance with recommended conditions. We
designed to provide common reference points and terminology
strongly suggest that test-takers complete the test to the
for learners, teachers, companies and other institutions to
best of their ability, with no references, aides or outside
describe achievement across diverse educational contexts.
help, in a single session. An organization that wishes to
Table 4 also shows how GlobalEnglish STEP+ scores relate
enforce these recommendations or require evidence of an
to CEFR levels and gives an example descriptive statement for
individuals identity or independent performance should
level. These relationships are intended as general guidelines,
set up a monitored, formal administration procedure or
not specific claims about the abilities of GlobalEnglish STEP+
use a respected third party administrator.
test-takers.

Table 4. STEP+ Scores and the Common European Framework of Reference

GlobalEnglish STEP+ GlobalEnglish CEFR CEFR

Score Level Level Description
0 - 120 0
Can understand and use familiar everyday expressions and very basic phrases
A1
aimed at the satisfaction of needs of a concrete type.
121 - 240 1

241 - 347 2
Can communicate in simple and routine tasks requiring a simple and direct
348 - 472 3 A2
exchange of information on familiar and routine matters.
473 - 620 4

621 - 768 5
Can understand the main points of clear standard input on familiar matters
769 - 916 6 B1
regularly encountered in work, school, leisure, etc.
917 - 1064 7

1065 - 1212 8
Can interact with a degree of fluency and spontaneity that makes regular
B2
interaction with native speakers quite possible without strain for either party.
1213 - 1399 9

Can use language flexibly and effectively for social, academic and
1400 - 1581 10 C1
professional purposes.

Can express him/herself spontaneously, very fluently and precisely,

N/A N/A C2
differentiating finer shades of meaning even in the most complex situations.

* For complete global descriptors for CEFR levels, please see the Common European Framework of Reference for Languages, available in English at
http://www.coe.int/t/dg4/linguistic/Source/Framework_EN.pdf

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus ( S T E P+ ) 11

Validation Reliability

As GlobalEnglish STEP+ is continuously monitored, Overall Test Reliability

evaluated, and revised, validation of the test is an ongoing
As GlobalEnglish STEP+ is a computer adaptive test with
process. The original test development team was comprised
a large pool of potential questions, it is not practical or
of English language teaching and assessment experts
useful to compute overall test reliability metrics used in
who drew on extensive knowledge of the Common
classical test theory (such as Cronbachs alpha). A modern
European Framework and other standards to create
alternative approach is Item Response Theory (IRT), which
items with appropriate content and difficulty. In 2011, a
considers the contribution of each question to the overall
respected external firm was hired to perform a review
tests capacity to give valid and reliable scores. IRT is
of item performance and the test overall; as a result,
a diverse family of [psychometric] models designed to
underperforming items were removed or revised in early
represent the relation between an individuals item response
2012. In late 2012, the speaking section of GlobalEnglish
and underlying latent trait, (Fraley, Waller and Brennan,
STEP+ was revised significantly in order to offer a more
2000) that is widely used in educational testing, and useful
complete evaluation of speaking ability, with more challenging
for modeling communicative competence in English (a latent
questions and a more comprehensive scoring process. To
trait).
understand the consequences of the changes to the test in
early and late 2012, data from that year was used to analyze As IRT analysis is designed for tests that measure a
test reliability and validity. single latent variable, a factor analysis of principal
components (PCA) was first performed on the four
Study Design GlobalEnglish STEP+ subscores (Grammar, Listening,
Data on test reliability and validity was analyzed from three Reading, Speaking) to ascertain unidimensionality across
samples: the test. As shown in the screeplot presented in Figure 4,
the analysis indicated a single-factor solution, with the first
For overall test reliability and item analysis using the
factor explaining 63% of the variance in subscores.
Item Response Theory (IRT) model, test results were
used from a three-week period in November and A factor map of the variables showed that while all
December 2012 (N = 2,862). subscores loaded positively on the first, primary factor,
the Speaking subscore also loaded positively on a second
For test-retest reliability analysis, test results were
factor (Figure 5).
taken from May 2012 through March 2013 and
included those test-takers who took the test twice in Figure 4.
one day (N = 384). Screeplot of Principal Components using STEP+ Section Scores

For the study on how test results related to other

2,000 4,000 6,000 8,000 10,000 12,000

constructs, participants (N = 62) were employed,

adult volunteers who took GlobalEnglish STEP+
in October and November of 2012. Participants
completed a short survey with demographic and other
Variances

questions after they finished the test. There were 16

men and 28 women in the sample (18 did not state
gender), with an average age of 33 (range: 20-55
years). Nine native English speakers were included
in the validation study for comparison purposes.
Participants were from a variety of countries in
Europe, South America, and Asia, and had a range Component 1 Component 2 Component 3 Component 4
of professions.

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus ( S T E P+ ) 12

Figure 5. Test-Retest Reliability
Variables Factor Map (Principal Components Analysis)
Another method for ascertaining test reliability is to
determine whether a person who takes the test more than
1.0

once will obtain a similar score (absent intervening study

Speaking or other factors). To perform this analysis, we examined a
Dimension 2 (15%)

0.5

sample of test-takers who voluntarily and spontaneously

chose to take a test composed of Reading, Listening and
Listening
0.0

Grammar items from the STEP+ pool (this test is available

Grammar
as the Business English Placement Test to GlobalEnglish
Reading
Edge subscribers). Among those who took both tests in
-0.5

the same day (N = 384), correlation between their two test

scores was positive and significant (r = .82, p < .001).
-1.0

-1.0 -0.5 0.0 0.5 1.0

Inter-rater reliability
Dimension 1 (63.24%)
When the GlobalEnglish STEP+ Speaking section was
revised in September 2012, the team of expert scorers
was extensively re-trained to address the addition of open-
For the validation study, we used the Rasch model,
ended questions. As part of the training, scorers listened to
which is commonly used in IRT for educational tests
sample responses, which represented a range of English
(Rasch, 1960/1980). To fit the model, we used the
language proficiencies, and discussed scoring issues until
maximum log-likelihood method as specified in the ltm
they reached agreement. An initial manual was composed
package of the R statistical analysis software (Rizopoulos,
to guide scoring for the validation period.
2012). As the IRT analysis with the full data set (767 items,
2,862 observations) produced unstable solutions due to a During validation, scorers were randomly assigned to
large number of missing values, the dataset was reduced to two groups, and each group scored the same 29 tests
the most frequent items (456 items, 60% of total) and the (which were also randomly assigned within each group),
most complete observations (2,053 observations, 72% of with a total of 394 Speaking items after unusable audio
total). Although a complete description of the IRT results is files were removed. After both groups had scored the tests,
beyond the scope of this white paper, using the IRT model items with significant score differences were identified
created with this dataset, a simulation of 1,000 possible test and reviewed. The GlobalEnglish STEP+ team discussed
forms was performed. Across the 1,000 test forms, average scoring issues with all scorers, and the scorer manual was
test information = 41.29, SEM = .16, and reliability = .97. revised accordingly. Finally, the 25 items with the greatest
The model also indicated that while the test was informative discrepancies were re-scored by their original scorers
(reliable) at all skill levels, it was most informative for following the revised guidelines, and inter-rater reliability
test-takers at the intermediate to high intermediate was computed using these final scores. Inter-rater reliability
proficiency levels. across the entire Speaking section was .84.

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus ( S T E P+ ) 13

Validity Native and Non-native Performance
Although the number of native speakers in the validation
Dimensionality
sample was small (n = 9), their performance on
One indication of validity is a tests ability to reveal multiple GlobalEnglish STEP+ was high, as expected. The average
dimensions of a particular construct, such as language, total GlobalEnglish STEP+ score for native speakers was
which is theorized to contain sub-skills or other secondary 1,414 (out of 1,581), with a standard deviation of 68 points.
components. While GlobalEnglish STEP+ tests Business For non-native speakers (n = 44), however, the average
English language proficiency generally, it also should allow was 1,012, with a much greater standard deviation of 300
for test-takers to have different levels of achievement in four points. These results, while preliminary, are evidence for
linguistic sub-skills: grammar, listening, reading and speaking. the construct validity of GlobalEnglish STEP+ as a
In Table 5, below, the correlations between sub-scores on reflection of English language ability. As we continue to
GlobalEnglish STEP+ show that while each sub-score was work on validation, more native speakers will be added to
positively and significantly related to the other sub-scores, this sample.
these relationships were of moderate strength (Pearsons
Relationship to Self-Rating
r-values of .42 to .57). Higher r-values would suggest that
GlobalEnglish STEP+ sub-sections were providing redundant As a further test of construct validity, we examined the
information on highly similar constructs, while lower r-values relationship between self-ratings and performance
would indicate too much disparity in sub-skills that are on GlobalEnglish STEP+. While certainly imperfect,
theoretically linked. It is also worth noting that receptive skills self-ratings are a reasonable way to approximate the
(grammar, listening and reading) are more strongly related to correspondence of a test with what it measures, in domains
each other than they are to productive skills (speaking), as where individuals can understand their own proficiency and
would be expected in standard models of language learning. report it with some degree of accuracy. Table 6 shows that
Overall, the moderate correlations demonstrated here among participants in the validation sample who had both
support the validity of GlobalEnglish STEP+ as a Business self-ratings and complete GlobalEnglish STEP+ scores (n =
English assessment with the ability to discriminate between 44), there were moderate, positive correlations between the
different secondary language skills. two variables, both overall and for different language skills
(corresponding to sections within the test).
Table 5. Correlations for STEP+ Section and Total Scores
(N = 2,862) Table 6. Correlation between STEP+ Section Scores and Self-
Ratings
Listening Reading Speaking Total
Correlation of Score
Grammar .57 .57 .48 .84 STEP+ Section
and Self-Rating

Listening .53 .48 .81 Overall .53**

Reading .42 .82 Grammar .38*

Speaking .70 Listening .42*

Reading .45*
All ps < .01.

Speaking .65*

* p < .01; ** p < .001

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus ( S T E P+ ) 14

Relationship to Previous Study reported average amount of time (in years) spent studying
The relationship of GlobalEnglish STEP+ scores with English across all categories was positively correlated with
previous English language study was also examined as overall performance on GlobalEnglish STEP+, Spearmans
evidence of construct validity. As language proficiency is a r (36) = .46, p < .001. Time spent studying English during
learned skill, the duration and quality of previous study should post-secondary education, graduate school, and private
have an impact on test outcomes. We asked validation study instruction were all significantly related to GlobalEnglish
participants to report how much time they spent studying STEP+ scores, while English language study during
English in various contexts, including primary, secondary, and primary and secondary school, or through self-study
post-secondary education, as well as independently through (including via the Internet) were not significantly related to
private instruction and self-study. Table 7 shows that the self- GlobalEnglish STEP+ scores.

Table 7. Previous Language Study Associated with STEP+ Scores

Educational Context Correlation with STEP+

N
(Years of English Study) Score

Significantly Related to STEP+

Private Instruction 27 .48*

Lifetime Average
36 .46**
(All Instruction Types)

Post-secondary Education 32 .41*

Graduate Education 32 .39*

Not Significantly Related to STEP+

Internet-based Study
28 .31
(not including GlobalEnglish Edge)

Primary Education 33 .28

Self-study 28 .20

Secondary Education 33 .15

* p < .05; ** p < .01

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus ( S T E P+ ) 15

Conclusion About the Authors
In todays global economy, Business English is vital to Dr. Lindsay Oishi is the Associate Product Owner for
organizational productivity. Companies and institutions need Progress and Measurement at GlobalEnglish Corporation.
to be able to assess Business English as part of critical She has a BA from Georgetown University, an MA from
human capital decisions such as recruiting the right person Oxford University, and a PhD in Educational Psychology
for the right job, handling promotions and raises, ensuring from Stanford University. Dr. Oishi has worked on
personnel skills development, and retaining employees. research, assessment and evaluation in both academic
GlobalEnglishs robust, online and adaptive assessment settings, at the Hasso Plattner Institute of Design and the
enables organizations to evaluate and monitor the language Stanford University Online High School, and in business
skills of their employees and job candidates with the end-goal settings, at Hewlett-Packard and Adobe Systems. She
of increasing organizational communicative and collaborative has written about online education and language learning
capacities. in publications for various organizations, including the
International Association for K-12 Online Learning, the
When used as recommended, GlobalEnglish STEP+ is an
California Foreign Language Project, and the American
effective diagnostic of Business English proficiency with
Educational Research Association.
significant advantages over alternative assessments. As
an online test, it is always available and does not require Dr. Silvie Liao is the ESL/EFL Learning Specialist at
travel to a test center at a specific date and time. Its breadth GlobalEnglish Corporation. She is an expert linguist,
of question types allows for a comprehensive picture of with an MA in TESOL from Teachers College, Columbia
Business English capability, yet because of its adaptive University, and a PhD in Linguistics from the University of
design, the test takes an hour or less. Test-takers receive California, Davis. Throughout her academic career,
detailed feedback on their performance, and Administrators Dr. Liao has published articles on second language
can easily access the score information and speech samples acquisition, sociolinguistics, language variation, and related
that they need. GlobalEnglish STEP+ is the best solution topics. She has over ten years of teaching experience with
for companies wanting fast, accurate, easy-to-implement students of all ages from diverse language and cultural
evaluations of Business English competence. backgrounds.

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus ( S T E P+ ) 16

 About GlobalEnglish
GlobalEnglish offers an innovative, cloud-based suite of Business English
solutions specifically designed to produce immediate productivity and
performance gains enterprise-wide. In blending the latest technology
innovations with research on how adults effectively acquire language,
GlobalEnglish provides a comprehensive solution: formal and informal
Business English learning, instant on-the-job support for business tasks in
English, enterprise collaboration, mobile productivity, adaptive Business English
assessments, and the ability to measure usage and proficiency improvements
across the company. GlobalEnglish experts located throughout the world help
companies maximize the value of their investment through custom analysis and
recommendations, coordinated program deployment, and ongoing support in
15 languages. Headquartered in Brisbane, California, GlobalEnglish has
partnered with more than 500 of the worlds leading corporations and
enterprises, including BNP Paribas, Capgemini, Deloitte, GlaxoSmithKline,
Hilton, John Deere, Procter & Gamble and Unisys. GlobalEnglish is owned by
Pearson, the worlds leading learning company.

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus ( S T E P+ ) 17

 References
Canale, M. (1983). From communicative competence to communicative language
pedagogy. In J. Richards and R.W. Smith (Eds.), Language and Communication.
New York: Longman.

Canale, M., Swain, M. (1980). Theoretical bases of communicative approaches to

second language teaching and testing. Applied Linguistics, 1, 147.

Council of Europe. (2001). Common European Framework of Reference for

Languages: Learning, Teaching, Assessment. Cambridge: Cambridge University
Press.

Council of Europe. (2009). Relating Language Examinations to the Common

European Framework of Reference for Languages: Learning, Teaching, Assessment
(CEFR): A Manual. Strasburg: Language Policy Division.

Hymes, D.H. (1966). Two types of linguistic relativity. In Bright, W. (Ed.),

Sociolinguistics. The Hague: Mouton.

Fraley, R.C., Waller, N.G., and Brennan, K.A. (2000). An item response theory
analysis of self-report measures of adult attachment. Journal of Personality and
Social Psychology, 78, 350365.

Rasch, G. (1960/1980). Probabilistic models for some intelligence and attainment

tests. Chicago: The University of Chicago Press.

Rizopoulos, D. (2012). Latent Trait Models under IRT. Retrieved from http://rwiki.
sciviews.org/doku.php?id=packages:cran:ltm.

W H I T E P A P E R : The Gl obal En g l i sh Stan dard Test f or E ng lis h Prof es s ionals Plus ( S T E P+ ) 18

053013