Brown 2005 - BRJ 29 2 Summer PDF
Brown 2005 - BRJ 29 2 Summer PDF
Brown 2005 - BRJ 29 2 Summer PDF
Abstract
This article reports findings from a study that investigated math
achievement differences between English language learners (ELLs)
and fully English proficient (FEP) students on a literacy-based
performance assessment (LBPA). It has been assumed that LBPAs
are superior to standardized multiple-choice assessments, but it
has not been determined if LBPAs are appropriate for measuring
the math achievement of ELLs. The most salient characteristic of
LBPAs is that students read multi-level questions and explain how
they solve math problems in writing. Thus, LBPAs place great
literacy demands upon students. Because most ELLs have
underdeveloped literacy skills in English, these demands put ELLs
at a great disadvantage. Analysis revealed that socioeconomic
status (SES) had a significant impact on all students, but the impact
was larger on FEP students than on ELLs; high-SES FEP students
outperformed high-SES ELLs, but there was no significant difference
between low-SES ELLs and low-SES FEP students. High SES
generally means more cognitive academic language proficiency,
because of the influence of non-school factors such as the presence
of a print-rich environment. High-SES ELLs did not do as well as
high-SES FEP students because of a lack of academic English. The
nature of the examination masked their true abilities. The finding
of no difference between low-SES ELLs and low-SES FEP students,
however, could be a result of the fact that neither group had the
advantage of high cognitive academic language proficiency; the FEP
students’ only “advantage” was superior conversational English,
of little use for performing academic tasks. This article concludes
that LBPAs, together with the current assessment-driven
accountability system, seriously undermine equal treatment for
ELLs.
The Study
This study focused on the achievement gap in math between ELL students
and FEP students on the Maryland School Performance Assessment Program
(MSPAP) using test scores from the year 2000.1 The MSPAP was chosen
because the Maryland State Department of Education (MSDE) created a unique
LBPA. The MSPAP differed from assessments used in other states in the
following ways:
1. One of the MSPAP content areas (the math communication subskill)
specifically measures students’ ability to communicate mathematical
knowledge in writing, thus challenging students to go well beyond mere
mathematical calculation;
2. This open-ended test asks students to construct written responses
throughout the entire testing program;
3. The entire math portion of the MSPAP consists exclusively of higher level
word problems (see sample test items in Appendix B);
4. Multi-procedure questions in math word problems require a high level of
reading comprehension; and
5. Connections between reading and writing across the curriculum reflect
the most salient characteristics of the LBPAs (see http://www.mdk12.org/
mspp/mspap/what-is-mspap for a detailed description of the MSPAP).
The MSPAP is given in Grades 3, 5, and 8. Third graders were chosen for
this study because there are a higher number of ELL students in Grade 3:
Young ELL students tend to exit ESL programs rather quickly. (Note that ELL
students, once exited from ESL programs, are not coded as ELL students.
They are reclassified and become part of the FEP population. Thus, there is a
high probability that reclassified ELL students were part of the FEP pool when
sampled.)
This study posed three research questions in relation to achievement
differences in math between ELL students and FEP students within the same
SES as measured by Free and Reduced Meals (FARMs) status. The SES
variable is held constant within each group to minimize its influence on the
test scores, since the SES is known to be the most influential determinant of
student achievement (Fernández & Nielsen, 1986). The research questions
were:
Instrumentation
MSPAP, a criterion-referenced test, assesses students’ achievement levels
in six content areas: reading, writing, language usage, math, science, and
social studies. It is constructed so that the scores from multiple content areas
can be cross-sectionally compared within a grade. The scaled scores, ranging
from 350 to 700, are designed to have a mean score of 500 and a standard
deviation of 50 (see http://www.mdk12.org/mspp/mspap/what-is-mspap for a
detailed description of the MSPAP, including administration and scoring).
Sampling
Test scores of the third graders from all 25 Maryland school districts were
selected, excluding students who received special education services
(language variables and exceptionalities related to special education have
confounding effects on the test scores). Random sampling for the ELL students
and stratified random sampling for the FEP students were planned according
to FARMs status to keep the SES variable constant. However, random sampling
for ELL students was not performed because of the contingency imposed on
the author by the MSDE due to the small percentage (1.1%) of ELL students’
participation in the MSPAP. Consequently, there were four subgroups:
(a) ELL students with FARMs, (b) FEP students with FARMs, (c) ELL students
with non-FARMs, and (d) FEP students with non-FARMs. Information
regarding participants’ prior educational backgrounds or formal schooling
was not available from the MSDE data set.
In 2000, a total of 65,536 third-grade students took the MSPAP; 742 of
them were identified as ELL students and the rest (64,794) as FEP students.
From the 742 ELL students, 90 students coded as special education were
excluded. From the remaining 652 ELL students, 492 (n1) students were
Data Analysis
Independent samples t-tests were selected to answer the first two research
questions, an investigation of performance differences between ELL students
and FEP students in the overall math examination, as well as the math
communication subskill. For the third research question, multiple linear
regression analysis was employed to determine which predictor variable
accounted for the largest proportion of the variance in the criterion variable,
math achievement.
Non-complete Scores Complete Scores (492) Non-complete Scores Complete Scores (53,025)
n1 % n2 %
American Indian 2 .4 2 .4
The .05 level of significance was chosen for the study; however, when
the same statistical procedures were performed more than once, the alpha
level was adjusted to a more conservative level (α = .01) in order to lower
the chances of committing a Type I error, that is, the error of concluding what
are actually non-significant findings as significant. For Research Questions 1
and 2, in addition to t-tests, a multivariate analysis of variance (MANOVA)
was conducted.
Results
Table 2
Means, Standard Deviations, and t-Test Results on Math
Non-Free and Reduced Meals groups
n1 M1 SD n2 M2 SD t p d
(462)
n1 M1 SD n2 M2 SD t p d
(518)
t
n1 M1 SD n2 M2 SD p d
(319)
t
n1 M1 SD n2 M2 SD p d
(366)
variables explained 64% of the total variance on math achievement (R² = .64,
F [4, 487] = 217.88, p < .05). The regression equation for ELL students and
FEP students are the following (see Table 4):
Y’1 (ELLs) = .41 reading + .26 language usage - .16 FARMs + .11 writing
Y’2 (FEPs) = -.38 FARMs + .32 reading + .17 writing + .11 language usage
As indicated by the findings, FARMs status was not only a significant
but also a strong predictor of math achievement for FEP students. FARMs
status was the strongest predictor of math achievement, followed by reading
skills. For ELL students, FARMs status was a statistically significant predictor
as well. It was, however, only the third strongest predictor for this group,
ranking behind reading and usage (see Table 4). Notably, reading was a stronger
predictor for ELL students than it was for FEP students. Correlations among
significant variables for ELL and FEP students can be found in Table 5.
Predictors B Beta t p
FEP students
math communication subskill (Wilks’s Λ = .95, p < .01 [see Figure 2]). Thus,
main effects were further analyzed. The MANOVA indicated that the overall
group difference among FARMs status students was not statistically
significant (Wilks’s Λ = .99, F [1, 366] = 1.78, p = .17 [see Table 6]). ELL
status explained only 1% of the variance associated with the dependent
variables (η² = .01). Among non-FARMs status students, however,
MANOVA results revealed a significant group difference (Wilks’s Λ = .71,
F [1, 319] = 66.16, p < .01 [see Table 7]). ELL status explained almost 30% of
the variance associated with the dependent variables, math and math
communication skills ( η² = .29). Table 8 reports descriptive statistics.
ELLs FEPs ELLs FEPs ELLs FEPs ELLs FEPs ELLs FEPs
Math 1.00 1.00 .61* .66* .52* .63* .56* .60* -.30* -.68*
560
560
540
540
520
520
500
500
480
0 1 0 1
FARMs FARMs
Figure 2. Interaction between free and reduced meals (FARMs) and English language
learner (ELL) status on math and the math communication subskill.
Note. “0” indicates non-FARMs group, and “1” indicates FARMs status. The broken
lines refer to the ELL students, the solid lines refer to the FEP students.
English language
learner status .99 1.78 2.00 365.00 .17 .01
Table 7
Summary of MANOVA of Non–Free and Reduced Meals
Students on Math and Communication Subskill
Effect Wilks's F Hypothesis Error p Eta2
Λ df df
English language
learner status .71 66.16 2.00 318.00 .00* .29
*p < .01.
Table 8
Group Means on Math and Communication Subskill of
Non–Free and Reduced Meals Students
Groups
n1 M SD n2 M SD
Educational Implications
Although this study is not comprehensive, its results illustrate a critical
aspect of how test formats could affect ELL students’ math achievement. The
American Educational Research Association (2000), on its Web site, expressed
its position regarding high-stakes testing by asserting that “appropriate
attention [should be given] to language difference among examinees” because
when the test scores of the ELL students are adversely affected by their
linguistic proficiencies, those scores cannot be considered an accurate
measurement of true ability.
Unfortunately, an assessment program created with good intentions can
jeopardize assessment equity for ELL students. Thus, policymakers must create
mechanisms that allow ELL students to be tested alternatively. One available
alternative, portfolio assessment, can show yearly progress and would free
schools and teachers to convert their energy from “teaching to the test”
toward helping students expand their knowledge.
In addition, implementing an assessment alternative such as portfolio
assessments would be the most meaningful way to include ELL students in
the accountability system. Portfolio assessments would help establish
accountability by allowing all ELL students to take part in the assessment
process, beginning from their first day of school. Then, achieving “adequate
yearly progress” would not be merely a federal mandate but a tangible and
meaningful goal for all stakeholders.
References
Abedi, J. (2004). The No Child Left Behind Act and English language
learners: Assessment and accountability issues. Educational Researcher,
33(1), 4–14.
Abedi, J., Hofstetter, C., & Lord, C. (2004). Assessment accommodations
for English language learners: Implications for policy-based empirical
research. Review of Educational Research, 74(1), 1–28.
Abedi, J., Leon, S., & Mirocha, J. (2003). Impact of student language back-
ground on content-based performance: Analyses of extant data (CSE
Tech. Rep. No. 603). Los Angeles: University of California, National
Center for Research on Evaluation, Standards, and Student Testing.
Abedi, J., & Lord, C. (2001). The language factor in mathematics tests.
Applied Measurement in Education, 14(3), 219–234.
Endnote
1
MSDE no longer uses the MSPAP to test students. The MSPAP did not
comply with the NCLB Act (2002) because it did not provide individual student
report cards. The MSDE developed a Maryland School Assessment that
consists of multiple-choice and constructed response items. The new test in
math, however, retains questions that require students to respond in writing,
in addition to multiple-choice items.
20. The table below shows how the chirping of a cricket is related to the
temperature outside. For example, a cricket chirps 144 times each minute
when the temperature is 76°.
144 76o
152 78o
160 80o
168 82o
176 84o
What would be the number of chirps per minute when the temperature
outside is 90° if this pattern stays the same?
Answer: ________________________
Explain how you figured out your answer.
________________________________________________
________________________________________________
________________________________________________
________________________________________________
________________________________________________
Step A
The zoo planner wants to have a small information center. They want to
cover the floors with tiles. Design a repeating pattern that could be used on
the floor in the information center. Show your work on the grid below.
Step B
Write a sentence or two explaining the pattern you chose.
Information Center
GO ON
________________________________________________
________________________________________________
________________________________________________
________________________________________________
________________________________________________
Note. This item is reconstructed based on the information available on the Maryland
State Department of Education Web site: http://www.mdk12.org/share/publicrelease/
plan_task.pdf