CH 4 Reliability
CH 4 Reliability
CH 4 Reliability
Name 1. 5+3 2. 7+2 3. 6+3 4. 9+1 5. 8+6 6. 7+5 7. 4+7 8. 9+2 9. 8+4 10. 5+6
Selam 1 1 1 1 1 1 1 1 1 1
Mulugeta 1 0 0 1 0 0 1 1 0 1
Linda 1 0 1 0 0 1 1 1 1 0
Lois 1 0 1 1 1 0 0 1 0 0
Ayuba 0 0 0 0 0 1 1 0 1 1
Andrea 0 1 1 1 1 1 1 1 1 1
Thomas 0 1 1 1 1 1 1 1 1 1
Anna 0 0 1 1 0 1 1 0 1 0
Amos 0 1 1 1 1 1 1 1 1 1
Martha 0 0 1 1 0 1 0 1 1 1
Sabina 0 0 1 1 0 0 0 0 0 1
Augustine 1 1 0 0 0 1 0 0 1 1
Priscilla 1 1 1 1 1 1 1 1 1 1
Tunde 0 1 1 1 0 0 0 0 1 0
Daniel 0 1 1 1 1 1 1 1 1 1
Solution
r =
KR20 ( )(
k
k-1
1–
Σpq
σ2 )
k = 10
Name 1. 5+3 2. 7+2 3. 6+3 4. 9+1 5. 8+6 6. 7+5 7. 4+7 8. 9+2 9. 8+4 10. 5+6
Selam 1 1 1 1 1 1 1 1 1 1
Mulugeta 1 0 0 1 0 0 1 1 0 1
Linda 1 0 1 0 0 1 1 1 1 0
Lois 1 0 1 1 1 0 0 1 0 0
Ayuba 0 0 0 0 0 1 1 0 1 1
Andrea 0 1 1 1 1 1 1 1 1 1
Thomas 0 1 1 1 1 1 1 1 1 1
Anna 0 0 1 1 0 1 1 0 1 0
Amos 0 1 1 1 1 1 1 1 1 1
Martha 0 0 1 1 0 1 0 1 1 1
Sabina 0 0 1 1 0 0 0 0 0 1
Augustine 1 1 0 0 0 1 0 0 1 1
Priscilla 1 1 1 1 1 1 1 1 1 1
Tunde 0 1 1 1 0 0 0 0 1 0
Daniel 0 1 1 1 1 1 1 1 1 1
Number of
1's 6 8 12 12 7 11 10 10 12 11
Proportion
Passed (p) 0.40 0.53 0.80 0.80 0.47 0.73 0.67 0.67 0.80 0.73
To calculate the proportion of the sample who answered the item Second, divided the number of students who
correctly, first counted the number of 1’s for each item. This gives the answered the item correctly by the number of
total number of students who answered the item correctly. students who took the test, 15 in this case.
• Next we need to calculate q for each item, the proportion
of the sample who answered each item incorrectly.
• Since students either passed or failed each item, the sum p
and q is 1 i.e p + q = 1.
Name 1. 5+3 2. 7+2 3. 6+3 4. 9+1 5. 8+6 6. 7+5 7. 4+7 8. 9+2 9. 8+4 10. 5+6
Number of 1's 6 8 12 12 7 11 10 10 12 11
Proportion Passed (p) 0.40 0.53 0.80 0.80 0.47 0.73 0.67 0.67 0.80 0.73
Proportion Failed (q) 0.60 0.47 0.20 0.20 0.53 0.27 0.33 0.33 0.20 0.27
I calculated the percentage who failed You will get the same answer if you
by the formula 1 – p, or 1 minus the count up the number of 0’s for each
proportion who passed the item. item and then divide by 15.
• Now that we have p and q for each item, multiply p by q for
each item.
Student Math Problem
Name 1. 5+3 2. 7+2 3. 6+3 4. 9+1 5. 8+6 6. 7+5 7. 4+7 8. 9+2 9. 8+4 10. 5+6
Number of 1's 6 8 12 12 7 11 10 10 12 11
Proportion Passed (p) 0.40 0.53 0.80 0.80 0.47 0.73 0.67 0.67 0.80 0.73
Proportion Failed (q) 0.60 0.47 0.20 0.20 0.53 0.27 0.33 0.33 0.20 0.27
pxq 0.24 0.25 0.16 0.16 0.25 0.20 0.22 0.22 0.16 0.20
10.
Name 1. 5+3 2. 7+2 3. 6+3 4. 9+1 5. 8+6 6. 7+5 7. 4+7 8. 9+2 9. 8+4 5+6 Total Exam Score
selam 1 1 1 1 1 1 1 1 1 1 10
mulugeta 1 0 0 1 0 0 1 1 0 1 5
Linda 1 0 1 0 0 1 1 1 1 0 6
Lois 1 0 1 1 1 0 0 1 0 0 5
Ayuba 0 0 0 0 0 1 1 0 1 1 4
Andrea 0 1 1 1 1 1 1 1 1 1 9
Thomas 0 1 1 1 1 1 1 1 1 1 9
Anna 0 0 1 1 0 1 1 0 1 0 5
Amos 0 1 1 1 1 1 1 1 1 1 9
Martha 0 0 1 1 0 1 0 1 1 1 6
Sabina 0 0 1 1 0 0 0 0 0 1 3
Augustine 1 1 0 0 0 1 0 0 1 1 5
Priscilla 1 1 1 1 1 1 1 1 1 1 10
Tunde 0 1 1 1 0 0 0 0 1 0 4
Daniel 0 1 1 1 1 1 1 1 1 1 9
Calculation of variance
Student Score Exam score Average Deviation Deviation sqr
Selam 10 6.6 3.4 11.56
Mulugeta 5 6.6 -1.6 2.56
Linda 6 6.6 -0.6 0.36
Lois 5 6.6 -1.6 2.56
Ayuba 4 6.6 -2.6 6.76
Andrea 9 6.6 2.4 5.76
Thomas 9 6.6 2.4 5.76
Anna 5 6.6 -1.6 2.56
Amos 9 6.6 2.4 5.76
Martha 6 6.6 -0.6 0.36
Sabina 3 6.6 -3.6 12.96
Augustine 5 6.6 -1.6 2.56
Priscilla 10 6.6 3.4 11.56
Tunde 4 6.6 -2.6 6.76
Daniel 9 6.6 2.4 5.76
Total=99 Total=83.6
Average=6.6
variance= sum of deviation sqr/n
variance =83.6/15=5.57
r =
KR20 ( )(
k
k-1
1–
Σpq
σ2 )
k = 10
Σpq = 2.05
σ2 = 5.57
r =
KR20 ( )(
10
10 - 1
1–
2.05
5.57 )
rKR20 = 1.11 * 0.63
rKR20 = 0.702
Split-Half Reliability - Spearman Brown formula
• Used for likert scale or another measure that does not have just
one correct answer
• The preferable statistic to calculate the split-half reliability is
coefficient alpha (otherwise called Cronbach’s alpha).
• However, coefficient alpha is difficult to calculate by hand.
Hence we can use the Spearman Brown formula instead of alpha
coefficient because it is much easier to calculate.
Spearman-Brown Formula
2rxy
r =
SB 1 + rxy
where rxy = Pearson correlation of
scores in the two half tests.
Example: self-esteem scale
A 10-item scale that measures a participant’s self-esteem at a given point in time. It is
designed to measure what you are thinking at this moment.
•NB: All items are answered using a 5-point scale (1= not at all, 2= a little bit, 3=
somewhat, 4= very much, 5= extremely).
1. I feel confident about my abilities
2. I am worried about whether I am regarded as a success or failure
3. I feel satisfied with the way my body looks right now.
4. I feel frustrated or rattled about my performance.
5. I feel that I am having trouble understanding things that I read.
6. I feel that others respect and admire me.
7. I am dissatisfied with my weight.
8. I feel self-conscious.
9. I feel as smart as others.
10. I feel displeased with myself
Assume that Fourteen participants took the test.
Questionnaire Item Number
Students 1 2 3 4 5 6 7 8 9 10
1 5 4 1 4 3 4 5 4 5 4
2 5 3 5 3 5 3 4 2 4 4
3 5 3 5 3 5 3 4 2 4 4
4 5 3 3 3 4 3 4 4 5 3
5 5 3 3 3 4 3 4 4 5 3
6 5 5 1 2 3 3 3 3 3 1
7 5 4 3 3 4 4 4 3 4 3
8 5 5 3 4 5 3 4 4 4 3
9 5 3 2 3 5 5 4 5 5 4
10 5 3 3 4 4 4 4 4 4 4
11 4 4 3 4 5 4 3 4 4 4
12 4 3 1 3 4 3 3 3 5 3
13 5 3 2 4 3 5 3 3 4 3
14 3 2 1 3 3 2 3 4 4 4
• The first step is to split the questions into half. The recommended
procedure is to assign every other item to one half of the test.
– If you simply take the first half of the items, the participants may
have become tired at the end of the questionnaire and the reliability
estimate will be artificially lower.
1 Half 2 Half
S/No 1 2 3 4 5 Total 6 7 8 9 10 Total
1 5 4 1 4 3 17 4 5 4 5 4 22
2 5 3 5 3 5 21 3 4 2 4 4 17
3 5 3 5 3 5 21 3 4 2 4 4 17
4 5 3 3 3 4 18 3 4 4 5 3 19
5 5 3 3 3 4 18 3 4 4 5 3 19
6 5 5 1 2 3 16 3 3 3 3 1 13
7 5 4 3 3 4 19 4 4 3 4 3 18
8 5 5 3 4 5 22 3 4 4 4 3 18
9 5 3 2 3 5 18 5 4 5 5 4 23
10 5 3 3 4 4 19 4 4 4 4 4 20
11 4 4 3 4 5 20 4 3 4 4 4 19
12 4 3 1 3 4 15 3 3 3 5 3 17
13 5 3 2 4 3 17 5 3 3 4 3 18
14 3 2 1 3 3 12 2 3 4 4 4 17
The second half total was calculated by The first half total was calculated by adding
adding up the scores for the first 5 items up the scores for last 5 items
• Now that we have our two halves of the test, we have to calculate the
Pearson Product-Moment Correlation between them.
Σ(X – X) (Y – Y)
rxy = √[Σ (X – X) ] [ΣY – Y) ]
2 2
Where ,
X = one person’s score on the first half of items
X = the mean score on the first half of items
Y = one person’s score on the second half of items
Y = the mean score on the second half of items.
Σ(X – X) (Y – Y)
rxy = √[Σ(X – X) ] [Σ(Y – Y) ]
2 2
1 Half 2 Half
S/No Total Total X-X Y-Y (X - X)(Y - Y)
1 17 22 -1.1 3.6 -3.96
Next we multiply (X – X) times (Y – Y).
2 21 17 2.9 -1.4 -4.06
3 21 17 2.9 -1.4 -4.06
4 18 19 -0.1 0.6 -0.06
5 18 19 -0.1 0.6 -0.06
6 16 13 -2.1 -5.4 11.34
7 19 18 0.9 -0.4 -0.36
8 22 18 3.9 -0.4 -1.56
9 18 23 -0.1 4.6 -0.46
10 19 20 0.9 1.6 1.44
11 20 19 1.9 0.6 1.14
12 15 17 -3.1 -1.4 4.34 After we have multiplied, we
13 17 18 -1.1 -0.4 0.44 sum up the products.
Σ(X – X) (Y – Y) = 12.66
√[Σ(X – X)2] [Σ(Y – Y)2] = 82.72
12.66
rxy = 82.72
rxy = 0.15
2rxy
r =
SB
1 + rxy
2 * 0.15
rSB = 1 + 0.15
0.3
rSB = 1.15
rSB = 0.26 The measure did not have good
reliability in my sample!
The steps for conducting Spearman-Brown in SPSS
1. The data is entered in a within-subjects fashion.
2. Click Analyze.
3. Drag the cursor over the Scale drop-down menu.
4. Click on Reliability Analysis.
5. Click on the first item to highlight it.
6. Click on the arrow to move the item into the Items: box.
7. Repeat Steps 5 and 6 until all of the survey items are in
the Items: box.
8. Click on the Statistics button.
9. In the Descriptive for table, click on the Item, Scale, and Scale if
item deleted boxes to select them.
10. In the mode, select split half
11. Click Continue.
12. Click OK.
Spss Result
Reliability Statistics
Total N of Items 10
xy x y
r n
( x) 2
( y)
2
x
2 . y
2
n n
Solution
Test 1st Test 2nd
Student ID (x) (y) XY X^2 Y^2
1 75 74 5550 5625 5476
2 50 53 2650 2500 2809
3 93 94 8742 8649 8836
4 80 79 6320 6400 6241
5 67 69 4623 4489 4761
6 88 89 7832 7744 7921
7 56 54 3024 3136 2916
8 71 72 5112 5041 5184
9 66 65 4290 4356 4225
Sum 646 649 48143 47940 48369
xy n
x y
r
x2
( x)
n
2
. y
2
(
n
y) 2
R=.993**
Steps for conducting test-retest reliability in SPSS
1.The data is entered in a within-subjects fashion.
2. Click Analyze.
3. Drag the cursor over the Correlate drop-down menu.
4.Click on Bivariate.
5.Click on the baseline observation, pre-test administration, or survey
score to highlight it.
6.Click on the arrow to move the variable into the Variables: box.
7.Click on the second observation, post-test administration, or survey
score to highlight it
8. Click on the arrow to move the variable into the Variables: box.
9.Click OK.
Spss output
Total scores for English, forms A and B are listed for each participant, as are
total scores for math's, forms A and B.
• calculate the Pearson's product moment correlation between English A and
English B. This is the parallel forms reliability coefficient for English.
• calculate the Pearson's product moment correlation between math's A and
math's B. This is the parallel forms reliability coefficient for math's.
Inter-Rater Reliability
1–41
Consider the African Geography quiz below:
1. The lake known by nick name “golden lake” is :
a. Tana b. ziway c. Langano
Is it reliable? – if you don’t
2. The most populated Ethiopian city is: know the answers, are you
a. Hawassa b. Adama c. AA going to suddenly know
them tomorrow or next
3. The largest region in Ethiopia is:
week? Probably not. We’ll
a. Oromia
say it’s reliable
b. Tigray
c. Amhara
4. The highest mountain is Ethiopia is:
a. Mt. Ras Dashen
b. Mt. Tulu Dimtu
c. Mt. Batu Is it valid?
This doesn’t look like a
5. The largest river in Ethiopia is measure of Africa
a. Abay Geography to me, it
focuses on Ethiopia . It’s
b. Awash not a valid measure of
c. Wabe shebelle African Geography.
Types of validity
Face validity
Content validity
Criterion-related validity
•Predictive validity and
•Concurrent validity
Construct validity
•Convergent
•Divergent validation
Face Validity