814545406.

xls

Diagnostic and Agreement St

This spreadsheet attempts to provide a comprehensive range of statistic
tables that are useful in evaluating diagnostic tests and inter-rater agreeme
Instructions
Only the shaded cells in the spreadsheet may be altered. The data to be e
arising from the cross-tabulation of the test being investigated against the
or absence of the disorder to which the test is supposedly sensitive. The c
as the “Gold Standard”.
As well as the data, the probability coverage of the confidence intervals
by entering a number between 0 and 1 (or a percentage between 0% and
next to the “Confidence Intervals” heading. For example, typing 0.90 i
90% confidence intervals.
Four Buttons control calculation:
Clear: zeroes all four cells in the table.
Calculate: Calculates statistics after data has been entered into cells.
T
slow machines and only functions if manual recalculation has been chosen
% and .0 toggle between percentage and decimal formats for appropriate s
Note that the spreadsheet will operate without the macros and these nee
concerned about viruses.
Statistics Produced
The majority of the statistics for diagnostic tests calculated by DAG_Stat
by Kraemer (1992). Where possible, the symbols she uses are employe
text is highly recommended as an introduction to this area and in
development of this spreadsheet. Inter-rater agreement and other statistics
of texts. The texts by Agresti (1993) and Fleiss (1981) are excellen
Cicchetti and Feinstein (1990) and Byrt, Bishop, and Carlin (1993) shou
who use Cohen’s kappa to assess interrater agreement.
Confidence intervals for indices that have a binomial distribution are ca
described by McKenzie et al. (1997). Other confidence intervals are co
means. References to each statistic are given in the pop-up comments tha
numbers refer to the list of citations below. Where more than one refere
reference to the coefficient while the second is a separate reference to its
reference exists, an asterisk appears. Examples of this are the predi
negative random tests, and the standard errors of Cicchetti and Feinstei
positive and negative agreement. These coefficients and formulae were

Citation Page 1
 814545406.xls

Citation

Registration
If you intend to use DAG_Stat, please register using the form on the w
purpose of registration is so that users can be contacted in the event th
spreadsheet. Registered users will also be informed if any improvements
References
1. Agresti, A. (1990). Categorical Data Analysis, New York, John Wiley
2. Armitage, P. & Berry G., (1994) Statistical Methods in Medical Resea
Blackwell.
3. Bishop, Y.M.M., Fienberg, S.E. & Holland, P.W. (1975) Discrete Mu
and Practice. Cambridge, Mass: MIT Press.
4. Brown, M.B. (1975) The asymptotic standard errors of some estimates
two-way contingency table. Psychometrika, 40, 291-295.
5. Brown, M. & Benedetti, J.K. (1977) Sampling behavior of tests for co
contingency tables. Journal of the American Statistical Association, 7
6. Byrt, T., Bishop, J. and Carlin, J.B. (1993). Bias, prevalence and kapp
Epidemiology, 46, 423-439.
7. Cicchetti, D.V. and Feinstein, A.R. (1990). High agreement but low k
paradoxes Journal of Clinical Epidemiology, 43, 551-558.
8. Digby, P.G.N. Approximating the tetrachoric correlation coefficient. B
9. Feinstein, A.R. Cicchetti, D.V. 1990 High agreement but low kappa: I
paradoxes Journal of Clinical Epidemiology, 43, 543-549
10. Fleiss, J.L., Cohen, J., & Everitt, B.S. Large sample standard errors of
kappa. Psychological Bulletin, 72, 323-327.
11. Fliess, J.L. (1981) Statistical Methods for Rates and Proportions. 2nd E
12. Kraemer, H.C. (1992). Evaluating Medical Tests. Newbury Park, CA:
13. Landis, J. R., & Koch, G. G. (1977). The measurement of observer ag
data. Biometrics, 33, 159-174.
14. McKenzie, D., Vida, S., Mackinnon, A.J., Onghena, P. and Clarke, D.
confidence intervals for measures of test performance. Psychiatry Res
Computer Requirements
Current versions of this spreadsheet should run on any version of Excel
Versions are available for Macintosh or Windows machines. If you wish t
note that some calculations are undertaken in invisible cells. Only modifi
other other cells must be unlocked before modification.
Copyright and Distribution
This work is copyright (2000) by Andrew J Mackinnon.
It may be distributed freely, but not resold for profit.
The author would be grateful for any corrections, suggestions or modifications to
Address
Andrew Mackinnon
Page 2
Mental Health Research Institute
Locked Bag 11
Parkville, Vic., 3052
 note that some calculations are undertaken in invisible cells. Only modifi
other other cells must be unlocked before modification.
Copyright and Distribution
This work is copyright814545406.xls
(2000) by Andrew J Mackinnon.
It may be distributed freely, but not resold for profit.
The author would be grateful for any corrections, suggestions or modifications to
Address
Andrew Mackinnon
Mental Health Research Institute
Locked Bag 11
Parkville, Vic., 3052
Australia
Tel: +61 3 388 1633
Fax : +61 3 387 5061

Page 3
 814545406.xls

greement Statistics
hensive range of statistics calculable from 2 by 2
ts and inter-rater agreement.

e altered. The data to be entered are the frequencies

g investigated against the criterion for the presence
pposedly sensitive. The criterion is often referred to

the confidence intervals calculated may be altered

rcentage between 0% and 100%) in the shaded cell
or example, typing 0.90 into this cell will produce

een entered into cells. This button is provided for

alculation has been chosen.
formats for appropriate statistics.
he macros and these need not be loaded if you are

calculated by DAG_Stat are defined and discussed

ols she uses are employed in the spreadsheet. This
on to this area and in many ways, spurred the
ement and other statistics are discussed in a number
eiss (1981) are excellent sources. The papers by
p, and Carlin (1993) should be read by researchers
ement.
nomial distribution are calculated using the method
onfidence intervals are constructed by conventional
he pop-up comments that describe them. Reference
here more than one reference is given, the first is a
a separate reference to its standard error. Where no
es of this are the predictive values positive and
of Cicchetti and Feinstein’s (1990) coefficients of
ients and formulae were developed specifically for

Page 4
 814545406.xls

using the form on the web site or by email. The

contacted in the event that an error is found in the
med if any improvements are made to the program.

is, New York, John Wiley & Sons.

Methods in Medical Research. 3rd

P.W. (1975) Discrete Multivariate Analysis: Theory

d errors of some estimates of unvertainty in the

40, 291-295.
ng behavior of tests for correlation in two-way
Statistical Association, 72, 309-315.
Bias, prevalence and kappa. Journal of Clinical

High agreement but low kappa: II. Resolving the

43, 551-558.
c correlation coefficient. Biometrics
greement but low kappa: I. The problems of two
43, 543-549
sample standard errors of kappa and weighted

tes and Proportions. 2nd Ed., New York: Wiley.

Tests. Newbury Park, CA:Sage.
asurement of observer agreement for categorical

nghena, P. and Clarke, D. (1997) Accurate

formance. Psychiatry Research, 69

on any version of Excel from Version 5 onwards.

ws machines. If you wish to modify the sheet, please
visible cells. Only modifiable cells are unlocked, so
cation.

estions or modifications to the spreadsheet.

Page 5
visible cells. Only modifiable cells are unlocked, so
cation.

. 814545406.xls

estions or modifications to the spreadsheet.

Page 6
 Index Principal Reference s.e.
Sensitivity Kraemer (1992) MacKenzie et al. (1997)
Sensitivity of a random test Kraemer (1992) —
Quality index of sensitivity Kraemer (1992) —
Specificity Kraemer (1992) MacKenzie et al. (1997)
Specificity of a random test Kraemer (1992) —
Quality index of specificity Kraemer (1992) —
Efficiency (Correct classification rate) Kraemer (1992) MacKenzie et al. (1997)
Efficiency of a random test Kraemer (1992) —
Quality index Kraemer (1992) —
Youden’s index Armitage & Berry (1994) *
Predictive value of positive test Kraemer (1992) MacKenzie et al. (1997)
Pred. value of a positive random test * —
Predictive value of negative test Kraemer (1992) MacKenzie et al. (1997)
Pred. value of a negative random test * —
False positive rate Kraemer (1992) MacKenzie et al. (1997)
False negative rate Kraemer (1992) MacKenzie et al. (1997)
Misclassification rate Kraemer (1992) MacKenzie et al. (1997)
Prevalence Kraemer (1992) MacKenzie et al. (1997)
Test level Kraemer (1992) MacKenzie et al. (1997)

Likelihood ratio of positive test Kraemer (1992)

Likelihood ratio of negative test Kraemer (1992)
Odds ratio Kraemer (1992)
Odds ratio (Haldane’s estimator) Agresti (1991) Agresti (1991)

Kappa and Related Indices

Cohen’s Kappa Agresti (1991)
Observed Agreement
Chance Agreement

Positive Agreement Cicchetti & Feinstein (1990) *

Negative Agreement Cicchetti & Feinstein (1990) *

Byrt’s Bias Index Byrt et al. (1993) —

Byrt’s Prevalence Asymmetry Index Byrt et al. (1993) —
Bias Adjusted Kappa Byrt et al. (1993) —
Prevalence & Bias Adjusted Kappa Byrt et al. (1993) —
Alternative Agreement Indices
Dice’s Index Agresti (1991) *
Gamma (Yule’s Q) Bishop et al. (1975) Bishop et al. (1975)
Phi Bishop et al. (1975) Bishop et al. (1975)
Scott’s agreement index —
Tetrachoric Correlation Digby (1983) Bishop et al. (1975)
Goodman & Kruskal’s tau (Crit. dep.) Agresti (1991)
Lambda(Symmetric) Bishop et al. (1975)
Lambda(Criterion dep.) Bishop et al. (1975)
Uncertainty Coefficient (Symmetric) Agresti (1991) Brown (1975)
Uncertainty Coeff. (Criterion dep.) Agresti (1991) Brown (1975)
Test
Pearson Chi Square Agresti (1991)
with Yate’s correction Agresti (1991)
Likelihood Ratio Chi Square Agresti (1991)

Test
McNemar’s Test Agresti (1991)
with Yate’s correction Agresti (1991)

* These indices or standard errors were developed by the author.

 Diagnostic & Agreement Statistics
Test Test
Positive Negative Positive Negative
Positive 473 81 554 Positive Row 85.38% Col. Row 14.62% Col.
95.56% 64.80% 89.35%
Criterion
Negative 22 44 66 Negative Row 33.33% Col. Row 66.67% Col.
4.44% 35.20% 10.65%

495 125 620 79.84% 20.16% N = 620

Confidence Intervals 95% No. Decimal Places 4

Index Symbol Estimate se Lower 95% CI Upper 95% CI Summary
Sensitivity SE 0.8538 0.0150 0.8216 0.8822 Sensitivity = 0.8538 (95% CI: 0.8216 - 0.8822)
Sensitivity of a random test SE_RAN 0.7984
Quality index of sensitivity k(1,0) 0.2748
Specificity SP 0.6667 0.0580 0.5399 0.7780 Specificity = 0.6667 (95% CI: 0.5399 - 0.7780)
Specificity of a random test SP_RAN 0.2016
Quality index of specificity k(0,0) 0.5825
Efficiency (Correct classification rate) EFF 0.8339 0.0149 0.8022 0.8623 Efficiency (Correct classification rate) = 0.8339 (95% CI: 0.8022 - 0.8623)
Efficiency of a random test EFF_RAN 0.7349
Quality index k(.5,0) 0.3734 0.0479 0.2795 0.4674### Quality index = 0.3734 (95% CI: 0.2795 - 0.4674)
Youden’s index J 0.5205 0.0599 0.4030 0.6379 Youden’s index = 0.5205 (95% CI: 0.4030 - 0.6379)
Predictive value of positive test PVP 0.9556 0.0093 0.9335 0.9719 Predictive value of positive test = 0.9556 (95% CI: 0.9335 - 0.9719)
Pred. value of a positive random test PVP_RAN 0.8935
Predictive value of negative test PVN 0.3520 0.0427 0.2687 0.4425 Predictive value of negative test = 0.3520 (95% CI: 0.2687 - 0.4425)
Pred. value of a negative random test PVN_RAN 0.1065
False positive rate FP 0.3333 0.0580 0.2220 0.4601 False positive rate = 0.3333 (95% CI: 0.2220 - 0.4601)
False negative rate FN 0.1462 0.0150 0.1178 0.1784 False negative rate = 0.1462 (95% CI: 0.1178 - 0.1784)
Misclassification rate 1-EFF 0.1661 0.0149 0.1377 0.2725 Misclassification rate = 0.1661 (95% CI: 0.1377 - 0.2725)
Prevalence P 0.8935 0.0124 0.8666 0.9167 Prevalence = 0.8935 (95% CI: 0.8666 - 0.9167)
Test level Q 0.7984 0.0161 0.7646 0.8293 Test level = 0.7984 (95% CI: 0.7646 - 0.8293)
Likelihood ratio of positive test LR+ 2.5614 1.1912 1.8178 3.6091 Likelihood ratio of positive test = 2.5614 (95% CI: 1.8178 - 3.6091)
Likelihood ratio of negative test LR- 4.5597 1.1441 3.5024 5.9361 Likelihood ratio of negative test = 4.5597 (95% CI: 3.5024 - 5.9361)
Odds ratio OR 11.6790 1.3331 6.6482 20.5166 Odds ratio = 11.6790 (95% CI: 6.6482 - 20.5166)
Odds ratio (Haldane’s estimator) OR' 11.4905 1.3299 6.5711 20.0929 Odds ratio (Haldane’s estimator) = 11.4905 (95% CI: 6.5711 - 20.0929)

Kappa and Related Indices

Cohen’s Kappa k 0.3734 0.0479 0.2795 0.4674 Cohen’s Kappa = 0.3734 (95% CI: 0.2795 - 0.4674)
This kappa indicates fair agreement. This kappa indicates fair agreement.
Test of Ho: Kappa=0: z=9.96, p =0.0000 t.t.t. Test of Ho: Kappa=0: z=9.96, p =0.0000 t.t.t.
Observed Agreement PO 0.8339 0.0149 0.8022 0.8623 Observed Agreement = 0.8339 (95% CI: 0.8022 - 0.8623)
Chance Agreement PE 0.7349 Chance Agreement = 0.7349 (95% CI: 0.0000 - 0.0000)
Positive Agreement PA 0.9018 0.0096 0.8829 0.9207 Positive Agreement = 0.9018 (95% CI: 0.8829 - 0.9207)
Negative Agreement NA 0.4607 0.0447 0.3730 0.5484 Negative Agreement = 0.4607 (95% CI: 0.3730 - 0.5484)
Byrt’s Bias Index BI 0.0952
Byrt’s Prevalence Asymmetry Index PI -0.6919
Bias Adjusted Kappa BAK 0.3625
Prevalence & Bias Adjusted Kappa PABAK 0.6677
Alternative Indices of Association
Dice’s Index p(s) 0.9018 0.0096 0.8829 0.9207 Dice’s Index = 0.9018 (95% CI: 0.8829 - 0.9207)
Yule’s Q (Gamma) g 0.8423 0.0418 0.7604 0.9241 Yule’s Q (Gamma) = 0.8423 (95% CI: 0.7604 - 0.9241)
Phi f 0.4001 0.0481 0.3059 0.4943 Phi = 0.4001 (95% CI: 0.3059 - 0.4943)
Scott’s agreement index p 0.3625 Scott’s agreement index = 0.3625
Tetrachoric Correlation r(t) 0.7576 0.0459 0.6676 0.8476 Tetrachoric Correlation = 0.7576 (95% CI: 0.6676 - 0.8476)
Goodman & Kruskal’s tau (Crit. dep.) t(asy) 0.1601 0.0385 0.0847 0.2355 Goodman & Kruskal’s tau (Crit. dep.) = 0.1601 (95% CI: 0.0847 - 0.2355)
Lambda(Symmetric) l(sym) 0.1152 0.0387 0.0394 0.1909 Lambda(Symmetric) = 0.1152 (95% CI: 0.0394 - 0.1909)
Lambda(Criterion dep.) l(asy) 0.0000 0.0000 0.0000 0.0000 Lambda(Criterion dep.) = 0.0000 (95% CI: 0.0000 - 0.0000)
Uncertainty Coefficient (Symmetric) U(sym) 0.1499 0.0339 0.0835 0.2163 Uncertainty Coefficient (Symmetric) = 0.1499 (95% CI: 0.0835 - 0.2163)
Uncertainty Coeff. (Criterion dep.) U(asy) 0.1861 0.0409 0.1060 0.2662 Uncertainty Coeff. (Criterion dep.) = 0.1861 (95% CI: 0.1060 - 0.2662)

Test Chi Square P

Pearson Chi Square 99.24 0.0000 Pearson Chi Square = 99.2431, df=1, p=0.000000
with Yate’s correction 96.04 0.0000 Pearson Chi Square with Yate’s correction = 96.0361, df=1, p=0.000000
Likelihood Ratio Chi Square 78.23 0.0000 Likelihood Ratio Chi Square = 78.2282, df=1, p=0.000000

Minimum Expected Frequency 13.31

Cells with Expected Frequency < 5 0 of 4 (0%)
Cells with Expected Frequency < 1 0 of 4 (0%)

Test Chi Square P

McNemar’s Test 33.80 0.0000 McNemar’s Test = 33.7961, df=1, p=0.000000
with Yate’s correction 32.66 0.0000 McNemar’s Test with Yate’s correction = 32.6602, df=1, p=0.000000