Organizational Research: Determining Appropriate Sample Size in Survey Research James E. Bartlett, H Joe W. Kotrlik Chadwick C. Higgins The determination of sample size i 2 common task for many organizational résearchers Unappropriate, accuracy. of research. This continuous and categeica! ‘lstsation of sample size Samui, smaller populations # included. excessive sample sizes continue to influence the quality and int describes the procedures for determining saimple size for vaciebles using Cochran's (1977) formulas. A discussion and luding the fonmula for adjusting the sample size for A table i provided that can be used to select the sample size for a research problem based on three alpha levels and a set error rate. Procedures for elermining the eneropriate sample sce for multiple repression and factor analysis, and common issues in sar Introduction A common goal of survey research is collect data representative ofa population. The researcher uses information gathered from the sarvey % generalize « findings from a drawn sample beck 0 2 p within the limits of redox 2 critiquing business education research, Wisch (1986) stated that “two ofthe mast consisont Saws included (1) distegerd for sempling error wi determining sample size, and 2) disregn fesponse and nonresponse bias” & 31) Within a quantitative sarvey deste, determining sample size and dealing wich nonresponse bies is essential. “One ofthe rec! advantages of quantitative methods i thet abi use smaller groups of peopie to make iferences about larger groups thet would b2 orcbibit expensive to stidy* Holton & Barnex. 167. hens, bow large 2 saxple is required to infer research findings beck population? Standard textbocit cuthers 2nd researchers offer tested methods that allow stacies mo ioke advantage of statistical meamarements, wich in tum glve researchers the upper hand i= information Tecinoloay Learning a ‘Journal, Vol. 79, 22 ceternination are examined, Non-espondent sampling issues determining the correct sample size. Sample size is one of the four interrelated features of a study design that can influence the detection of significant differences, relationships or interactions (Peers, 1996). Generally, these survey designs try to ‘minimize both alpha error (finding a difference that does not actually exist in the population) and beta error (failing to find a difference that actualy exists in the population) (Peers, 1996). However, improvement is needed. Researchers are learning experimental statistics from highly competent statisticians and then doing their best to apply the formulas and approaches James E. Bartlett, it's Assistant Professor, Department of Business Education and Office Administration, Ball Sate University. Muncie. Indiana, Joe W. Kotrlk fs Professor, School of Vocational Education, Lousiana State Univesity, Baton Rouge, Louisiana, Chadwick C. Higgins isa doctoral student: School ‘of Vocatlonal Education, Louisiana State Univesity, Baton Rouge, Lovsiana pring 2001 43they learn to their research design. A simple survej-of published manuscripts reveals numerous terrors and questionable approaches to sample size selection, and se: ves as proof that improvement is needed. Many researchers could benefit from 2 real-life primer on the tools needed to properly conduct research, including, but not limited to, sample size selection. ‘This manuscript will describe common procedures for determining sample size for simple random and systematic random samples. It will also discuss alternatives to these formulas for special situations. ‘This manuscript is not intended {o be a totally inclusive treatment of other sample size issues and techniques. Rather, this manuscript will address sample size issues that have been selected as 2 resull of observing problems in published manuscripts As a part of this discussion, considerations for the appropriate use of Cochran's (1977) sample size formula for both continuous and categorical data will be presented. Krejcie and Morgan's (1970) formula for determining sample size for categorical data will be briefly discussed because it provides identical sample sizes in all cases where the researcher adjusis the {value used based on population size, which is required when the population size is 120 oF Jess. Likewise, researchers should use caution when using any of the widely circulated sample size tables based on XKrejeie and Morgan's (1970) formula, as they assume an alpha of .05 and a degree of accuracy of 05 (discussed later). Other formulas are available; however, these two formulas are used more than any others. s Foundations for Sample Size Determination Primary Variables of Measurement ‘The researcher mus! make decisions as to which variables will be incorporated into formula calculations, For example, ifthe researcher plans (o use a sevenpoint scale to measure a continuous variable, e.g, job satisfaction, and also plans to determine if the respondents differ by certain categorical variables, e.g., gender, tenured, educaticnal level, etc., which variable(s) should be 132 - important because the use of gender asthe p researcher can simply use the largest nas the = Bartlett, Kotrlik, used as the basis for sample size? Thi variable will result in a substantially larger: size than if one tised the seven-point scale primary variable of measure. : Cochran (1977) addressed this issue by that “One method of determining sample specify margins of error forthe items that a regarded as most vital io the survey. Ane of the sample size needed is first made se for each of these important items” (p. 81). these calculations are completed, researchers have a range ofn's, usually ranging from small a's for scaled, continuous variables, to larger for dichotomous or categorical variables. ‘The researcher should make sampling Gecisions based on these data. If the n's for the: variables of interest are relatively close, the sample’size and be confident that the Sample size will provide the desired results. ‘More commonly, there isa sufficient variation among the n's so that we are reluctant to choose the largest, either from budgetary considerations or because this will give an overall standard of precision substantially higher than originally ‘contemplated. In this event, the desired ° standard’ of precision mey be relaxed for cerlain of the items, in order to permit the use ofa smaller value of n (Cochran, 1977, p. 81). ~The researcher may also decide to use this information in deciding whether to keep all ofthe: variables identified in the study. “In some cases. the 's are so discordant that certain of them must be dropped from the inquiry: ..." (Cochran, 1977, p. 81). Error Estimation, Cochran's (197) formula uses two key factors: (1) the risk the researcher is willing to accept in the study, commonly called the margin of error, or the error the researcher is willing to accept, and (2) the alpha level, the level of acceptable risk the researcher is willing to accept that the true marginOrganizational Research: Determining Appropriate Sample Size in Survey Research of error exceeds the acceptable mangin of error; ic., the probability that differences revealed by statistical analyses relly do not exist; also known as Type | error. Another type of error will not be addressed further here, namely, Type I error, also ‘now 2s bela error. Type Il error occurs when statistical procedures result in 2 judgment of no significant diferences shen these differences do indeed exist. ‘Alpha Level. The aipha level used in determining sample size in most educetional Tesearch studies is either .05 & Rezwvieh, 1996). In Cockren's formula, the alpha level is incorporate utilizing the tvalue (eg, tvalue for alpha level of 05 is 1.96 for sample sizes above 120). Researchers should ensure they use the correct + value when th ‘esearch involves smaller populations, e.g, tvalue for pha of 05 and 2 population of 60 2.00. In general, an alpha level of.05 is eccepteble for most research. An alp y used if the researcher is more interested in ‘dentifying marginal relationships, differences or other statistical pt rena 2 2 precursar tb Further studies. An zlpba level of 01 may be used in those cases where decisions based on the tesearch are critcel and errors may cause substantial financial or personal harze, e., major programmatic changes. Acceptable Margin of Erroc. The general rule relative to acceptable margins cf exer in educational and social research i 2s follows: For margin of ror is acceptable, categorical da and, for continuous date, 39% margin of error is acceptable (Krefcie & Morgan, 1970). ‘example, a 3% margin of ecror would resul fn the ‘thatthe true mean of @ (.03 times seven points onthe scale) ofthe meen calculated from the researcher being c seven point scale is wit Tesearchsample. For a dichotomous variable being confident who were mele was when a higher degree of precision is needle ii (Ary, Jacobs, 2 5% margin of error would result in the rescarcher the propertion of respondents ofthe proportion caleulated from the research sample. Researchers may increse these values when a higher matin of error acceptable or mey decrease these values Variance Estimation A caitical component of sample size formulasis the estimation of variance in the primary variables of interest in the study. The researcher does not have direct control over Variance and must incorporate variance estimates into research design. Cochran (1977) listed four ways of estimating population variances for sample size determinations: (1) take the sample in two steps, and use the results of the first step to determine how many additional responses are needed to altain an appropriate sample size based on the variance observed in the first step data; (2) use pilot study results; (3) use data from previous studies of the same or a similar Population; or (4) estimate or guess the structure of the population assisted by some logical mathematical results. The first three ways are logical and produce valid estimates of variance: therefore, they do not need to be discussed further. However, in many educational and social research ‘studies, itis not feasible to use-any of the frst three ways and the researcher must estimate variance using the fourth method. A researcher typically needs to estimate the variance of scaled and categorical variables, To ~ estimate the variance ofa scaled variable, one must ‘determine the inclusive range ofthe scale, and then divide by the number of standard deviations that would include all possible values in the rarge, and then square this number. For example, if a researcher used a seven-point scale and given that six standard deviations (three to each side of the ‘mean) would capture 98% of all responses, the calculations would be as follows: 7 (sumber of points on the scale) 6 (number of standard deviations) ‘When estimating the variance of a dichotcimous (proportional) variable such as gender. Krejeie and ‘Morgan (1970) recommended that researchers should use .50 as an estimate of the population Proportion, This proportion will result in the ‘maximization of variance, which will also produce the maximum sample size. ‘This proportion can be ‘used to estimate variance in. the population, For ‘example, squaring .50 will result in @ populationvariance esiimate of .25 for'a dichotomous variable. Basic Sample Size Determination Continuous Data Before proceeding with sample size calculations, assuming continuous data, the researcher should determine ifa categorical variable will play a primary role in data analysis. Ifso, the categorical sample size formulas should be used. If this is not the case, the sample size formulas for continuous data described in this section are appropriate. ‘Assume that a researcher has sel the alpha level a priori at .05, plans to ue a seven point scale, fia set the level of acceptable error al 3%, and has eslimated the standard deviation of the scale as 1.167. Cochran's sample size formula for continuous data and an example of its useis presented here along with the explanations as to how these decisions were made. (1.967(1.1677 a = 118 (r+.03 each tail = 1.96 (the alpha level of .05 indicates the level of risk the researcher is willing to-take that true margin of error may exceed the acceptable ‘margin of error) Where Sigeestimenmetaerdisincsnanengin the population = 1.167. (estimate of variance deviation for 7 point scale calculated by using 7 {inclusive range of scale) divided by 6 [number of standard deviations that include almost all (approximately 98%) of the possible values in the range)) Wiles € RReCaES SST REE RTT 2 being estimated = .2] (qurber of potnts on primary scale * acceptable margin of error, points on primary scale = 7: SCT argh of rors 8 researcher is willing to excepil). ‘Therefore, for a population of 1,679, the required sample:size is 118, However, since this 134, Bartlett, Kotrik,& Hagia sample size exceeds 5% of the population (1,879*.05=84), Cochran's (1977) correction formula should be used to calculate the final sample size. These calculations are as follows: me ais) B= 1 (+f Population) -- 1 1181679 ‘Where population size = 1,679. ‘Where mo = required return sample stze according to Cochran's formula= 118, ‘Where ni = required retum sample size because sample >°5% of population. ‘These procedures result in the minimum retumed sample size. Ifa roscarcher has a captive audience, this sample size may be attained easily. However, since many educational and social research studies often use data collection methods such as Surveys and other voluntary participation sethods, the response rates are typically well below 100%. Salkind (1997) recommended oversaripling when he stated that “If you are . Tpalling out surveys or questionnaires, .”.. . count ‘on increasing your sample size by 40%-50% to account for lost mail and uncooperative subjects” (p. 107). Fink (1995) stated that “Oversampling ‘can add costs to the survey but is often necessary” (p. 36). Cochran (1977) stated that "A second consequence i, of course, that the variances of estimates are increased because the sample actuaily obtained is smaller than the target sample.. This factor van be allowed for, at east approximately, in selecting the size of the sample” (p. 398). However, many researchers criticize the use of jovér‘sampling to ensure that this minimum sample size is achieved and suggestions on how to secure the minimal sample size are scarce. Ifthe researcher decides to use oversampling, four methods may be used to determine the anticipated response rate: (1) take the sample in two steps, and tise the results ofthe first step to estimate how.many additional responses may be ‘expected from the second step; (2) use pilot study Fesults; (3) use responses rates from previous studies of the same or a similar population; or (4) » gatimate the response rate. The first three ways are « ‘Mogical and will produce valid estimates of response. “aOrganizations Reszerch: Deternining Appropriate Sample Size in Survey Research rates; therefore, they do not need to be discussed. further, Estimating response rates is not an exact science. A researcher may be 2ble to consult other researchers or review the research literature in similar fields to determine the response rates that have been achieved with similar and, if necessary, ‘Based’ on prior recearch expenence. Chena “Tequired minimum semcie sie ecrrected) of 111, the following calcu the drawn semple sz minimum samp! Where anticipated return rote = 6556. ‘Where me = sample size adjusted for response rate. ‘Where migimum saxie sie (cecrected) = 111. ‘Therefore! me = 111/85 = 1712 Categorical Datz variations do exist. Assume 2 researcher has set the alpha level 2 price! 2x05 proportional varisble, basses the level of standard deviation of he sealeas S. Cochran's sample size formula for cotegorical date and an example ofits use is presented here along with ‘explanations 2s to how these decisions were made, (1.865) {the alpha lev! of the researcher's wiling wo take that true margin of ervr may exteed the acceptable margin of ero:). Where (p)(q) = esimate of variance = .25. “(1+ ne/ Population) (maximum possible proportion (§) * 1- maximum possible proportion (5) produces ‘maximum possible sample size) Where d = acceptable margin of error for proportion being estimated = .05 (error researcher is willing to except). ‘Therefore, for a population of 1,679, the required sample size is 384, Holvever, since this sample size exceeds 5% of the population (1,679*.05=84), Cochran's (1977) correction formula should be used to calculate the final sample size. These calculations are as follows: (384) 313 (a + 384/679) Where population size = 1.679 ‘Where no = required réturn sample size according to Cochran's formula= 384 Where mi = required return sample size because sample > 5% of population ‘These procedures result in @ minimum returned sample size of 313. Using the same oversampling procedures 25 cited in the continuous data example, and again assuming a response rate ‘of 65%, a minimum drawn sample size of 482 should be u8ed. These calculations were based on the following. Where anticipated return rate = 65% Where ne = sample size adjusted for response rate. Where minimum sample size (corrected) = 313 ‘Therefore, me = 313/.65 = 482 Sample Size Determination Table Table J presents sample size values that will be appropriate for many common sampling problems. ‘The table includes sample sizes for both continuous ‘end categorical data assuming aipha levels of .10. 05, or 01. The margins of ertor used in the table were .03 for continuous data and .05 forthe margin of error show appropriate Considerations Regression Analysis. Siwations exist where the procedures described in the previous paragraphs will not salisly the needs of a study and two examples will be addressed here. One situation is when the researcher wishes 'o use multiple regression analysis ina study. To use multiple regression analysis, the ratio of observations to independent variables should not fall below five. IF this rninimum is nat followed, there isa risk for overfittng, + making the results too ‘specific to the sample, thus lacking generalizability” (Hair, Anderson, Tatham, & Black, 1995, p. 105). A more Conservative ratio, of ten observations for each independent variable was reported optimal by Miller and Kunce (1973) and Helinski and Feldt (1970). ‘These rtias are especially critical in using regression analyses with continuous data because sample sizes for continuous data are typically ‘much smaller than sample sies for categorical data ‘Therefore, there is @ possibilty that the random sample will not be sufficient it multiple variables are used in the regression analysis. For example, in the continuous categorical data. Researchers may use this table if is appropriate lar their siudy; however, the appropriate sample size must bé calculated if these error rates are not Other Sample Size Determination ‘Table 1: Table for Determining Minimum Returned San Gata illustration, a population of 1 and it was determined that a migi ‘sample size of 11] was req the recommendations cited in the ph paragraph, uses both the five to one and Population Size for Continuous and Categarieal Data aple Size for 2 Grea Sample se Continuous daa (rargin of error=.03} Populatesi alpha 05 sie 11.96 wo [a6 $5 ofa 200-59 5 102 us fie Vase 30s 35 13 3 fies Yaar 40 [es 92 13 we [ise [250 so * [72 98 “7 we fas | 86 eo [73 100 155, wr [235 [316 os 102 181 195 zu + an 2076 104 165 vos [250 | 368 so 76 105 170 zoo favo ~ | 302, 1000 | 77 108 173 as [ze | 309 100 [79 10 183 zo [306 200 | a3 ne 19 EE 400 | 63 ne 198 24 | 351 600 | 83 na 208 250. | 362, 800 | 83 ng 209, ze [ser fis wooo | 83 ne 28 2a [aro | 62s NOTE: ‘The margins of error used inthe tble were 03 for continuous data and 06 for ‘categorical data. Researchers may use tis tbl fhe margin of eor shown s appropiate lor thelr study; however, the appropriate spl size must be calculated ithese ero rats. ‘are not appropriate, Table developed by Batt, Kol, Higorganizational Resex: determining Appropriate Sanple Size in Survey Research y “Table 2: Minisem Nenber of Regressors? Paes Tr Sepsag Emmet | Maxam number of | epee fetes Sample sas for [ser :]rows Comins jz [x Caogorcléere= 31s [se [at Asshi optimal ratio often to one with continuous data, the number of regressors (independent verizles) in the multiple ee model would be limited to 11. Larger numbers of regressors could be used with the other situations shown. It should be noted that if'a variable such 2s ethnicty is incorporated into the categorical exemple, this variable must be dummy coded, which wil resat in muitiple variables utilized in the model rater than 2 single variable. One variable for each etimic group. e,, White, Black, Hispanic, Asian, American Indian would each be coded as 1 =yes and 2. regréssion model, which would resalt in five variables rather than one in the regression model. In the continuses dete exemple, if2 researcher Planned to use 14 verizbles in a multiple regression ‘analysis and wished to use the optimal ratio of ten toone, the retumed seranle size mast be increased from 111 to 140. This sample size of 140 would be calculated from taking the number of independent varizbles to be entered in the regression (fourteen) and multiplying them by the ‘number ofthe ratio (ten). Caution shotld be used when making this decision beceuse raising the Sicated by the sample ease the probability of Type T the researcher plans to use factor analysis the seme ratio Considerations discussed! under maltinle regression should be used, with one additional criteria, namely, that fcter analysis should not be done with less than 100 observations. Itstiould be noted thal an increase in semple size will decrease the level at which an item Iceding on 2 factor is significant. For exemple, essoming en alpha level 137 ‘of .05, @ factor would have to load ata level of 75 + to be significant in a sample size of 50, while a factor would only have'to load at a level of 30 to be significant in a sample size of 350 (Hair etal, 1995). ‘Sampling nonrespondents. Donald (1967), Hagbert (1968). Johnson (1958), and Miller and ‘Smith (1983) recommend that the:researcher take a random sample of 10-20% of non-respondents to use in norerespondent follow-up analyses If non- respondents are treated as a potentially different population, it does not appear that this, recommendation is valid or adequate, Rather, the researcher could consider using Cochran's formula to determine an adequate sample of non- respondents for the nob-respandent followup response analyses. Budget time and other constraints. Often, the researcher is faced with various constraints that ray force them to use inadequate sample sizes because of practical versus statistical reasons. ‘These constraints may inchide budget, time, Personnel, and olher resource limitations. In these cases, researchers should report both the appropriate sample sizes along with the sample sizes actually used in the study, the reasons for using inadequate sample sizes, and a discussion of the effect the inadequate sample sizes may have on the results ofthe study. ‘The researcher should exercise caution when making programmatic recommendations based on research conducted with inadequate semple sizes, Final Theughts Although it s not unusual for researchers fo have different opinions as to how sample size should be calculated, the procedures used in this process should always be reported, allowing the reader lo make his or her own judgments as (0 whether they ‘accept the researcher's assumptions and procedures. In general, a researcher could use the ‘Standard factors identified in this paper in the sample size determination process. ‘Another issue is that many studies conducted with entire population census data could and probably should have used samples instéad. Many of the studies based on population census data achieve low response rates. Using an adequatesample alung with high quahty data collection elforts will result in more reliable, valid, and {generalizable results; it cuuld also result in other sesource savings. The bottom line is simple: research studies take substantial time and effort on the part of researchers. This paper was designed as a too} that a researcher could use in planning and conducting quality research. When selecting an appropriate sample size for a study Is relatively easy, why wouldn't a researcher want to do it right? References ‘Ay. D., Jacobs, L. C... & Razavieh, A. (1996). Jeteedcton to esearch in education Fort Werth, ‘TX: Harcourt Brace College Publishers. Cochran, W. C. (1977). Sampling techniques (3% ed). New York: John Wiley & Sons. Donald, M. N. (1967). Implications of non-response for {he interpretation of mail questionnaire data. Public Opinion Quantely, 241), 99-114. Fink, A. (1998). The survey handbook. Thousand Oaks, CCA: Sage Publications. Hagbert, £. C, {1966}. Valldty of questionnaire dat Reported and observed atlendance in an adult exlucation program. Public Opinion Quactery, 25, 493-456. : Dou Evert, J 138 ‘Material published as part of this journal, eiher online or in print, s copyrighted by the | Organizalional Systems Research Association, Permision to make digital oF paper copy of | ator all of these works for personal or classroom use is granted without fe provided that | the copies ae not made or distributed for profit or commercial advantage AND that copes 1) bcar this nvice in ful anc 2) give the Tull cation. Tis permissible to abstract these works so lug as credit i given. To copy in all other cases orto republish orto poston 9 sever or (a rodistrilate to lists requires specific permission and payment of a fee. Contact esen@norehead-s.edu to request redistribution permisin, Bartlett, Kotrlik, & Hagin Tatham, R., & Black, W. (1995). Multivariate data analysis (2 ed). Upper Saddle River, NJ: Prentice Hall. Halinsk, R. S. & Feldt, LS. (1970). The selection ot variables in multiple regression analyses. Journal of Eciucational Measurement, 7(3); 151-158. Holton, E.H., & Burnet, MB. (1997). Qualitative research methods. In R. A. Swanson, & E. F, Holton (Eds), Human resource development researc ‘handbook: Linking reseaich and practice. San Francisco: Berret-Koehler Publishes. Johnson, P. 0. (1958). Development of the sample survey aa sclentilerhethodology. Journal of Experiential Education, 27, 167-176. Krejee, R, V., & Morgan, D. W. (1970). Determining sample size for research activities. Educational a Peychological Measurement, 30, 607-610. Miller, D. E., & Kunce, J.T. (1973). Prediction and statistical overil revisited. Measurement and Evaluation in Guidance, (3), 187-163. Miller. L. E, & Smith, K. L. (1883). Handling * nonresponse issues. Joumal of Extension, 21, 45- +50 Peers, I. (1996). Statistical analysis for education and poychology researches. Bristol, PA: Falmer Press. Salling, N. J. (1997). Hsploning research (8 ed). Upper Saddle River, NJ: Prentice Hall ‘Wunsch, D. (1986): Survey research: Determining | sample size and representative cesponse. Business Education Forum, 40(5), 31-34. j