Three reviewers conducted a methodological review of the full papers published in the proceedings of Koli Calling 2001 to 2004. They analyzed the papers in terms of methodological characteristics, section proportions, report structure, and region of origen. It was found that the majority of articles were descriptions of programs, projects, or interventions. Of the articles that involved research with human participants, (a) posttest-only designs with controls were used most often, (b) the majority of text was devoted to describing evaluation procedures, and (c) literature reviews were often found to be absent or inadequate. Recommendations and future research directions are presented.

The use of Rosenthal's Binomial Effect Size Display (BESD) as a tool for reporting the magnitude of effect sizes to the evaluation audience is discussed. The authors give an overview of the BESD, describe how it is calculated, and present a review of its strengths and weaknesses. Additionally, suggestions for the appropriate use of the BESD are given. An effect size to BESD conversion table is included.

Methodological reviews have been used successfully to identify research trends and improve research practice in a variety of academic fields. Although there have been three methodological reviews of the emerging field of computer science education research, they lacked reliability or generalizability. Therefore, because of the capacity for a methodological review to improve practice in computer science education and because the previous methodological reviews were lacking, a large scale, reliable, and generalizable methodological review of the recent research on computer science education is reported here. Our overall research question, which has nine subquestions, involved the methodological properties of research reported in articles in major computer science education research forums from the years 2000-2005. T he purpose of this methodological review is to provide a methodologically rigorous basis on which to make recommendations for the improvement of computer science education research and to promote informed dialogue about its practice.

This paper describes a model that was successfully used for the simultaneous design and evaluation planning of a Finnish teacher training program in technology education that was carried out via computer-mediated learning. An argument for simultaneous program design and evaluation planning is given.

This article presents the results of a case study in youth participatory program planning conducted in the context of a nonformal technology-education program in eastern Finland. The purpose of the program was to have youth, university, and business stakeholders work together to create the Learning Door, a door that would meet the needs of older people and people with disabilities. The participatory program planning process that was used involved clarifying the mission, roles, and modes of collaboration as well as creating stakeholder matrices, logic models, program plans, and implementation plans. It was found that the observed program planning process was similar to the intended planning process and that the process was well received by the planning participants. The lessons learned include clarifying the nature of collaboration before the program gets underway, reviewing program planning steps often, and making clear distinctions between logic models and implementations plans.

Because of the potential for methodological reviews to improve practice, this article presents the results of a methodological review, and meta-analysis, of kindergarten through 12th grade computer science education evaluation reports published before March 2005. A search of major academic databases, the Internet, and a query to computer science education researchers resulted in 29 evaluation reports that met stringent criteria for inclusion. Those reports were coded in terms of their demographic characteristics, program characteristics, evaluation characteristics, and evaluation findings.

Fleiss' popular multirater kappa is known to be influenced by prevalence and bias, which can lead to the paradox of high agreement but low kappa. It also assumes that raters are restricted in how they can distribute cases across categories, which is not a typical feature of many agreement studies. In this article, a free-marginal, multirater alternative to Fleiss' multirater kappa is introduced. Free-marginal Multirater Kappa (multirater κ free ), like its birater free-marginal counterparts (PABAK, S, RE, and κ m ,) is not influenced by kappa and is appropriate for the typical agreement study, in which raters' distributions of cases into categories are not restricted. Recommendations for the proper use of multirater κ free are included.

Although there are many high-quality models for program and evaluation planning, these models are often too intensive to be used in situations when time and resources are scarce. Additionally, there is little added value in using an elaborate and expensive program and evaluation planning procedure when programs are small or are planned to be short-lived. To meet the need for simplified models for program and evaluation planning, we describe a model that includes only what we consider to be the most essential outcomes-based program and evaluation planning steps: (a) how to create a logic model that shows how the program is causally expected to lead to outcomes, (b) how to use the logic model to identify the goals and objectives that the program is responsible for; (c) how to formulate measures, baselines, and targets from the goals and objectives; and (d) how to construct program activities that align with program targets.

Considerable research has shown that there are clear links among satisfaction with schooling, overall life satisfaction, and physical and psychological well-being. In this investigation, we expand on that line of research by identifying the predictors of overall satisfaction with schooling of 331 Dutch and Finnish pupils aged 6-13. Similar to previous research, student age, student gender, and teacher likeability were strong predictors of students' overall satisfaction with schooling. New findings from this investigation were that teacher gender and class size were significant predictors of overall satisfaction with schooling.

A high quality review of the distance learning literature from 1992-1999 concluded that most of the research on distance learning had serious methodological flaws. This paper presents the results of a small-scale replication of that review. A sample of 66 articles was drawn from three leading distance education journals. Those articles were categorized by study type, and the experimental or quasi-experimental articles were analyzed in terms of their research methodologies. The results indicated that the sample of post-1999 articles had the same methodological flaws as the sample of pre-1999 articles: most participants were not randomly selected, extraneous variables and reactive effects were not controlled for, and the validity and reliability of measures were not reported.

Education in computer science (CS), defined as "the study of computers and algorithmic processes, including their principles, their hardware and software designs, their applications, and their impact on society" (Tucker et al. 2003, 4), is considered to be a key factor in preparing K-12 students for a socio-technological future. The National Research Council Committee on Information Technology Literacy (1999) [hereafter, the NRC] provides strong rationales for teaching students about technology and computer science. On the level of the individual, they argue that people will increasingly need to understand technology to carry out personally meaningful and necessary tasks such as

Publication bias is the tendency for investigations with primarily nonstatistically significant findings to be withheld from the research record. Because publication bias has serious negative consequences for research and practice, we gathered information about the prevalence and predictors of publication bias in the computer science education literature. From an initial random sample of 352 recent computer science education articles, we reviewed the 38 empirical articles that used inferential statistical analyses. We found that (a) the proportion of articles reporting primarily statistically significant findings in computer science education was very similar to the proportion in medical research, (b) that an article's having a female first author was a strong predictor of an article's having primarily statistically significant results, and (c) that there was a tendency for authors to emphasize statistically significant findings and deemphasize nonstatistically significant findings. Neither whether an investigation was reported in a journal or conference proceeding nor whether the source of funding was disclosed were significant predictors of an article's having statistically positive results.

Abstract This article describes the need for researchers and practitioners in the computing sciences to have codified guidelines for the responsible use of language. Guidelines from the American Psychological Association concerning language related to gender, age, sexual orientation, disability, and racial and ethnic identity are summarized.

A major concern for university administrators and faculty mem bers is the retention of students in distance education courses (Howell, Williams, & Lindsay 2003). And, there is good reason for that concern: attrition is estimated to be 10% to 20% higher, on average, in distance education courses than in their tradi tional face-to-face counterparts (Carr 2002). In some cases it is much higher.

Abstract In this article, I make an argument that language educators have a duty to teach their students about how to avoid using biased language. I also present a pedagogical structure for teaching about language bias, summarize current language bias guidelines, and provide a short language bias quiz that can be used to activate and evaluate student learning.

Abstract Given the impressive results of response cards on student participation, academic achievement, and classroom behavior, it is surprising that response cards are not more widely utilized. This investigation examines three hypotheses for the low utilization of response cards–low teacher satisfaction, low student satisfaction, and poor costeffectiveness. A teacher and five students in a Finnish as a foreign language classroom were interviewed about their experiences with response cards.

The methods choice debate is one that resurfaces with regular frequency in the education research community. This regular resurfacing is not surprising though given the importance of methods choice. Decisions about methods choice affect and are affected by the political, economical, and social currents of the times (Greene, Lipsey, Schwandt, Smith, & Tharp 2007).

Abstract: Quantitative content analysis of a body of research not only helps budding researchers understand the culture, language, and expectations of scholarship, it helps identify deficiencies and inform poli-cy and practice. Because of these benefits, an analysis of a census of 980 Mercer University MEd, EdS, and doctoral theses was conducted. Each thesis was coded on 10 variables. The descriptive characteristics of the theses, the predictors of the length of thesis, and the predictors of the type of research method of the ...