1. What is Policy Evaluation?

What is Policy Evaluation?

By Shane Hall, eHow Contributor

Print this article

Public policy making does not end with the passage of legislation and the implementation of programs authorized by the new law. The next questions are whether the initiative achieved its objectives, what the effects were and whether any policy changes are needed. Policy evaluation answers these and related questions.

Other People Are Reading

How to Evaluate Policies & Procedures How to Evaluate Public Policy

1. Identification
o

Policy evaluation is a systematic process for assessing the design, implementation and outcomes of public policies. Evaluation uses social science research methods, including qualitative and quantitative techniques, to examine the effects of policies. Some policy scholars, such as political scientist James Anderson, describe policy making as a sequential process marked by distinct steps, such as agenda-setting, policy formulation, adoption and implementation. For Anderson and others, evaluation is the final step in this process. However, they caution that the public policy process is ongoing, with evaluation often resulting in policy changes, which are then implemented and evaluated again.

Function
o

Policy evaluation enables all participants in the policy process, including legislators, executives, agency officials and others, to measure the degree to which a program has achieved its goals, assess the effects and identify any needed changes to a policy. In addition, many state and local governments fund programs with federal grants, which have evaluation requirements.

Sponsored Links Life Insurance USD7.5million in Life Cover & years of stability for you & your family. www.jwseagon.com

History
o

As early as the 1930s, social scientists advocated the use of empirical research methods to assess the effects and consequences of government policies and programs. Policy evaluation grew as an activity after World War II because the growth in the number of programs boosted the need for evaluations. Over time, policy analysis and evaluation became established fields of study.

Types
o

The two main types of policy evaluation are formative and summative evaluation. Formative evaluation examines the operations of the program, usually for the purpose of improving the program and assessing its implementation. For example, a formative evaluation of a tutoring program would consider such measures as the number of tutors, the number of students who participated, enrollment procedures and the amount of tutoring students received. Summative evaluation asks whether the program achieved its intended goals. If the tutoring program's goal was to raise student test scores in math, a summative evaluation would include an analysis of math scores for students who participated. Often, the best policy evaluations employ a comprehensive approach that uses both formative and summative techniques.

Considerations
o

Policy evaluation is rarely as simple or straightforward as some politicians suggest. Factors that complicate evaluations include identifying goals, measuring performance and isolating the effects of policy from those of other factors. In addition, although it attempts to assess policy in an objective manner, evaluation activities occur within a political environment. Policy-makers often want immediate information on policy effects, but many programs have long-term effects that will not be known in the short term.

References

James Anderson, Public Policymaking, 1994

Evaluation: A Systematic Approach, Peter Rossi, Howard Freeman, and Mark Lipsey, 1999.

Cart | Account | Help

Home Literature Writing Foreign Languages Math Science More Subjects Test Prep College Cliffs Films Shop

Home > More Subjects > American Government > The Policymaking Process The Constitution Federalism Congress The President The Judiciary The Bureaucracy Public Opinion The Mass Media Political Parties Voting and Elections

Interest Groups Civil Liberties Civil Rights Public Policy

The Policymaking Process Politics and Policymaking Policymaking in Action

Economic Policy Foreign Policy Related Topics: Criminal Justice U.S. History I U.S. History II 41

The Policymaking Process

Public policy refers to the actions taken by government its decisions that are intended to solve problems and improve the quality of life for its citizens. At the federal level, public policies are enacted to regulate industry and business, to protect citizens at home and abroad, to aid state and city governments and people such as the poor through funding programs, and to encourage social goals.

A policy established and carried out by the government goes through several stages from inception to conclusion. These are agenda building, formulation, adoption, implementation, evaluation, and termination.

Agenda building
Before a policy can be created, a problem must exist that is called to the attention of the government. Illegal immigration, for example, has been going on for many years, but it was not until the 1990s that enough people considered it such a serious problem that it required increased government action. Another example is crime. American society tolerates a certain level of crime; however, when crime rises dramatically or is perceived to be rising dramatically, it becomes an issue for policymakers to address. Specific events can place a problem on the

agenda. The flooding of a town near a river raises the question of whether homes should be allowed to be built in a floodplain. New legislation on combating terrorism (the USA Patriot Act, for example) was a response to the attacks of September 11, 2001.

Formulation and adoption

Policy formulation means coming up with an approach to solving a problem. Congress, the executive branch, the courts, and interest groups may be involved. Contradictory proposals are often made. The president may have one approach to immigration reform, and the oppositionparty members of Congress may have another. Policy formulation has a tangible outcome: A bill goes before Congress or a regulatory agency drafts proposed rules. The process continues with adoption. A policy is adopted when Congress passes legislation, the regulations become final, or the Supreme Court renders a decision in a case.

Implementation
The implementation or carrying out of policy is most often accomplished by institutions other than those that formulated and adopted it. A statute usually provides just a broad outline of a policy. For example, Congress may mandate improved water quality standards, but the Environmental Protection Agency (EPA) provides the details on those standards and the procedures for measuring compliance through regulations. As noted earlier, the Supreme Court has no mechanism to enforce its decisions; other branches of government must implement its determinations. Successful implementation depends on the complexity of the policy, coordination between those putting the policy into effect, and compliance. The Supreme Court's decision in Brown v. Board of Education is a good example. The justices realized that desegregation was a complex issue; however, they did not provide any guidance on how to implement it "with all deliberate speed." Here, implementation depended upon the close scrutiny of circuit and appeals court judges, as well as local and state school board members who were often reluctant to push social change.

Evaluation and termination

Evaluation means determining how well a policy is working, and it is not an easy task. People inside and outside of government typically use cost-benefit analysis to try to find the answer. In other words, if the government is spending x billions of dollars on this policy, are the benefits derived from it worth the expenditure? Cost-benefit analysis is based on hard-to-come-by data that are subject to different, and sometimes contradictory, interpretations. History has shown that once implemented, policies are difficult to terminate. When they are terminated, it is usually because the policy became obsolete, clearly did not work, or lost its support among the interest groups and elected officials that placed it on the agenda in the first place. In 1974, for example, Congress enacted a national speed limit of 55 miles per hour. It was effective in reducing highway fatalities and gasoline consumption. On the other hand, the law

increased costs for the trucking industry and was widely viewed as an unwarranted federal intrusion into an area that belonged to the states to regulate. The law was repealed in 1987.

What is Policy Evaluation? The consequences of such policy programs are determined by describing their impacts, or by looking at whether they have succeeded or failed according to a set of established standards. Several evaluation perspectives are:

Evaluation is the assessment of whether a set of activities implemented under a specific policy has achieved a given set of objectives. Evaluation is the effort that renders a judgment about program quality. Evaluation is information gathering for the purposes of making decisions about the future of the program. Evaluation is the use of scientific methods to determine how successful implementation and its outcomes have been.

(Theodoulou and Kofinis, 2004, p. 192)

page 8 of 17

Four Types of Policy Evaluation

Most models of policy evaluation ground their analytical perspective in the logical process used to determine the disparity between what was conceptualized by the initial policy goals and what has actually been accomplished by the policy or program as implemented. However, many other models focus their analysis on different objectives such as what is the true purpose of the evaluation, what is the role of the evaluator in the process, how broad or narrow should the scope of the evaluation be, and finally how should the evaluation be organized and conducted, for example what measurement instruments should be employed to determine success or failure. Theodoulou and Kofinis (2004) identify four generic types of the most commonly used policy evaluation typologies and they are: process evaluation, outcome evaluation, impact evaluation, and cost-benefit analysis. Types of Policy Evaluation

Process Evaluation Outcome Evaluation Impact Evaluation Cost-Benefit Evaluation

(Theodoulou and Kofinis, 2004, pp. 193-194)

Process Evaluation:

As its name implies this type of evaluation analyzes how well a policy or program is being administered. This type of evaluation is employed more often by program managers to determine what can be done to improve the implementation, the aspects of service delivery, of the program. It does not directly address whether or not the policy or program is achieving the desired outcome or impact on the target population. Types of Policy Evaluation: Process Evaluation

Determine why a program or policy is performing at current levels. Identify any problems. Develop solutions to the problems. Improve program performance by recommending how solutions should be implemented and evaluated once carried out.

(Theodoulou and Kofinis, 2004, pp. 193-194)

Outcome Evaluation:

Theodoulou and Kofinis somewhat confuse the concepts of outputs and outcomes in their discussion of outcome evaluation. Outputs are measures of government activity such as the number of tax returns processed or the number of social security checks sent out each month. Whereas outcomes are normally considered to be the impact that a policy has on a target population, for example, did the policy produce the desired behavioral change initially sought. In this case, the authors state that outcome evaluation is concerned with outputs. For example, if the stated goal of a welfare policy is to reduce the number of people receiving welfare benefits then a determination is made to see if less people are receiving welfare benefits after program implementation then before. However, what this type of evaluation does not indicate is what happened to the people who used to receive the welfare benefits and who have been forced off the system because they are no longer eligible. Did they find employment? Did they find other means of charity? Have they migrated to a life of crime? Outcome evaluation as described by Theodoulou and Kofinis focuses more on the readily available and tangible results of policy. The actual impact of the policy is the subject of the next type of evaluation. Types of Policy Evaluation: Outcome Evaluation

Legislative intent Program goals Program elements and indicators Measures of indicators Program outcomes (positive or negative)

(Theodoulou and Kofinis, 2004, p. 194)

page 9 of 17 Impact Evaluation:

This type of evaluation is what is more commonly perceived as a policy evaluation. The objective of this type of evaluation is to determine whether or not a given public policy or program is in fact achieving the intended impact as visualized by the various policy actors who either supported or opposed the given policy. Using the welfare example once again as discussed above this type of evaluation would answer the question what happened to the people who used to receive the welfare benefits and who have been forced off the system because they are no longer eligible. Did they find employment? Did they find other means of charity? Have they migrated to a life of crime? In comparison [with outcome evaluation], impact evaluation is concerned with assessing whether the target population is being affected in any way by the introduction and implementation of the policy. There is also concern with the impact of the program on the original problem being addressedfor it is important for both [policylevel managers and policy designers] to ascertain whether target populations are appropriately receiving delivery of a program (Theodoulou and Kofinis, 2004, 194). Types of Policy Evaluation: Impact Evaluation

Theoretical goals of the program or policy. The actual goals. Program or policy objectives. Program or policy results and whether they are intended or unintended, positive or negative, in effect.

(Theodoulou and Kofinis, 2004, p. 194)

Cost-Benefit Analysis:

Simply stated, a cost-benefit analysis is the comparison of the costs associated with a policy or program to the benefits generated by the policy. Continuing with the welfare example cited above in the two previous definitions, the tangible cost of a new welfare policy with a goal to reduce the number of welfare recipients could be accurately evaluated to include agency operating costs and the actual monetary cost of the benefits that are provided to welfare recipients. However, the cost-benefit analysis technique is controversial because it is extremely difficult to calculate the intangible costs such as those borne by those individuals who are no longer eligible to receive welfare benefits as a result of the new policy to restrict the number of people on the welfare roles. Additionally, the intangible costs borne by society such as the potential for increased crime or increased public health care costs or the long-term loss of

employment opportunity and participation in the market place by those individuals who may have been able to pull themselves out of poverty if they were still eligible for welfare assistance are extremely difficult to calculate. All too often the cost-benefit analysis technique is used because actual real-world costs are easy to obtain, quantify, evaluate, and contrast against a variety of metrics or other policies or programs. Unfortunately, many intangible benefits, such as the advantages gleaned by a well-educated society, may not be readily visible for many years to come, and some intangible benefits are impossible to quantify such as the quality of life. Policy evaluators must constantly be aware that the costs and benefits used in any evaluation may not accurately, if at all, represent the real impact of a given policy or program. Instead, a cost-benefit analysis should be employed as one of several methods used to determine the efficacy or efficiency of government action. Types of Policy Evaluation: Cost-Benefit Analysis

A method with which to evaluate and assess the effectiveness of a policy's costs, benefits, and outcomes For certain types of programs, such as education or the environment, one could argue that the real benefits do not materialize for years or decades.

(Theodoulou and Kofinis, 2004, p. 194)

page 10 of 17

Seven Important Barriers to Effective Policy Evaluation

As increasing pressures are brought to bear on the public sector to perform its role more effectively and efficiently, evaluation will probably become an even greater source of conflict. Negative evaluations of a programs effectiveness and efficiency now will be more likely to lead to the programs termination than in more affluent times. The content of an evaluation, the values that are contained in it, and event he organization performing the evaluation will all affect the final assessment. Evaluation research is now a major industry involving numerous consulting firms (beltway bandits), universities, and organizations within government itself. These evaluative organizations will have their own perspectives on what is right and wrong in policy and will bring those values with them when they perform an analysis (Peters, 2007, p. 175). As such, Peters identifies seven important barriers to effective policy evaluation that can impede the seemingly simple but operationally difficult process of determining what actually occurred as a result of government action and more specifically determining the level of performance of a public policy of program. It is equally important for both policy evaluators and the consumers of policy evaluation to understand the basic nature of these difficulties. Specifically, anyone engaged in policy evaluation must appreciate the degree of bias and unintentional, albeit sometimes intentional, confusion that can occur as a result of the difference between what is being measured during a policy evaluation and what policy-makers thought the policy should achieve. All too often evaluators, administrators, and the public focus on measurement statistics

that are easy to obtain but have no real relationship between what has been accomplished compared to what the original intent of the public policy was. The following text block identifies the seven barriers as enumerated by Peters and it is followed by a brief description of each. Barriers to Effective Policy Evaluation
1. 2. 3. 4. 5. 6. 7. Goal Specification and Goal Change Measurement Targets Efficiency and Effectiveness Values and Evaluation Politics Increasing Requirements for Evaluation