_______________________________________________________________________________
Title: Program Evaluation Issues
Subtitle:
Report No.: GAO/OCG-93-6TR Date: December 1992
_______________________________________________________________________________
Author: United States General Accounting Office
Office of the Comptroller General
Addressee: Transition Series
This file contains the text of a GAO report. Delineations within the text
indicating chapter titles, headings, and bullets are preserved. No attempt has
been made to display graphic images or pagination of the typeset report.
Footnotes appear in brackets at the reference point in the text. Underlined
text is indicated by underscore characters (_Introduction_). Superscript
characters are preceeded by a backslash (\a). Figures may be omitted or
replaced with tables. Tables may not resemble those in the printed version.
A printed copy of this report may be obtained from the GAO Documents
Distribution Facility by calling (202) 512-6000 or faxing your request
to (301) 258-4066 or writing to P.O. Box 6015, Gaithersburg, MD 20877.
_______________________________________________________________________________
CONTENTS
The Importance of Program Evaluation
Rebuilding Capacity
The Effects of Many Important Programs Are Unknown
- Education
- Health and Human Services
- Health and Human Services
- Defense
Some Agencies Are Poorly Informed About Program Targeting and Outreach
- Health and Human Services
- Education
- Agriculture
- Aging
Agencies Sometimes Rely Upon Flawed Studies and Ignore or Misuse Sound
Analyses
- Environmental Protection Agency
- Multiple Agencies
- Defense
- Agriculture
- Transportation
Some Promising Initiatives Do Exist
- Centers for Disease Control
- Agency for Health Care Policy and Research
- Office of Personnel Management and Merit Systems Protection Board
Related GAO Products
- Importance of Program Evaluation
- Program Effects Often Unknown
- Program Targeting Often Unknown
- Failure to Use Quality Studies
- Promising Evaluation Efforts
Transition Series
- Economics
- Management
- Program Areas
Office of the Comptroller General
Washington, DC 20548
December 1992
The Speaker of the House of Representatives
The Majority Leader of the Senate
In response to your request, this transition series report discusses a topic
that is critical to the effective oversight of government programs: the need
for sound, evaluative information on how programs are operating and what they
are actually accomplishing. This report, unlike our 1988 transition series
report on this topic, cites some examples of good work being done within
executive branch agencies. More generally, however, we feel that the attention
being paid to evaluation issues is inadequate either for managing programs
efficiently or for providing the Congress with the data necessary for informed
program oversight.
The GAO products upon which this report is based are listed at the end of the
report.
We are also sending copies of this report to the President-elect, the
Republican leadership of the Congress, the appropriate congressional
committees, and the designated heads of the appropriate agencies.
THE IMPORTANCE OF PROGRAM EVALUATION
-------------------------------------------------------------------------------
Over the next few years, the federal government will face powerful opposing
pressures: the need, on the one hand, to reduce the federal deficit, and the
demand, on the other, for a federal response to some potentially expensive
domestic problems (expanding health insurance, restoring the economy, and the
like). These pressures are likely to intensify concern with the effective
management of federal programs and with the availability of objective
information on the results of federal investments. In other words, are the
federal officials who administer programs adequately informed about the
implementation and the results of those investments? And can they, in turn,
adequately inform the President, the Congress, and the nation about what has
been accomplished?
Program evaluations contribute systematic information to federal
decision-making that has been useful in a variety of ways, such as the
following:
-- Recent welfare-to-work evaluations funded by the Department of Health and
Human Services showed a relationship between the increased employment of
program participants and savings to the federal and state governments
achieved through reduced welfare payments.
-- Evaluations of chemical weapons demonstrated gaps in the military's
capacity to manage and use these weapons and played a major role in the
termination of the Bigeye Bomb program and in the successful completion
of ongoing arms control negotiations on chemical warfare.
-- Evaluations conducted some years ago showed the effectiveness of the Job
Corps program in preparing disadvantaged young men and women for employment
and were a major contributor to the reauthorization of this expensive
intervention.
-- The Safe Medical Devices Act of 1990 included a number of provisions that
were the direct results of findings in a series of medical device
evaluations. Among other things, the act provides for increased recall
powers for the Food and Drug Administration and for improved
information reported to the Congress.
It is important to recognize that an objective and systematic evaluation
function not only serves to protect an agency against wasted resources in the
form of inefficient or ineffective programs. There may also be elements of
government programs that are in fact harmful to the well-being of some
segments of society--unintended effects that a well-conducted program
evaluation could prevent or detect. For example, an evaluation of the likely
impacts of proposed immigration reform legislation--suggesting that the
proposal would result in long waiting lists and delay the reunification of
families--led to appropriate changes in the bill before its enactment in 1990.
If the nation is to have strong, well-managed federal programs that can deal
efficiently and successfully with our domestic and international problems and
if the President and the Congress are to be adequately informed of progress in
meeting those challenges, the numbers and quality of the program evaluations
conducted by executive branch agencies must be improved.
In our 1988 transition series report, we found that there had been a
22-percent decline in the number of professional staff in agency program
evaluation units between 1980 and 1984. A follow-up study of 15 units that had
been active in 1980 showed an additional 12-percent decline in the number of
professional staff between 1984 and 1988. Funds for program evaluation also
dropped substantially between 1980 and 1984 (down by 37 percent in constant
1980 dollars). We have not repeated this survey, but discussions with the
departments and the Office of Management and Budget offer no indication that
the executive branch investment in program evaluation showed any meaningful
overall increase from 1988 to 1992.
Apparently, the effort to rebuild the government's evaluation capacity that we
called for in our 1988 transition series report has not been carried out. As
in 1988, executive branch agencies have often failed to conduct the program
evaluations that would provide officials with knowledge about the
effectiveness of their programs. As in 1988, the Congress continues to turn to
us and our sister agencies--the Congressional Budget Office, the Office of
Technology Assessment, and the Congressional Research Service--to do studies
that might more appropriately be conducted by executive branch agencies. It is
our mission to provide credible information to the Congress and to help ensure
that the reports the Congress receives are not limited to those from special
interest groups. However, we should not, and indeed cannot, do it all.
A first step in improving capacity is for agencies to review the adequacy of
their current funding for evaluation. Some agencies, like the Department of
Commerce and the Administration on Aging, devote few resources to evaluating
their programs. Other agencies--like the Department of Education and the
Public Health Service--dedicate major resources to evaluation. In these
agencies, the task may be less one of rebuilding overall capacity than of
strengthening areas of weakness. For example, the Department of Education
conducts many evaluations of its elementary and secondary programs, but the
investment at the postsecondary level is irregular in spite of some major
ongoing problems. Similarly, some Public Health Service agencies do a great
deal of evaluation, whereas others, like the National Institutes of Health,
spend only a small percentage of available funds for program evaluation.
Our evaluations help fill the gaps in the information available to the
Congress, but there is no substitute for a systematically planned, ongoing
effort for an agency to evaluate its own programs. The next sections
demonstrate that the limited capacity for program evaluation in the executive
branch has some important consequences:
-- agencies lack information on the effectiveness of their programs;
-- agencies lack data on the targeting and outreach of their programs; and
-- agencies need to improve their capacity to make sound decisions on the use
of data for policy-making.
We are, however, able to conclude with some examples of promising agency
initiatives.
THE EFFECTS OF MANY IMPORTANT PROGRAMS ARE UNKNOWN
-------------------------------------------------------------------------------
One of the most significant gaps in program evaluation information from the
executive branch concerns program effectiveness. Program effectiveness
evaluations show what, if anything, has changed as a result of implementing a
program. Limited budget dollars can be more concentrated among programs that
have demonstrated effectiveness, while programs with little evidence of
effectiveness can be cut or reformed and restructured.
Do participants in federally funded elementary and secondary education
programs for disadvantaged children show improvements in educational
achievement similar to that of other children? Are federal housing vouchers
shown to be effective in helping needy persons who would often not have
adequate housing without the vouchers?
In short, is there evidence of some concrete benefit that results from a
program that would not have occurred without the program? Program
effectiveness evaluations estimate the effects of federal programs using
statistical analysis of outcomes (such as educational achievement test scores
or condition of housing) for groups of persons receiving program services
compared with similar groups of nonparticipants.
Our response to congressional requests over the last 4 years has yielded the
following information on the effects of federal programs.
Despite increased attention in recent years to removing the barriers that
prevent the full involvement of persons with disabilities in work and other
activities, the Department of Education has not evaluated the effectiveness of
its $1.8 billion-per-year program of vocational rehabilitation. Our evaluation
using confidential income tax data showed only very modest overall long-term
gains in earned income, in contrast to the dramatic short-term employment
effects often cited by the program. The Congress has strengthened evaluation
in the recent reauthorization of the program. The Department of Education
should conduct such studies and help establish the overall impact of the
legislation by identifying, for example, how people with certain disabilities
have been helped by the program while those with other disabilities may need
different assistance than the program has provided.
===============================================================================
HEALTH AND HUMAN SERVICES
The Department of Health and Human Services regularly makes public recognition
awards to "promising" drug abuse prevention programs on the basis of reviews
that required no hard evidence of program effectiveness. The problem here is
that people in other communities could base new programs upon weak models
while other, more effective programs go unrecognized. In response to our
report, the agency agreed to begin seeking such evidence.
===============================================================================
HEALTH AND HUMAN SERVICES
To correct perceived widespread abuses of foster care, federal reforms were
enacted in 1980 to ensure that the necessity and appropriateness of each
foster care placement was periodically reviewed and that families received
needed services. A 1989 evaluation found that these reforms had not been
completely carried out and that no national evaluations had been performed.
This evaluation gap means that children placed in foster care may still be
unnecessarily at risk of abuses such as needless delays in returning them to
their natural parents.
We evaluated the Department of Defense's (DOD) methods for selecting recruits
for training in technical occupational specialties and for assessing the
effectiveness of the training. DOD's major selection instrument, the Armed
Services Vocational Aptitude Battery, which has been extensively researched
over the years, was moderately successful in predicting classroom performance
in these more demanding training courses. But in most cases, testing of actual
field performance--the end point of the training program--was either
nonexistent or inadequate, making it impossible to evaluate the overall
effectiveness of the services' training program. Thus, much of the investment
that the Department had made in program evaluation was in this case
inefficient because the final loop--showing which training programs produce
the top performers and thus make the best use of DOD's human resources--had
not been closed.
SOME AGENCIES ARE POORLY INFORMED ABOUT PROGRAM TARGETING AND OUTREACH
-------------------------------------------------------------------------------
A second type of program evaluation that is useful--but often not available
from federal agencies--concerns questions of program implementation. Agencies
should evaluate different aspects of program implementation, such as the
proper targeting of programs and their outreach--that is, whether they reach
some or all of those eligible.
Analysis of program targeting demonstrates how well the program is reaching
its intended recipients (such as determining the extent to which Chapter 1
aid, under the Elementary and Secondary Education Act of 1965, is targeted for
and reaches educationally disadvantaged children). Studies of outreach examine
participation in federal programs.
Such evaluations help agencies understand why their outreach may not be
successful and what barriers may need to be overcome before participation can
increase. There are important gaps in agencies' knowledge of the targeting and
outreach of their programs.
The limited executive branch evaluations of targeting and program outreach led
to congressional requests for us to do work such as the following.
===============================================================================
HEALTH AND HUMAN SERVICES
The National Cancer Institute is responsible for disseminating information on
treatments proven to be effective in the treatment of cancer in experimental
situations. An evaluation revealed various blockages in the processes that the
Institute used to move breakthrough therapies from clinical trials to the
patient. Recommended treatments had not been adopted for many patients in the
samples we studied. Consequently, these patients had not received what the
Institute considered state-of-the-art treatment.
Federal student aid is especially intended to help persons of modest means
gain access to postsecondary education, but sound information on school costs
and the availability of aid has to reach potential recipients in time for them
to make crucial decisions about their higher education plans. An evaluation
documented both unawareness and incorrect understanding of the program that
could have significant effects on the decisions students and parents make. The
Congress acted on these findings in revising student aid laws in 1992 to
require the Department to make improved information available and to evaluate
its impact.
We found in reviewing the Department of Agriculture's Food Stamp program that
less than half of the households eligible for food stamps participated in the
program in the mid-1980s. Evaluators found evidence of a variety of outreach
problems. About half of the eligible nonparticipants incorrectly thought that
they were ineligible. Almost two-thirds of the eligible nonparticipants cited
either a lack of information or program barriers, including administrative
"hassles," as the reason for their nonparticipation. These findings suggest
that the program should be changed to make food stamps more available to
eligible persons who need them.
The Older Americans Act mandates that in the provision of services, the
Administration on Aging should target older individuals with the greatest
economic or social needs and give particular attention to low-income minority
individuals. A program evaluation found that the data collection instrument
and methodology used by the agency did not permit the generation of accurate
counts of all participants, including targeted populations, in mandated
programs and services. The data are thus inadequate to answer fundamental
congressional questions about the degree to which agency programs target
resources to persons with the greatest economic or social needs.
AGENCIES SOMETIMES RELY UPON FLAWED STUDIES AND IGNORE OR MISUSE SOUND
ANALYSES
-------------------------------------------------------------------------------
In 1992, we found some new consequences of the deterioration of the program
evaluation capacity of the federal government in addition to the absence of
information. In some cases, agencies _have_ conducted evaluation studies, but
the information produced is either flawed or improperly used for policy
purposes. The studies may be based upon problematic data or analysis, or they
may be properly conducted but ignored or misused in the formulation of policy.
These problems suggest that management improvements are necessary. Agencies
need to review the quality of their data and research more carefully and
better integrate the findings from program evaluations and other analyses into
agency decision-making.
Examples of inappropriate use--or neglect--of evaluation evidence by federal
agencies in recent years include the following.
An analysis of the shared responsibilities of the Environmental Protection
Agency (EPA) and state environmental agencies for managing a national program
of hazardous waste found that important information gaps remain. Problematic
measurement and data collection procedures limit the quality of some of the
information produced, and the biennial reporting system still does not ensure
that the states will collect or report to EPA all necessary data. The result
is that it is not possible to determine whether the hazardous waste reduction
goals are
eing met.
We conducted a comprehensive evaluation of four executive branch
agencies--EPA, the Food and Drug Administration, the Occupational Safety and
Health Administration, and the Consumer Product Safety Commission--to
determine how well they were protecting the public from exposure to
reproductive and developmental toxicants. There were major gaps in the
evaluative information available to these agencies. First, because no accepted
federal list of these toxins has been developed, such as that mandated for
carcinogens, these agencies have had no index of the most important hazards to
reproduction and development. Second, risk assessment for these toxicants has
been based upon a flawed threshold assumption: that is, that there is a
specific dose level below which no problems occur. However, well-known
hazards, such as lead and radiation, are dangerous at any dose level.
Therefore, it appears that current standards for exposure to lead and
radiation could be resulting in more developmental problems in children and
reproductive problems in adults than would occur under an alternative,
nonthreshold approach.
Many unproven assumptions weaken DOD's decision-making in important national
security areas. Our evaluation of the U.S. strategic triad found several major
examples of assumptions that we found to be either inaccurate or unsupported
by available data. In some areas, such as the threat posed by the former
Soviet Union, the assumptions grossly _over_stated what the data actually
support. In other areas--for example, the performance of weapon
systems--available data instead show _under_stated assumptions. In still other
cases, such as specific assertions made by officials, no supporting data were
available. Over the past 30 years, DOD has not conducted any comprehensive
evaluation of the strategic nuclear triad. The lack of realistic assessments
of the threat and the lack of rigorous analysis of the relative performance
and merit of the weapon systems has resulted in the questionable development
and procurement of multiple costly modernization programs.
A series of studies examined the accuracy of various price, production, and
supply forecasts made by the U.S. Department of Agriculture. Long-term
commodity forecasts had large and systematic error rates over the period
1981-88. These errors contributed to a significant underestimate of the
commodity program outlay estimates that were made in the President's 1990
budget submission to the Congress. Accurate forecasts are also important for
administering such agency programs as acreage reductions and export
enhancements. The Department agreed with recommendations to improve
forecasting and set up a process to identify, report, and correct errors when
they occur.
We reviewed the quality and completeness of the analytical efforts supporting
the testimony of the National Highway Traffic Safety Administration regarding
the impact of continued requirements for automobile downsizing upon highway
safety. The agency's finding that more than 1,300 fatalities each year can be
attributed to the automobile weight reduction efforts that began in the 1970s
was not supported by available data. Instead, our analyses showed that the
automobile weight reductions have had virtually no net effect on total highway
fatalities. On the one hand, the very lightest cars have higher fatality rates
than the very heaviest. On the other hand, the decreased number of heavy cars
on the highways accounts for much of the total car weight reduction that
occurred during the time period, diminishing the danger to the occupants of
other cars.
SOME PROMISING INITIATIVES DO EXIST
-------------------------------------------------------------------------------
This year, in contrast to 1988, we identified a number of agency evaluations
that were well done or work in progress that seems promising.
===============================================================================
CENTERS FOR DISEASE CONTROL
The Congress requested an independent review of the investigation by the
Centers for Disease Control (CDC) into the causes of the HIV infections found
among patients of a Florida dentist. We concluded that CDC's
research--especially that on genetic sequencing--was well done and our review
supports CDC's conclusions that five patients became infected as a result of
receiving care from the dentist with AIDS, although the mode of transmission
remains uncertain. This review suggests that CDC made an exemplary effort to
find and use the best information available at the time.
===============================================================================
AGENCY FOR HEALTH CARE POLICY AND RESEARCH
We reported that despite considerable changes in the management of breast
cancer treatment since the 1970s, there was no observable improvement in
survival. That we were the only organization to examine nationally how well
cancer patients fared underscored the fact that there was no federal agency
charged specifically to examine health outcomes. That situation changed in
1989 with the establishment of the Agency for Health Care Policy and Research.
Currently, this agency has within its mandate the broad areas of determining
the effectiveness of medical interventions, creating medical practice
guidelines, and disseminating information on outcomes. This offers the promise
of advances in the quality and quantity of information that will be available
for making informed, future health care policy decisions.
===============================================================================
OFFICE OF PERSONNEL MANAGEMENT AND MERIT SYSTEMS PROTECTION BOARD
In response to concerns in the Congress and among such groups as the National
Commission on the Public Service that the quality of the federal professional
and technical workforce was declining in the 1980s, we examined the available
data and found no significant evaluation of workforce quality either in the
major agencies or in the Office of Personnel Management (OPM). However, in
response to our 1988 report, OPM and the Merit Systems Protection Board have
in the last 4 years developed significant evaluation programs involving
measurements of quality among those recruited and retained in a number of key
occupations, and an expanded effort is being planned on the basis of a
national advisory panel's work.
