OMB should rethink research evaluation tool

EPA's research programs have suffered because of OMB's efficiency assessment tool.

A National Research Council (NRC) report released on January 31 recommends that the White House Office of Management and Budget (OMB) modify the way it evaluates the efficiency of federal agency research programs. The report suggests that OMB move away from its current management-by-the-numbers approach and rely more on evaluations based on outside expert review.

OMB uses a set of questions called the Program Assessment Rating Tool (PART) to evaluate research programs at the U.S. EPA, the National Science Foundation (NSF), and other federal agencies. EPA commissioned the NRC report in 2006 after years of having difficulty meeting OMB's requirements. "During my time, EPA's STAR [Science to Achieve Results] grant program for ecological research was cut due to a poor evaluation, and this led to congressional hearings," says Paul Gilman, former head of EPA's Office of Research and Development and a member of the NRC committee. The program was cut from $32 million in fiscal year 2002 (FY '02) to about $8 million in FY '05, according to NRC.

The report criticizes PART for focusing on ultimate outcomes—measuring lives saved, for example—instead of immediate outcomes, such as whether the research resulted in the dissemination of new tools or added to a body of knowledge. "Researchers can't be judged on ultimate outcomes," says Gilman, currently head of the Oak Ridge Center for Advanced Studies. Such outcomes depend on decisions made by program offices and by the states, all of which are beyond the control of researchers, he says.

The report also criticizes OMB for applying inconsistent standards to different agencies. "In an effort to satisfy OMB, EPA succumbed to using the number of papers each scientist publishes as a measure of efficiency," says Gilman. "That's really a crude measure, a lowest-common-denominator measure. But even then, OMB accepted this metric from some agencies and rejected it from EPA." In 2006, OMB faulted EPA's Water Quality Research Program for using publications to measure efficiency. In 2007, the PART Appeals Board ruled that EPA could use the publications provided that the water program develops an outcome-oriented efficiency measure in the future.

A representative from NASA told the NRC committee that PART examiners vary widely in attitude and experience. This strikes a chord with an EPA scientist, who says, "For PART, the [analysts with] MBAs arrive and expect us to tell them how we're making progress on something we said we'd spend money on 5 years ago. But with research, you follow leads and surprises. That doesn't correspond to the plan from 5 years ago." The scientist, who was not authorized to speak on the subject and therefore requested anonymity, added, "You can't really win with PART. If you do badly, you lose money. If you do great, you don't get any extra money. Your budget just stays the same."

The NRC report recommends that OMB evaluate research in two different ways. An expert panel should evaluate "investment efficiency", or how well an agency's R&D portfolio, including the budget, is relevant, is of high quality, matches the agency's strategic plan, and is adjusted as new knowledge and priorities emerge. For example, EPA currently uses its Board of Scientific Counselors and its Science Advisory Board to perform such expert reviews.

"Process efficiency" or traditional input–output issues, such as the people, funds, and facilities as well as the services and grants delivered, can be handled through numerical assessment. For example, NSF asks external committees to conduct a portfolio-level assessment, called a merit review, every 3 years. NSF also tracks efficiency by measuring the time taken to reach a decision on research awards—70% of applicants are informed within 6 months.

In testimony before the NRC committee, representatives from the Department of Energy and the National Institutes of Health (NIH) emphasized that the substantial time and costs of complying with PART take away from these organizations' primary purpose. NIH found that approximately 250 high-level staff worked full-time for 3 months to comply with PART for the NIH extramural program.

OMB has yet to fully evaluate the report, according to a spokesperson, who says that the findings will be reviewed and taken into consideration. —REBECCA RENNER