Demystifying the NIH proposal review process.

Introduction

Victoria J. Molfese, Ph.D., holds the Ashland/ Nystrand Chair in Early Childhood Education in the University of Louisville's Center for Research in Early Childhood. Prior to this appointment, Dr. Molfese served as Director of the Office of Research Development and Administration at Southern Illinois University for 13 years and was a member of the Psychology faculty. She was elected President of the Society of Research Administrators International in 1998 and has held SRA's Distinguished Faculty designation since 2002.

Dr. Molfese's research focus is on how children learn and factors that influence learning, such as children's home environments and family background characteristics; how schools, teachers, and curriculum influence learning in preschoolers; and how to assess evidence of learning in children from infancy through early elementary grades. Her research has been supported by grants from private and government agencies, including the March of Dimes, the National Institutes of Health (NIH), and the U.S. Department of Education's Institute of Education Science (IES).

She has served as a peer reviewer on several study sections, including review panels for the IES, NIH, NIMH (National Institute of Mental Health), the International Dyslexia Association, the National Foundation/March of Dimes, Networks of Centers of Excellence of Canada, the Ontario Mental Health Foundation, and the U.S. Department of Agriculture.

In this article, Dr. Molfese answers a series of questions developed by the co-authors to reveal the interpersonal dynamics of an NIH study section and the nitty-gritty details of how an NIH proposal is reviewed. Dr. Molfese's "voice of experience" as an active researcher and distinguished research administrator provides a candid insider's view of this often mysterious process.

Question: You have served as a principal investigator (PI) as well as a research administrator (RA). How do these two roles differ?

Answer: PIs are in charge of conducting a research project over which they usually have quite a bit of control--after all, they designed the project and now have a chance to conduct it using grant funds. While all projects involve unexpected events--nothing ever turns out as perfectly as we think it will--most projects tend to involve components that the PI has done before (possibly on a smaller scale) and, therefore, most of the components are familiar. An RA depends on others--PIs or prospective PIs - to set job duties in motion. Because there appear to be endless variations (or variants!) of PIs, RAs encounter projects that often are completely unfamiliar to them. Even projects that could be familiar have PIs who put their personal spin on the project, which tends to make the familiar once again unfamiliar. RA and PIs have to learn to work together with their different motivations. The RA wants to get the proposal submitted to the agency with all compliance issues resolved in plenty of time to make the deadline, while the PI wants to continually rewrite the proposal until the last possible minute to get it perfect before the submission deadline, with no worries about compliance issues.

Question: When did you begin reviewing proposals for the NIH?

Answer: I began reviewing in 1994, at the suggestion of a friend. I had been asking questions about how people became reviewers and learned that a person can ask NIH to consider them as a possible reviewer. So, I sent my vita and a letter of interest to my program officer, and he contacted me to be part of a standing study section dealing with projects related to childhood development.

Question: How did you get selected to serve as a peer reviewer for NIH?

Answer: After the initial time I requested to be a reviewer, I found that I was asked by other agencies and other branches of the same agency to be a reviewer for their proposals. Clearly, there is a mechanism by which reviewers are "shared" by program officers and scientific review administrators (the people who lead the proposal review sessions). One program officer told me that she had worked with a federal grants program where the program officers provided a written evaluation of each reviewer, and that served as a basis by which people were asked to be or not to be reviewers in the future. Imagine that--reviewers being reviewed! This may be why some people appear frequently as reviewers, but another explanation is that people with grants and people who have strong research credentials in a particular field (based on publications and reputation) often are asked to be reviewers.

Question: What were your specific responsibilities as a peer reviewer?

Answer: In each case, I was given as many as 10 proposals to read and a set of guidelines against which to produce a written review. Most agencies have fairly similar guidelines--they ask about the significance of the project (how will the project contribute to the solution of a problem that agency is interested in or move a field of science forward?), the soundness and feasibility of the study design, the ability of the PI and other key project personnel to execute the project, the reasonableness of the budget, and the extent to which human subject protections are in place. Usually proposals are mailed to the "primary" reviewers (usually there are 2 to 3 per proposal) with between 2 to 4 weeks before the study section is scheduled to meet. The primary reviewers are assigned to proposals by the scientific review administrator of the study section based upon the expertise needed to cover certain aspects of the proposal (e.g., content, methodology, or statistics). Before the scheduled meeting, written reviews and preliminary scores for each proposal have to be posted to a website by each of the primary reviewers, At the scheduled meeting, primary reviewers each summarize their reviews of each proposal and discuss its strengths and weaknesses. After the discussion, all the reviewers (those assigned as primary reviewers of a proposal and other reviewers on the panel) discuss the proposal and provide an overall vote on the proposal.

Question: Do NIH peer reviewers get paid to review proposals? How much? If not, why do people do it?

Answer: Yes, they do get paid. NIH pays $200 to 300 per day plus a fixed per diem rate, and transportation; they cover hotel nights directly. IES pays $2,000 for the review, certain per diem costs, and transportation, and they also cover hotel nights directly. The pay does not offset the time-consuming and pressured nature of the review process. However, the nonmonetary payoff is that reading proposals can be interesting and educational, plus the other reviewers are very interesting to get to know.

Question: Where does the peer review take place?

Answer: Review meetings are almost always held in Washington, DC for federal agencies. Occasionally, when agencies conduct site visits to large program grants, the review takes place where the program grant is sited (e.g., New York City). But site visits occur rarely now (too expensive) and agencies usually have review sessions where they are located (usually Washington, DC). Sometimes reviewers don't have to be physically present at the review--they can do a phone review. This is done when the reviewer can't come or only has 1 or 2 proposals to review. I have also done a review in an airport hotel, which was very convenient for a large group of reviewers and program people flying in from different cities.

Question: How many proposals are you required to review?

Answer: As many as they send you! On average, I receive around 10 proposals, but occasionally I only have 1 or 2 (and then I can usually do a phone review). Sometimes, with triage in which the proposals with poor scores are not discussed, I will only have a few proposals that are actually discussed at the review session, but I still have to provide the full written review and evaluations for all the proposals. (Note: A triage occurs when a proposal is reviewed by the 2 to 3 "assigned" reviewers with a preliminary numerical score of >2.5, on a 1 best to 5 worst scale. Proposals with a score greater than 2.5 by the primary reviewers are rarely if ever competitive for funding, and therefore, they are not discussed during the limited study section meeting time. Those proposals do receive the full written comments by the reviewers. If the assigned or other study section reviewers request it, a >2.5 proposal may still be discussed, but I have never seen one receive a competitive score after the discussion. Usually there is at least one fatal or near- fatal flaw with the proposal.)

Question: How many peer reviewers are there for each round of proposals?