Statistical Research Grants from the NIH
This column highlights research activities that may be of interest to ASA members. This article includes information about new research solicitations and the federal budget for statistics. Comments or suggestions for future articles may be sent to the Amstat News managing editor at email@example.com.
Contributing Editor Jeremy M G Taylor is a professor of biostatistics at the University of Michigan. He completed his education at Cambridge University and the University of California at Berkeley. He is co-editor of Biometrics.
Most statisticians with faculty positions at universities and research organizations expect to undertake original research. While it may be possible to achieve this with funds from your own institution, obtaining funds from external sources almost certainly will enhance your research. In addition, such funding is likely to be an important element in criteria for promotion.
There are many governmental and other organizations that can provide support for statistical research. While the National Science Foundation is probably the most common source of support for more theoretical research, the National Institutes of Health (NIH) is by far the biggest supporter of more applied methodologic research motivated by applications in health, medicine, and biomedical science. The purpose of this article is to provide an overview of how statistics grant applications are generally reviewed at the NIH and guidelines for preparing a grant application.
Assistant professors who are engaged in biomedical research tend to start thinking about writing grant applications 2–3 years after their first appointment. At first, they may partner with more senior investigators and act as a co-investigator on someone else’s grant application, either a statistical methodology grant or a nonstatistically focused scientific project. This is an important activity for statisticians and a useful learning experience, but the focus here is statistical research grants from the NIH as a principal investigator.
After a humble start in 1887 as a one-room laboratory of hygiene, today’s National Institutes of Health (NIH) comprises 27 institutes and centers. Among the larger institutes are Cancer; Allergy and Infectious Diseases; and Heart, Lung and Blood. Most NIH offices are headquartered just outside Washington, DC, in Bethesda and Rockville, Maryland.
The NIH supports biomedical research through both intra- and extramural programs. Intramural research is that done by scientists employed by the various institutes within NIH. However, the bulk of the NIH budget goes into its extramural programs, which fund researchers at various universities and research institutions throughout the United States, and even internationally.
The NIH supports biomedical research through many kinds of grant mechanisms. Sometimes individual institutes or centers may solicit applications to undertake research in specific areas that are high priority for that institute or center. NIH would issue a request for applications (RFA) on a program announcement (PA). These RFAs and PAs will tend to have specific rules and instructions for submission. They may or may not have specific funds set aside to support them. They may exist for long periods of time, or may only be available for a short period. RFAs and PAs may be reviewed differently from unsolicited grant applications; sometimes they will be reviewed by a regular study section, and sometimes a special review panel will review them. Although RFAs and PAs targeting statistical methods are relatively rare, some will express interest in methodological issues that might include statistical methods. A listing of RFAs and PAs can be found on the NIH website. Visit the StatFund website for information about possible funding opportunities for statistical research.
Unsolicited research grant applications can be of a number of types, including R01, R03, R15, R21, and K99/R00. R01s (research project grants) are the most common mechanism used by NIH to fund individuals to pursue their own research ideas. NIH accepts new R01 applications in February, June, and October. R01 grants are for well-conceived and novel research ideas with important and probably multiple related aims, and would require preliminary data or preliminary research.
There are several other mechanisms worth considering. For example, R15 grants (academic research enhancement awards) are designed to support research by faculty located in institutions that have not been major recipients of NIH research grant funds. R15 grants are required to involve students, as well.
The R03 grant mechanism (small grants) is worth consideration as a means of obtaining short-term funding with a small budget to enable you to pursue a project that could then lead to a full-scale proposal. The R03 grant mechanism might be useful for a young investigator’s first grant application.
R21 grants (exploratory/development research grants) are for a two-year period with a fixed and limited budget. They are designed to support high-risk projects and require less preliminary data than a typical R01.
While all institutes in NIH will fund R01 grants, the policies regarding support of R03, R15, and R21 grants differ between institutes, and not all institutes accept unsolicited R03 and R21 applications.
K99/R00 grants (Pathway to Independence awards) are designed to facilitate the transition of a researcher from a mentored postdoctoral period (K99) into a faculty position (R00). These grants are personal fellowships that include a significant research project. They are open to statisticians at some institutes, but because the criteria by which they are assessed differ from usual research grants, they will not be discussed here.
In a recent change to the grant application process, all applications are now submitted in response to a funding opportunity announcement. NIH has developed Omnibus PAs for use by applicants who wish to submit what were formerly termed “unsolicited” applications. This process does not diminish the interest of NIH institutes in investigator-initiated, unsolicited research grant applications. NIH continues to welcome “unsolicited” applications in statistical research in the same way it does in any other area of biomedical research.
General Review Process
Nearly all applications are processed through the Center for Scientific Review (CSR) and receive a careful initial screening to ensure practical requirements have been met (e.g., page limits, font sizes, required forms, signatures, and budgetary information). Once successfully through this step, grant applications receive two critically important assignments. One is to an institute that will fund the grant, should it pass the rigorous standard of peer review. The other is to a “study section,” which will provide the independent peer review to assess the scientific quality of the proposal.
In general, institute and study section assignments are separate. A particular study section will review the scientific merit of applications with a common disciplinary theme, but that may have a variety of institute assignments. Presently, we’ll talk about the specific study section (BMRD) in which most statistical methods grants are reviewed. The assigned institute uses the outcome of the study section to help decide whether to fund the proposal.
While the review from the study section and the score it assigns is strongly associated with the likelihood of funding, it is not the only factor that determines funding. The discussion at the study section and written review will not mention funding. NIH has a dual peer review system, with the first review by the study section and the second by the institute. Different institutes have different budgets, their own priorities, and their own policies regarding allocation of resources. The leadership at each institute will ultimately decide which grants are funded, with input from the scientific staff at the institute.
An investigator can influence the institute and study section assignments. For example, it is a good idea to include a cover letter requesting assignment to a specific study section for review and institute for possible funding. The CSR and institute generally accommodate these requests, if possible. It is also important to choose an appropriate title for the proposal and to write a clear, concise abstract that allows a reader to get a sense of the proposal immediately.
Because biostatistical method is a relatively small and specialized field, it often will take some time and effort to identify the appropriate target for your application. Your proposal will have the best chance of success if it focuses on the development and application of statistical methods of direct relevance to one or possibly two institutes. For example, someone interested in working on methods for evaluating surrogate endpoints in HIV vaccine studies might choose to target their proposal to the National Institute of Allergy and Infectious Disease. Someone interested in methods for processing and integrating genetic and genomic data might draw the interest of the National Human Genome Research Institute.
Choosing the general focus for your proposal is a critical first step. It is a good idea to talk to colleagues in your department about how their work is funded. Attending workshops, such as those sponsored by ENAR, provides another opportunity to receive feedback on how to frame your research proposal. It is a good idea to check the websites of potential institutes to identify current areas of priority and to make contact with the program officers who work in the extramural research divisions of those institutes to talk about your ideas and gain feedback.
Program officers can provide valuable advice about strategies to enhance your proposal’s likelihood for funding. The program officer will generally be someone who understands statistics. They also will understand the mission of their institute and be a strong advocate for well-conceived research grants that align with that mission. The program officer may observe the review of your grant application at the study section, but they play no part in that review or in determining the score for your application. Their feedback after the review, however, could be valuable to you.
NIH designates the principal investigator (PI) of an application as a new investigator if they have not previously competed successfully for a significant research award, such as an R01. An early stage investigator is a new investigator who is within 10 years of completing his or her terminal research degree or residency. The new investigator and early stage investigator designations can be helpful to a grant application, both in how it is reviewed and—for R01 applications—its chances of funding.
A list of websites that provide information for would-be grantees can be found on the NIH website.
Writing the Proposal
Once you have decided on the general focus for your proposal, you should start planning the details. Assuming you already have an active research program under way, you should allow at least six months of relatively focused effort to put your first proposal together.
It is important to submit the most polished application you can for your first submission. Testing the water by throwing in many ideas to see which the study section like with the idea of better developing them in the revision is probably not a good idea. You would be wise to ask a senior colleague who is experienced in grant submissions and reviews to carefully read and critique a draft of your grant application. Colleagues at other institutions also may be willing to give you advice about the specifics of your research, but you should be aware that they then would be considered in conflict with your application and not able to review it at a study section.
NIH has strict guidelines for how your grant should be formatted. For example, you must include all required sections, adhere to page limits, and use clear, readily legible fonts that satisfy strict minimum size requirements. Required sections such as your face page, project description, key personnel, table of contents, budgets, and biographical sketches for key personnel need to be filled out on specific forms. While it is important to have all the administrative details correct, the two most important parts of your proposal are your project description and specific research plan.
The project description should summarize your proposal succinctly and in such a way to give a clear sense of what you will do and why it is important. This short write-up provides you with an opportunity to catch the attention and interest of reviewers and convince them that something exciting is coming along. Having a well-written project description also is important because it may be used by staff in the Center for Scientific Review to assign your proposal to a study section and institute. The project description should be written in a way that a scientifically trained, but not necessarily statistically trained, person can understand. If your proposal is funded, the project description will be made publicly available, as well.
The specific research plan consists of a number of sections, including specific aims, research strategy, bibliography and references cited, plus other sections of an administrative nature. The permitted length of the research strategy section differs depending on the funding mechanism. For R01 applications, it is 12 pages; for R03 and R21 grants, it is six pages.
This is where you outline the list of concrete problems you plan to tackle. This section would generally begin by briefly describing the context of the problem and articulating why the type of research you are proposing is important. It is a good idea to arrange your specific aims into three or four main groups, with each containing two or three related and specific sub-aims. You should establish a numbering system for your aims and sub-aims (e.g., 1a, 1b, 1c, 2a, etc.) Data sets you will be using in your research are important to mention in the specific aims. If you plan to develop software for others to use, that should be mentioned and could be one of the specific aims. This section is restricted to one page.
This should have three subsections: significance, innovation, and approach.
Significance: This subsection should convince the reviewer that the problems listed in your specific aims section are strongly grounded in and motivated by important issues in health sciences research. You must convince reviewers that by accomplishing those aims, you will contribute significantly to the advancement of biomedical research. This section also needs to convince reviewers that you know the literature and your specific aims have not been accomplished by someone else already. Hence, it is important to include the appropriate references in this section. This section also can be used to describe applied work or data analysis that has served as the motivation for the proposed methodological research.
As you then work your way through the motivation for each specific aim and sub-aim, try to keep a consistent numbering system. Remember, an important goal is to make your proposal as readable as possible for reviewers. You want your aims and logic to be clearly articulated all the way through.
You should provide details about the applications that motivate your proposed research, along with data sets that will be used in your work. Referring to letters of support from subject matter specialists will generally be helpful. For most data sets or biomedical research applications, there are already existing methods that could be used. Thus, it is important for you to convince the reviewer that your methods are sufficiently different and likely to lead to different conclusions.
This section also should describe the likely impact of your work. Ideally, your proposed research should have an impact on both statistical methodology and the advancement of knowledge in an area of application. Many people use this section to describe the background experience of the PI and other key personnel, outlining any key papers or results that might serve as the basis for the methods to be developed if the grant is funded. It is a good idea to list relevant publications—these will help demonstrate that you are well qualified to accomplish your research aims. You are not permitted to include papers that have not been accepted for publication.
The significance section will typically be 3–5 pages long.
Innovation: In this section, you explain the novel aspects of your proposed research. The novelty may be in the aims, themselves, or they may be in the details of the approach you will be taking. It also may be because sophisticated statistical approaches have not been applied to data of this type before. Since innovation is an aspect of the proposal that can be scored, it is important that the reviewers understand where the novelty is in your proposed research. It may be helpful to list the novel aspects of each aim and sub-aim explicitly.
Approach: This section is where you describe in detail what you want to do and how you are going to do it. To the extent possible, without sacrificing coherency, the approach section should not overlap with the significance section. As you write this section, ask yourself whether this material should be moved into the significance section if you find yourself starting to justify why the methods are important.
Using the same numbering system as used in your previous sections to denote aims and sub-aims, the approach section should immediately start laying out the details of your proposed research. This might include how the models will be formulated, the estimation method or algorithm that will be used, how variances will be calculated, how you will evaluate your ideas, or to what alternatives it will be compared. If you propose to undertake theoretical proofs or simulation studies, a brief outline of how you will go about that should be included. It may be a good idea to identify possible pitfalls and how you will address them if they arise.
Many applications will have a software development and dissemination plan, so some detail of the language you will use and how it will be disseminated should be given. The target users of your software should be clear (e.g., other statisticians, health practitioners and other scientists) and your dissemination plan should be appropriate for this group.
Many people find that a good strategy is to provide a more detailed description for the first few specific aims and sub-aims (perhaps the strongest ones), then have only the broad steps outlined for the others. Reviewers understand some aims will be better developed than others will. It is usually a good idea to describe your strongest aims first, while retaining a logical flow.
The number of pages available to describe details of your approach for all your aims is quite limited. Recent changes in the review criteria for grant applications place more emphasis on the impact and significance of your proposed research, and less on the details of how you are going to achieve it. Thus, you will have to choose carefully what details to show. Providing excessive detail on background material or algebraic derivations is probably not helpful or necessary.
Maintaining a clear, simple writing style throughout the proposal is an essential ingredient for success. As you write, keep in mind the goal of making the application as easy as possible for reviewers to understand and appreciate. While the importance of clear writing cannot be overemphasized, the most important determinant of success will be the nature of the proposed research, itself.
There is a fine line to walk in deciding on your general focus and specific aims. On the one hand, you want to demonstrate creativity by tackling unusual or nonstandard problems. On the other, you want to tackle problems you can realistically solve, given your background and expertise. In general, it is a good idea to write a grant application on a topic with which you are very familiar. If you are not confident that an idea is a good one, don’t include it as a major aim. Tackling methodological problems that arise from consulting or project work is often an ideal solution, provided the methodology has broader applicability than just that project. Reviewers will generally have a favorable impression of a proposal that demonstrates a good knowledge of the underlying scientific context and questions. Balancing theory and application is generally important. While it is good to propose research problems that are statistically interesting and innovative, reviewers will not be convinced of their importance unless you also can argue for the practical importance of what you are proposing to do.
A common problem with statistical methods applications is that they provide a narrative of a research area, describing the issues and general approach that will be taken, but they lack a clear explanation of what the investigators will actually do during the period of the grant. You should not assume the reviewers have as much expertise or in-depth knowledge as you in the focus of your grant application. A clear distinction between what has already been developed and what you plan to do with grant is likely to improve your review.
The approach section also should include a brief timeline that describes in which years of the grant each aim and sub-aim will be worked on.
The investigators are an important review criterion. The group of investigators will include the PI; co-investigators; and potentially graduate students, post-docs, research assistants, programmers, and consultants. Who is included and for what amount of effort should be driven by the needs of the research. For each of the key personnel, the grant application will include a biosketch. This document is limited to four pages and shows the person’s education, positions held, awards and honors, publications, other grants that support them, and a personal statement describing their expertise relevant to the grant and the role they will play in the research.
The recommended limit for the number of publications is 15. These publications should be selected carefully and focus on those that are more recent and more relevant to the research. This is particularly important for the PI, who needs to demonstrate to the review committee that they are an active researcher and have the specialized expertise needed to complete the proposed research successfully. The list of publications also can help highlight that the members of the research team have successfully collaborated before.
The personal statements are also an opportunity to demonstrate that the investigators are well suited to make the research successful through their role in the proposed research.
It can be useful to include letters of support with your grant application. If your proposed research is focused on a particular application, including a letter from the person who will provide or “owns” that data will be important. This solidifies that you really do have access to the data and increases the likely impact of your work.
You should start thinking about budget well ahead of the submission due date. Most departments will have a financial administrator who can help you prepare the needed numbers and documentation. You will need to think about the percent effort you wish to devote to the grant, yourself, along with the percent efforts of any co-investigators, associated computing and other expenses, the grant start date, and its duration. These decisions help determine the “direct costs,” for your application.
Your grants administrator will then figure out the “indirect costs,” which are generally a percentage of the direct costs that go to your school or research institution to help pay for the resources you need to do your work (e.g., your office, heat, administration, etc).
Mid-level and senior investigators typically apply for 10–30% annual effort for themselves for a 3–4-year period. Faculty who are paid on a nine-month basis might ask for summer salary. While some applicants request five years of funding, there is a certain risk in doing so, since the proposed research agenda will need to be particularly strong to justify such a long period of support.
It is common to request funds for one or two statistical co-investigators (say at 20% effort), along with a postdoctoral fellow and/or a graduate student. Sometimes, it is a good idea to include a subject matter collaborator for a small percent effort, especially if your proposal is strongly grounded in applications. The appropriateness and amount of effort will depend on the details of the research, role of the collaborator, and maybe the type of grant mechanism.
For example, a proposal focused on statistical methods for cancer clinical trials might include as a co-investigator an oncologist who would provide data and subject matter advice, with 5% effort allocated. More junior investigators, particularly those applying for their first grant, will typically ask for less in terms of postdoc or student support, but usually request a larger proportion of their own salary (40–50% is common for first-time applicants).
Preparing a budget for an NIH grant application is usually relatively simple. R03 and R21 have fixed direct cost budget totals. R01s will nearly always be “modular grants.” Basically, any grant application with a total direct cost of less than $250,000 per year can use the “modular grant” format. This means you do not have to provide many details about the budget and funds can be requested in increments of $25,000. A typical, basic statistical methods grant will have an annual direct cost of $150,000–$200,000. More complex interdisciplinary grants involving medical co-investigators or multiple co-investigators will generally be more expensive.
People sometimes ask for consultant expenses. Usually, this will be a relatively modest amount (less than $5,000), which will enable you to invite a colleague to visit you and work on one of the specific aims outlined in your proposal. As indicated above, reviewers often like to see the involvement of a consultant who is a subject matter specialist.
Your NIH budget can be used to purchase needed equipment. For most statisticians, this means computers. It is reasonable for your budget to include modest funds for “supplies,” as well as travel funds for yourself and perhaps one co-investigator to attend one meeting each year. Once again, submitting a modular grant has the major advantage that you don’t need to provide a detailed budget justification, but instead outline it in broad terms.
While you do not have to provide dollar amounts for each investigator, there is a section called personnel justification. This section lists the number of calendar months each person will devote to the research each year. It is important that the level of personnel support requested be commensurate with the proposed research agenda. This section also should show in which aims each person will be involved and what their role will be. If the application includes students, post-docs, or research assistants, you should indicate who would supervise them.
The NIH Review Process
How does the review process work? For many years, most biostatistical grant applications were reviewed in a so-called “special study section” that focused on statistical methods grants. About 15 years ago, the Center for Scientific Review undertook a major reorganization of its study sections. One goal of the reorganization was to reduce the number of narrowly focused study sections and streamline the review process to reflect the increasingly interdisciplinary nature of biomedical research more effectively. Because the number of biostatistical grants submitted does not warrant the establishment of its own study section, it was decided that biostatistical grants would be reviewed in the Social Sciences, Nursing, Epidemiology, and Methods [SNEM-5] Study Section.
About nine years ago, the Center for Scientific Review deemed there were sufficient biostatistical grant applications and established the Biostatistical Methods and Research Design (BMRD) Study Section. The NIH website gives the description of the types of research BMRD reviews as follows:
The specific topics are the following:
- High-dimensional data methods such as those arising from genomic technologies, proteomics, sequencing, and imaging studies; development and applications of methods for data mining; statistical innovation in decision support, statistical machine learning, Bayesian networks, neural networks and outcome prediction; statistical methods for high-throughput data; biomarker identification
- Novel analyses of existing data sets: Innovative application of existing or development of new statistical and computational methodologies; application of methods in substantially new areas of application; innovative, non-routine data analysis strategies including combinations of existing methods rather than de novo development of new methods; development and evaluation of novel analytic tools to address new questions within existing data sets
- Research design: Development and innovative application of randomized trial designs; adaptive designs; sample size determination; design issues for experimental and observational studies; methods to improve study design efficiencies; methods for survey sample design; methods for comparative effectiveness studies
- Data collection and measurement: Development and adaption of methods to estimate and improve data precision, reliability, and validity; methods to estimate and adjust for bias, measurement error, confounding, sampling and nonsampling error; psychometric methods
- Data analysis and modeling: Development of statistical theory, analytic methods and models, computational tools, and algorithms for the analysis and interpretation of data from clinical studies, randomized trials, observational studies, epidemiological studies, human genetic association studies, environmental studies, complex surveys, large databases, and registries; methods to handle data features and anomalies such as correlation, clustering, and missing data; risk prediction and forecasting methods; causal modeling
Tomas Drgon is the scientific review officer (SRO) for BMRD. The role of the SRO is to organize the review of the submitted grants, assign the reviewers, and be the contact person for applicants regarding the review process.
Although most statistics grant applications are reviewed in BMRD, there are other specialized study sections in which statistics grants can be reviewed. For example, grants focused on statistical methods for AIDS research might be reviewed at the AIDS Clinical Studies and Epidemiology Study Section (ACE).
The permanent members of the BMRD study section, who have terms of 4–6 years, are mainly statisticians and biostatisticians who have NIH grant funding. Permanent members come from different regions of the United States and are selected by the SRO. As a group, they will have a broad range of expertise in statistical methodology and application areas. At each study section meeting, there may be temporary members who also review grants, depending on the number of submissions and particular topics for which expertise is needed.
Study sections generally meet three times a year to review proposals. Reviewers receive the packet of grant applications to be considered, along with specific assignments to review a subset (usually 8–10) in detail. Reviewers will need to declare themselves in conflict of interest for applications from anyone from their same institution or from any close colleague. Reviewers prepare a written critique of each assigned proposal.
Usually, each proposal will be assigned a primary, secondary, and tertiary reviewer. The written critiques from these three reviewers are available to the other study section members a few days before the meeting. Prior to the meeting, each assigned reviewer also posts a preliminary overall score and preliminary scores for each of the following five criteria: significance, investigators, innovation, approach, and environment. While all five criteria matter, their order of importance goes from most important to least important. When submitting a grant application, it is important to bear this order of importance in mind and help the reviewers understand (i) why your proposed research is significant, (ii) that the investigators have strengths relevant to the proposed research, and (iii) that the work is innovative.
Significance: Is the work important? If successfully accomplished, will the proposed research have an important effect on biomedical science? The NIH website describes the assessment of significance as follows:
Reviewers will be influenced by how well you have written your significance section in preparing this part of their report.
Investigator: Is the team of investigators well qualified to accomplish the proposed research? If early stage investigators or new investigators, or in the early stages of independent careers, do they have appropriate experience and training? If established, have they demonstrated an ongoing record of accomplishments that have advanced their field(s)? If the project is collaborative, do the investigators have complementary and integrated expertise? The reviewers will be paying close attention to the biosketch(es) and any preliminary results in your proposal.
The study section will understand that junior investigators are likely to have less of a record of accomplishment of research than more established investigators and make allowances for that in their review. They will, however, still need some track record of relevant research to convince the reviewers that they can successfully achieve the proposed research.
Innovation: Is the proposed work new? Is it creative? For innovation, the NIH website states the following:
Approach: Is the planned approach well reasoned, appropriate, and likely to lead to accomplishment of the specific aims? The reviewer will pay close attention to what you have outlined in your approach section.
Environment: Is the infrastructure at the institutions where the research will be undertaken appropriate for what is required? The reviewers will be looking at the resources page in the grant application to assess this. They also will consider any unique features of the scientific environment, subject populations, or collaborative arrangements.
Prior to the meeting, each of the assigned reviewers provides written critiques and a score for each of the five criteria. The critiques are in the form of bullet points and focus on the strengths and weaknesses of the proposal. The five scores are intended as a guide to the reviewer and help them arrive at the overall impact score. There is no formula for using criterion scores to calculate overall impact score. This is the score that matters. Each reviewer will write a paragraph on the overall impact of the proposed research.
The overall impact will be an assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the five review criteria.
At the study section meeting, each committee member gives their final overall impact score after the application is discussed, including issues related to human subjects, and the committee has a good understanding of the strengths and weaknesses of the application. The average of these scores (multiplied by 100) is what the applicant sees. They do not see the impact score from each reviewer.
Only after the application is scored is the budget discussed. The reviewers will assess whether the budget is reasonable relative to the work to be accomplished and whether the length of time requested is appropriate.
During the study section meeting, the applications are reviewed in batches. Roughly speaking, new investigators are in one batch, smaller grants are in another batch, and experienced investigators are in another batch. The preliminary scores from the three reviewers will be used to decide whether the application will be discussed. Approximately the top 50% of the applications in each batch are discussed.
A few days after the meeting, the final score will be made available online to the PI. Around 2–4 weeks later, the written critiques from the three reviewers are posted online for the PI to see. These critiques will be updated versions of the preliminary critiques prepared by the three reviewers. There also will be an overall summary of the discussion that highlights its main strengths and weaknesses and should provide the PI with an understanding of why the grant application received the score it did. The overall summary is written by the SRO.
The possible score for each component and the overall impact from each reviewer is an integer from 1–9, with 1 representing perfect. Despite efforts by the Center for Scientific Review to harmonize scores between different study sections, the typical scores do differ. Roughly speaking, scores of 1 and 2 are outstanding and scores of 7, 8, and 9 indicate substantial problems with that aspect of the application.
The overall impact scores are assigned a percentile. In BMRD, grants are percentiled based on scores received by other grants reviewed at BMRD in the current and two previous cycles. For R01 and most other grants, it is the percentile that is used to help determine the funding, whereas the institutes use a score for some other grants. The relationship between scores, percentiles, and funding decisions varies considerably by institute and grant mechanism, depending largely on budgetary issues specific to each institute and its individual funding priorities. Some institutes also have policies that favor new investigators and early stage investigators by applying a different threshold for funding for R01 grants.
The percentiling occurs for most study sections at NIH. There are two obvious, but often unrecognized, effects of this. One is that it does not make any difference to the number of grant applications scored in the fundable range if the study section is one that tends to give harshly worded reviews or more gently worded reviews. The other is that if a study section reviews a large number of applications, the absolute number of grants from that study section that will be in the fundable range is greater than for a study section that reviews a smaller number of applications.
The percentile necessary to gain funding also will vary over time, dependent on the current economic situation. In lean times, an 8th percentile or less would have a high chance of funding, regardless of institute. In plentiful times, percentiles less than the 20th will have a reasonable chance of funding.
After you receive your critiques, you can be cautiously optimistic if your percentile is in the top 8%. If you are in the range of 10th–20th percentile, you are in a “gray zone,” in which funding is uncertain. Sometimes an institute will fund a grant that is just above its funding cutoff in terms of percentile, if it is in an area of particularly high scientific priority. Percentiles higher than 25% will be funded rarely.
The process by which reviewers assign scores is hard to describe. In general, it is fair to say that people assign scores by keeping in mind these rough guidelines on how scores translate to funding decisions. If three reviewers all independently give similar scores, it is reassuring to the rest of the committee and the final score will generally be similar. If the three reviewers start out with different scores, then considerable discussion might be needed before the committee is comfortable voting on a particular application.
If your application is in the 10–25% range, you can still have some hope of funding, but should start to think about a resubmission. Proposals considered to fall into the lower half of those reviewed typically will be “not discussed” and no overall impact score given. If you get a bad score or your application is “not discussed” for your first submission, don’t despair. It happens to most people. Give yourself a week or two to get over your disappointment, and then start to evaluate your critiques carefully. Try to determine whether the reviewers think your proposal shows promise and can be fixed easily. Talking to a senior colleague can be helpful at this stage. Your program officer also may be able to give you advice.
If you decide to fix the application and reapply, respond explicitly to each of the weaknesses in your critique, indicating how and where you have revised your application. This response should be laid out clearly in a section called “Introduction.” If you disagree with the reviewer on certain points, state your arguments in a logical manner, but avoid criticizing the reviewer—this tactic is likely to backfire on you! Add and point out any additional improvements. It can be useful to highlight revisions to the proposal, itself, by using a different font or italics. The more you can convince reviewers you have responded thoroughly and thoughtfully to the previous critique, the greater your chance of success on your resubmission.
Note that you will only have one opportunity to resubmit your grant. If you wish to apply for NIH funding for this line of research after that, you will have to make substantial modifications to the aims of your grant application so it can be considered as a new proposal.
How grant applications are reviewed and funding decisions are made at the NIH is a complex process. Much effort goes into thinking about how it should be organized and conducted to be objective and fair and in such a way that enhances the mission of the NIH. No applicant can be expected to understand all the intricacies of the system. If you are not funded, you can console yourself by recognizing you are part of the majority and understanding that the system will hopefully enable you to have a better chance at funding for your next application.
The grant review process has some similarities to the review of an article submitted to a journal; however, there are important differences. If the editor invites you to revise and resubmit a paper, the referees’ reports are usually fairly clear about what needs to be done to make the paper acceptable for publication. If you successfully do those things, there is a good chance the paper will be accepted. For a resubmission of a grant, the critiques you received are an evaluation of the original submission’s strengths and weaknesses; they are unlikely to include specific suggestions about how to make the grant application better.
Another difference concerns the review of resubmissions. It is likely that one, if not two, of the main reviewers will not have been reviewers for your original submission. While they will see the original critiques and your response to them, they are expected to use their expertise to assess the strengths and weaknesses of the new application, including the possibility of identifying different strengths and weaknesses than were noted for the original submission.
Mathematical Statistician, and Program Official
Biostatistics Research Branch,
Division of Clinical Research
6700-B Rockledge Drive-MSC 7609
Bethesda, MD 20892-7609
Michelle C. Dunn
Mathematical Statistician and Program Director
Surveillance Research Program
Division of Cancer Control and Population Sciences
National Cancer Institute
6116 Executive Blvd.
Bethesda, MD 20892-8316
Scientific Review Officer
Center for Scientific Review
6701 Rockledge Drive
Bethesda, MD 20892
Writing a grant application sounds like a lot of work, and it is. However, even if your grant is not successful the first time, the effort is not wasted. Many describe the experience of grant writing as positive. It is an opportunity to think through your research priorities and focus your thoughts. For many researchers, writing the grant is integral to doing the research.
You can even think of a grant application as a compilation of a series of half-written papers. The significance section contains the introductions to the various papers, while the approach section contains an incomplete draft of the main results of the papers.
Winning your first RO1 grant can be the first step in a long-term, satisfying relationship with the NIH. So long as you accomplish your aims, publish your work, and generate new ideas for each successive renewal, there is no reason why your RO1 grant can’t stay with you throughout your career. During your earlier years, you might be the one doing all the calculations and programming needed to work out the problems outlined in your proposal. As you become more senior, the problems can be shared with your students and junior colleagues. Eventually, you will be guiding fresh new investigators applying for their own grants.
I want to emphasize the importance of the BMRD study section to our profession. BMRD is the only study section devoted to statistical methodology, hence every application reviewed by BMRD will be assessed by statisticians and every funded grant reviewed by BMRD is a statistical grant. The percentiling system means that the more grant applications reviewed by BMRD, the more that will be in the fundable range. In the recent past, BMRD has not been reviewing as many applications as most other study sections, leading observers to question the need for it. So, I hope this article will be helpful to you in putting together an NIH grant application and requesting in your cover letter that it be reviewed by BMRD.
Editor’s Note: This article was commissioned by the ASA Committee for Funded Research. It borrows information from and updates a similar article written by Louise Ryan that appeared in the March 2002 issue of Amstat News.