Tips for Writing a Statistical Analysis Plan
This column highlights research activities that may be of interest to ASA members. This article includes information about new research solicitations and the federal budget for statistics. Comments or suggestions for future articles may be sent to the Amstat News managing editor at firstname.lastname@example.org.
Contributing Editor Amy Herring is professor and associate chair of biostatistics at The University of North Carolina at Chapel Hill. She is PI of an R01 for developing statistical methods with applications in birth defects epidemiology and co-investigator on numerous projects in public health and medicine.
In the first of a series of articles commissioned by the ASA Committee on Funded Research, Jeremy Taylor provided an overview of the review process for statistical methodology grants in last month’s issue. This month, we consider important facets of writing statistical sections for NIH grants not primarily focused on development of new statistical methods. We assume readers are familiar with last month’s overview, particularly the description of the NIH, its review process, scoring of proposals, and other important issues.
In particular, we present tips for writing an excellent statistical analysis plan or biostatistical core for a biomedical or public health research grant with a primary focus outside of biostatistics. We will focus our attention on R01 research project grants and multi-project awards (e.g., P01, P30, P50, U19).
Multi-project awards support a multidisciplinary research team or group of investigators that focuses on a common research topic. They generally fund shared resources and facilities across multiple smaller research projects, and a biostatistics, data management, and/or bioinformatics core facility is often part of these proposals.
In an R01 proposal that does not involve statistics as a primary focus, the statistical portions of the grant usually contribute to the scores under the categories “Investigators,” “Approach,” and “Overall.” In the investigator category, reviewers are looking for evidence that the statistician or statistical analysis team has the skill and experience to evaluate the hypotheses in the specific aims. Your relevant skills and experience are judged based primarily on the information you provide in your biosketch(es) and the quality of the study design and analysis plan in the grant.
If you are a new researcher with relatively few publications, you should consider engaging a more senior biostatistician as a consultant or investigator on the grant to ensure the reviewers are comfortable with the level of statistical support. Reviewers will express their comfort with the planned study design and statistical analysis in the approach section of the grant, and both your credentials and statistical analysis plan/study design may affect the grant’s overall score.
For multi-project awards, a biostatistics, data management, and/or bioinformatics core facility is often scored as either acceptable or unacceptable, rather than using the typical 1–9 scale described in last month’s article. Unacceptable scores are not rare, so this scoring scheme does not mean the statistical section is less important than in other proposals.
Tips for Meeting Scientific Goals
Match the specific aims of the grant to your statistical analysis plan. Every hypothesis laid out in the specific aims should have a corresponding section in the analysis plan clearly describing how the hypothesis will be tested or otherwise evaluated. It is critical that the analysis plan is specific about how the investigator’s aims will be translated into hypotheses that you will then evaluate. It will be helpful if you use the same numbering system in the analysis plan that is used for the specific aims.
Know your audience. It is important to learn about the NIH review group that will score your application. Expectations for a statistical analysis section or biostatistics core may vary greatly across fields. Suppose the grant examines interactions between individuals’ genetic profiles and their diet in predicting cancer. Your grant may be reviewed by epidemiologists, bench scientists, or clinicians, and each type of reviewer would have different expectations of an excellent analysis plan. Current and previous review group rosters can be found at http://era.nih.gov/roster and can provide valuable information about the expertise (and expectations) of the review group. Previous grant reviews from the same review group can help you learn more about the review group’s expectations, even if they are from different proposals.
Provide something for everyone, explaining statistical concepts in clear, concise language that is accessible to nonstatisticians as well as to statisticians. As Taylor mentioned in last month’s issue, it is critical to “keep in mind the goal of making the application as easy as possible for reviewers to understand and appreciate.” Maybe you do really need that complex structural equations model or new methodology for dynamic treatment regimes to evaluate the specific aims; if so, it needs to be in the grant. However, in a nonstatistical review group, you may get three grant reviewers who have relatively little statistical knowledge. The reviewers may criticize an analysis plan if it comes across as overly involved or too ambitious. Be sure to take a little space or add a figure to explain the basic ideas of any complicated methods so a reviewer with a minimal background in statistics will get the big picture and understand why something more than a t-test is required.
Be specific. Don’t use boilerplate or a standard template for every grant you write. Reviewers will be looking to see how your analysis plan addresses the specific aims of the proposal. Have you addressed pertinent issues in the study at hand (e.g., a particular missing data problem, measurement error, or potential biases)? In a multi-project grant, reviewers will look to see whether the biostatistics core has the specific expertise to achieve all aims in the component grants. In some multi-project grants, you will need to show broad statistical expertise across biostatistics core members, as many multi-project grants devote considerable resources to helping new researchers get new projects off the ground.
Trust your collaborators. If they have a concern about the analysis plan, it is likely to be shared by the reviewers. Incorporating their feedback to improve the analysis plan will generally lead to a superior final product. But not always. Stick to your guns if you really think your collaborators are going in the wrong direction (e.g., using medical students for data cleaning is not acceptable, even if this has worked well for them in the past).
A second pair of eyes is often helpful. Offer to look over a colleague’s grants in exchange for having your colleague review yours.
Data cleaning and reproducibility are two critical concerns. Be sure you have addressed these issues in your proposal and have budgeted appropriately.
Keep your eyes open for methodological opportunities. Many statisticians are successful at getting their own grants (as PI) based on interesting methodological issues that arise in collaboration.
Operational Aspects Critical to Meeting Scientific Goals
Set expectations early so there are no unpleasant surprises at the time of submission. Will you be a co-investigator (this is standard) or dual PI (uncommon but appropriate if there is a large statistical component)? Often, the roles “biostatistician” or “statistician” are used, and these generally indicate more basic support, rather than doctoral-level scientific leadership, with a few notable exceptions (one that comes to mind is the biostatistics core of a multi-project grant, in which multiple researchers may be listed with minimal support just in case their expertise is needed). What percent effort will be required of the statistical team? Do you need graduate student support and computing resources? Who will be responsible for data entry, data management, and archiving code? What is the time frame for grant writing?
Be realistic. Don’t promise too much work for too little time. Nobody is happy if you cannot meet the goals you set. When the analysis is extensive or involves some new methodological territory, be sure your percent effort is substantial. For many projects, 1.5 months of effort plus a graduate student research assistant will be appropriate in analysis years, with adjustment of the effort required if early years of the grant do not involve any data analysis (however, you would generally still want around 0.6–1.0 months of funding yourself if you expect to provide input on the study design and other important issues that may arise early in the study).
Along these lines, be aware that sometimes grants will face cuts, either by the PI right before submission (to get the budget within pre-specified limits) or by NIH at the time of funding, and the PI generally has wide latitude in how to apply the cuts. You should not be afraid to put your foot down if your 10% effort plus a graduate student is cut to only 4% of your own time with no graduate student. In this case, you would explain how much of your time is available on that limited basis (e.g., 4% may be only enough to support your participation in a single 1.5 hour meeting per week, with no statistical analysis included, and, in that case, you may prefer to spend your time on projects that will provide you with more interesting work) and negotiate to obtain enough effort to support the work needed. Your department chair (or a senior faculty member, if you are a junior faculty member) can be helpful in such negotiations. Don’t be afraid to refuse to work on a project if the % effort is truly inadequate (though you should check with colleagues to be sure your version of inadequate is not out of line).
Read the review criteria before writing your sections of the grant. For some grants, the review criteria specifically address statistical analysis plans. Responsiveness to these criteria can greatly enhance your chances of success in the peer review process.
Cores in multi-project awards can be tricky to write. Sometimes, a reviewer may be assigned to review only your core, and sometimes a reviewer may review the entire grant. Thus, the core should be responsive to the research projects in the grant while also standing alone for review. Biostatistics cores have special requirements beyond statistical analysis plans of R01s. A core needs a specific leader who will be responsible for all core activities. The application must explain the organization of the core and clearly describe how it operates, including how researcher requests to use the core will be prioritized. Core services will vary based on the goals of the multi-project award but typically provide expertise for the planning, conduct, analysis, and reporting of studies; scientific computing; data management; manuscript preparation; and training of core users (reviewers often look favorably on cores that incorporate a training component by providing relevant workshops and seminars). Often, a strong case can be made to include time for methodological research by core biostatisticians when the multi-project aims would benefit from enhanced statistical methodology. It is always a good idea to provide specific names for all personnel (including programmers and graduate students), rather than budgeting for unnamed individuals in these applications.
Be committed. Carefully tailor the personal statement on your biosketch and the accompanying list of publications to the grant application at hand. For example, you will want to include papers that are co-authored with your collaborators on the current application and other publications that show you have already worked in areas relevant to the grant. If the grant requires statistical assistance in an area in which you have no expertise, you may want to bring another statistician onto the team as co-investigator or a consultant to the grant. If you do not show you have the skills to carry out the proposed analysis, or if you do not look fully engaged with the grant, the grant may get less favorable scores for the investigators, approach, and overall components.
Block off time for last-minute changes well in advance. Your colleagues may have others inside the university review the grant before submission, and an aim may be replaced at the last minute. This could require a new analysis plan, new power calculations, etc. While major last-minute changes should not be a regular occurrence, this happens periodically, even with outstanding collaborators, and you should not be surprised to have requests for 11th-hour edits.
Meetings such as ENAR and JSM often offer roundtable discussions on writing statistical components of non-statistical grants. These discussions are a great way to share good (and bad!) experiences with colleagues to increase the probability of success in the future. Good luck!