Statistical Consulting in a University Research Setting
This column is written for statisticians with master’s degrees and highlights areas of employment that will benefit statisticians at the master’s level. Comments and suggestions should be sent to Megan Murphy, Amstat News managing editor, at firstname.lastname@example.org.
Gregory J. Stoddard, a master’s degree–level biostatistician, is co-director of the Study Design and Biostatistics Center at the University of Utah, which provides statistical consultations on applied research problems throughout the university.
My first job after graduation was as a biostatistician for a medical device subsidiary of Becton Dickinson, Inc. The director of statistics from the corporate headquarters commented to me that he found working as a statistician in the medical device industry much more exciting than his experience in the pharmaceutical industry. In the pharmaceutical industry, he explained, it takes several years to complete a clinical trial, while for medical devices, the clinical trials are completed in less than a year. So, you get to see the results of your work much faster and then move on to a new problem. That seems like an amusing contrast now. In the university setting, I frequently complete consulting projects in less than a week, and the projects just keep coming.
Here are three hypothetical situations you might find yourself in, so you can evaluate your skill level and interest.
A researcher from the surgery department comes to you with the consultation request, “I have some data, and I now need some help with the analysis.” The researcher gives a short description of the study. It involves a retrospective data collection from n=50 patient records who underwent Therapy A and n=50 patients from the same clinic who underwent Therapy B. It was not a randomized study, but a series of patients who went through the clinic during the past year. The investigator wants to show that Therapy A is better. You ask to see the study protocol. In the statistical section, it states simply, “The data will be analyzed with chi-square tests and analysis of variance. The sample size of n=50 per group provides adequate power10.” The reference number 10 is a surgery research article, not an article on sample size determination.
A researcher from the art department comes to you at the design stage of a study. She says, “I have developed an art therapy program that I think will help children with muscular dystrophy. As you probably know, muscular dystrophy is a disease in which muscles of the body get weaker and weaker and may slowly stop working. I think if I can get the kids meaningfully engaged in expressing themselves through drawing and painting, it will slow their disease progression. Can you help me with this study?”
A researcher from the sociology department comes to you with the request, “I have a grant that is due in one week. I would have come to you sooner, but I just found out about the grant opportunity one month ago and I wanted to have a good part of it done so you could see what it is about.”
Common Theme One
The common theme of these situations is that you are now on the spot. You have to display confidence and take charge, or you will lose your credibility as an expert. These researchers are depending on you to make their projects successful. You are now in a very applied situation, which requires you to think way outside of the statistical theory box. You have to be much more than a statistician—you have to be a researcher and think like a scientist. You have to not be intimidated by the research client’s field, which is unfamiliar to you. Even with your own field, which is statistics, you cannot get away with specializing in just one or a few topic areas. Your consulting clients rely on you to be an expert in all areas of applied statistics, providing the best solutions available and tailoring these statistical methods to their specific research problems.
That is pretty much how your job interview will go, as well. The interviewer, who is trying to predict your success in an academic applied statistical consulting, will provide you with a research problem and ask how you would solve it. Your statistical theory classes are essential training, but questions about theory are not likely to come up in the job interview. To prepare, if you are still in school, you should take as many applied statistics and research design classes as you can from departments all across campus. Also, getting work experience while earning your degree is crucial.
Now let us test your skill at grasping what is involved with these three scenarios.
Situation One Solution:
From the sample size determination of the protocol, you can tell a statistician was not involved in designing the study. The researcher apparently found an article in a surgical journal, not necessarily involving the same therapies the researcher is studying, in which a significant difference was observed using a sample size of n=50 per group. It is surprisingly frequent for nonstatisticians to think this qualifies as a sample size determination. The statistical methods of the protocol appear to be some statistical tests thrown in to sound credible, so it will be up to you to suggest better statistical approaches.
One thing you should bring up in your meeting with this investigator is whether it would be sufficient to show that Therapy A is no worse than, or noninferior to, Therapy B. If the two therapies are essentially identical in effectiveness and complications, but Therapy A is less expensive, a noninferiority study approach is a better study design than the traditional superiority, or “find a difference,” approach. Most researchers do not think about this, so it is up to you to bring up the topic before looking at the data.
Something else you should recognize is that since the study was not randomized, being an “observational” study, confounding is going to be present. Since it is a case series, some form of selection bias is likely to be present, as well. The concepts of confounding and bias are taught in epidemiology courses—every statistician should take an epidemiology class, or at least read one epidemiology textbook. The control for confounding is going to require some version of a multivariable regression model. Addressing bias might require a sensitivity analysis.
Situation Two Solution:
Since a degree in art does not require research training, you will probably have to take the lead with all research aspects of this project. The study is not a bad idea, as some benefit to the children with this disease could come out of it. It is not likely that art therapy will successfully slow the disease progression, or at least the effect will be very small. It would require an impractically large sample size to have adequate statistical power. So, you should make physical function a secondary variable, being content to show a “trend in the right direction” descriptively. Then, make something like “life satisfaction” or “self-esteem” your primary outcome, using an existing standardized scale, and power the study for that outcome.
What to use for a control group will take some thought. What will help with that decision is determining what the researcher would want to say in her article when she publishes the result. It is possible that the improvement in self-esteem is due largely to the attention the researcher gives the children during the art therapy sessions, rather than the art activity. If that is fine with the researcher, then you would only need to compare against a control group in which no art therapy is given. If it is important to the researcher to conclude that the effect can be attributed to the art activity itself, then a control group in which the researcher is giving the children the same amount of attention, but with a different type of activity, would be required. This is a good example of the statistician needing to think way beyond statistical tests.
Situation Three Solution:
Grant writing is a very advanced statistical consulting skill. Frequently, you have to help the researcher revise his study aims as the first step, which requires that you be a better researcher than the research client to even recognize that the aims are not correctly conceived. In the end, the grant reviewer for the funding agency, in deciding if the project should be funded, will look to see if there is perfect alignment between aims, testable hypotheses, statistical method, and sample size determination.
You cannot just cut and paste statistical methods into the grant; they have to be perfectly tailored, and you have to convince the reviewer why those statistical methods are the best methods for testing the hypothesis. It is almost always a rushed process, so you do not have time to learn as you go—you have to be an expert from the start. Since funds are limited, the reviewer must find critical faults with 90% of the grants, so these can be eliminated from the competition. A lot is riding on your performance, frequently more than $1 million in potential funding. If you make a mistake, that money will be lost, since the statistical sections of a grant are an easy target for the reviewer to find an excuse for not funding the study.
Common Theme Two
A second common theme of these scenarios is that there is a lot more to consulting than just analyzing data. The analysis part is there, to be sure, but you cannot get by with just crunching numbers. You have to have a “big picture” perspective and expertise of the whole research process. If this sounds like a fun challenge and you do okay with deadline stress, then statistical consulting is a great career path.