Jake Basson

This column is written to inform ASA members about what the ASA is doing to promote the inclusion of statistics in policymaking and the funding of statistics research. To suggest science policy topics for the ASA to address, contact ASA Director of Science Policy Steve Pierson at pierson@amstat.org.

Jake Basson was born in Milwaukee, Wisconsin, and earned his BS in biology from the University of Wisconsin-Madison. His graduate work at Washington University in Saint Louis was in statistical genetics of complex traits with a focus on gene-gene interactions in blood pressure.

I became interested in science policy and motivated to improve data literacy after completing my PhD in biostatistics and witnessing polarizing discussions about such challenging issues as climate change and the alleged link between vaccines and autism. I chose to pursue a Science and Technology Policy Fellowship with the American Association for the Advancement of Science through their Big Data and Analytics program. I took a position at the National Institute for General Medical Sciences (NIGMS) in a newly revamped analytics office that is leading the charge toward data-driven decision-making at NIGMS. What I’ve found here is encouraging to any statistician hoping for a more active role for analytics in policy formulation, implementation, and evaluation.

Situated within the office of the NIGMS director, the Office of Program Planning, Analysis, and Evaluation (OPAE) grew rapidly in 2015, going from two to 10 full-time employees. This growth in both personnel and resources—achieved through a combination of new hires and reassignments of existing positions—reflects the NIGMS commitment to ensuring a heightened level of stewardship and maximal return on investment for tax-payer dollars.

To that end, the OPAE focuses on performing data-driven analyses and evaluations of both NIGMS research programs and funding mechanisms and on communicating the results of such analyses to key decision-makers within the institute and National Institutes of Health (NIH). Its team of multidisciplinary data scientists puts the power of data analytics and statistics to use in their advisory capacity to the institute’s leadership, reporting on such matters as trends in grant applications, funding decisions, research outcomes such as publications and citations, and other data relevant to the efficient and effective administration of programs and policies.

Most exciting to me, I’ve observed that OPAE analyses are not rejected, ignored, or shelved. Instead, NIGMS leadership actively embraces the role of analytics as a crucial part of a data-driven, accountable decision-making process. As a result, data generated through the OPAE’s analyses are consistently used to inform the forward momentum of the institute against an ever-changing landscape of science, technology, fiscal responsibility, and legislative policy.

Informing Policy

One example of how the OPAE’s data analyses contribute to decision-making can be observed in its recent analyses of laboratory productivity as a function of the amount of funding received from both NIGMS and NIH sources.

Building on a prior analysis that focused on outcomes from a single year, this work looked at scientific funding versus productivity over a three-year period and used a broader set of metrics than were considered in earlier analyses. OPAE’s analysis illustrated the law of diminishing returns, with funding beyond a certain threshold not producing proportional gains in productivity as measured by publication and citation rates.

NIGMS director, Jon Lorsch, presented these data to the institute’s advisory council in September of 2015. In particular, the data provided firm footing to defend the institute’s “750k policy” that stipulates grants to investigators who would have more than $750,000 in funding from all sources undergo an additional round of review by the institute’s advisory council.

Some stakeholders mistakenly perceive the policy as a hard cap on funding, while others focus on specific counter-examples of highly funded investigators who are also perceived to be highly productive.

Having concrete data has been key to NIGMS’ decision to continue implementation of its 750k policy. In addition, these results suggest funding more labs at lower levels may achieve better returns than funding fewer labs at higher levels, providing support for NIGMS’ decision to diversify rather than concentrate its portfolio. This insight informed the design of the recently launched Maximizing Investigator’s Research Award funding mechanism, which was developed in part to contribute to the breadth and diversity of the institute’s research portfolio while maximizing scientific creativity and flexibility.

Not only has this productivity analysis played an important role in informing policy formulation within NIGMS, but the institute’s leadership is making a concerted effort to communicate and disseminate the findings more broadly. For instance, these results were shared with the directors of other NIH institutes and centers to help inform considerations related to the effective funding of investigators in challenging or unpredictable fiscal times. A short video also was created by Lorsch for the biology outreach site iBiology, presenting these data and his thoughts about their implications to the scientific community at large and the general public.

Evaluating Programs

As I noted above, the institute’s primary role is to maximize the potential for important scientific discoveries while ensuring a return on taxpayers’ investments in the areas of fundamental biomedical research and training. To this end, OPAE is developing metrics and techniques for evaluating the institute’s research programs and funding mechanisms. In the last year, OPAE conducted several focused data analyses to support outcomes evaluations of two major NIGMS programs: the Biotechnology Research Resources program, which funds numerous centers to develop novel technologies and provide research resources across a wide range of scientific fields, and the National Centers for Systems Biology program, which has provided more than $350 million since 2003 to promote development of multidisciplinary research, training, and outreach focused on systems-level biomedicine.

Bearing in mind the collaborative nature of science, the OPAE also recently used geographic analysis of collaboration networks to evaluate the efficiency and synergy of program project grants and the contributions of team-based versus individual science. Further, the OPAE used yet a different approach to determine whether the R37 MERIT awards are achieving their intended goal.

The R37 MERIT award is a modification of the traditional R01 research project grants and is awarded to highly meritorious investigators to provide an extended period of stable funding to allow them to take more research risks. Using medical subject headings (MeSH terms), the OPAE measured changes in scientific focus over the life of these awards to assess the extent to which R37-funded investigators branched out into more diverse areas of research relative to their R01-funded counterparts.

Supporting a Data-Driven Institute

As the above examples illustrate, the OPAE contributes to ensuring that robust, reproducible, and meaningful data inform the institute’s pursuit of its mission. Currently, the office is in the process of creating a performance monitoring dashboarding system so program officers and other staff can readily access data related to the institute’s performance in the pursuit of the goals listed in its strategic plan. Development of this dashboarding capability is part of a broader effort to expand the capacity of the institute to access, analyze, communicate, and use different data types.

Staff members within the OPAE are also working to build relationships with other NIH institutes and centers and with institutions outside the NIH to access a wider range of data and broaden portfolio analysis techniques. Representatives from the OPAE are also participating in a new NIGMS initiative to assess the ways in which collaborative science is currently supported and whether development of alternative mechanisms is warranted.

Collectively, these examples illustrate that the curation, analysis, visualization, and communication of robust, reproducible, and meaningful data play an important role in many of the NIGMS’ functions. Statisticians rejoice!