Both agencies acknowledge help of ASA forensic science committee

The U.S. Government Accountability Office (GAO) released in December 2014 its report on the scientific approaches in the FBI investigation of the 2001 anthrax attacks. As the title indicates, ANTHRAX: Agency Approaches to Validation and Statistical Analyses Could Be Improved, the GAO recommended that the FBI develop a framework for validation and statistical approaches for future investigations, with which the FBI agreed.

Early in 2013, the GAO requested help from the ASA Ad Hoc Advisory Committee on Forensic Science in reviewing statistical aspects of the GAO’s planned methodology for their investigation and a draft of the final report. The committee’s role is acknowledged in the GAO report.

Karen Kafadar, chair of the ASA forensic science committee, welcomed the GAO engagement: “The fact that an important government agency like GAO contacted the ASA for its statistical expertise is a positive development for our profession. And the willingness of our subcommittee to provide this expertise, amidst their other duties, is highly commendable. Statisticians are valuable in all areas of science, including forensic science, which this interaction demonstrates.”

The GAO report also discusses the FBI laboratory’s efforts to increase its statistical expertise. They cite an FBI official who described efforts to build formal forensic statistical expertise both internally and externally through a visiting scholar program, a working relationship with members of the ASA forensic science committee, and engagement with other federal agencies.

Kafadar and Hal Stern, committee vice chair, acknowledged this working relationship, which included two visits in 2013 to the FBI laboratory and subsequent discussions. Stern said, “Committee members have enjoyed interacting with FBI scientists. We have expressed our willingness to provide statistical support for forensic science research projects and hope that the FBI will take us up on our offer.”

After a 2011 National Academy of Sciences (NAS) study—which the FBI commissioned—concluded it was not possible to definitively determine the origin of the bacterium used in the 2001 attacks, members of Congress asked the GAO to independently investigate scientific and technical issues the NAS was not asked to review.

The GAO’s focus was the FBI’s genetic test development process and statistical analyses. As the GAO report summarizes, “The genetic tests that were conducted by the Federal Bureau of Investigation’s (FBI) four contractors were generally scientifically verified and validated and met the FBI’s criteria. However, GAO found that the FBI lacked a comprehensive approach—or framework—that could have ensured standardization of the testing process.”

The specific recommendations are for the FBI to work with the Department of Homeland Security on verification and validation framework to be applied at the outset of an investigation involving any microbial pathogen. “It should (1) incorporate specific statistical analyses allowing the calculation of statistical confidence for interpreting the results and specifying the need for any additional testing to fully explore uncertainties relative to the type of genetic test being validated and (2) [be] applied and adapted to a specific scenario and employ multiple contractors.”

The report also recommends the FBI “establish a general statistical framework that would require input from statistical experts throughout design and planning, sample collection, sample processing, sample analysis, and data interpretation that can applied and adapted to address a specific scenario involving an intentional release of B. anthracis or any other microbial pathogen.”