Probability Statements Should Be Introduced into Scientific Testimony
A probability of a match, unless that probability is practically zero or one, would be a key first step to improving forensic science. In the case of the convicted Martin Luther King Jr. assassin, the James Earl Ray rifle was determined to be an inconclusive match to the assassination bullet by all of the several examiners used by the House Select Committee on Assassinations (HSCA). A finding of an 80%, 50%, or 20% chance of a match, along with an interval of uncertainty for that chance, would have been informative to Congress, the public, and possibly the judicial system. In current murder cases, it would provide a measure of the strength of evidence.
Current presentations in court state that when the match of the defendant’s gun to a murder bullet is inconclusive, the prosecutor can state that the bullet could well have come from the defendant’s gun. A probability for a match would be helpful to assess a proposed match. In addition, matches should be specific to a subclass of weapons. The uniqueness of firearm toolmarks has never been established. Using probability statements that can be defended is in the best interests of the defense and prosecution, and it will help the courts to make correct decisions more of the time. Such statements also are supported by the National Resource Council’s Strengthening Forensic Science in the United States: A Path Forward, which recommends establishing a National Institute of Forensic Sciences.
Methods that produce probability statements have yet to be tested and will require the cooperation of law-enforcement agencies. I was recently part of a group invited to submit a full proposal to the National Institute of Justice to test a method in District of Columbia (DC) criminal cases that would produce viable probability statements that could be defended through an application of Bayes’ theorem. While the DC Public Defender’s Service agreed to participate in the proposed research, the DC Metropolitan Police Department did not, so the full proposal did not go forward. Without the cooperation of law enforcement, having bullets and weapons available for experimentation is impossible without case-by-case litigation.
A successful test of this method would be a good precursor to trying the method on the bullet fragment recovered from Martin Luther King Jr. It would provide the probability of a match to the James Earl Ray rifle and rifles with similar subclass characteristics.
While automated identification in our proposal was not available at the time of the HSCA study, Bayes’ theorem and Bayesian methods were. Had there been input from statisticians, it is likely that information that is more useful would have been provided to Congress.
Much needs to be done to make forensic science more scientific. In “Probability Statements Should Be Introduced into Scientific Testimony,” I make the case for introducing probability statements in science testimony. That faces resistance, however, in that a key police agency has declined participation in a study of one proposed method.
The NRC report Strengthening Forensic Science in the United States: A Path Forward had many recommendations, one of which I’ll highlight. To address the identified lack of rigorous certification programs for forensic scientists, of strong standards and protocols for analyzing and reporting on evidence, and of peer-reviewed journal articles on scientific bases and reliability of many forensic methods, the report called for the establishment of a National Institute of Forensic Sciences (NIFS). Requirements for the NIFS include the following:
The NRC also noted that no federal agency exists that meets these well-considered and important criteria.
The Innocence Project, based in New York City, is taking a coordinating role in implementing many of the NRC forensic science recommendations through a coalition called Just Science. I have signed their petition to Congress to establish an agency like NIFS and would encourage readers to do the same.
It is time for forensic science to become part of the mainstream scientific community. Probability statements in scientific statements are one step. NIFS is another. Statisticians should be integrally involved by educating the public about the weaknesses of forensic science, urging and developing solutions, and working with forensic scientists.
I agree that forensic scientists should characterize their findings in a quantitative manner, but I don’t understand what the term “chance of a match” means. If the “chance of match” is 80% does that mean there is an 80% chance that the matching items have a common source? (If so, how would a forensic scientist be able to make that assessment without considering evidence beyond the forensic analysis?) Or does it mean there is an 80% chance of a match IF the matching items have (or do not have?) a common source? The latter interpretation seems more tenable, but will members of a jury understand it?
The chance of a match here is to to a common source. The toolmark examiner in these court cases has already declared a match, by methods that are not based upon verifiable statistical foundations (as stated by the NRC). The standard testimony is that the match is a practical certainty to a single weapon. Some courts have modified that to a better than 50% chance (and I believe they mean to the alleged murder weapon.) A proper error statement should have verifiable error rates. One error rate for matches that should have been found that were not. Another for matches that are declared but are really not matches. Two NRC panels found that there is no basis for claiming a bullet matches to a unique weapon. A suggestion is that an estimate be made of the number of weapons that could match toolmarks on a bullet. This can be done by sampling, by modeling the manufacturing, and wear process, and by searching computer records of toolmark images. Much as the NRC committee did for compositional bullet lead analysis (CBLA) a match should be stated to a class of weapons (or for CBLA bullets with common or coincidentally similar sources.) With research a reasonable upper bound to this number may be estimated. It will depend upon weapon, ammunition used etc. Thanks for your question