Practice What We Preach
RPFHS concluded by citing ASA/CB, which appears to be the most comprehensive study of AP statistics so far. I reatly applaud this much needed effort by ASA/CB, and appreciate its careful discussions of various limitations and difficulties. I also particularly appreciate the various steps taken during the study to reduce biases, especially the non-response bias, and its call for further studies in order to make causal inferences about the impact of the AP statistics. Real-life causal assessment/inference is never easy, especially for a large and complex program such as AP statistics. And the closer a study is to real life, the easier it is for us academics to criticize, especially for those of us who never get our hands “dirty” with real data, something I have been learning as my hands have been getting increasingly dirty. I therefore want to preface my discussion of ASA/CB study with a grand disclaimer: the discussion is not intended as criticism of any sort for any individual involved in the study nor the study as a whole. Rather, the sole purpose here is to put its findings into perspective based on the evidence provided by the report itself, to further caution ourselves to not put more faith in the findings than the quality of the data can possibly support.
As explicitly acknowledged as a serious sampling bias, due to the lack of contact information, the ASA/BC survey population was those who took the AP Statistics Examination, not those who have taken AP statistics classes. Furthermore, as given in the Appendix of ASA/BC’s full report, the response rates varied from 9.3% to 23.7% across all subpopulations examined, and there is strong evidence of non-ignorable nonresponse mechanism. For example, among the five exam grades, the response rates went from 9.3% for the lowest buy ativan us pharmacy Grade 1 to 22.9% for the highest Grade 5, essentially in a monotone fashion. Indeed, ASA/BC reported that the results from adjusting nonresponse using the demographic and exam performance data “differ substantially” from the adjusted ones, and “the unadjusted approach overestimates the proportion of students who indicated a greatly or somewhat increased interests in statistics as a response to the AP Statistics course, and underestimates the proportion of students whose interests greatly or somewhat decreased.” This finding is consistent with our intuition that those who did well on the exam are more likely to be those who have positive experiences with AP classes, just as our intuition would suggest that those who took the course but did not take the exam are more likely to have had more negative experiences with their AP statistics courses than those who took both the courses and the exam.
Whereas ASA/BC correctly emphasized the need to adjust for the non-response bias, it can only do so for the non-responses biases that can be explained by the covariates measured. Therefore, the ASA/BC’s recommendation that “Because of the corrections above, it is appropriate to consider these results as representative of the entire population of approximately 230,000 examines” is based on a yet-to-be-tested assumption that the non-response mechanism can be adequately captured by the covariates measured (the reported ones are year of exam, exam grade, gender, and six regions defined by the College Board). As we all know, the wonder of statistics is that we can infer quite reliably the opinions of a population of hundreds of millions based on a sample of a few thousands or even few hundreds, if the sample is a probabilistic one without any serious self-selection mechanism in play. In contrast, inferences of a population of 230,000 based on a highly self-selected 408 respondents (the actual sample size of the ASA/CB study) are not something that most (any?) professional statisticians would be willing to vouch for without carefully laying out all the heavy assumptions.
As a matter of fact, even if by luck our inferences for the 230,000 population are dead on, they still provide little information about how many have been so turned off by poorly taught AP classes that they decided not to take the AP exam or touch statistics ever again. Most critically, thinking from the causal inference perspective, even with 100% response from everyone who ever took AP Stat classes, we still need to work hard to construct an appropriate “control group”, perhaps via propensity score matching, in order to come close to assessing the real impact of AP statistics. Again, ASA/BC correctly acknowledged, repeatedly, that this lack of comparison/control groups makes it “difficult to make comparisons and impossible to draw causal inferences.”
There is of course nothing special about AP statistics in this regard; the same caution and rigor should be exercised when evaluating other AP programs, and indeed any other educational program. However, in responding to my point that it is our professional responsibility to serve as a (not “the”) police of science (Meng, 2009), one reader wrote to me that we should self-police, namely to critically examine how we statisticians invoke statistical evidence and arguments in our own work before we police others. I, of course, completely agree: only when we practice what we preach can we convince others to take our advice seriously. But most critically, even if we don’t care about how sound our arguments are or how others think of us, this is a case where we really want to know how effective the AP program is in real terms for the sake of our own future. I therefore very much appreciate the many cautions taken by ASA/CB in presenting its findings. Indeed, the full report as posted on the College Board website suggests that the overall conclusion of the study is substantially more cautious than the impression the quotes in RPFHS might generate.
As a matter of fact, the abstract of the ASA/CB full report reads
The abstracted was dated 2/28/2009, the same month the full report was issued as Patterson, 2009; and it was downloaded on Oct 31, 2009 here
In its “Conclusion” section, which was reproduced in the Amstat News (May, 2009), the entire paragraph was kept except that the word “not” was removed from the first sentence, and a more qualified “not” message was appended at the very end of the above paragraph: “It may not, however, affect students’ choice to pursue statistics as a major.” This contradiction between the abstract and the conclusion could simply be an oversight or could reflect a compromise perhaps between different versions/revisions to prevent misquotations of the study.
There is indeed at least one obvious discrepancy or typographical error in the full report; the text mentioned several times that response rate is higher for female, hereas the appendix listed that male’s response rate is 23.7% and the female’s rate is 15.0%.
From a critical self-policing perspective, adding “not” or not brings to mind the quote by Mark Twain, which was allegedly switched from “Some congressmen are …” to “Some congressmen are not …” – the limitations of the sample are just too great to draw any statistically sound general conclusions in either direction. I therefore fully agree with ASA/CB’s emphasis that “There is certainly more research needed in order to make causal inferences about the issues presented in this analysis.”
Leave a Reply