Hal Stern and Len Stefanski

The September issue features two articles (one in Theory and Methods and one in Applications and Case Studies) that were presented and discussed at the 2011 Joint Statistical Meetings in Miami, Florida.

Theory and Methods

What good is an estimate without some measure of its uncertainty? Most statisticians—frequentist and Bayesian—would answer “not much.” Indeed, providing a measure of uncertainty with our answers helps distinguish statistical data analysis from nonstatistical data analysis. This ‘world view’ provides the motivation for Eric Laber and Susan Murphy’s article, “Adaptive Confidence Intervals for the Test Error in Classification,” which constructs confidence intervals for the estimated test error of a learned classifier.

The authors eschew the common frequentist approach of calculating a point estimate and then approximating its sampling distribution and assess the variability in an estimate of the test error via direct construction of approximate upper and lower confidence bounds. The authors note that the usual point-estimation/sampling-distribution approach is problematic for assessing estimated test error variability because some classification algorithms result in a classifier that is a non-smooth functional of the training set and, even when the classifier is a smooth functional of the training set, the test error is the expectation of a non-smooth function of it. The lack of smoothness is the culprit in the failure of standard approaches to provide reliable confidence intervals.

Laber and Murphy start by constructing smooth, data-dependent upper and lower bounds on the test error. These bounds are sufficiently smooth that the bootstrap can be used to construct valid confidence intervals for them. Although conservative in general, the bounds are adaptive in the sense that exact coverage is delivered under certain settings, thus the appellation adaptive confidence intervals.

The article is nicely complemented by a series of discussion articles that offer modifications of the adaptive confidence interval approach that ameliorate its conservative tendencies, extensions of the approach to handle particular classes of classifiers, and alternative strategies for achieving the same goal.

Variable selection, model identification, and model uncertainty cover a lot of territory in contemporary statistics research. In “Rao-Blackwellization for Bayesian Variable Selection and Model Averaging in Linear and Binary Regression: A Novel Data Augmentation Approach,” Joyee Ghosh and Merlise Clyde note that advances in computational algorithms have made it possible to take a Bayesian approach to these key issues, but that success has been limited in large problems. As they write, “While it is straightforward to formulate the model uncertainty problem under the Bayesian paradigm, its implementation often becomes nontrivial in large problems, particularly with highly correlated predictors.”

For an orthogonal design (i.e., one with uncorrelated predictors) with suitable prior distribution, the posterior probability of any model can be calculated without exhaustive enumeration, and computation scales linearly with the predictor dimension p. Keying off the computational advantages of the orthogonal design case, the authors propose a novel data-augmentation strategy method wherein an observed nonorthogonal design is enlarged by, at most, p new rows to obtain a new design matrix having orthogonal columns. Then, the data are further augmented by generating the ‘missing’ response variables corresponding to the added rows of the design matrix. The authors show that their data augmentation scheme does not alter the original posterior distribution. Methods for estimating quantities of interest such as the posterior probability of any model are derived using Rao-Blackwellization.

Applications and Case Studies

The Sleep Heart Health Study is a landmark study of sleep and its effect on health. The multicenter study measured multiple physiological signals during sleep (including EEG) to identify potential biomarkers of disease and study the association between such biomarkers and health outcomes. The study is one example of many biomedical studies that challenge statisticians because of the large amount of image data collected.

In “Population Value Decomposition: A Framework for the Analysis of Image Populations,” Ciprian Crainceanu, Brian Caffo, Sheng Luo, Vadim Zipunnkkov, and Naresh Punjabi use a matrix decomposition approach to analyze data from the study. They focus on two-dimensional (time by frequency) image representations of the EEG data for each of two visits for more than 3,000 subjects. If the original images are represented as two-dimensional matrices, then the population value decomposition (PVD) approach uses a matrix factorization to identify population-level features and then a smaller subject-specific matrix that represents the individual image relative to the population-level features. Both the population-level features and the subject-specific representations provide useful information about the sleep study. Elements of the subject-specific matrices can be used in subsequent analysis, for example as predictors for health outcomes. The paper is supplemented with four discussion papers that address the benefits and limitations of the approach and suggest avenues for future research in the analysis of large imaging data sets.

The initial phase of clinical trials for new oncology drugs involves determining the largest dose that can be used in subsequent studies with an acceptable risk of toxicity (a target is usually set, perhaps 30% probability of toxicity). The continual reassessment method (CRM) is a popular approach in which a dose-toxicity relationship is estimated using a sequence of patient cohorts with the goal of identifying the dose with toxicity probability closest to the target. One challenge is that toxicity outcomes for previous cohorts may not be known when the time comes to assign the new cohort to a dose. Ying Yuan and Guosheng Yin present an approach that addresses this limitation in their article, “Robust EM Continual Reassessment Method in Oncology Dose Finding.” The authors propose to treat the unobserved toxicity outcomes as missing data and use the familiar EM algorithm to obtain maximum likelihood estimates of the parameters in the dose-toxicity model. The authors also show how to make the results more robust by considering a range of dose-toxicity models and using model selection or model averaging approaches to inference. Simulation studies show that the EM-CRM approach reliably finds appropriate dose levels and requires fewer patients than alternative approaches that wait for the missing outcomes to resolve. The method is applied to a prostate cancer trial.

There are many other interesting Theory and Methods and Applications and Case Studies articles, as well as a set of informative book reviews. The full list of articles and a list of the books under review can be viewed on the JASA website.