Ron Wasserstein, ASA Executive Director

Assessing the quality of academic programs has become an increasingly important issue. Such assessments can have major impact on choice of graduate program, careers of individual researchers, funding opportunities, and strategic decisions. University administrators, politicians, and thought leaders are looking for ways to assess the quality of programs out of concern for cost and societal return on investment for higher education.

However, as Nick Fisher, a statistician who specializes in performance measurement for enterprises and programs, noted, “None of the existing methodologies developed to carry out these assessments seems to be satisfactory.” According to Fisher, some of the limitations derived from failure to be specific about the diverse quality requirements of different target audiences, which meant there was no sound basis for identifying suitable indicators of the quality of graduate programs. He thinks bringing statistical ideas and methods to bear on these issues could lead to greatly improved assessments.

With this in mind, the ASA conducted a two-day workshop, funded through the ASA’s Member Initiative Program, during which Fisher and 16 others evaluated methodologies and developed an agenda for further research. They determined that at least the following five elements are worthy of note:

Creativity in discovery. An effective academic institution fosters an environment in which the creative component of discovery is enhanced.

Quality versus productivity. Quantitative approaches for ranking departments or programs rely largely on measures of productivity because they are readily available and less susceptible to the criticism of subjectivity. But the quality of departments and programs is more inherently qualitative.

Impact of measurement on behavior. Setting productivity targets, rather than continually improving the workplace, may well be counterproductive. There is evidence that this also may be true for graduate programs.

University goals sharply differ from corporate goals. There is a tendency for many to view universities as corporations and to assess a university’s quality and effectiveness as they would that of a corporation. However, insisting on the narrow corporate model risks diminishing the intellectual environment that nurtures creativity and discovery.

Transparency is better than rankings. The “inputs” and “outputs” of academic institutions and programs should be publicly available. However, transparency should be moderate, lest it stifle frank discussion.

Workshop participants evaluated a number of methodologies for ranking graduate programs and developed an agenda for further research. Read the entire report.