Organized and Chaired by Xiaoyue Maggie Niu, The Pennsylvania State University

Mary Batcher (Mary) recently retired from Ernst & Young, where she began a statistical practice that started with $30,000 in revenue in 1997 and was more than $4.5 million when she retired in 2015. After leaving Ernst & Young, Batcher teamed with two other retired statisticians to start a consulting company, BDS Data Analytics.

James L. Rosenberger (Jim) is professor of statistics at Penn State and director of the Statistical Consulting Center and Outreach and Online Programs. He served as vice president of the ASA from 2013–2015, is a member of the board of directors of the National Institute of Statistical Sciences, and served for two years as a statistics program officer at the National Science Foundation.

Nathaniel Schenker (Nat) is associate director for research and methodology at the National Center for Health Statistics (NCHS) and an adjunct professor at the University of Maryland. Before starting at NCHS in 1999, he was a faculty member at the University of California, Los Angeles, Department of Biostatistics, and, before that, a mathematical statistician at the U.S. Census Bureau. He was president of the ASA in 2014.

Introductory Statements

Jim: This panel provided a great opportunity to think about my meandering path from a math major in college to an academic applied statistician with 40 plus years of experience. It has been a very rewarding journey having collaborated with many interesting researchers on challenging projects.

Upon graduating from college with just one statistics course and one FORTRAN computer-programming course, I landed a job at NYU Medical Center as a data analyst. I was responsible for the data analysis for multiple heart and stroke studies, creating random access databases, writing statistical programs, and using BMD programs. This was collaboration at its best. I was also fortunate to work with investigators who shared coauthorship on this joint work, which stimulated me to pursue additional education in statistics. After four years of working and earning a master’s degree, I was motivated to return to full-time graduate study at Cornell University, in statistics with a biometrics orientation.

After graduate study, I accepted a tenure-track faculty position at Penn State, which included teaching our statistical consulting course and working with faculty researchers in the college of agriculture. However, my early interactions with medical research inspired me to accept a postdoctoral position at Harvard University with Fred Mosteller, who—as my mentor—invited me to participate on various projects with participants including John Tukey, Tom Louis, Dave Hoaglin, John Emerson, and John Bailar. It was a heady year for a young academic with broad interests in applied statistics. Fred demonstrated a model for mentoring, which I wish I could emulate with the many students and colleagues I work with.

One project I worked on that year was a study for the editors of the NEJM to analyze the quality of the statistical reporting of articles published in the journal. This resulted in new guidelines for authors in studies that were to be published, highlighting the importance of randomization, blinding, and treatment of outliers for ethical reporting on the analysis of data.

My own work at Penn State was characterized by collaboration with researchers from a wide range of disciplines. Most of these research collaborations began with researchers coming to the Statistical Consulting Center for advice and, if the problem required more than a straightforward application of statistical methods, led to further investigation and joint funding and publications.

The Statistical Consulting Center at Penn State exists for three purposes, in order of importance:

1. Teach statistics students the art and science of statistical consulting through engaging in real-world research projects
2. Provide excellent statistical advice to clients to improve the quality of research at Penn State
3. Find new and interesting challenges requiring further statistical methodological research

This mission statement has guided the center over many years, which continues to serve an important educational role in the department of statistics.

Mary: My experience includes 11 years of internal consulting in the government, 17 years of business consulting, and one year of independent consulting. My government experience was at the IRS in Statistics of Income, where we provided statistical support for new initiatives and quality audits. At Ernst & Young, we worked through our tax and audit colleagues to serve their corporate clients, providing primarily sampling services. Now, as an independent consultant, I do not have a readily available network to rely on to connect me to clients.

Common Elements

There are several elements common to statistical consulting in the three settings, along with some differences. The common elements include the consulting process and the need to market statistics. In all three settings, it is necessary to work with clients, both internal and external, to help them become good consumers of statistics and partners in data collection. It is important in all three settings for them to gain an appreciation of statistical variation so they do not over-react to small changes.

The statistical consulting process also contains the same basic elements, regardless of the setting. Although there will be a series of interactions throughout the process, there is generally an initial meeting to plan the project. The initial meeting should be carefully planned; the meeting plan should include specification of roles, key information to be gleaned, and development of an agenda for the client meeting.

Project Process

The basic project process starts with planning the initial client meeting, followed by a discussion of the problem with the client. It is important to listen carefully and to draw the client out. It is not uncommon for the stated problem or need to not be the actual need, which will come out in the discussion with some probing questions. Offering a quick solution to the initial problem described may prevent a discussion of the broader problem.

It is important to gain a solid understanding of data provided by the client. In my experience, it is not unusual for the data to include extraneous elements like data outside of the specified date range or out-of-scope elements. It is advisable to compute basic frequencies of all the variables in the data set, ask the client to review them and provide guidance about data anomalies, and approve the data before beginning work.

In addition to agreeing about the data, there should be a written agreement with the client about the timeline and fees for the project, as well as the scope of work to be done by the statistician and the client’s responsibilities to support the project’s completion.

As statisticians, it is very tempting to try to impress the client with the technical difficulty of our work. This is to be avoided; it might make us feel good, but it does not improve the client relationship. Similarly, we should avoid jargon as much as possible. When it is necessary to use a technical term, we should explain it as simply as possible. Often, the client does not want to have all the details of the statistics; he or she just wants to know that the work will be done correctly and in a timely fashion and will meet any applicable regulatory requirements.

Once the work is finished, a process check should be done. This can be as simple as a phone call to ask the client how the project went and whether there are any suggestions to improve the process.

Differences

In the three settings in which I have worked, there are some differences worth noting. Business is subject to fewer rules and regulations than government, so it is generally more flexible and can move faster, but it still has restrictions. As an independent consultant, the only restrictions are on the client side. Because of that lack of restrictions and oversight, the statistician bears all the responsibility for the quality of the work. It is important to build in checks and duplicate computations and to be an ethical service provider. This is critical, as the clients are often not able to tell good from bad statistical work.

Although, in my experience, it was necessary to market statistics to potential clients in all three settings, it is markedly different as an independent consultant. In government and business, I had internal clients to approach. As an independent consultant, I do not have a built-in client base. In fact, I am prohibited from working for companies for which I worked before retirement. I therefore have to be opportunistic and explore different avenues of outreach. It is also important to carefully target marketing efforts to reach the most likely clients.

I am still learning how to be an independent consultant, but I will pass on a few lessons learned. Initially, there were structural decisions to be made. I teamed with two other retired statisticians to form a statistical consulting company, BDS Data Analytics. We had to choose a name and decide whether to incorporate and under what structure—partnership or corporation. There were tax implications to consider in the decision. You can hire an attorney or accountant, but both cost money. If you are diligent and thorough, you can work it out yourself, but you have to do careful research and be compulsive enough to read it all carefully.

In building your business, it is important to set realistic goals for business growth. Unless you have a built-in client base, it is helpful to have other income to start. That could be a spouse’s income or retirement income. In building the business, decide on your statistical areas and likely clients. Do whatever market outreach makes sense—emails, articles in trade publications, a website. In addition and most importantly, use your network.

Finally, it is very exciting to build a new business, but you should do it with realistic goals, expecting it to take time to bring in clients and to let word of mouth help. And you should have fun and love what you do.

Nat: As the “government representative” on the panel, I’ll focus my comments on the most recent 17 years of my career, which I’ve spent as an “in house” researcher and consultant at the National Center for Health Statistics. Since many readers may not be familiar with government statistical work, I’ll describe NCHS and the work done there.

NCHS is the principal health statistics agency of the United States and one of the nation’s 13 principal statistical agencies. It is also part of the Centers for Disease Control and Prevention. NCHS’s mission is to provide statistical information that will guide actions and policies to improve the health of the American people. Such information is used, among other purposes, to document the health status of the U.S. population and selected subgroups; track the impact of major policy initiatives, including the Affordable Care Act; document access to and use of the health care system; identify disparities in health status and use of health care by race and ethnicity, socioeconomic status, other population characteristics, and geographic region; monitor trends in health indicators; support biomedical and health services research; and provide data to support public policies and programs.

To produce statistical information on health, NCHS has several, primarily survey-based, data-collection systems. Sources of the data include birth and death certificates, patient medical records, personal interviews (in households and by phone), standardized physical examinations and lab tests, and facility information. NCHS has four “data divisions,” each of which focuses primarily on one or two of those data sources. It also has two divisions that focus on cross-cutting analyses and research, developing methodology, consulting, and dissemination. This organizational structure is similar to that of several other federal statistical agencies, which have some units focusing on specific types of data or subject matter and others focusing on cross-cutting issues.

I’ve spent my 17 years at NCHS in one of its cross-cutting divisions, the Division of Research and Methodology (DRM), serving as senior scientist for 11 years and then as division director for the last six years. DRM is the central methodological research, development, and consulting unit in NCHS. It provides technical expertise to NCHS and other organizations while avoiding duplication of effort across NCHS; conducts research on and evaluation of methods for use in NCHS and the broader scientific community; promotes sharing of information and methods across NCHS; and advances the state of knowledge and application in the core areas of measurement, collection, analysis, and dissemination of data. DRM has three branches: the Center for Statistical Research and Survey Design; the Center for Questionnaire Design and Evaluation Research; and the Research Data Center, which provides analysts with secure access to restricted-use data.

Serving the public as an employee at the nation’s principal health statistics agency is rewarding in itself, but I’d like to mention several aspects of the job that I’ve particularly enjoyed. One is working on “real-life” problems and thereby making sure my research is relevant. I like to do theoretical and methodological work, but focusing on applications helps to keep me “grounded.” I often find myself working in teams with people having diverse skills, backgrounds, and interests. Such teamwork helps me learn about many topics. I’ve also found that explaining statistical issues to nonstatisticians prods me to think about statistical issues in various ways, which adds to my understanding.

Another aspect I enjoy is working on large-scale projects that are of interest to a lot of people, such as researchers and policymakers. I feel a real sense of accomplishment when such a project is completed. The projects often involve complicated statistical problems that have nonstandard solutions, and thus they necessitate conducting theoretical and methodological research, making assumptions, and using approximations. At NCHS, we often have to “push the envelope” with regard to methodology, under time and resource constraints, while maintaining the integrity of the data. This results in a job I would describe as “exciting and sometimes even scary.”

A final aspect that I’d like to cite is mentoring. I’ve had opportunities to work with and advise several more junior staff members, and such activities have been rewarding as both a teaching and learning experience for me. I also feel that one is never too senior to be a mentee, and I’ve never hesitated to reach out to more experienced people when I’ve needed advice, such as when I became a division director, which in some ways was my first true management position.

I’d like to conclude my introductory remarks by mentioning a few projects in which I’ve been involved at NCHS to give you a flavor of such projects and some of the statistical issues involved. One activity in which DRM staff are often involved is the sample redesign for NCHS’s surveys. For example, our flagship National Health Interview Survey recently underwent a sample redesign to incorporate new information from the U.S. Census and also to achieve new efficiencies. Because of increasing expenses of having field staff identify and list housing units, NCHS has now adopted an “address-based sampling” design using commercially available address lists. It has been necessary to evaluate the accuracy of those address lists and decide which areas of the country still require traditional fieldwork for listing, subject to cost constraints. The redesign also involved mathematical statistical work to inject flexibility to accommodate potential changes in emphasis of the survey, such as focus on demographic subgroups versus focus on geographic subgroups, for which different types of designs are more efficient.

Another project had the goal of bridging the transition between single-race reporting and multiple-race reporting in federal data collections. Standards published by the Office of Management and Budget in 1997 specified that respondents to questions on race would be allowed to choose multiple race groups, rather than the traditional single race group. While this gave respondents more flexibility and allowed for more accurate reporting, it also caused incomparability of reporting over time and across data systems. NCHS fitted bridging models using data on people categorized under both reporting systems and, in collaboration with the Census Bureau, applied those models to recent county-level census counts by age and race. his resulted in publicly available counts under the newer reporting system, along with bridged estimated counts under the older reporting system.

Missing data are a pervasive issue in statistics, including in government surveys. NCHS applied multiple imputation for missing income data in the National Health Interview Survey and for missing body scan (DXA) data in the National Health and Nutrition Examination Survey and released the multiply imputed data publicly. Both projects involved careful modeling and devising methods to deal with issues such as the need to transform variables, modeling survey data at both the family and individual levels, and handling missing data on a large number of variables in one survey and missing data that were not completely at random.

The participants in the above projects, besides having the opportunity to work on important, interesting problems, also are co-authors on various publications from the projects, many of them in peer-reviewed outlets.
Looking into the future, an important area of research and application will be the use of alternative data sources to supplement NCHS data systems. NCHS has an active data-linkage program that has linked its survey data to various administrative data sources. Together with the National Cancer Institute, it has combined information from two surveys to create small-area estimates of the prevalence of cancer risk factors and screening. Finally, it is investigating possibilities for using other data sources, such as web surveys and nonsampled electronic health records, to supplement its current data systems.

Questions and Answers

Q: What are the most different features that distinguish your environment from the other two? What kind of person/personality do you think would enjoy your work?

Jim: Effective collaboration requires someone who is sufficiently curious to learn the other person’s jargon and discipline. Academia provides an invigorating environment with a steady stream of new students to mentor and new potential collaborators to work with.

Nat: One of the two most different features is that, although statisticians at NCHS consult and collaborate often with people from other organizations, our main responsibility is to work with other “in-house” staff. A second feature is that we typically don’t need to write proposals to obtain funding for our work. A person who enjoys public service, who is curious and interested in both methodological work and subject-matter problems, who likes to work with others, and who is patient and friendly would enjoy my work. Additional important characteristics would include a solid training in statistics, good communication skills, and character and integrity.

Mary: I think a person with the requisite statistical skills who likes working with people and enjoys variety in the work and the challenge of solving new problems would enjoy my work.

Q: What is the most difficult thing you have to do on a routine basis? When have you felt like crying (and/or laughing) at work?

Mary: The most difficult and frustrating experience at work for me both in government and at Ernst & Young was doing annual performance evaluations, with the imposition of quotas on the different rating levels. This sometimes meant I had to give a person a lower rating than they deserved.

Nat: Like Mary, I’d also say conducting annual performance evaluations. I find it hard to use numerical scores to evaluate the work of my staff members, and I find that most employees (myself included!) like to think of themselves as above average, which is, of course, a logical impossibility. Moreover, it is difficult for a supervisor to give, and a supervisee to receive, constructive criticism, although that is important and useful. I much prefer regularly working with staff on improving their performance without having to evaluate them.

Jim: Mentoring students who haven’t grasped the importance of treating each client with respect. I have had to suppress dismay and anger when observing a student’s response to a consulting client, who had not taken the client’s goals seriously and simply produced a superficial recommendation to close out the case.

Q: How have the organizations you have belonged to expressed their appreciation of your contributions?

Mary: I received cash awards and annual bonuses, as well as verbal recognition, all of which I greatly appreciated. At Ernst & Young, I also had a pot of money I could use to make cash awards throughout the year to deserving employees for special efforts.

Nat: NCHS has expressed its appreciation in several ways. These include favorable performance evaluations, raises, and bonuses. NCHS also has an extensive award program, and I’ve been very happy to receive, for example, awards for science and leadership. I believe my recruitment to direct the Division of Research and Methodology over six years ago, which represented a new opportunity and learning experience, was an expression of appreciation (unless the agency thought my statistical work wasn’t very good and therefore wanted to move me into management! :)). Finally, a very meaningful expression of appreciation, which would apply in any organization, is the occurrence of requests from other staff for collaborations!

Jim: Early in my academic career, I was told by colleagues I should focus more on single-authored publications. Instead, I enjoyed the collaborative work with researchers in other disciplines and made it my focus. Despite this choice, I was granted tenure and promotion, though perhaps at a slower pace. In more recent years, collaborative research has become widely recognized and indeed essential for pursuing big science challenges.

Q: Do you see yourself more as a statistical consultant or collaborator? What do you feel is the difference?

Nat: I see myself and my staff more as collaborators than consultants. I think of consulting more as short-term assistance with solving a specific problem, and collaboration more as long-term teamwork on a project from beginning to end. I think statisticians can do a better job, even on specific problems, if they are concerned with the entire project and feel they are part of the team. I also prefer for project leaders to think of statisticians as team members, rather than simply “helpers.” Of course, as internal staff members at NCHS, all employees are part of a team that collaborates to monitor the nation’s health.

Mary: I think I am both. To me, a collaborator is someone who understands the business and advises more broadly on general problems. Sometimes, there is a narrowly defined statistical task and little opportunity to collaborate. The collaboration occasions are very rewarding and can lead to a longer-term relationship with the client.

Jim: I have worked in both roles. Mostly when engaged in shorter-term interactions that do not lead to publications, I consider myself in a consulting role. However, when collaborating with someone longer term with a challenging and worthwhile problem, the work is more satisfying and leads to a mutually beneficial relationship.

Q: How can academia better prepare students to become effective statistical collaborators?

Jim: I prefer to have the industry/government panel members answer this question. However, I feel we can do better than we have in the past by utilizing more of the tools of feedback. We have students view video of themselves after their interactions with clients, and we use peer observers during the consultant/client meetings. Then, we follow up by evaluating the students’ reports and recommendations.

Nat: Three important areas of preparation would be experience in using statistics to solve nonstandard, applied problems, experience at working in teams, and communication skills.

Q: What have you learned from your statistical collaboration experience that you apply outside the field of statistics (outside your job)?

Jim: I think the direction is mostly reversed. My broad interests in science and engineering (how things work) have made me a much better consultant and collaborator. I encourage statisticians to take courses, read, and explore interests in areas beyond statistics.

Mary: For me, it goes the other way; a lifetime of different experiences has taught me patience, listening skills, and problem solving ability that make me a better statistical collaborator.

Nat: I’ll also turn the question around and answer the following question: “What courses outside of statistics especially helped you as a statistical collaborator?” My answer would be the two literature courses I took as a freshman in college. They helped me learn how to write clearly and logically, which has been an important success factor in my career.

Q: What aspects of your job do you regularly trust to your colleagues/assistants?

Jim: Whenever possible, I try to have my assistants and students tackle a problem first and use me as a sounding board, or mentor, to discuss the approach and either confirm it or point in another direction.

Mary: As a manager, I trust the process we put into place for following established procedures and reviewing work. An initial discussion of the project approach and a final review, with agreement that I would be consulted on arising issues, were sufficient involvement for me with an experienced and talented group.

Q: Discuss your work/life balance and how you maintain it?

Nat: My family is very important to me, and I’ve always tried to follow the principle of “family first.” that can have different meanings, however. For example, as the sole wage earner in my immediate family, I’ve had to work diligently enough to keep and advance in my job so I can support my family. That occasionally has involved working late or on weekends. Primarily, though, “family first” has meant to me that I should be sure to have time to spend with my family. I’ve found several ways to accomplish that. One is to learn to say “no,” that is, not to take on so much work—even if it is attractive and interesting—that it forces me to spend most of my time away from the family. Another is to choose jobs with flexibility in scheduling and/or an appreciation of the importance of family. I’ve found both my academic and government jobs to satisfy those criteria. Also helpful is to choose job locations close to the extended family. For example, when we left the West Coast, where my wife’s parents live, we moved to the East Coast, where many of her extended family members live and my parents lived. Finally, it has sometimes been necessary to work during “abnormal hours” to spend “normal hours” with my family. For example, I needed to serve as ASA president outside my government job, so I often made use of saved vacation time or late-night hours to carry out my ASA duties.

Jim: Early in my career, I found myself working too much at home late at night. Preparing lectures, grading papers, etc., can easily spill into home life. I now try to put my academic work into focused time during the day, but confess to failing far too often.

Questions from the Audience

Q: If I want to be a statistical consultant, shall I get a master’s or PhD?

Nat: I think a master’s degree can prepare you very well for statistical consulting If you also want your job to involve theoretical and methodological research, as is often the case, for example, in my division at NCHS, then the research experience inherent in earning a PhD is helpful. Having a PhD can also be helpful, although not required, if you want to supervise other people who have PhDs.

Mary: You can be successful with either, but some situations require a PhD.

Jim: I think you should seek experiences as early in your career as possible, though internships and working on projects, which provides insight into what kind of work gives you most joy. If your peers and mentors encourage you to seek a higher degree, listen to them, since they may see more potential in your abilities than you are aware of yourself.

Q: How did you make the decision about career changes?

Nat: I’ve made two major career changes, and the second was far more complicated than the first. When I was at the Census Bureau immediately after earning my PhD, I knew I would eventually like to try my hand at an academic job. I heard it would difficult (though not impossible) to get a first job in academia if I waited too long after graduate school, so I decided to leave the Census Bureau after working there for three years. I applied for a wide variety of faculty jobs advertised in Amstat News, visited every institution that invited me for an interview, and chose the one (UCLA) that felt like the best fit. Life was simple!

My decision to leave UCLA and move to NCHS involved many more considerations. Life wasn’t as simple! I had been at UCLA for 11 years, had tenure in the department of biostatistics, and was well settled in there. I had married a woman from L.A. whose parents lived there, and I had a three-year-old son. Why would I leave such a secure situation in beautiful, sunny California? Well, my wife and I felt we could use a change in location, my son was young enough that a move wouldn’t be too hard on him, and as mentioned above (in answering the question on work/life balance), a move to the East Coast would place us close to other family members. I also missed aspects of working for the government such as those discussed in my introductory remarks, and there were three attractive senior-level job openings in the DC area. I applied and interviewed for all of them and chose the one (NCHS) that felt like the best fit.

Mary: It was easy for me. The fun and challenge had gone out of my work at the IRS Statistics of Income, so it was the right time to make a change. I retired from Ernst & Young to do independent consulting when it felt right to do so.

Editor’s Note: Nathaniel Schenker’s findings and opinions are his own and do not necessarily reflect the views of the National Center for Health Statistics, the Centers for Disease Control and Prevention, or the U.S. government.