Chaitra Nagaraja and Mark Glickman
In response to the White House’s Request for Information on the Development of an Artificial Intelligence Action Plan, the ASA’s Scientific and Public Affairs Advisory Committee and the Committee on Data Science and Artificial Intelligence orchestrated the ASA’s submission highlighting the essential role of statistical science in developing, optimizing, and evaluating AI systems. The committees’ five recommendations, summarized below, aim to ensure such systems are both efficient and minimally burdensome.
Establish voluntary best-practice guidelines based on statistical quality assurance methods. A major concern for AI development is managing risk without slowing the pace of innovation. Statistics provides an effective means of achieving this balance by offering methodologies for testing and validating AI systems. Model overfitting, for example, which occurs when AI models perform well on training data but poorly in real-world applications, can be prevented using well-established statistical techniques such as cross-validation and bias-variance tradeoff assessments. Statistical uncertainty quantification ensures AI models provide not only predictions but also an estimate of confidence in those predictions, allowing end-users to make better-informed decisions. Furthermore, companies that incorporate these principles into their development processes will gain a competitive edge, as consumers and businesses will naturally gravitate toward AI tools that produce fair, transparent, and accountable results. Rather than relying solely on external oversight, statistical best practices enable AI to be self-improving while allowing room for necessary safeguards to ensure responsible deployment.
Standardize statistical benchmarks for AI model performance. A targeted initiative to develop standardized statistical benchmarks for AI performance assessment would provide a reliable framework for evaluating AI models by application. By defining statistical metrics for fairness (one of the principles in the Ethical AI Principles for Statistical Practitioners), accuracy, and uncertainty quantification, policymakers can create a foundation for AI evaluation that balances innovation with responsible deployment. This approach ensures AI models meet high performance standards without imposing rigid constraints that may stifle progress.
Incentivize AI system developers to document and share data integrity and model validation procedures. Another important step is incentivizing AI developers to document and share data integrity and model validation procedures, including evidence of maintaining the privacy and confidentiality of individuals. Accountability and transparency are two principles of ethical AI practice described in the Ethical AI Principles for Statistical Practitioners. By fostering transparency in AI model development, organizations can build trust in their systems while allowing external stakeholders to assess performance and fairness objectively. Encouraging companies to disclose key statistical validation metrics in AI reporting would create a more competitive and trustworthy AI marketplace.
Establish partnerships and collaborations across government, business, and academia. Establishing public-private partnerships and other types of collaborations among AI developers, statisticians, and policymakers can accelerate advances in AI model development and evaluation, including explainability, fairness, and risk assessment. These collaborations would ensure statistical expertise is applied effectively to AI systems, creating a foundation for more robust AI applications. Supporting research initiatives that bring together statisticians and AI practitioners across government, business, and academia would also drive methodological innovations that improve AI performance and accountability.
Supporting the continued collaboration between the National Artificial Intelligence Research Resource Pilot and National Secure Data Service Demonstration is a good example of a policy action that can be integrated into the new action plan. These two efforts have their roots in legislation passed during the previous Trump administration. NAIRR is a promising initiative to harness government and other data for AI methodological and substantive research, with a statutory focus on leveraging publicly available government data. NSDS is a statutorily authorized program that complements the federal statistical agencies in harnessing government data restricted by law or regulation for privacy and other reasons for research and other statistical purposes. The NSDS builds on decades of robust statistical system data governance with strong ethical, methodological, and legal frameworks. There are several collaborative efforts between NAIRR and NSDS already underway, including a technical project to make a safe version of restricted data available for AI research.
Additional joint methodological and data governance projects would advance safe and appropriate access to government data for cutting-edge AI research.
Promote statistical literacy among AI system developers and practitioners. A final action item is the promotion of statistical literacy among AI developers and practitioners. Ensuring that those designing and implementing AI systems have a fundamental understanding of statistical principles will improve model reliability and prevent common pitfalls such as overfitting, biased decision-making, and unreliable predictions. Investing in statistical training programs and integrating statistical coursework into AI-related education would significantly enhance the quality of AI systems being deployed and support implementation of the other policy actions proposed in these comments.
To conclude, AI is a transformative technology with the potential to revolutionize industries, drive economic growth, and strengthen national security. Ensuring successful implementation requires a commitment to sound statistical practice. The role of statistics in AI is not to create obstacles, but to provide tools that produce AI models that can be efficiently governed and are robust, reliable, and adaptable. If policymakers incentivize the private sector to integrate statistical best practices, it will ensure AI continues to evolve in ways that benefit both industry and society, along with maintaining US dominance in the field.
These recommendations reflect the ASA’s ongoing commitment to elevating the profile of statisticians and data scientists in policymaking. Members of the Scientific and Public Affairs Advisory Committee and the Committee on Data Science and Artificial Intelligence anticipate more opportunities to provide input on AI policy and welcome feedback on their recommendations. To share ideas for future commentaries, email ASA Director of Science Policy Steve Pierson
Leave a Reply