Praveen Gupta Sanka
Machine learning teams have become good at measuring models. We track accuracy, precision, recall, calibration, latency, and business impact. We compare new models against production models, design experiments to detect small effects, and debate whether a marginal gain is worth the added complexity.
Yet many of the same teams have a weaker operating model for the data that feeds those models.
That gap matters. A model can appear stable while the data beneath it slowly changes. A feature can remain populated but become less meaningful. A source table can pass freshness checks while upstream definitions shift. A training dataset can look statistically reasonable while certain groups, geographies, or behavioral segments become underrepresented. In these cases, model accuracy is not enough. The question is not only, “How well did the model perform?” It is also, “How trustworthy was the data used to train, evaluate, and operate it?”
This is the motivation behind the Data Reliability Score, a framework I developed with Vidya Sagar Minukuri in our Joint Statistical Meetings 2025 conference proceeding, “Ensuring Model Performance Reliability Through a Data-Centric Approach.” The idea is simple: As machine learning teams use quantitative metrics to decide whether a model is ready for deployment, they should also use quantitative metrics to decide whether the data is reliable enough to support that decision.
The Data Reliability Score evaluates data through six practical comprehensive pillars: lineage; completeness; consistency; bias; frequency; and accuracy. Each pillar captures a different failure mode. Lineage asks whether the data source, transformation path, and ownership are traceable. Completeness asks whether expected fields and records are present. Consistency asks whether values follow stable definitions over time. Bias asks whether the data represents the populations or conditions for which the model will be used. Frequency asks whether the data arrives at the required cadence. Accuracy asks whether recorded values reflect the underlying reality.
The value of this framework is not that every organization must use the same formula. Different domains will weigh these pillars differently. A fraud-detection system may place high weight on frequency and accuracy. A healthcare model may emphasize bias, lineage, and completeness. A credit-risk model may depend on the consistency of external reporting sources. The broader point is that data reliability should become an explicit part of model governance.
For applied statisticians and data scientists, this shift is natural. Statistical practice has always emphasized measurement quality, study design, missingness, sampling, bias, and uncertainty. Modern machine learning systems have not replaced those concerns; they have scaled them. When a model is trained and deployed across millions of decisions, small data-quality problems can become large operational risks.
A practical implementation does not need to begin with a complex platform. Teams can start by identifying the 10 or 20 most important features or tables behind a high-impact model. For each, they can define checks tied to the six pillars, set thresholds, and monitor changes over time. The resulting score does not replace model metrics. It complements them by making visible the conditions under which those metrics should be trusted. This also changes how teams respond to model degradation. Instead of asking only whether the algorithm needs retraining, teams can ask whether the data itself has become less reliable. Has a source changed? Did a field become sparse? Did a segment disappear from the training sample? Did the refresh cadence slip? These are statistical questions as much as engineering questions.
The future of reliable machine learning will not be achieved through model accuracy alone. It will require treating data as a measurable, monitored, and governed asset. For ML teams, the next frontier is not just better models. It is better evidence about whether the data behind those models can be trusted.
Leave a Reply