From Scattered PDFs to Analysis-Ready Data: Putting an End to Time-Consuming Data Collection, Cleaning

Thiyanga S. Talagala, Department of Statistics, Faculty of Applied Sciences, University of Sri Jayewardenepura

Cyclone Ditwah brought heavy rains to Sri Lanka. Amid this national emergency, Sri Lankans living around the world demonstrated the strength and unity of the nation by organizing various programs, such as relief supplies distribution programs, clean-up and restoration campaigns, and missing persons identification services. 

While these responses delivered lifesaving support on the ground, the disaster also revealed a critical gap in Sri Lanka’s disaster information ecosystem. That is, much of the essential data needed for data analysis and decision-making was often locked in PDF files across different websites. 

There was no centralized repository to access the available data, which was also unstructured and messy—not ready to analysis. When researchers spend most of their valuable time collecting and cleaning data, it can delay analysis and modeling. In response to this information gap, our team developed open-source R programming packages that provided access to tidy data sets related to rainfall, river level, and impact counts, along with interactive dashboards for visualizing data and early warnings of landslides during the Ditwah cyclone period.

Data Is Everywhere, but Is All Data Useful?

When the cyclone Ditwah struck, vast amounts of data were generated every day, sometimes every hour. For example, during the cyclone, the Department of Meteorology and the Disaster Management Centre published (mostly in PDF format) weather data, river water level, flood warning level data, and landslide early warning data. Furthermore, the Hydrology and Disaster Management Division of the Irrigation Department provided, through a dashboard, real-time water level information for major rivers across Sri Lanka. These efforts helped the public stay informed during the emergency. However, we found these data sets were not immediately ready for statistical data analysis. The primary reasons: reports were distributed across multiple sources—PDFs or HTML files—and they followed different naming conventions. They also varied in granularity. 

To make the data analysis-ready, it must be converted into a tidy format, meaning:

• Each variable forms a column: Every column represents a single variable.
• Each observation forms a row: Each row corresponds to a single observation or measurement.
• Each value must have its own cell: Each cell contains a single value corresponding to that variable for that observation.

Our Contributions

We developed two R packages, transforming landslide and flood warning reports, real-time water level data, situation counts, and rainfall data into tidy data sets. The data was published by the Disaster Management Centre, the Hydrology and Disaster Management Division of the Irrigation Department, and the Department of Meteorology, respectively. The R package ditwahLandslide also includes a tidy data set on early warnings of landslides that allowed researchers and practitioners to quickly access, filter, and analyze landslide-related information. 

The R package Ditwah includes rainfall, flood warning levels, and river water level data, which allows users to efficiently study flood risks and hydrological patterns. This package also contains situation-related data, such as numbers of affected families, deaths, and safety centers. By converting scattered, heterogeneous, and often non-machine-readable reports into standardized, tidy data structures, these packages facilitate reproducible analyses, visualizations, and decision-making. They also serve as a foundation for developing forecasting models, risk assessments, and other data-driven applications for disaster management.

In addition to providing tidy data sets, these packages offer functionality to visualize data. The packages also offer access to interactive data visualization dashboards. A data dashboard is a visual display of the most important information in the data. This tool helps researchers quickly see what is happening, understand trends, and make decisions without needing to look at raw data.

By centralizing all the data sets we developed, we created an online website called Ditwah Cyclone Data and Analysis Hub, where researchers can easily explore, visualize, and get data download instructions from a single platform. By bringing diverse data sets into a single online platform, this hub enables researchers to easily discover, explore, and reuse data. Open, transparent, and accessible practices support the development of science. 

Key Benefits

This work contributes to achieving the Sustainable Development Goals, particularly SDG 3 (Good Health and Well-Being), SDG 4 (Quality Education), and SDG 13 (Climate Action). Key benefits include the following:

1. Timely and accurate weather forecasts and early warnings: Structured data allows researchers to focus on data analysis and model building, without the need to manually download and clean data from dozens of PDF files. By streamlining these processes, this work helps strengthen national preparedness for climate-driven emergencies.
2. Open-access centralized disaster data hub: A single location—where all Ditwah-related data is organized, stored, and made publicly accessible—helps promote research transparency by allowing others to verify, reproduce, and build upon available data analyses. Furthermore, it helps avoid duplicating data cleaning efforts and encourages data reuse and sharing, which is crucial for research sustainability.
3. Stronger education and data literacy: Beyond immediate disaster response, the packages are designed for educational purposes. Schools, universities, and community groups can now use real Sri Lankan data to learn about disasters and patterns. 
4. Enhance research collaboration: Data packages support research collaboration, particularly during large-scale events like natural disasters. When data is shared in a structured, standardized, and open format, collaboration becomes faster, more effective, and more inclusive.
5. Supporting interdisciplinary work: Data packages can be used by statisticians, geographers, climate scientists, public health experts, social scientists, and educators. A single, well-prepared data set enables diverse perspectives and integrated solutions.

The Way Forward

More data from cyclone Ditwah will be added in the coming months, including post-impact data sets and regular updates as new information becomes available. We invite researchers, educators, journalists, and the public to visit the Ditwah Cyclone Data and Analysis Hub to access the data, explore practical examples, provide feedback, and contribute to the continued growth of this open-access initiative. By leveraging existing information, this initiative supports disaster response, research, and education.

Thiyanga S. Talagala

Department of Statistics, Faculty of Applied Sciences, University of Sri Jayewardenepura

Thiyanga S. Talagala is a senior lecturer in the department of statistics, faculty of applied sciences, University of Sri Jayewardenepura in Sri Lanka. She holds a PhD in mathematics and statistics from Monash University, Australia. Her research focuses on developing new statistical machine learning tools to help both practitioners and theoreticians make more open, explainable, and reproducible data-driven discoveries. She develops open-source software tools to support reproducible research and teaching. She’s authored several R packages available on CRAN.