‘My Rome started to be built today’: Assessing and enhancing government data quality, theory to practice

A Rome that started to be built

‘My Rome started to be built today.’ These were the parting words of a Namibian government data producer who had just participated in a workshop that had two main objectives:

1. to share the findings and recommendations of a government data quality assessment with the relevant data producers and with the primary user of government administrative data, namely the directorate that held overall responsibility for national monitoring and evaluation (M&E); and
2. to not only enhance data producers’ capacity to implement the recommendations but also support them on how to conduct their own data quality assessments.

Though the workshop was held several years back, I can still sense the energy that pulsated in the room where these data producers and data users had gathered to engage with the subject matter. The exercises that formed part of the workshop had been tailormade for the audience’s practical use in their work, and the discussions were lively as the participants leaned into the collaboration to improve their data. From my perspective as funding and implementation partner, the enthusiasm in the room was as electric as the thunder rumbling in the Namibian sky outside the meeting room during that rainy season.

Much more important than my perspective, however, the participants themselves affirmed the achievement of the workshop’s goals, as reflected in the ‘my Rome’ statement that opened this article. The 10-step approach to assessing and enhancing government data quality that had been presented during the workshop appeared to have been well accepted and the possibilities that it presented well understood. [PRACTICAL TIP #1: You can apply these 10 steps in the assessment of the quality of the data used in your project’s M&E system. I'd love to hear from you about the experience.]

Before outlining what those 10 steps are, it may be worth highlighting why all this matters in the first place.

Why it matters: a given, but still worth noting

Government administrative data are used by, among others, (a) public authorities to track progress towards national development objectives, (b) many development projects in their M&E systems, and (c) evaluators and researchers as a secondary data source for their studies. All of these users, in their various ways, seek to determine the extent to which policies, programs, and projects aimed at creating positive social impact are delivering the desired results.

For that determination to be credibly made, the underlying data need to be of a sufficiently high level of quality; thus, quality considerations should underpin any data collection effort. Several theoretical frameworks have been developed for determining whether data are ‘fit for purpose’. However, it is not always possible – or necessary – to have the highest quality data, given limited resources, capacity, and a myriad of other contextual factors.

This heightens the importance understanding how to interpret these sometimes-imperfect data. So instead of asking ‘Are the data fit for purpose?’ based on theoretical frameworks, the question becomes ‘What purpose(s) are the data fit for?’ [PRACTICAL TIP #2: Try asking the question in that way when assessing the quality of your own project’s M&E data, whether or not they include government administrative data.]

A related question is ‘How can we support data producers in enhancing the quality of their data?’

This takes data quality assessments from being theoretical exercises to being grounded in practical use.

Origins of the 10 steps

The 10-step approach presented in the above-mentioned workshop – and now updated and presented in a working paper that I cowrote with my collaborators Nils Riemenschneider and Jesse McConnell – is one proposition for how this theory-practice divide might be bridged.

The approach was born in the process of assessing the quality of the government administrative data used in the M&E system of a USD304.5 million project in Namibia. The project spanned the sectors of agriculture, education (standard education at all levels as well as vocational training), land, and tourism, and the data quality assessment ultimately answered the following questions for the project:

• ‘Which elements of these data can be used and for what purposes?’
• ‘How might we best support local capacity to ensure quality data?’

As the project’s Director of Monitoring and Evaluation, I obviously felt it was important to verify the quality of the data being used for the project's M&E system. [PRACTICAL TIP #3: Assess the quality of the data used in your project’s M&E system at least once in 3- to 5-year project period and more often as necessary.] The findings and recommendations that came out of the data quality assessment were useful to that end, and appropriate changes were made to the system and documented accordingly (watch this space for more on how to document changes to an M&E system).

It was of equal importance, however, to be sensitive to the needs of other data users who would be continuing to use the data long after the project ended and, perhaps most importantly, to be responsive to the needs of the data producers as they sought to address data quality gaps. This made it critical to not only engage in a discussion with the relevant data producers and the primary user of government administrative data about the findings and recommendations of the data quality assessment, but to also take the initial steps towards supporting them in conducting such assessments themselves. This is what was done in the above-mentioned workshop, with the intention of serving the broader interest of supporting Namibia in the realization of its own aspirations.

The 10 steps to assessing and enhancing government data quality

The updated version of the 10 steps to assessing and enhancing government data quality that were presented at the workshop in Namibia are now available in the form of a Oxford Policy Management (OPM) working paper that I’m pleased to have cowritten with two wonderful collaborators. The approach is designed to be pragmatic in its required level of effort and is applicable across a variety of data sources and types. We presented it in the paper in the hopes that it would be helpful to others confronting the same type of questions that we faced starting out.

Here is an outline of the 10 steps, which fall across three phases of the process of assessing and enhancing government data quality:

Phase I: Preparatory and planning work

• Step 1: Clarify agenda for DQA and establish the purpose for which the data are being used
• Step 2: Clarify definitions
• Step 3: Review documentation on the data’s creation and use
• Step 4 (iterative): Formulate guiding questions

Phase II: Embarking on a ‘voyage of discovery’ to understand the data collection process

• Step 5: Consult data producers to understand the data collection process and the context(s) in which it is carried out
• Step 6: Observe the data collection process in action
• Step 7: Test the data
• Step 8: Cross-check understanding and, if necessary, conduct follow-up consultations

Phase III: Drawing conclusions and developing recommendations

• Step 9: Draw conclusions and identify actionable recommendations
• Step 10: Begin the process of strengthening local DQA capacity in line with local priorities and drawing on existing capabilities

The detailed version of the 10 steps can be read in the full working paper, available on OPM’s website here.

It is my hope that, with the help of this approach – which will no doubt be improved in future iterations as learnings emerge from its use –, Rome will start to be built for other data producers today, too. (And perhaps the overall timeline for Rome’s completion will be drastically reduced as well!)

Please shared your thoughts with me on LinkedIn.

[The working paper can be cited as follows: Riemenschneider, N., McConnell, J., Shejavali, K. (2021, May). Assessing and enhancing government data quality, from theory to practice. Working paper. Oxford Policy Management: Oxford, UK. https://www.opml.co.uk/publications/assessing-enhancing-government-data-quality-theory-practice]