New Data Assessment Tool: Mitigating Risk of Bias – Ethnicity and Migration

At UK Prevention Research Partnership (UKPRP) funded consortium VISION we have dedicated a workstream to studying the data gaps, analytical biases, and systemic blind spots that arise around questions of race, ethnicity and migration. We acknowledge that all research data is socially constructed. Asking questions about that process of construction can help researchers be aware of the biases and distortions that will arise when data are used uncritically and unreflectively.

Over the first two years of the project, we have considered the gaps in our own data, drawn on expertise and insight from our research team and associates, and drawn on a diverse array of methodological and disciplinary backgrounds. This research has allowed us to develop a tool to support researchers on our project and beyond to mitigate the risk of introducing or reproducing bias regarding ethnicity and migration in data, analysis and reporting findings. The tool initially responds specifically to administrative and survey data but can be adapted for use with any dataset.

The tool was conceived following consultation with VISION colleagues regarding the construction of ethnicity and migration data across the consortium. The objective was to produce the best quality data and analysis possible that could account for diversity and diverse experiences in the population. The VISION consortium is concerned with measuring violence: ethnicity and migration status are key areas where violence is underreported, which creates gaps in data. We found that existing datasets are unable to properly reflect diversity and inequality in the population and therefore cannot fully explain different experiences. Thus, we sought ways to prevent reproducing biases and data gaps within our analyses.

A collaborative workshop held by the Ethnicity and Migration research group initiated an iterative process of tool development. A meeting with the UKPRP Community of Practice for Race and Ethnicity specified feedback to the tool design. Extensive research of existing literature took place between September 2022 and March 2023 and was summarised into a companion document, both providing citations and further information.

A draft tool and companion document was shared with specialist service IMKAAN (a by-and-for service for minoritized women and girls at risk of violence) in July 2023 for consultative feedback.

I am pleased to announce the tool is now ready for circulation and use.

The tool comprises three parts:

1. The first part offers guiding questions to assess the quality of a data set at the adoption stage, with regard to how well the dataset mitigates the potential biases that may be produced during data collection regarding ethnicity and migration. Researchers can apply the questions within this section to any dataset they are adopting, although it was designed particularly with survey data and administrative data in mind because these are the main datasets used in the VISION project.
2. The second part of the tool guides the researcher in a reflective process that is intended to allow the researcher to assess the potential for internally held biases or structural and systemic biases to which they have been exposed affecting the data analysis.
3. The third part asks researchers to consider the impact of reporting findings. The wording of publications might affect how the finds are interpreted or reported in the media, cited by other researchers, or circulated. Findings might also be misused. This final section of the tool offers some techniques for mitigating the misinterpretation or misuse of findings (although we acknowledge that this is often outside of researcher control).  

This document will evolve over time, therefore, we are keen for your feedback. If you download the tool and guide and use them, please let us know how you get on! Is the tool easy to use? Is the guide clear? How effective were they in helping you mitigate bias when working with your data? Please let us know by contacting Andri at Alexandria.innes@city.ac.uk. We look forward to hearing from you!

Download the assessment tool

Download the companion guide