Inside a Fraud Prediction Algorithm
Obtaining the source code Rotterdam uses to predict benefits cheating
Every year, hundreds of intimate details relating to every welfare recipient in the Dutch port city of Rotterdam are fed into a machine learning algorithm. These details — which range from address to mental health history to hobbies — ultimately result in each recipient being assigned one number: a fraud risk score.
This risk assessment can have life-changing implications for the people being scored even though they themselves have no knowledge of how it is calculated. Rotterdam is one of many European cities that are turning to algorithms to inform critical decisions that shape the allocation of public resources.
A joint investigation by Lighthouse Reports and Dutch public broadcaster VPRO obtained the source code, inputs and outputs, as well as technical documentation of Rotterdam’s predictive algorithm. Disclosed documents reveal that since 2018, the municipality of Rotterdam has automatically put thousands of welfare recipients with high risk scores under investigation; hundreds have had their benefits terminated.
Blending data journalism with traditional reporting methods, Lighthouse is working to understand as concretely as possible the technical and human dimensions of Rotterdam’s predictive algorithm.
In a welcome step, the city of Rotterdam voluntarily disclosed some of the code and technical information required for an independent assessment of their algorithm to Lighthouse and VPRO.
Unfortunately, key material — such as training data — necessary to conduct a comprehensive technical analysis have not been disclosed by Rotterdam as of writing. Lighthouse’s data scientists have so far conducted a partial audit based on the disclosed material evaluating data quality, reliability, transparency, and accuracy.
In order to better understand the human consequences of a high risk score, we have spoken to welfare recipients either definitively or potentially flagged for investigation by the algorithm. Some of these welfare recipients have submitted subject access requests under the GDPR to understand how their personal data has been processed.
To keep up to date with Lighthouse investigations sign up for our monthly newsletter
Our investigations don’t end when we publish a story with media partners. Reaching big public audiences is an important step but these investigations have an after life which we both track and take part in. Our work can lead to swift results from court cases to resignations, it can also have a slow-burn impact from public campaigns to political debates or community actions. Where appropriate we want to be part of the conversations that investigative journalism contributes to and to make a difference on the topics we cover. Check back here in the coming months for an update on how this work is having an impact.