Towards an integrated regulatory approach of the use of Automated Decision-Making systems in the public sector

Partners: Tilburg University, Utrecht University, Delft University of Technology
Type: PhD
Duration: 2023—2026

Algorithms and AI are increasingly infiltrating social and political life. Ranging from the social media content we are exposed to and the products we are recommended to calculating eligibility for certain benefits or determining our credit score, algorithms are constituting a critical factor in decision-making within more and more domains. This also impacts long-standing public values such as equity, solidarity, and human dignity, making it important to understand how these values will interact with the increasing use of algorithms. Although the use of algorithms can be beneficial in efficiently allocating resources and relieving civil servants of administrative burdens or more mundane tasks, it is imperative to consider the risks associated with the use of automated decision-making systems (ADS) in the public sector.

On the one hand, reliance on ADS can lead to the delegation of complex, political decisions to machines (Ruha, 2019). This happens on two levels: First, it is often left to private companies to develop these systems without the same rigid oversight that public institutions are scrutinized by. Second, the actual delegation of decisions from public officials to ADS exacerbates these shortcomings of democratic oversight. Whereas so-called automation bias often generates the assumption that the use of algorithms produces more precise, logical, and neutral decisions, an increasing body of academic literature demonstrates how normative and political considerations are still embedded in the code, data, infrastructure, weighing of values, etc. (Grimmelikhuijsen and Meijer, 2022; Citron and Pasquale, 2014). However, these normative and political questions are not negotiated within democratic deliberations but rather brushed over by a select cadre of technical experts and civil servants who may not be aware of the severity of their choices (Rekenkamer Amsterdam, 2023).

This ties into the second issue, namely, that these biases often aggravate already existing disadvantages experienced by minorities and marginalized communities. Recognizing that the data used by ADS is distorted in the sense that definitions of what is considered positive or valuable are influenced by the status quo consequently means that other qualities or traits that are less dominant will not receive equal weighing (Ruha, 2019). While this can have several reasons, such as certain demographics not being represented in the data or discriminatory classifications of certain traits, the consequence remains the same: Marginalized communities will experience disadvantages in some contexts (Burrell and Fourcade, 2021).

In light of this, the aim of this research is to understand the process of regulating ADS in the public sector, highlighting which public values are at stake and the requisite balancing thereof, and examining possibilities of including relevant stakeholders back into these deliberations. This leads to the following central research question: How can core values in the context of regulating automated decision-making systems (ADS) in the public sector be (democratically) negotiated by including relevant stakeholders – those directly involved in applying, and those being affected by, ADS?

The overall goal of this PhD project is to understand how marginalized communities could be affected by the use of ADS and to make deliberations on their use, and on the safeguarding of relevant public values, more inclusive. Instead of delegating these political decisions to technocratic circles, it is important to empower everyone to express their opinions and find a way for these voices to be heard. The project will incorporate various approaches and theories to account for the diversity in perspectives, such as socio-legal, public administration, STS, techno-feminist, and socio-technical angles. As such, the project strongly aligns with the overarching research question of AlgoSoc: how to realize public values in the algorithmic society by investigating who is currently in a dominant position to frame the debate on algorithms, their risks, and the values at stake. With this information in mind, an attempt will be made to identify pointers or clues that could help in transferring these decisions back into more democratic arenas.