Methodology

For this article, we conducted eleven unstructured interviews with twelve professionals in computer vision in the field of security. The participants were purposively recruited based on their direct experience of and involvement in at least one of the following activities: (1) the design and development of computer vision algorithms and models; (2) the practical deployment, operational integration, and routine management of these systems in real-world security contexts; or (3) active critique, resistance, or contestation at a policy level of the implementations of computer vision technologies in contexts of (in)security.

Given practical constraints such as the specialised nature of this field and the limited pool of experts, we utilised convenience and snowball sampling techniques to recruit interviewees, primarily guided by participant availability, ease of access, and strategic relevance to the research question. Our selected interviewees should not be understood as statistically representative of a broader population or as ‘illustrative representatives’ of the field (Mol and Law, 2002, pp. 16 –17). Rather, through purposive sampling, interviewees were deliberately chosen to ensure variation across cultural and institutional affiliations, professional roles, and types of engagement with algorithmic practices.

The interviews were conducted in English in the Netherlands (6), Hungary (3), Germany (1) and Poland (1) by Ruben van de Ven and Ildikó Plájás, and subsequently by Clemens Baier. We employed a predominantly unstructured interviewing approach, initiating each conversation with two guiding prompts aimed at eliciting initial reflections. We began by asking the interviewees if they use diagrams in their daily practice. We then posed the prompt: ‘When we speak of ‘security vision’, we speak of the use of computer vision in a security context. Can you explain, from your perspective, what these concepts mean and how they come together?’ Initial interviews revealed emergent thematic patterns, prompting the adaptation and refinement of subsequent interview questions to explicitly address these recurring themes.

At the practical level, we presented our interviewees with a large, A3-sized, digital tablet, and asked them to draw a diagram while answering our questions. Ruben van de Ven programmed an interface that could record conversation in drawing and audio. The participants could not delete or change their drawings, so their hesitations and corrections were preserved.² Breaks appear in the conversation as the interviewees need to think about how to draw (Bravington and King, 2019: 509). Sometimes, the participants seemed uncomfortable about their ability to draw figuratively (see also Copeland et al., 2012). These moments allowed us to clarify that our interest is not so much in the figurative as much as in the relations they drew. Some of the participants therefore opted to diagram with words, rather than figures.

Some of the diagrams that were drawn during the conversation show similarity with the kind of figures the practitioners work with in their own practices. Diagramming is indeed a key method in the field of technology, most notably in the conceptualisation and design of computational practices (Mackenzie, 2017). This can primarily be seen in the use of flow diagrams (Soon and Cox, 2021: 221), that represent different states or stages of computational processing as data ‘travels’ through a system. In our conversations, this style of drawing is particularly visible with those who develop or manage systems for algorithmic security vision. One interviewee explicitly mentioned he was reproducing his slideshow presentation in drawing as he went to explain some of the key concerns of his project. Many of the figures that appear in drawing thus do not exist purely for the conversation, but resemble or reproduce, both in form and content, the figures used by the practitioners in their daily practices. However, as the interviewees draw them on the tablet, they invite reflections that provided crucial insights. For example, one interviewee, a legal scholar and activist, drew a face-database, after which he reconsidered the visual representation: face-databases do not necessarily contain images of faces, but facial features in numeric form. While he initially drew on a common cultural representation of face recognition and surveillance, his technical expertise made him go against such a way of visualising these technologies (van de Ven and Plájás, 2022). Thus, the interviewees tweak the images by drawing new relations, adding layers of explanation, and, as the conversation progresses they refer back to what they drew before, ‘hyperlinking’ the conversation back to earlier concerns.

In the phase of data analysis, the software we developed allows the diagrams to be annotated, creating short video clips, facilitating comparisons among the conversations. All participants except two provided explicit consent for identification by name³. The multimodal tool was created and used to explore the possibilities offered by diagrams to elicit the main three affordances described earlier : (1) relational thinking; (2) graphical manipulation of abstraction; and (3) generative open-endedness in discussing the heterogeneous relations that constitute algorithmic security vision (see van de Ven and Plájás, 2022 for a more detailed account of the methodology and its affordances). Integrating temporal and dynamic aspects of diagrammatic practices was particularly significant for capturing not merely static visual representations, but also the performative processes through which these relational features emerge and are enacted. It is thus important to note that while the PDF version of this article contains only static image representations of the diagrams, the multimodal version allows accessing the full audio-visual excerpts of the diagrams⁴. Our analysis refers to the audio-visual material.

1. From photographic to cinematic vision

The first finding of our diagrammatic interviews is a noticeable shift in the temporal dimension of the uses of computer vision in security technologies. With the advancements in algorithmic models and computational power, more systems than ever are able to assist security decisions based on video data. This shift from still images to video has implied an exponential growth in data which necessitates advanced infrastructure and analytical techniques capable of handling and interpreting continuous streams of visual information. As Gerwin van der Lugt notes, ‘Most cameras do about 30 frames a second. So if you have for example [in] Amsterdam 300 cameras [with] 30 frames per second for what is it? This is basically the number of frames generated each week. Yes, about 100 million’.

This, in our view, marks a shift from what we could call a ‘photographic’ mode of algorithmic analysis, such as biometric imaging (Pugliese, 2010), facial, iris and fingerprint recognition (Møhl, 2021), and body scanners (Leese, 2015). While the former type of systems works on the premise that it can identify individuals on the basis of unique and immutable features of their body (a fingerprint, an eye iris), the new, ‘cinematic’ systems work instead from the assumption that temporary anomalous states, grounded in suspicious movements or emotions, for instance, are what allows the identification of suspects, regardless of their intrinsic features. In the ‘cinematic’ mode of analysis, suspicion is thus not established only inside a single frame, but across a succession of them. This does not mean that both logics cannot be used simultaneously within an integrated system (for example looking in a crowd at movements of people previously identified through facial recognition), but rather that the ‘cinematic’ mode is a relatively new feature that comes with its own affordances and politics⁵.

Two diagrams represent this shift particularly well. Diagram 2, by Ádám Remport⁶, illustrates the photographic mode. It depicts how photo snapshots produced by several surveillance cameras equipped with face recognition technology analysed over a longer period of time, can identify the same person visiting a bar, a church, an NGO. Remport spatialises the functioning of a facial recognition system, which constructs movement through the placing of cameras along the route of an observed individual. Here, the diagram quite literally maps the ways in which a series of static data points (images) can be used to infer and analyse movements and life habits over time and space.

Diagram 3, by Gerwin van der Lugt⁷, shows how the alternative is ontologically different, through a different graphical spatialisation of the relations between the same entities. The system van der Lugt develops is meant to analyse the behaviour of people by looking specifically at movements and distinguishing different behavioural patterns. This system, as he explains, ‘can just detect things like punching, hitting, kicking, dragging someone or pushing someone and then combine those detections and at some point conclude that there is an ongoing violent incident or aggressive behaviour’.

The drawing on the top of diagram 3 represents the photographic paradigm: when processing video data, the frames (f₁, f₂, f₃) are passed to an object detector (YOLO), which runs independently on each frame, one after the other. In the second diagram, the video frames are no longer processed as individual frames but coalesced into a single data unit of 10 frames (f₁–f₁₀) that is passed to the algorithmic model (M). Multiple 10-frame units are then analysed by an additional process that looks at how these groups of frames relate to each other. This allows models to identify and categorise patterns across frames that happen in time, for example a hand gesture. By using the diagrammatic schemata to highlight the distinction between the two systems, the diagrams reveal that it is not a variation of the previous, photographic logic, but a profoundly different one.

This shift has political consequences in how models perform suspicion. The cinematic logic no longer relies on the inscription of supposed ‘indelible’ truths in biological features of individuals (faces, fingerprints, retinas); instead it operates by categorising unfolding movement as suspicion (a suspicious gesture, gait, facial expression), encoded across the frames. The advent of the cinematic dimension thus constructs suspicion as time-based, ephemeral and changing. Suspects no longer need to be categorically present in or absent from a space—it does not matter if they enter the NGO office—but bodies are marked by their movements through space, they can be passing, traversing, entering, or escaping. Thus, while earlier studies on biometric technologies have located the operational logic on identification, verification, authentication, in other words, knowing the individual (Ajana, 2013; Muller, 2010), the cinematic vision finds its operational logic in the mobility of embodied life (see Huysmans, 2022), and its governance through the dynamism of ever-changing ‘real-time’ clusters (Isin and Ruppert 2020).

2. Synthetic data

A second major theme that emerged from our diagram-interviews is the growing significance of synthetic data in reshaping AI training pipelines which raises new ethical concerns. Over the past two decades, much of the critique of algorithmic security systems has centred on the tension between the ‘reality’ they claim to capture and the processes through which those realities are rendered computable. Central to this critique is Haggerty and Ericson’s notion of the ‘data double’ (2000), which highlights the constructed, arbitrary and always imperfect nature of data representations. This process of translation is indeed never neutral; it involves choices about what to include, what to exclude, and how to categorise, fundamentally shaping the reality that the system perceives and acts upon. Once encoded, these data profiles become the basis for social sorting (Gandy 1993, Lyon 2003), where realities are increasingly shaped by data-driven inferences and interventions, potentially leading to new forms of normativity and control (Bigo, Isin and Ruppert 2019). The data used to train self-learning systems—which functions as ‘ground-truth’—is thus never a neutral or ‘raw’ representation of the world (Gitelman, 2013), but rather a product of selective encoding that risks perpetuating societal biases and representational shortcuts (Malevé, 2020).

Our research shows that synthetic data complicates these debates even more, appearing to actors both as a possible remedy and a further destabilising element to the question of representation of reality. In diagram 4, Sergei Miliaev⁸ describes the process through which an AI model is trained. The cycle begins with data acquisition, signalled by the words ‘data’, followed by ‘annotation’, ‘prototyping’, ‘deployment’, and an arrow back to ‘prototype; to emphasise the feedback-based refinements. The AI models are typically using real-world datasets, explains Miliaev in the interview, to generate inferences which can be acquired through ‘web scraping’, by relying on ‘partner data’, and open-source footage (see the upper left corner of Diagram 4).

Yet sourcing such real-world datasets—as shown in the controversies around large language models (LLMs) and copyright issues (Cyphert 2024)—is a key challenge for the AI industry. Miliaev’s diagram allows us to pinpoint that this issue emerges from the very beginning of the AI training workflow; while hidden behind a sequence of processing steps—indicated by arrows in the diagram—the deployment of algorithmic security vision relies on a range of data sources, all of which raise particular concerns. Acquiring ethically sourced, high-quality data—especially for scenarios that require very specific data, like video—is increasingly difficult due to privacy laws and annotation costs. Moreover, these datasets—when they exist—either contain very little data, are of bad quality, or do not adhere to the requirements of the model.

To circumvent the challenges posed by the availability of reliable, varied and open access data Miliaev adds, thus introduces, another data source in the lower left corner of his diagram: synthetic data. The diagrammatic format allows to easily lay out the structure of relations graphically, evaluate it, and insert a new node (synthetic data) within the existing configuration. Synthetic data, which consists of computer-generated images created via 3D modelling, or by means of generative AI (genAI), can be a useful ‘workaround’, so explains Miliaev. Synthetic data enables the creation of large, labelled datasets tailored to specific scenarios, from suspicious activity to pre-crime behaviours. Crucially, because it is generated rather than captured, synthetic data bypasses some of the most stringent privacy restrictions. Furthermore, manipulated data can, for example, simulate diverse conditions—lighting, actions, faces—that are otherwise hard to capture at scale. Miliaev explicitly cites Microsoft’s work on training facial recognition models entirely on synthetic faces (Bae et al., 2023) as proof of this potential.

The adoption of synthetic data thus forces us to problematise the politics of ‘ground truthing’, i.e. the practice of establishing what the models should consider as ‘true’. One is fidelity: the challenge of ensuring synthetic scenes accurately reflect the variability and messiness of real-world environments. Poorly generated data—lacking realism in lighting, weather, or human behaviour—can lead to models that fail when deployed. Another issue is identity representation. While synthetic datasets allow designers to manipulate ethnicity, gender, age, and other attributes in a hope to de-bias the collection, Miliaev notes the difficulty of modelling long-term identity changes like ageing or behavioural variability. If the parameters used to generate this data are biased, the resulting models will inherit and even amplify social biases, as synthetic data does not escape the representational politics that shape captured data.

The diagram thus reveals how synthetic data destabilises conventional epistemologies. While conventional data sources for machine learning, such as operational data or web-scaping, have long been considered to provide access to a ‘raw’ or unfiltered truth, years of critique have established that data are never raw; they are constructed, interpreted, and made intelligible through a relational network of institutional, technical, and epistemic filters (e.g. Gitelman 2013). This critique has now been taken up by many who develop algorithmic technologies, presenting itself as a problem in need of a technical solution. To this end, Miliaev annotates synthetic data with a moral assessment—‘fairness’—which is to be obtained by balancing various ‘attributes’ of the faces in the dataset different from what would be found in previously considered raw data. The new flow diagram that emerges no longer consists of merely technical description, but becomes indicative of moral and ethical assessments that lie at the heart of data gathering and processing. In what way, however, is this fairness established? Does the introduction of these debates imbue the diagram with ethical considerations, or is the ethics rendered as a technical concern? The ‘mean images’ of synthetic data—visual renderings of a non-existent, probable median—reproduce normative conceptions of reality, effectively complexifying the claims to indexicality of lens-based images (Steyerl 2023). Instead of referencing empirical events, synthetic datasets rely on speculative associations and probabilistic reasoning, reframing our understanding of evidence, ‘ground truth,’ and credibility in machine learning practices.

3. Managing error: from the sublime to the risky algorithm

The question of the ‘ground truth’ is indissociable from the third central theme emerging from the diagrammatic engagements: the constitutive role of error in algorithmic security systems. While much literature addresses algorithmic prediction through self-learning systems (Azar et al., 2021), most of it has focussed on how these systems produce risky subjects (Amicelle et al., 2015; Amoore and De Goede, 2005; Aradau et al., 2008; Aradau and Blanke, 2018), based on environmental and individual features (Calhoun, 2023). In this critical body of literature, practitioners working with algorithmic technologies are often critiqued for understanding software as ‘sublime’ (e.g. Wilcox, 2017: 3), meaning that they are thought to believe that their systems ‘work’ in what they claim to do.

The diagrams we collected force us to rethink this premise. Models cannot be assessed through a binary criterion—working/not working – they are instead assessed through a series of error metrics. This is not a flaw that actors aim to fix once and for all; it is the asymptotic quest towards a complex mix of lowest error rates that drives the work of algorithmic security vision. Our diagrammatic interviews show both the importance of the error rates, but also the complexity to account for them and explain precisely what they are.

Diagram 5, by CTO Gerwin van der Lugt is particularly enlightening in this regard. As he explains, the most prominent way in which error figures in model development is in its quantified form of the true positive rate (TPR) and false positive rate (FPR). Van der Lugt stresses the significance and definition of these metrics by marking them as the key variables to optimise, on the top right corner of the page (Diagram 5). Van der Lugt initially describes the FPR is the number of false positive classifications (FP) relative to the ‘input’, the number of video frames being analysed. When he continues to explain the TPR, he similarly marks it on the tablet as being equal to the number of true positive classifications (TP) over all inputs. As he goes on however, he realises that this is slightly incorrect. In fact, it should be ‘true incidents’. Looking back at the tablet again, he adjusts the original FPR equation: it should be specified as FP over the number of ‘false inputs’ (and not just all inputs). ‘These are quite tough to define but they are quite important, as these need to be really low’. The diagrams work here as an important support for van der Lugt to manipulate the abstract notions of errors through graphical inscription: by first noting them, then correcting them, the diagramming helps refine the explanation of a key yet complex notion to explain, even for the developer of the system. Moreover, as he corrects his definition, van der Lugt comments not only on the difficulty of these definitions, but also on the importance of having them right, as the these definitions determine the work of his development team, the ways in which his security operators engage with the technology, and how the system is evaluated by his clients.

As he goes on, van der Lugt ponders that algorithmic security vision is inherently prone to error, which affects the implementation of security practices. He illustrates this with Oddity.ai’s violence detection system, questioning whether playful fighting (stoeien) should be classified as violence. He argues for distinguishing between false positives—errors in evaluation—and errors in how the security problem is operationalised. He offers two reasons. The first reason is that excluding stoeien could reduce the algorithm’s TPR, disrupting the balance between TPR, FPR, and other performance metrics. The developers aim for fewer than 100 false positives per 100 million frames weekly, since too many alerts can desensitise human operators. The second reason is that excluding stoeien might introduce subtle biases—for instance, the algorithm might infer violence based on age rather than behaviour. Van der Lugt warns that such discrimination is both undesirable and hard to detect. In this view, error is not just mathematical but something to pre-emptively manage, placing responsibility on developers to anticipate and mitigate failure.

Another aspect concerns the mutable nature of threats and susceptibility to adversarial attacks. András Lukács⁹ notes that detection models ‘can be cheated by quite simple tricks’. Even data-rich systems remain vulnerable to intentional evasion, highlighting the challenge of maintaining relevance in evolving threat landscapes. John Riemen¹⁰ adds that poor input quality undermines facial recognition—for instance, the wrong still from a video may fail to identify correctly.

Practitioners suggest several strategies to respond to these issues. One is narrowing the algorithm’s scope to high-impact, clearly defined crimes. Guido Delver¹¹ recounts moving from detecting vague ‘suspicious behaviour’ to focusing on burglary. Van der Lugt’s firm similarly focuses on weapons, vandalism, and physical violence. This narrowing reduces ambiguous boundary cases—both an ontological and an operational shift. Another strategy is continuous retraining. Dirk Herzbach¹² explains how operators feed annotated alerts back into the system. Parameter tuning to balance false positives and negatives is also key. This balance is political: as Jeroen van Rest¹³ notes, communities vary in their tolerance for error—some may accept high false positives for safety, others may not. Thus, the acceptable error threshold becomes a negotiation site between developers, publics, and institutions.

In this light, error is not external to algorithmic systems—it is their condition of possibility. As Pasquinelli (2019) puts it, machine learning is based on ‘formulas for error correction’; error is not a failure to remove, but the mechanism through which learning occurs. Amoore (2019) similarly notes that ‘it is precisely through these variations that the algorithm learns what to do.’ Error, initially a mathematical concept, permeates every discussion of algorithmic security vision. What emerges is a double articulation of risk. While risk is usually treated as external to, and produced by, security technologies, technologies of algorithmic security vision themselves become objects of risk management. The inevitable errors of machine learning algorithms place risk not outside, but at the core of the security apparatus.

4. Reconfiguring the human-in-the-loop

The understanding of algorithmic security vision as a practice of managing error has significant consequences for the role of the human within these assemblages—our fourth main theme. The critique of algorithmic security has often construed the human-in-the-loop as one of the last lines of defence to the inevitable erroneous outcomes of automated systems (Markoff, 2020). However, critical security studies have questioned this representation, emphasising the visualisation of algorithmic predictions in graphical interfaces (Aradau and Blanke, 2018) and how the operator's embodied decision-making is intertwined with the algorithmic system of which they are part (Wilcox, 2017). Furthermore, operators may find themselves uncertain about the system's functioning when errors occur (Møhl, 2021). In other words, when considering questions of responsibility, a system's operator cannot be seen as distinct from the algorithmic assemblage (cf. Hoijtink and Leese, 2019). The question in a system's design thus is not whether a human operator makes autonomous decisions but rather, how they are to interface algorithmic predictions, and how issues of agency and responsibility are negotiated. The diagrams illustrate how this leads to differentiated processes of (in)visibilisation.

Looking across the various diagrams reveals a spectrum of system designs concerning the autonomy of security operators and the technical expertise they are expected to bear. In a first kind of designs, the human operator is central to the decision-making processes, acting as the interface between, and external to, algorithmic systems and surveillance practices. For example, Dirk Herzbach explains that when the Mannheim police is alerted to an incident by the system, it is the operator who decides whether to deploy a patrol car. Here, the human-in-the-loop has full agency and responsibility for operating the (in)security assemblage, with the capacity to evaluate and utilise algorithmic systems selectively and with care. Herzbach therefore prefers an operator to have patrol experience, so they can best assess which situations require intervention. He is concerned that knowledge about algorithmic biases might interfere with such decisions. In other words, the security operator is considered the expert of the subject of security and is expected to make decisions independently from the information that the algorithmic system provides.

However, in a second kind of design, the human operator is considered an integral part of the algorithmic security vision system, influencing and influenced by its operation. Such designs acknowledge that it might be more complex for the human-in-the-loop to perform as the primary counter to algorithmic error. This is illustrated by Guido Delver, project manager for the Burglary-Free Neighbourhood project in Rotterdam, which incorporates autonomous systems into street lamps. In Diagram 6, Delver maps out the different ‘stakeholders’ of the project on a flat canvas, using the diagram as a way to spatialise relations of a heterogeneous set of key entities. He envisions the neighbourhood as a site where the public and private space—the residents and government—meet, and which also involves suppliers of technology and research institutions. By rethinking the traditional top-down hierarchy of the surveillance apparatus, Delver aims to counter government hegemony. Delver uses this rendition of the various stakeholders to ask the question ‘who [of these parties] owns the data?’. However, upsetting the traditional hierarchies comes with its own risks. Delver illustrates his point with a scenario in which the algorithmic signalling of a potential burglar may have dangerous consequences: ‘Does it evoke the wrong behaviour from the citizen? [They could] go out with a bat and look for the guy who has done nothing [because] it was a false positive.’ In this case, the worry is that the erroneous predictions will not be questioned. Thus, while often deemed to be ultimately responsible for the judgement that follows from the algorithmic process, human participation or ‘interference’ in the operation can likewise be rendered as harmful.

As a means to mitigate these concerns, in Delver’s project, the goal was to actualise an automated decision system (ADS), ‘with as little interference as possible’. Placed in the centre of his drawing, different stakeholders interact with this system in different ways. Thus, figuring the human-in-the-loop—whether police officer or neighbourhood resident—as risky can lead to the relegation of direct human intervention. For Delver, access to the data—both the data that makes inferences possible, as well as the outcomes produced by the device—needs to be curated.

The curation of data flows—acknowledging the fallibility of human decision making—is even more prominent in another diagram. In Diagram 7, John Riemen (dis)assembles the process of facial recognition as it is employed by the Dutch police. After drawing three human figures, he stresses the relations among these multiple humans-in-the-loop, and how some data should be allowed to pass (indicated by the arrows in the diagram; for example, from a database expert who performs the search to an investigator of the case), while other flows of information should be blocked. For facial detection to be accurate, the human interaction with the algorithm needs to be sanitised of personal information.

By drawing the process as series of actors, all of whom are biased, Riemen sketches out how they are embedded in, and thereby influenced by, a set of mutual relations that needs to be regulated, here the diagrams capture the relational and sequential nature of these links.

The diagrams thus challenge the simplistic view of the human-in-the-loop as a fail-safe against algorithmic errors by highlighting the complex network of sociotechnical relations they are enmeshed in. For both Delver and Riemen, the inherent error—whether performed by a person or a computer—has to be mitigated by curating the transfer of information from one node to the next. More than multiplying the number of humans-in-the-loop, these diagrams encourage us to think in terms of ‘curatorial pipelines’ (Malevé, 2023) that mediate the different human and algorithmic operations.

5. Delegating accountability to benchmarks

The final theme of our diagrams concerns the question of accountability. Literature on the ethical and political effects of algorithmic vision has notoriously focussed on the distribution of errors, raising questions of ethnic and racial bias (e.g., Buolamwini and Gebru, 2018). This critique has now been internalised in the field of AI development; virtually all of our correspondents acknowledge the concern with bias.

To mitigate it, many AI developers—including some of our respondents—have come to rely on an external reference against which the error is measured: benchmarks. John Riemen, for example, who is responsible for the use of forensic facial recognition technology at the Centre for Biometrics of the Dutch police, describes how their choice of software is driven by a public tender that demands a ‘top-10’ score on a benchmark for facial recognition vendors, maintained by the American National Institute of Standards and Technology. Often shortened to the acronym NIST (e.g., Diagram 7), this benchmark ranks facial recognition technologies of different companies by their error metric across groups. The mitigation of bias is thus outsourced to an external, and in this case US-based, institution.

By examining where John Riemen’s diagram ends (see Diagram 7)—that is, where entities are no longer broken up into its component parts, but rather bracketed and considered as a whole—we get a sense of how issues are ‘black boxed’. As Riemen describes the process of facial recognition, he is able to explain and draw it in much detail. Therefore, when asked about bias mitigations, he is able to precisely locate the various measures of filtering that take place, drawing the ‘walls’ in between the different people (see the vertical line in the diagram). On the lower part of the drawing, the NIST database does not have the same detail in process, nor scrutiny. In Riemen’s diagram it becomes a single entity, represented by an acronym. As a benchmark, the facial recognition vendor test is implicitly assumed to have already implemented the measures that are required for its use by the Dutch police. The diagrams here provide a graphical illustration of the abstraction and erasure of particular processes, showing only the inputs and outputs within a general network. Put in more stark terms, where the diagram stops, the respondent’s responsibility stops. With NIST, the prevention of algorithmic bias comes to rely on a single benchmark: a de-facto standard that is managed by an external institution.

The problem is, while a particular kind of algorithmic bias (i.e., ‘demographic differentials’) is rendered central to the NIST benchmark, the mobilisation of this reference obfuscates questions on how that metric was achieved. For example, the NIST benchmark datasets are known to include faces of wounded people (Keyes, 2019). Moreover, questions about training data are invisibilised, even though that data is a known site of contestation; the Clearview company is known to use images scraped illegally from social media, and IBM uses a dataset that is likely in violation of European GDPR legislation (Bommasani et al., 2022: 154). Pasquinelli (2019) argues that machine learning models ultimately act as data compressors: enfolding and operationalising imagery of which the terms of acquisition are invisibilised.

This attention to the invisibilisation of and in/between processes reveals a discrepancy between the developers and the implementers of these technologies. On the one hand, the developers we interviewed expressed concerns about how their training data is constituted to gain a maximum FPR/TPR ratio; while showing concern for the legality of the data they use to train their algorithms—propelling for example the use of generative AI (see section 2). On the other hand, questions about the constitution of the dataset have been largely absent from our conversations with those who implement software that relies on models trained with such data. Occasionally this knowledge was considered part of the developers' intellectual property that had to be kept a trade secret. In such cases, for the implementers, a high score on the benchmark is enough to pass questions of fairness and to forgo any further inquiry into the inequalities propagated by the algorithmic system and the data through which it learns, thus legitimising its use. While the deployment of algorithmic models indirectly relies on the training data, it is not deemed relevant in the consideration for one particular model over another.

The configuration of algorithmic vision’s bias across a complex network of fragmented locations and actors, from the dataset, to the algorithm, to the benchmark institution reveals the selective processes of (in)visibilisation. It illustrates well how accountability is bracketed through the invisibilisation of the dataset as it is ‘compressed’, in Pasquinelli’s terms, into an algorithmic detection model, with the formalisation of guiding metrics into a benchmark. One does not need to know how outcomes are produced, as long as the benchmarks are in order.