humans and autonomous machines. While mid-twentieth-century cyberneticians like Wiener saw the feedback loop essentially as a mechanism of control producing stability within complex systems, they ‘did not quite foresee its capacity to generate emergent behaviours’ (Amoore 2019: 11). In the words of the literary theorist Katherine Hayles: ‘recursivity could become a spiral rather than a circle’ (2005: 241, cited in Amoore 2019).
Consider as an example a simplified portrait of product recommendations on e-commerce platform Amazon. Input data about platform users’ purchasing behaviour are fed in real time into an algorithmic model, which considers two products as ‘related’ if they are frequently bought together (Smith and Linden 2017; Hardesty 2019). By learning from customers’ datafied behaviour, the system generates as output a personalized list of items related to the browsed product. On the other side of the screen, millions of Amazon customers navigate recommended products, and decide whether to purchase some of them, or not. It is estimated that automated recommendations alone account for most of Amazon’s revenues (Celma 2010: 3). Since users largely rely on the algorithm to decide what to purchase next, and the algorithm analyses users’ purchasing patterns to decide what to recommend, a feedback loop is established: the model attempts to capture user preferences without accounting for the effect of its recommendations and, as a result, input data are ‘confounded’ by output results (Chaney, Stewart and Engelhardt 2018; Salganik 2018). This techno-social process has implications that go well beyond the engineering aspects of the system. Feedback loops in recommender algorithms are believed to lead to the path-dependent amplification of patterns in the data, eventually encouraging the formation of filter bubbles and echo chambers (Jiang et al. 2019). The idea here is that the very same digital environment from which the algorithm learns is significantly affected by it. Or, in the words of STS scholars MacKenzie and Wajcman (1999), the social shaping of technology and the technological shaping of society go hand in hand. Which leads to our two main sociological questions.
The first one is about the social shaping of algorithmic systems, or the culture in the code (Chapter 2). Platform algorithms like the one in the example above can autonomously ‘learn’ from users’ datafied discourses and behaviours, which carry traces of the cultures and social contexts they originated from. For instance, in 2017, Amazon’s recommender algorithm proposed as ‘related items’ the ingredients for making an artisanal bomb (Kennedy 2017). The recommendation system suggested to customers a deadly combination of products, most likely following the scary shopping habits of a bunch of (wannabe?) terrorists. That was one of the (many) cultures inscribed in the platform data, then picked up by the algorithm as a supposedly innocent set of correlational patterns. Far from being an isolated case, this incident is only one in a long list of algorithmic scandals covered by the press. Microsoft’s infamous chatbot ‘Tay’, which eventually started to generate racist tweets in response to interactions with social media users (Desole 2020), or the ‘sexist’ algorithm behind Apple’s credit card – allegedly offering higher spending limits to male customers (Telford 2019) – are other examples of how machine learning can go wrong.
The main way in which the critical literature surveyed above has dealt with these cases is through the notion of bias. Originated in psychology, this notion indicates a flawed, distorted and ‘unfair’ form of reasoning, implicitly opposed to an ideal ‘neutral’ and ‘fair’ one (Friedman and Nissenbaum 1996). Researchers have rushed to find practical recipes for ‘unbiasing’ machine learning systems and datasets, aiming to address instances of algorithmic discrimination. Still, these attempts are often ex post interventions that ignore the cultural roots of bias in AI (Mullainathan 2019), and risk paradoxically giving rise to new forms of algorithmic censorship. As Završnik puts it: ‘algorithms are “fed with” data that is not “clean” of social, cultural and economic circumstances […]. However, cleaning data of such historical and cultural baggage and dispositions may not be either possible or even desirable’ (2019: 11). While the normative idea of bias has in many cases served to fix real-life cases of algorithmic discrimination and advance data policies and regulations, it hardly fits the sociological study of machine learning systems as social agents. In fact, from a cultural and anthropological perspective, the worldviews of any social group – from a national community to a music subculture – are necessarily biased in some way, since the socially constructed criteria for ultimately evaluating and valuating the world vary from culture to culture (Barth 1981; Latour and Woolgar 1986; Bourdieu 1977). Hence, the abstract idea of a ‘bias-free’ machine learning algorithm is logically at odds with this fundamental premise. ‘Intelligent’ machines inductively learn from culturally shaped human-generated data (Mühlhoff 2020). As humans undergo a cultural learning process to become competent social agents – a process also known as ‘socialization’5 – it can be argued that machine learning systems do so too, and that this bears sociological relevance (Fourcade and Johns 2020). Here I propose to see Amazon’s controversial recommendations and Tay’s problematic tweets as the consequences of a data-driven machine socialization. Since user-generated data bear the cultural imprint of specific social contexts, a first open question is: how are algorithms socialized?
A second question raised by techno-social feedback mechanisms is about the so-called ‘humans in the loop’, and how they respond to algorithmic actions. This concerns the technological shaping of society (MacKenzie and Wajcman 1999), or what in this book I call code in the culture (Chapter 3). The outputs of recommender systems, search engines, chatbots, digital assistants, information-filtering algorithms and similar ‘calculative devices’ powerfully orient the everyday lives of billions of people (Amoore and Piotukh 2016; Beer 2017; Esposito 2017). We know that individuals massively – and, to some extent, unwittingly – rely on algorithmic systems in their decision making. For instance, Netflix’s recommendation system is estimated to influence choice for ‘about 80% of hours streamed’, with ‘the remaining 20%’ coming ‘from search, which requires its own set of algorithms’ (Gomez-Uribe and Hunt 2015: 5). Similar figures can be found for other platforms, including Amazon, and they explain the spectacular marketing success of recommender algorithms (Celma 2010; Konstan and Riedl 2012; Ansari, Essegaier and Kohli 2000). Myriad feedback loops like the one sketched above constellate our digitally mediated existence, eventually producing self-reinforcement effects that translate into a reduced or increased exposure to specific types of content, selected based on past user behaviour (Bucher 2012a). By modulating the visibility of social media posts, micro-targeted ads or search results, autonomous systems not only mediate digital experiences, but ‘constitute’ them (Beer 2009), often by ‘nudging’ individual behaviours and opinions (Christin 2020; Darmody and Zwick 2020). What happens is that ‘the models analyze the world and the world responds to the models’ (Kitchin and Dodge 2011: 30). As a result, human cultures end up becoming algorithmic cultures (Striphas 2015).
Critical research has mainly dealt with this ‘social power’ of algorithms as a one-way effect (Beer 2017), with the risk of putting forward forms of technological determinism – implicit, for instance, in most of the literature about filter bubbles (Bruns 2019: 24). Yet, recent studies show that the outputs of autonomous machines are actively negotiated and problematized by individuals (Velkova and Kaun 2019). Automated music recommendations or micro-targeted ads are not always effective at orienting the taste and consumption of platform users (Siles et al. 2020; Ruckenstein and Granroth 2020; Bucher 2017). Algorithms do not unidirectionally shape our datafied society. Rather, they intervene within it, taking part in situated socio-material interactions involving both human and non-human agents (Law 1990; D. Mackenzie 2019; Burr, Cristianini and Ladyman 2018; Orlikowski 2007; Rose and Jones 2005). Hence, the content of ‘algorithmic culture’ (Striphas 2015) is the emergent outcome of techno-social interactional dynamics. From this point of view, my deciding whether or not to click on a recommended book on Amazon represents an instance of human–machine interaction – which is, of course, heavily engineered to serve the commercial goals of platforms. Nonetheless, in this digitally mediated exchange, both the machine learning algorithm and I maintain relative margins of freedom. My reaction to recommendations will be immediately measured by the system, which will behave differently in our next
