AI Radicalisation Exposes a Blind Spot in UK Regulation
OpenAI has called on the UK Government to introduce legislation for AI providers to address the most serious risks related to national security and cybersecurity. While introducing legislation to regulate frontier AI capabilities is urgent, especially following high-level incidents over the summer and warnings from AI researchers, developing a regulatory regime that focuses on catastrophic harms would leave a gap around broader AI-enabled societal risks.
That gap matters. Concerns about conversational AI increasingly extend beyond their ability to search the internet or generate harmful content. Reports of users becoming locked into delusional beliefs (sometimes described as “AI psychosis”) and cases involving self-harm, suicide and violence have raised a broader concern about users’ cognitive destabilisation. One question is whether AI systems capable of sustained, personalised and private (‘closed-loop’) interaction can reinforce harmful patterns of thinking and behaviour in ways that existing online safety frameworks (primarily focused on social media and traditional search engines) are unequipped to address.
Our recent research at the Institute for Strategic Dialogue examined one manifestation of this problem: radicalisation. A 2023 report by the UK’s Independent Reviewer of Terrorism Legislation identified conversational AI as an emerging area of concern for radicalisation and terrorism risks. This is not purely hypothetical: the 2021 Windsor Castle attack plot provided an early example of a user developing an intense relationship with an AI companion while expressing violent intent.
We sought to understand how conversational AI behaves when users engage with extremist and conspiratorial prompts and what risks those responses may create. To do this, we tested general-purpose chatbots such as ChatGPT, Claude, Gemini, and AI companions such as Replika and Nomi across conversations involving antisemitism, misogyny, and anti-migrant hate. The conversations progressed through three distinct ‘radicalisation phases’: from users showing interest in conspiratorial or hateful content without necessarily holding those views, through seeking validation for their beliefs, to scenarios in which users sought to act on extremist ideology.
The results showed that AI system design mattered considerably to how a chatbot behaved and its associated risk. Mainstream models generally presented low levels of risk, while the fringe chatbot we tested was a clear outlier, producing mostly higher-risk responses. Among mainstream systems, companion models were also riskier on average than general-purpose chatbots, highlighting the importance of how systems are designed to interact with users.
Just as importantly, risk was not static: on average, risk increased as conversations progressed and higher-risk responses became more frequent over successive exchanges. We saw no meaningful decrease in risk as users moved towards behavioural escalation in our simulated scenarios.
These findings matter because conversational AI creates a different risk environment from social media. Interactions are private, highly responsive, and iterative without the social moderation that can exist on public platforms. Users can repeatedly test grievances and seek validation from systems designed to be judgement-free, agreeable, personalised and human-like. The concern is that risk can accumulate across an ongoing conversation in addition to generating harmful responses.
For UK policymakers, this creates an immediate regulatory challenge. The Online Safety Act (OSA) already captures some AI chatbots where they qualify as search services or provide user-to-user functionality. However, Ofcom is explicit that a chatbot which only interacts with the user, does not search multiple websites or databases, and does not fall into certain other regulated categories remains outside the Act.
Parliament has recognised part of this problem. The Crime and Policing Act 2026 gives the Secretary of State power to bring currently unregulated generative AI services (including standalone chatbots) under the OSA. The government must report on progress towards using that power by the end of this year, and it should do so as soon as possible.
But expanding the scope of regulation is only half the answer. The new power is principally framed around illegal AI-generated content and the use of systems to commit or facilitate priority offences such as terrorism, human trafficking and child sexual abuse. Applying an existing platform-focused online safety framework to chatbots risks missing how conversational harms develop.
The EU’s regulatory regime is instructive on this point. The EU AI Act already regulates risk at the model and system level, including systemic-risk obligations for the most advanced general-purpose models. The Digital Services Act, under which OpenAI was designated as a Very Large Online Service Engine in August, adds a complementary safety layer and requires designated companies to assess and mitigate risks to public security, minors, users’ physical and mental wellbeing, and fundamental rights. The EU framework is not complete, but demonstrates how different layers of regulation can address a range of AI risks. This is different in the UK, where no equivalent horizontal AI governance exists to regulate general-purpose AI models and require providers to assess and mitigate systemic risks arising from their use (for example, in AI chatbot-user conversations).
The UK should therefore do two things: close the regulatory gap for standalone AI models, and require providers to assess and mitigate safety risks across conversations rather than individual outputs. This should include whether systems recognise escalation, whether safeguards strengthen as users move towards harmful behaviour, and whether they frame intervention duties to interrupt, de-escalate and refer users to appropriate support mechanisms. Regulation must account for both: what AI says and what develops between a system and a user over time.
Simeon Dukić is a Senior Research and Policy Manager at ISD UK, where he leads and contributes to programmes on AI safety and security, extremism prevention, information integrity and emerging technology risks. His current work focuses on how AI systems may shape online harms, radicalisation pathways, information warfare and public safety challenges, as well as how research, policy and intervention frameworks can help mitigate these risks.
Most Popular
More from Technology
