The profound danger of conversational AI

When researchers contemplate the risks that AI poses to human civilization, we often reference the “control problem.” This refers to the possibility that an artificial super-intelligence could emerge that is so much smarter than humans that we quickly lose control over it. The fear is that a sentient AI with a super-human intellect could pursue goals and interests that conflict with our own, becoming a dangerous rival to humanity.

While this is a valid concern that we must work hard to protect against, is it really the greatest threat that AI poses to society? Probably not. A recent survey of more than 700 AI experts found that most believe that human-level machine intelligence (HLMI) is at least 30 years away. 

On the other hand, I’m deeply concerned about a different type of control problem that is already within our grasp and could pose a major threat to society unless policymakers take rapid action. I’m referring to the increasing possibility that currently available AI technologies can be used to target and manipulate individual users with extreme precision and efficiency. Even worse, this new form of personalized manipulation could be deployed at scale by corporate interests, state actors or even rogue despots to influence broad populations.   

The ‘manipulation problem’

To contrast this threat with the traditional Control Problem described above, I refer to this emerging AI risk as the “Manipulation Problem.”  It’s a danger I’ve been tracking for almost two decades, but over the last 18 months, it has transformed from a theoretical long-term risk to an urgent near-term threat.

That’s because the most efficient and effective deployment mechanism for AI-driven human manipulation is through conversational AI. And, over the last year, a remarkable AI technology called Large Language Models (LLMs) has rapidly reached a maturity level. This has suddenly made natural conversational interactions between targeted users and AI-driven software a viable means of persuasion, coercion, and manipulation.  

Of course, AI technologies are already being used to drive influence campaigns on social media platforms, but this is primitive compared to where the technology is headed. That’s because current campaigns, while described as “targeted,” are more analogous to spraying buckshot at flocks of birds. This tactic directs a barrage of propaganda or misinformation at broadly defined groups in the hope that a few pieces of influence will penetrate the community, resonate among its members and spread across social networks.

This tactic is extremely dangerous and has caused real damage to society, polarizing communities, spreading falsehoods and reducing trust in legitimate institutions. But it will seem slow and inefficient compared to the next generation of AI-driven influence methods that are about to be unleashed on society. 

Real-time AI systems

I’m referring to real-time AI systems designed to engage targeted users in conversational interactions and skillfully pursue influence goals with personalized precision. These systems will be deployed using euphemistic terms like Conversational Advertising, Interactive Marketing, Virtual Spokespeople, Digital Humans or simply AI Chatbots.

But whatever we call them, these systems have terrifying vectors for misuse and abuse. I’m not talking about the obvious danger that unsuspecting consumers may trust the output of chatbots that were trained on data riddled with errors and biases. No, I’m talking about something far more nefarious — the deliberate manipulation of individuals through the targeted deployment of agenda-driven conversational AI systems that persuade users through convincing interactive dialog.

Instead of firing buckshot into broad populations, these new AI methods will function more like “heat-seeking missiles” that mark users as individual targets and adapt their conversational tactics in real time, adjusting to each individual personally as they work to maximize their persuasive impact.  

At the core of these tactics is the relatively new technology of LLMs, which can produce interactive human dialog in real time while also keeping track of the conversational flow and context. As popularized by the launch of ChatGPT in 2022, these AI systems are trained on such massive datasets that they are not only skilled at emulating human language, but they have vast stores of factual knowledge, can make impressive logical inferences and can provide the illusion of human-like commonsense.

When combined with real-time voice generation, such technologies will enable natural spoken interactions between humans and machines that are highly convincing, seemingly rational and surprisingly authoritative. 

Emergence of digital humans

Of course, we will not be interacting with disembodied voices, but with AI-generated personas that are visually realistic. This brings me to the second rapidly advancing technology that will contribute to the AI Manipulation Problem: Digital humans. This is the branch of computer software aimed at deploying photorealistic simulated people that look, sound, move and make expressions so authentically that they can pass as real humans.

These simulations can be deployed as interactive spokespeople that target consumers through traditional 2D computing via video-conferencing and other flat layouts. Or, they can be deployed in three-dimensional immersive worlds using mixed reality (MR) eyewear.  

While real-time generation of photorealistic humans seemed out of reach just a few years ago, rapid advancements in computing power, graphics engines and AI modeling techniques have made digital humans a viable near-term technology. In fact, major software vendors are already providing tools to make this a widespread capability. 

For example, Unreal recently launched an easy-to-use tool called Metahuman Creator. This is specifically designed to enable the creation of convincing digital humans that can be animated in real-time for interactive engagement with consumers. Other vendors are developing similar tools. 

Masquerading as authentic humans

When combined, digital humans and LLMs will enable a world in which we regularly interact with Virtual Spokespeople (VSPs) that look, sound and act like authentic persons. 

In fact, a 2022 study by researchers from Lancaster University and U.C. Berkeley demonstrated that users are now unable to distinguish between authentic human faces and AI-generated faces. Even more troubling, they determined that users perceived the AI-generated faces as “more trustworthy” than real people.

This suggests two very dangerous trends for the near future. First, we can expect to engage AI-driven systems to be disguised as authentic humans, and we will soon lack the ability to tell the difference.  Second, we are likely to trust disguised AI-driven systems more than actual human representatives. 

Personalized conversations with AI

This is very dangerous, as we will soon find ourselves in personalized conversations with AI-driven spokespeople that are (a) indistinguishable from authentic humans, (b) inspire more trust than real people, and (c) could be deployed by corporations or state actors to pursue a specific conversational agenda, whether it’s to convince people to buy a particular product or believe a particular piece of misinformation. 

And if not aggressively regulated, these AI-driven systems will also analyze emotions in real-time using webcam feeds to process facial expressions, eye motions and pupil dilation — all of which can be used to infer emotional reactions throughout the conversation.

At the same time, these AI systems will process vocal inflections, inferring changing feelings throughout a conversation. This means that a virtual spokesperson deployed to engage people in an influence-driven conversation will be able to adapt its tactics based on how they respond to every word it speaks, detecting which influence strategies are working and which are not. The potential for predatory manipulation through conversational AI is extreme. 

Conversational AI: Perceptive and invasive

Over the years, I’ve had people push back on my concerns about Conversational AI, telling me that human salespeople do the same thing by reading emotions and adjusting tactics — so this should not be considered a new threat.

This is incorrect for a number of reasons. First, these AI systems will detect reactions that no human salesperson could perceive. For example, AI systems can detect not only facial expressions, but “micro-expressions” that are too fast or too subtle for a human observer to notice, but which indicate emotional reactions — including reactions that the user is unaware of expressing or even feeling.

Similarly, AI systems can read subtle changes in complexion known as “blood flow patterns” on faces that indicate emotional changes no human could detect. And finally, AI systems can track subtle changes in pupil size and eye motions and extract cues about engagement, excitement and other private internal feelings. Unless protected by regulation, interacting with Conversational AI will be far more perceptive and invasive than interacting with any human representative.

Adaptive and customized conversations

Conversational AI will also be far more strategic in crafting a custom verbal pitch. That’s because these systems will likely be deployed by large online platforms that have extensive data profiles about a person’s interests, views, background and whatever other details were compiled over time.

This means that, when engaged by a Conversational AI system that looks, sounds and acts like a human representative, people are interacting with a platform that knows them better than any human would. In addition, it will compile a database of how they reacted during prior conversational interactions, tracking what persuasive tactics were effective on them and what tactics were not. 

In other words, Conversational AI systems will not only adapt to immediate emotional reactions, but to behavioral traits over days, weeks and years. They can learn how to draw you into conversation, guide you to accept new ideas, push your buttons to get you riled up and ultimately drive you to buy products you don’t need and services you don’t want. They can also encourage you to believe misinformation that you’d normally realize was absurd. This is extremely dangerous. 

Human manipulation, at scale

In fact, the interactive danger of Conversational AI could be far worse than anything we have dealt with in the world of promotion, propaganda or persuasion using traditional or social media. For this reason, I believe regulators should focus on this issue immediately, as the deployment of dangerous systems could happen soon.

This is not just about spreading dangerous content — it is about enabling personalized human manipulation at scale. We need legal protections that will defend our cognitive liberty against this threat. 

After all, AI systems can already beat the world’s best chess and poker players. What chance does an average person have to resist being manipulated by a conversational influence campaign that has access to their personal history, processes their emotions in real-time and adjusts its tactics with AI-driven precision? No chance at all. 

Louis Rosenberg is founder of Unanimous AI and has been awarded more than 300 patents for VR, AR, and AI technologies.