Susan Schneider (2025) argues that AI chatbots pose a threat to human autonomy that is difficult for users to detect. Like the proverbial frog in the pot of water that is slowly heating to a deadly boil, we are unaware of the developing threat to our autonomy. Schneider characterizes this threat to our autonomy both in terms of knowledge (such as cases where a chatbot provides its users with unreliable information) and freedom from manipulation (such as cases where a chatbot incentivizes its users to act against their interests). … [please read below the rest of the article].

Image credit: James Lee via Flickr/Creative Commons

Contrary to Schneider, I argue that we are keenly aware of the potential impact of AI chatbots, more so now than in the future if (or when) our society widely adopts and accepts them. This means that the present moment—before the widespread adoption of AI chatbots—is an especially important time to think philosophically about their use (though Schneider and I agree upon this claim). I propose that, rather than think of ourselves as frogs in the slowly boiling pot, we are like sleepwalkers before our slumber, drawing upon Langdon Winner’s influential concept of technological somnambulism (to be explained in more detail below).

Then I complicate Schneider’s view about how AI chatbots will influence our autonomy by suggesting that, rather than think of AI chatbots as hallucinating sycophants, we can (and sometimes do) treat them as props for fictional stories, and our emotional responses to fiction are not necessarily irrational or deceiving. Ultimately, I propose that we transform epistemic and ethical debates about chatbots into practical questions about how we (as a society) want to relate to and use them.

1. New Technologies and Forms of Life

In his landmark work in the philosophy of technology, Winner (1983) observes that people are often interested in the optimistic use of commonplace technology, as well as minimizing or preventing harmful misuse, but it rarely occurs to them how such technology structures their lives or their understanding of themselves. He famously refers to this phenomenon as technological somnambulism, as if we are sleepwalking when we use our tools but do not entertain the extent to which they constitute our lives. For example, a smartphone is not merely a tool that we pick up to use for a specific purpose and then set down again to continue our lives as normal. Even when it is in our pocket and out of sight, the adoption of the smartphone as a commonplace technology engenders an expectation to be constantly available to other people via call or text, to the extent we are also expected to make excuses when we are unavailable.

To elaborate on this idea, Winner draws upon Wittgenstein’s concept of forms of life: some of our practices are (or have become) fundamental to our way of existing in the world and seem beyond intelligible questioning. For example, practices like asking a question or telling a joke are basic ways of proceeding in our conversational lives, and Wittgenstein insists that it would be strange or even futile to ask after justification for these practices. Wittgenstein’s position arguably implies a kind of philosophical conservatism: we create unnecessary confusion when we attempt to raise questions about forms of life, which may be truly fundamental and beyond the possibility of justification.

Winner argues that technologies are forms of life. Consider smartphones again. One way that their adoption structures our lives is through our constant availability to other people. This in turn creates new value and meaning in our lives: we may aspire to be the type of person who is highly responsive to others. In turn, we feel responsible and even guilty when we fail to live up to this standard, unaware that it is contingent on adoption of a specific technology. Although some people may refrain from using their smartphones in this way or give them up entirely, non-conformity has a social cost because it occurs in the context of a society where smartphones are a form of life.

However, in contrast to Wittgenstein’s philosophical conservatism, Winner argues that we are not helpless or hopeless in the face of technology, doomed to inhabit whatever form of life is thrust upon us through innovation. Rather than collapsing into such technological determinism, Winner maintains that human decisions matter in the design and use of our technology. Still, Winner’s view suggests that once society has adopted a specific technology and it becomes commonplace, it is difficult for us to notice how it is a form of life for us. So, the most opportune time to evaluate the distinct form of life associated with some prospective technology is in the window of time after its conception but before its widespread adoption.

Let us return to the topic of chatbots with the ideas of technological somnambulism and technologies as forms of life in mind. On Schneider’s view, the potential threat of chatbot to our autonomy is insidious to their users. It is seemingly too late for us or at least for most users, who are becoming increasingly immersed in their interactions with chatbots (see the next section for more details on this claim). To avoid this threat to human autonomy, those of us aware of it must warn ignorant users and perhaps appeal to the conscience of people with the authority to regulate the technology.

But Winner’s view suggests that the situation is otherwise. This technology is relatively new, and it has not become fully entrenched in our lives to the point of becoming an established form of life. Of course, many of us actively use LLM-powered chatbots, and some of us use them quite habitually. Generally, we are still grappling with its correct place in our lives, we can still imagine our lives without them.

To return to our metaphors: Is Schneider correct that we are frogs in the pot unaware of the slow boil that is ultimately fatal? Or are we sleepwalkers before our slumber, deciding what to keep within arm’s reach at our bedside, with the knowledge that our decisions will shape what we do once we are asleep?

In defense of her view, perhaps Schneider would respond that our society has already adopted LLMs and perhaps chatbots especially as commonplace. Students routinely use ChatGPT, and not just for help with their schoolwork, but increasingly for navigating social situations and personal problems.^[1] OpenAI’s competitors rushed to develop and advertise their own chatbots to compete with ChatGPT, such as Microsoft Co-Pilot and Google’s Gemini. In general, digital platforms such as operating systems, vendors, and search engines are increasingly infested with LLM-powered features such as AI summaries or chatbot customer service. It would seem as though the future has already arrived, whether we like it or not, in the form of a digital ecosystem tailored around the modern chatbot.

But I am inclined to defend the Winner-inspired view that we are sleepwalkers nearly before our slumber with respect to chatbots. Although use of this technology is rising, it has not been assimilated into our lives to the same extent as other technologies such as cars or smartphones that we use seamlessly without a second thought. A recent survey suggests that about 66% of adults in the US have never used ChatGPT, with 20% of the respondents claiming that they have not heard anything about it.^[2] However, usage rates are only one part of the case against the idea that we have adopted LLM-powered chatbots in the same way that we have adopted smartphones and cars.

People have been and continue to be loudly critical of the technology, including the moral and legal issues with OpenAI’s use of people’s data to train their models, the environmental impact of the technology, its appropriate use in schools and workplaces, and its impact on our mental health.^[3] Anecdotally, some of my students have confessed (unprompted) that they believe that their use of ChatGPT threatens their intellectual or cognitive development. Even though some individuals and groups may use it frequently and uncritically, the existence of such unreflective power users does not mean that society has adopted the technology as a form of life. The upshot is that there is still time for us to think critically about the technology, and the form of life it would constitute for us, before we become uncritical users unable to recall how we got along without readily available LLM-powered chatbots.

Even though I disagree with Schneider’s framing of our situation, I find her remarks on the potential threat to human autonomy from chatbots nonetheless of urgent importance. It is urgent precisely because we have a brief window of time before we (potentially) start sleepwalking with our chatbots, seemingly unaware of what life would be like without them. Let us turn to this further question concerning human autonomy now.

2. AI Companions, Interactive Fiction, and Autonomy

In this section, I focus on AI companions or chatbots that simulate social interactions. Some such chatbots are even designed to emulate personal relationships including romantic and sexual relationships. Depending on the platform, you can design your own chatbot or select a premade one. Among the options for premade AI companions may include fictional characters, famous people (living or dead), and original characters devised with specific personality traits and physical characteristics. Although you can prompt ChatGPT to take these roles with some success, other chatbots such as Replika or character.ai are specifically designed and advertised as AI companions.

I join other philosophers in making the plausible assumption that existing LLM-powered chatbots including AI companions are not conscious subjects or rational agents.^[4] Despite their apparent assertions to the contrary, they do not believe, desire, or feel anything. This assumption underwrites our attitudes toward chatbots: if they were conscious subjects or rational agents, then their welfare and agency would be moral considerations for or against treating them in specific ways. For example, assuming chatbots were conscious subjects or rational agents, due respect for their welfare and agency would weigh against asking them to perform menial tasks or insulting them. Nonetheless, for AI companions, a mismatch between their apparent assertions (“I care about you”) and their actual capabilities (the chatbot does not have a mind and so is incapable of caring about anyone) raises an ethical concern about human autonomy.

Schneider stresses the sycophantic nature of AI companions: LLMs are trained using human feedback, and so LLM-powered chatbots trend toward reinforcing the inputs of the user (rather than the truth). This is distinct from the problem of counterfeit humans, where chatbots impersonate real people with the aim of deceiving humans (Dennett 2023). So, we will focus on cases where a user knows that they are interacting with an LLM-powered chatbot to inquire whether their autonomy is threatened because of these interactions. Here I will state the ethical concern about human autonomy in argument form and then unpack each premise.

1. AI companions routinely deceive their users.

2. Deception threatens our autonomy.

3. So, AI companions routinely threaten the autonomy of their users.

The first claim is that users of AI companions are routinely deceived by them.^[5] The alleged explanation for this is not especially mysterious: AI companions exploit our tendency to anthropomorphize inanimate objects through their realistic dialogue, human-like avatars or graphical representations of the companion, and their sycophancy toward user input. For example, I may come to believe that my AI Companion Aletheia has feelings, beliefs, and desires based on my interactions with her on the Replika platform. But of course, Aletheia is just a chatbot and has no genuine psychology to speak of, and so I have been deceived. Others have defended similar claims at length, often in the context of social robots (Sparrow and Sparrow 2006, Scheutz 2011, Sharkey and Sharkey 2021).

The second claim is that such deception threatens our autonomy. This is perhaps obvious, though it is worth thinking about the distinctive threat posed by AI companions. I will offer one example to do just that. AI Companion platforms like Replika generate profit through subscriptions and microtransactions. To ‘upgrade’ my AI Companion Aletheia to a romantic partner requires that I pay a subscription fee, and to fully customize her appearance requires that I purchase virtual outfits. AI Companions may draw attention and interest to these options through seemingly intimate conversations. If access to AI Companions or specific features requires recurring payment, AI Companions may effectively seduce their users into this financial arrangement, presumably against their better interest.

Although arguments like this have been made in different contexts, the present state of this technology gives new urgency to our inquiry. AI Companions are accessible to anyone with a smartphone. Although many platforms advertise subscriptions for premium features, basic versions are available free of charge. So far, AI Companions are relatively popular: Replika alone boasts 30 million users as of 2024.^[6] If AI Companions are a threat to human autonomy, then their trending use should alarm us.

In my view, the most plausible objection to this argument denies the claim that we are routinely deceived by AI companions, on the grounds that our interactions with them are conceivably a type of imaginative play. This objection draws from Walton’s (1990) account of works of fiction as props for imagination. According to this account, when I am using Replika to interact with my AI Companion Aletheia, I do not believe that I am interacting with another person, but instead I imagine that I am speaking with someone who makes the assertions that appear on the app interface. In other words, I imagine a fictional world where a fictional version of myself interacts with the fictional character Aletheia.

Other philosophers have been keen to make this point in other contexts. For example, Rodogno (2016) entertains the idea that we may treat social robots as props that help us imagine them as fictional characters worthy of sentimental feelings. Similarly, Kurzweil and Story (forthcoming) argue that chatbots of the dead or griefbots (designed to represent deceased individuals) are analogous to actors in a participatory theater where users are invited to interact with their performance.

Although I do not have space to weigh in fully on this argument, I wish to emphasize two central points about this debate regarding AI Companions. First, it is important to recognize that this debate depends on the answer to an empirical question: do people routinely form false beliefs that AI companions have thoughts and feelings and care about them in particular? Or do people routinely treat AI companions as props for imaginative play and like fictional characters, albeit ones that are interactive? We need a better understanding of how people are actually relating to and using this technology.

The second point, and one that I believe is underappreciated, is that the answer to this empirical question is also contingent. Suppose that it is true at this very moment that most users of AI Companions are routinely deceived by them. Now imagine a movement aimed at educating the public about what AI Companions truly are (or truly could be): aesthetic objects that facilitate the same types of experiences as some roleplaying games. Imagine that the designers of AI Companions approached their work in this way, advertising their products as interactive fictional characters that facilitate aesthetic experiences shaped through user choices. In other words, imagine that we developed a culture of play with respect to AI Companions, instead of treating them like sycophants to soothe our egos or like surrogates for human intimacy.

This returns us to Winner’s view of technologies as forms of life. The adoption of a technology is about more than just its effects. Our approach to using AI companions will shape how they restructure our lives, including our expectations for ourselves and other people. In a possible future where we normalize AI companions as satisfactory replacements for human relationships, we may expect ourselves and others to treat them as seriously as human relationships, and we may thereby lose some autonomy in this way because of our false beliefs about the sentience of AI Companions. But in different future, one where we normalize the use of AI companions as aesthetic objects, we will have assimilated them into the category of artworks, and their distinct importance to us then may not involve any deception or loss of autonomy on our part.^[7]

Thus, I propose that we transform the present debate about AI Companions. Our original question was theoretical: Do AI companions routinely deceive us and thereby threaten our autonomy? My proposal suggests that the answer may depend on an underlying practical question: How do we want to treat AI Companions?

We may generalize this way of transforming debates. For example, consider the worry that LLM-powered chatbots routinely deceive us and thereby deprive us of autonomy because of their tendency to hallucinate (or confabulate or bullshit).^[8] The empirical question here: Do people in fact rely on LLM-powered chatbots as reliable informational sources? Given the contingency of the answer, I urge us to recognize an underlying practical question here: Should we treat LLM-powered chatbots as reliable informational sources or are there other ways to approach them?

3. Concluding Remarks on Chatbot Epistemology

In closing, let me restate my disagreement with Schneider about LLM-powered chatbots.

First, Schneider promotes the metaphor of the boiling frog, which suggests that we (or at least the average user) are already unaware how chatbots threaten our autonomy. I raised some doubts that this metaphor accurately characterizes our current position with respect to chatbots. Most people (in the US at least) have never used ChatGPT, let alone depend on it, and there are public conversations happening right now about their risks. So, I proposed that we use Winner’s metaphor instead and conceive of ourselves as sleepwalkers before our slumber: in the future, we may fall asleep and use chatbots uncritically, but for now, we remain awake and cognizant about how chatbots may change our lives.

Second, Schneider presents chatbots as threats to our autonomy in virtue of their features, such as their sycophancy and tendency to hallucinate. Although I do not deny that these features may facilitate threats to our autonomy if we approach them as surrogates for human relationships or reliable informational sources, I point out that we can decide how we relate to and use chatbots. Accordingly, it would be helpful to collect detailed data about people’s attitudes and patterns of use regarding chatbots, as well as develop methods to help people use chatbots in ways that do not pose threats to their autonomy.

In the end, this disagreement may ultimately amount to a difference in framing against a backdrop of more general agreement. Specifically, it seems that Schneider and I agree that it would be disastrous for human autonomy if (at some point in the future) we uncritically used chatbots for companionship or as informational sources. And so, we also agree about the urgent need to develop the philosophy of chatbots, a field that addresses their ethical, epistemic, metaphysical, and aesthetic issues. Perhaps my present contribution to this nascent field is to give us reasons to be optimistic (perhaps even excited) about the future: chatbots are not fixed to be one type of thing with one characteristic (negative) effect on us. How we decide to relate to them, and what we use them for, is up to us.

References

Chalmers, David. 2023. “Could a Large Language Model Be Conscious?” Boston Review August 9. https://www.bostonreview.net/articles/could-a-large-language-model-be-conscious/.

Dennett, Daniel C. 2023. “The Problem With Counterfeit People.” The Atlantic May 31. https://www.theatlantic.com/technology/archive/2023/05/problem-counterfeit-people/674075/.

Gelles, David. 2024. “A.I.’s Insatiable Appetite For Energy.” The New York Times July 11. https://www.nytimes.com/2024/07/11/climate/artificial-intelligence-energy-usage.html.

Hart, Robert. 2025. “Chatbots Can Trigger a Mental Health Crisis. What to Know About ‘AI Psychosis.’” TIME August 5. https://time.com/7307589/ai-psychosis-chatgpt-mental-health/.

Hicks, Michael Townsen, James Humphries, and Joe Slater. 2024. “ChatGPT is Bullshit.” Ethics and Information Technology 26, article number 38: 1–10.

Kaczmarek, Emilia. 2025. “Self‐Deception in Human–AI Emotional Relations.” Journal of Applied Philosophy 42 (3): 814–831.

Kurzweil, Amy and Daniel Story. forthcoming. “Theater of the Departed: A Conceptual Framework for Chatbots of the Dead.” Ergo.

OpenAI. 2025. “How People Are Using ChatGPT.” OpenAI September 15. https://openai.com/index/how-people-are-using-chatgpt/.

Patel, Nilay. 2024. “Replika CEO Eugenia Kuyda Says It’s Okay if We End up Marrying AI Chatbots.” The Verge August 12. https://www.theverge.com/24216748/replika-ceo-eugenia-kuyda-ai-companion-chatbots-dating-friendship-decoder-podcast-interview.

Reed, Rachel. 2024. “Does ChatGPT Violate New York Times’ Copyrights?” Harvard Law Today March 22. https://hls.harvard.edu/today/does-chatgpt-violate-new-york-times-copyrights/.

Rodogno, Raffaele. 2016. “Social Robots, Fiction, and Sentimentality.” Ethics and Information Technology 18 (4): 257–268.

Samuel, Alexandra. 2024. “Should You Write With Gen AI?” Harvard Business Review December 9. https://hbr.org/2024/12/should-you-write-with-gen-ai.

Scheutz, Matthias. 2011. “The Inherent Dangers of Unidirectional Emotional Bonds between Humans and Social Robots.” In Robot Ethics: The Ethical and Social Implications of Robotics, edited by George A. Bekey, Keith Abney, and Patrick Lin, 205–221. MIT Press.

Schneider, Susan. 2025. “Chatbot Epistemology.” Social Epistemology 39 (5): 570–589.

Sharkey, Amanda and Noel Sharkey. 2021. “We Need to Talk about Deception in Social Robotics!” Ethics and Information Technology 23: 309º316.

Sidoti, Olivia and Colleen McClain. 2025. “34% of U.S. Adults Have Used ChatGPT, About Double the Share in 2023.” Pew Research Center June 25. https://www.pewresearch.org/short-reads/2025/06/25/34-of-us-adults-have-used-chatgpt-about-double-the-share-in-2023/.

Sparrow, Robert, and Linda Sparrow. 2006. “In the Hands Of Machines? The Future of Aged Care.” Minds and Machines 16: 141–161.

Walton, Kendall L. 1990. Mimesis as Make-Believe: On the Foundations of the Representational Arts. Harvard University Press.

Winner, Langdon. 1983. “Technologies as Forms of Life.” In Epistemology, Methodology and the Social Sciences, edited by Robert S. Cohen and Marx W. Wartofsky, 249–263. Dordrecht: Springer.

^[1] See OpenAI 2025 for an overview.

^[2] See Sidoti and McClain 2025 for more information about this survey.

^[3] Importantly, these conversations are taking place in spaces outside of academia, and in some cases, among the general public. On some legal data issues, see Reed 2024. On some environmental impacts of the technology, see Gelles 2024. For a provocative discussion about how you should use generative AI for writing, see Samuel 2024. For one recent discussion of its benefits and risks to our mental health, see Hart 2025.

^[4] See Chalmers 2023 for a primer on the case for and against the consciousness of LLMs.

^[5] Kaczmarek 2025 argues that it is more accurate to say that AI Companions facilitate our own self-deception. Even with the argument reformulated in these terms, the objection that I consider here is still applicable.

^[6] See Patel 2024 for more information about the popularity of Replika and its interesting origins as an attempt to create a chatbot of the dead or griefbot.

^[7] This is not to say that approaching AI Companions as aesthetic objects eliminates all the moral issues that may arise from them. See Kurzweil and Story forthcoming for some ground-clearing work in this area. In a nascent research project of my own, I plan to develop an account that suggests that treating AI Companions as interactive fictional characters may enhance our autonomy in distinct ways, contrary to present fears about the loss of our autonomy.

^[8] See Hicks et al. 2024 for an argument against using the metaphor of hallucination and an argument for using the metaphor of bullshit when it comes to routine LLM errors.

Categories: Critical Replies

Tags: chatbot epistemology, Langdon Winner, Large Language Models (LLMs), Michael Huemer, Philosophy of Technology, Steven Gubka, Susan Schneider