When it comes to AI and how it will change our life, public comments often make me think of that Timbuk 3 song, “The future’s so bright, I gotta wear shades”[1]. Voices pleading for some caution are easily drowned out as too overly pessimistic. Many are still convinced that AI is just another tool, not able to “think” and, therefore, not able to make any decisions on its own. While the idea that we’re in the driver’s seat can be very comforting and reassuring, it is time for a wake-up call.[2]
Poem "The Station" by Ernest Claes (Belgian author, 1885-1968)
AI agents
Sometimes reality catches up with us faster than expected. The past few years, each time when I brought up the idea of AI systems developing ‘agency’ with my students, I was met with blank stares or frowning. – Today, the term ‘AI agent’ has become a new buzzword overnight. Only a couple of weeks ago[3], OpenAI – one of the leading AI research organizations – announced that they are planning to charge up to $20,000 per month for specialized so-called AI ‘agents’[4]. The prices vary from $2,000 a month for an AI agent at the level of a “high-income knowledge worker”[5], a software developer agent will cost you about $10,000 a month, and at the top we find a ‘Phd-level research’[6] agent which will cost you no less than $20,000 a month.
“AI isn’t a tool – it’s an agent” (Yuval Noah Harari)
Be that as it may, one thing’s for sure: developments in AI are making giant leaps at an ever-increasing pace. Only half a year ago, in September 2024, the well-known historian Yuval Noah Harari published Nexus, on information networks from early history to AI today. In Nexus, he warns us: “AI isn’t a tool – it’s an agent”[7]. He meant to say that AI has the capacity to replace humans in the making of decisions.[8] Harari was heavily criticized for this, being put away by some as a doomsday prophet.[9] People reproached him that his book was “based on shallow scholarship”.[10] Someone even bluntly suggested that the real lesson we can learn from Harari is that “there’s an incredible amount of money to be made with doomsday predictions”[11].
“AI systems might be more intelligent than we know” (Geoffrey Hinton)
But even Geoffrey Hinton, the so-called godfather of AI, who shared the 2024 Nobel Prize in Physics with John Hopfield, warns us sternly. “We’re moving into a period when for the first time ever we may have things more intelligent than us.” - “AI systems may be more intelligent than we know and there’s a chance the machines could take over”.[12] He even left Google a year before, in May 2023, because of his concerns about the many risks of AI.[13]
If even the very brightest and the most well-versed among us in the field of AI yield warnings about the possible unforeseen devastating consequences of AI, maybe it is time to listen to what they have to say. Maybe it is time to postpone our judgement, even if only for a moment.
Vincent Ginnis, professor of mathematics, physics and artificial intelligence at the Free University of Brussels-VUB and at Harvard University, is very concerned about the disconcerting lack of concern about the possible dangers of AI. At a recent AI safety conference in Paris[14], no one seemed to be concerned anymore with the dangers of AI. Instead, it was all about PR and power struggles.
Once, Ginnis says, AI safety was about risks for humanity. Millions of people possibly losing their jobs, misinformation and manipulation spreading at a scale undermining democracy, AI systems possibly one day taking decisions we no longer comprehend, let alone control.[15] Instead, at the conference in Paris it was all about power. Who will get AI? Who controls it? Who is running ahead in the race? The focus, according to Ginnis, shifted from risks to geopolitics.
The question which intrigues me the most is not whether or to what extent Harari, Hinton, Ginnis and others are right about their warnings. What intrigues me the most, as a philosopher of science, is: why are so many people so adamant and resolute in brushing aside all these warnings? Why do we cling so tightly to the reassuring idea that AI is just a mere tool, totally under our control? We are in the driver’s seat, remember? What are the tacit assumptions which apparently make it very hard for us to find the blind spots in our rosy vision on AI?
The Chinese room thought-experiment
When I was still a graduate student in philosophy, way back in the 1980s, I remember we were discussing John Searle’s thought-experiment of the Chinese room. Remember that 40 years ago, AI was still a very remote theoretical possibility, a popular theme in science fiction movies perhaps, but not something to be taken very seriously. Nevertheless, some philosophers gave it their attention, more often than not in order to concoct sophisticated arguments to show that artificial intelligence would not be possible. A machine, a computer, could never have a mind in the same way human beings can be said to have minds.
In brief, John Searle’s argument goes as follows.[16] Someone is sitting in a room and receives papers with Chinese characters from the left (through a slit in the wall). This person then meticulously follows a detailed instruction table, like a computer program, to transform the Chinese messages into other messages with different characters on another sheet of paper. Once the conversion is complete, this person then puts this new paper out to the right (likewise, through a slit in the wall). To an outside observer, it seems as if the person inside the room understands Chinese. However, in reality, the person inside the room is just following instructions, he does not need to understand a single word of Chinese himself.
Searle wanted to show with this thought experiment that two people can be functionally identical: a native Chinese speaker and the person inside the Chinese Room. They both provide perfect answers to questions posed in Chinese. Yet, they have completely different mental states (one understands Chinese, while the other does not understand it at all). The bottom line is: a computer will remain fundamentally different from a human being. A computer system will never “truly” understand what it is doing, whereas human beings obviously can.
Mary in the black-and-white room
Another famous example comes from the Australian philosopher Frank Jackson. In 1986 he published his famous article, What Mary didn’t know[17]. Even though it is almost forty years old, it is about a thought-experiment which is still used today in discussions about the possibility of artificial general intelligence (AGI).
This experiment is about a fictional scientist in a distant future, Mary. In that distant future, both physics and neurophysiology have reached a final state. That means that Mary knows everything these sciences have to say about perceiving colors.
But there is something special about Mary: she lives in a completely colorless room. Everything is black or white. So, Mary has never seen a color before in her entire life. Her knowledge of colors is therefore purely based on books about physiology, neurology, and the biochemistry of color perception.
Now it gets exciting. One day, Mary finds a secret door to the outside world. The first thing she sees when she steps outside is a red apple. Jackson asks: does Mary learn something new when she sees this apple?
He answers: yes, she learns a new fact — she learns what it is like to experience the color 'red’. This phenomenon is referred to in philosophy of mind as ‘qualia’ = ‘individual instances of subjective, conscious experience’ (qualia is Latin and is the plural of quale, which literally means ‘such as’).
Jackson aims to conclude that knowledge about qualia fundamentally relies on subjective experiences—unlike knowledge about physical states in our brains. By subjective, we mean how someone experiences or judges something from a personal perspective.
Should AI possess consciousness in order to be intelligent?
Jackson’s argument was originally meant to question the philosophical position of physicalism (in short, the view that “everything is physical”, that there is nothing “above” the physical[18]). But his thought experiment has also become a classic in discussions about AI.[19] In short: it is used to illustrate the idea that there will always be an unbridgeable gap between AI and the human mind. AI is Mary stuck in the black-and-white room, never having seen a color in her entire life. The human being is Mary when she walks into the outside world and sees a red apple. Apparently, some people desperately want to hold on to the idea that their "self-awareness" is some kind of mystical property that fundamentally distinguishes them from animals and intelligent machines.
That brings me to a very intriguing observation. The tacit assumption underlying all these approaches seems to be that AI should possess consciousness in order to be called “truly” intelligent. All these thought experiments and arguments serve the same strategy: machines, artificial intelligence, cannot possibly attain consciousness or “self-awareness”, if you’d like, and, hence, machines, artificial intelligence, cannot possibly be called “really” intelligent.
The risk of our blind spot
It is exactly this tacit assumption that intelligence somehow presupposes consciousness that I want to put into question. This is crucial to my argument, so I want to state it as explicitly as I can. For some reason or another, whatever it might be, some people still see computers as inanimate, dumb machines which are incapable of thinking or feeling anything. And, so the reasoning goes, because computers cannot think or feel anything, they are not capable of making any decisions on their own.
Why, you might ask, is that so important? Because we run the risk of creating a blind spot. As long as we keep on seeing AI systems as a mere tool, as a “dumb” instrument, passively waiting for human instructions to do something, we put ourselves in jeopardy. Why? Because we risk overlooking the fact that these systems are getting more and more agency overnight, that is to say, more and more autonomy and decision-making capabilities. If those rapidly increasing decision-making capabilities of AI systems remain in our blind spot, we run a serious risk of continuing to outsource human decision-making to the point where it is no longer under our control.
“I’m sorry Dave, I’m afraid I can’t do that.” (HAL 9000)
When it comes to artificial general intelligence (AGI), the risk of the blind spot impeding us to see the real potential of danger becomes even greater. When bringing up the mere possibility of artificial general intelligence, you easily get your share of disbelief and laughter, conjuring up the famous image of the super-intelligent HAL 9000 computer on board of the spaceship U.S.S. Discovery in the motion picture 2001: A Space Odyssey (1968, Stanley Kubrick), refusing to open the air-lock, in that very calm, soothing tone: “I’m sorry Dave, I’m afraid I can’t do that”[20].
By brushing aside the mere possibility of artificial general intelligence, we run the risk of turning a blind eye to the real dangers of the rapid development of current AI technology. The position we are in right now remembers me of the horrifying images of the 2004 tsunami in Sout-East Asia. Remember the footage of the receding water from the beaches, with people laughing at this strange phenomenon, small boats, sloops, suddenly lying dry at the sea floor, maybe some fish flopping in a remaining shallow pool, children still playing around. Even when they saw the wall of water looming in the distance, people’s warning systems still didn’t seem to be triggered, they curiously kept on staring in the distance, instead of deciding to make a run for it to higher grounds as quickly as possible.
“AI is set to surpass us in speed and understanding.” (Geoffrey Hinton)
We must face the facts that some recent developments are taking an ugly turn. At a recent conference, Geoffry Hinton made a very interesting point, which strengthens me in my argument that we are put on the wrong track, set on the wrong foot, so to speak, when we continue to convince ourselves that the current AI systems are just dumb tools, not capable of “really” understanding what they are doing. Hinton’s point was that even with today’s large language models (LLMs), there is a key difference between them and the way human memory works, and that is “AI’s unmatched ability to share knowledge”[21]. Human beings pass information in small pieces, whereas AI has the ability to synchronize trillions of bits in the blink of an eye:
“(…) It’s no competition,” he said. If intelligence is about learning and sharing knowledge, AI is set to surpass us in speed and understanding. It’s a “very scary conclusion,” Hinton said—a warning that highlights the need for consensus on AI’s capabilities. (…)” [22]
The danger is coming from us
Now, let us not get carried away by popular science fiction ideas of evil AI systems taking over the world. The danger of AI does not come from systems like HAL 9000, the super-intelligent computer, eliminating all humans on board of the U.S.S. Discovery. The danger is coming from us. It is we who are putting ourselves in danger. Because it is we who are turning a blind eye to the growing agency of AI systems. And we are doing so because we are held captive by an image, the iconic image of a computer as a simple box with a keyboard. It is this image which makes us see AI as a mere tool. Because we tacitly assume that agency presupposes consciousness, and since we are firmly convinced that a simple box with a keyboard cannot possibly acquire consciousness, AI cannot possibly acquire autonomous agency.
Even now, as we speak, AI systems analyze tremendous amounts of data, they take decisions in fractions of a second, more and more without any human intervention. Just think about all the sophisticated algorithms which manage the content feed on social media. Think about the trading by algorithms on financial markets. The point is: we rely more and more on AI systems to make decisions that once were the sole province of human beings.
The principle of ‘least effort’
As we are getting more and more comfortable towards these new technologies, as we are embracing them more deeply, we tend to become less vigilant over the technology, not to say downright lazy. The principle of ‘least effort’ is a strong predictor of human behavior.[23] It makes me think of my former life, when I was still working for IT companies. In those days, you had to be able to write sophisticated and elegant search queries in SQL-type languages to retrieve relevant information from a database. Nowadays, we just shout “Hey, Google!”, followed by some vague query, far from anything elegant, and we accept the result at face value, without giving it a second thought. The near infinite capacity of human beings to take things for granted will never fail to amaze me. Aren’t we lulling ourselves to sleep too quickly?
Where do the US “reciprocal tariffs” come from?
A striking example of how this lazy attitude, blindly accepting what AI systems regurgitate, might lead us straight into disaster is reported by Bastian Leibe, professor at the RWTH Aachen University in Germany.[24] When President Donald Trump from the United States announced his reciprocal tariffs on April 2nd, 2025, a number of people noticed that these proposed tariffs are not related to the actual tariffs these countries charge on imports from the United States. Instead, they have been shown to correspond to the United States’ trade deficit divided by the United States import volume from that country, which, according to economists, does not make sense at all from an economic perspective.[25] Since April 2nd, a lot of economists have been trying to find out how on earth someone could come up with such an insane strategy. Until, Leibe continues his argument, someone found out that if you pose the question of tariff tables to current LLMs (like ChatGPT version 4o, Gemini 2.5pro et al.), they all propose tariffs which turn out to be very close to the tariffs in the list of president Trump. Leibe concludes that the most likely conclusion is that the Trump administration simply based its tariffs “on the unchecked outputs of an LLM”[26]:
“This has real-world consequences. It is already sending economies into turmoil and it will cause worldwide harm and suffering. And it is sadly nothing that AI safety research could have prevented -- because the problem lay in front of the screen.” [27]
We are lured into outsourcing our decision-making and autonomy to AI
We see it happening all around us, as we speak. Society is becoming more and more complex. We are delegating human decision-making to sophisticated systems which involve storing our digital data and automating decision rules at an ever-increasing pace.
We are lured into outsourcing our cherished decision-making and autonomy to AI systems step by step. As these smart systems, powered by sophisticated machine-learning, will rapidly augment their sophistication level in the next, say, ten years from now, we might lose the ability to keep on making decisions independently of these systems. Barry Chudakov, founder and principal of Sertain Research, sees the relationship between human beings and AI systems as
“(…) a struggle between the determined fantasy of humans to resist (‘I’m independent and in charge and no, I won’t give up my agency!’) and the seductive power of technology designed to undermine that fantasy (‘I’m fast, convenient, entertaining! Pay attention to me!’)” (…)” [28]
We will have to face difficult questions
What is the way forward? Is the future so bleak as some of these findings tend to suggest? – No, I think not. But it is time that we realize that we will have to face difficult questions, now that there is still time to do so. Janna Anderson and Lee Rainie sum it up succinctly: what are the things we really want agency over? Under what conditions will we turn to AI to help us in making decisions? And under what conditions and tight control mechanisms are we prepared to outsource certain precisely defined decisions to AI systems?[29]
Guarding the transition towards human-AI cooperation
We do not have the luxury of turning a blind eye to these difficult questions, as we must see to it that the transition towards human-AI cooperation does not become too disruptive. Because this could present not only a threat to an inclusive and just division of labor, but also possibly leading to destabilizing society.[30] When tackling these thorny issues, our focus should remain on the really important question: how can we contribute to building a more inclusive and fairer world? From this perspective, we plea for the need of continuing policy debates on AI regulation taking explicitly into account the phenomenon of growing AI agency.
References
Anderson, J. & Rainie, L. (2023). The Future of Human Agency. In Pew Research Center, https://www.pewresearch.org/internet/2023/02/24/the-future-of-human-agency/
Baron, S. (2025). Are a Machine's Thoughts Real? The Answer Matters Now More Than Ever. In Science Alert, https://www.sciencealert.com/are-a-machines-thoughts-real-the-answer-matters-now-more-than-ever
Douglas Heaven, W. (2023). Deep learning pioneer Geoffrey Hinton quits Google. In MIT Technology Review, https://web.archive.org/web/20230501125621/https://www.technologyreview.com/2023/05/01/1072478/deep-learning-pioneer-geoffrey-hinton-quits-google/
Edwards, B. (2025). What does “PhD-level” AI mean? OpenAI’s rumored $20,000 agent plan explained. In Ars Technica, https://arstechnica.com/ai/2025/03/what-does-phd-level-ai-mean-openais-rumored-20000-agent-plan-explained/
Ferguson, N. (2025). The Doom Nexus. In Niall Ferguson’s Time Machine, https://niallferguson.substack.com/p/the-doom-nexus
Foreman, J. T. (2024). How to Make it as a Doomsday Prophet. In The Metaphor, https://www.taylorforeman.com/p/how-to-make-it-as-a-doomsday-prophet
Ginnis, V. (2025). Is er nog íémand bekommerd om de gevaren van AI? In De Standaard, 15 February 2025, https://www.standaard.be/cnt/dmf20250214_96655287 [In Dutch; English title: Is there still anyone concerned about the dangers of AI?]
Harari, Y. N. (2024). Nexus. A Brief History of Information Networks from the Stone Age to AI. Vintage Publishing, Kindle Edition.
Jackson, F. (1986). What Mary Didn’t Know. In The Journal of Philosophy, Vol. 83, No. 5 (May, 1986), pp. 291-295.
Leibe, B. (2025). Post on LinkedIn, https://www.linkedin.com/feed/update/urn:li:activity:7313873939691130880/
Lim, D. (2024). Why Yuval Noah Harari’s AI Doomsday Prophecies Are Misleading. In Medium, https://medium.com/@don-lim/why-yuval-noah-hararis-ai-doomsday-prophecies-are-misleading-5541504ec3ab
Molek, N., Pulinx, R. & van Biezen, A. (eds.)(2024). Analysis of the State of the Art on the Future of Human Workforce. Scientific Report. Transform, European Union.
Palazzolo, S. and Weinberg, C. (2025). OpenAI Plots Charging $20,000 a Month For PhD-Level Agents. In The Information, https://www.theinformation.com/articles/openai-plots-charging-20-000-a-month-for-phd-level-agents
Pelley, S. (2024). "Godfather of Artificial Intelligence" Geoffrey Hinton on the promise, risks of advanced AI. In CBS News, https://www.cbsnews.com/news/geoffrey-hinton-ai-dangers-60-minutes-transcript/
Renard, V. et al. (2024). Mary Steps Out: Capturing Patient Experience through Qualitative and AI Methods. In NEJM AI, Vol. 1 No. 12, https://ai.nejm.org/doi/10.1056/AIp2400567
Saso, E. (2025). The path to safe, ethical AI: SRI highlights from the 2025 IASEAI conference in Paris. In Schwarz Reisman Institute for Technology and Society, University of Toronto. https://srinstitute.utoronto.ca/news/the-path-to-safe-ethical-ai
Searle, J. (1980). Minds, Brains and Programs. In Behavioral and Brain Sciences, 3, pp. 417-517.
Searle, J. (1984). Minds, Brains and Science. Cambridge, Mass., Harvard university press.
van Biezen, A.F. (2016). A Case for Naturalism. In van Biezen, A.F., The Torch of Discovery, http://alexanderfvanbiezen.blogspot.com/2016/05/a-case-for-naturalism.html
van Biezen, A.F. (2022). Top-Down Cosmology and Model-Dependent Realism. A Philosophical Study of the Cosmology of Stephen Hawking and Thomas Hertog. Brussels, VUB Press.
van Biezen, A. (2024). Emerging Skills for the Future Workforce. In Analysis of the State of the Art on the Future of Human Workforce. Scientific Report. Molek, N., Pulinx, R. and van Biezen, A. (eds.)(2024), Transform, European Union.
Wang, X. (2023). The Possibility of Artifical Qualia. In Communications in Humanities Research, https://doi.org/10.54254/2753-7064/6/20230083
Wiggers, K. (2025). OpenAI reportedly plans to charge up to $20,000 a month for specialized AI ‘agents’. In TechCrunch, https://techcrunch.com/2025/03/05/openai-reportedly-plans-to-charge-up-to-20000-a-month-for-specialized-ai-agents/
[1] Quote from the lyrics “The Future’s So Bright, I Gotta Wear Shades” from Timbuk 3, https://www.imdb.com/title/tt6891542/trivia/
[2] This blog post is based on a keynote speech given at the international conference ‘Transform: The Future of Human Workforce’ on 27 March 2025.
[3] At the beginning of March 2025.
[4] See Palazzolo, S. & Weinberg, C. (2025).
[5] Wiggers (2025).
[6] See Edwards (2025).
[7] Harari (2024), p. XXII.
[8] See Harari (2024), p. XXII.
[9] E.g. Lim (2024).
[10] Ferguson (2025).
[11] Foreman (2024).
[12] See Pelley (2024).
[13] See Douglas Heaven (2023).
[14] Ginnis (2025).
[15] Ginnis (2025).
[16] The following account is based on Searle (1980) and Searle (1984).
[17] Jackon (1986).
[18] For a more elaborate discussion, see van Biezen (2016) and van Biezen (2022).
[19] See e.g. Wang (2023), Baron (2025), Renard (2024).
[20] Quote from the motion picture 2001: A Space Oddysey (1968) from Stanley Kubrick, https://www.imdb.com/title/tt0062622/quotes/
[21] Hinton in Saso (2025).
[24] See Leibe (2025).
[25] See Leibe (2025) for further references.
[26] Leibe (2025).
[29] See Anderson & Rainie (2023).
[30] See also van Biezen (2024) and Molek, N., Pulinx, R. & van Biezen, A. (eds.)(2024).
