Series Order
Part 1 | Is AI Conscious? The Risks of Technology Turning Human Nature into Data
By Layne Hartsell
We live in an era of unprecedented technological scale-up and convergence — sometimes called NBIC (nano-bio-info-cogno). Alongside a golden age of biotechnology that has brought healthspan extension, mRNA technology, CRISPR, personalized medicine, and biohacking, artificial intelligence has surged past every other technology to the center of public attention, taken up almost immediately for routine mental tasks, art and play, among countless other uses. Biotechnology, by comparison, remains esoteric and laboratory-bound. More importantly, AI is even perceived as a "cognitive" technology: something similar to us, something conscious. Almost as soon as AI arrived on the open Internet in late 2022, it produced new threats, crimes, and tragedies, while magnifying older ones — war among them — to the point that there is now consensus across executives, governments of every political stripe, academics, and the public that AI needs technical and social restraints immediately, restraints that should have existed a couple of decades ago.
AI ethics has been rejuvenated. Key figures in the field have even called for the technology to be shut down, and have spoken openly about its risks. Imagine an airline executive publicly discussing on mainstream media that the airline's own planes are already crashing — and investors continuing to pile into airline stock anyway, inflating the largest airline bubble in history. Such a hypothetical is tiny next to the real threats actual executives and scientists are discussing: the destruction of the economy, the social fabric, or even a bioweapon capable of wiping out humanity. These are not idle musings.
While the need for regulation is extremely serious, we've also seen a further detachment from reality, driven by belief — specifically, belief in machine sentience and consciousness. It is this belief I want to examine because a real issue essential to the discussion over what AI is and may become is a more basic consideration: what do we mean when we call a machine intelligent or conscious and what might we, and the machines, become? The widespread attribution of consciousness to AI is central to a broader technification of human nature, which is an effort that reduces human beings to computational processes, with profound ethical, mental, social, political, and spiritual consequences. The most obvious place to start is with the claims themselves: AI's intelligence, sentience, and consciousness are extraordinary claims requiring rigorous substantiation — a standard many forget the moment an AI generates the text they want to hear.
A famous materialist-scientist, in particular, recently has spoken fondly of this software, which leads me to think that treating machines as conscious would be consistent with and, to him, confirm his materialist theories of mind and humanity; though he would otherwise demand extraordinary evidence for extraordinary claims. Normally, the substantiation would be the minimum to require for public discussion on AI consciousness and intelligence. However, the basic requirement of substantiation has largely been ignored, which is why I think we need a public discussion of erroneous belief and hype that is producing real social outcomes. Put simply, the claim is that AI is conscious, intelligent, and essentially the same as us, so the real issue is willingness to abandon basic evidentiary standards in favor of indulging that belief — a new belief system insidious to both society and knowledge itself.
AI is not human according to how we have understood ourselves historically, including subjective experience, selfhood, intentionality, embodiment, relationality, and so on. Empirically, what we observe is that AI systems generate text through token prediction in ways that mimic aspects of human language. A demonstrated capability is evidence of that capability; it is not automatically evidence of a constellated, unobserved subjective state. This distinction is consistent with Hilary Putnam's critique of inferring a full mental capacity from a particular behavioral or functional attribute at the time of the Cognitive Revolution last century. Empiricism doesn't establish the phenomenon of the mind or consciousness though we do a version of this to establish a theory of mind, or the reasoning that others have a mind as we do; hence, the relative efficacy of the Turing Test, and the fact that many accept that consciousness and mind are the realm of a constellation of attributes based on embodied reality or experience.
The techno-social discussion around this technology is better understood as an attempt at the technification of human nature: an effort to turn humanity and consciousness into engineering problems, with the effect of reducing people to something like automatons. Social media, paired with AI, had already begun this process, which was poorly understood by the public, but understood very well by those who stood to gain from the hype, in the market through the quantification of human traits. The significance of this debate becomes clearer when we look at what happens when human beings themselves are understood as probability indicators for prediction markets. Harvard emeritus professor, Shoshana Zuboff, has documented and warned about people being "controlled remotely," without public or individual awareness, well before AI was openly available on the internet. The mining of desire, the directing of attention, and the prediction of behavior were pioneered a decade ago by Google and the infamous example of Pokémon Go for location-based behavioral data extraction and commercial surveillance (economies of action).
Then, there is also the manufacturing of political opinion pioneered by Cambridge Analytica and Facebook's behavior modification experiments all around the same period of time. AI dramatically increases the scale and depth with which human intentions can be inferred, modeled, and acted upon. The first phase of the digital and knowledge economy competed for our attention. The emerging AI economy can compete for—and potentially shape—our intentions. The upcoming U.S. elections will be core studies for early use of AI in this way.
Zuboff argued in the last decade that these tactics have scaled into what she calls surveillance capitalism. Today, the manufacturing of intention will become routine as software advances, computing power increases, and surveillance reaches into ever more intimate mental and physical spaces via AI. Another consideration is what remains for a human being to sell in the market, as discussed by a prominent investor recently on a top podcast (19mn followers): people only have themselves to sell. He went on, “your only asset is yourself,” and each has to be good at selling: “how do you sell yourself.” Later in the podcast, he discussed that once the body and the mind are replaced, “what is it that you have to sell?”
What seems to come out of this position of market enthusiasm is that people would have to be at least kept alive such that emotions and intuitions can be sold, and intentions and attention can be extracted, as most all other jobs will be taken. His is not far from reality — it is, more or less, a realization of the Matrix. This investor says that he loves capitalism, but there is a reality, and then went on to muse about a scenario where a human would be a masseuse because a robot probably is not very good at it. Are we back to mechanical work of the muscles again, but this time with “affection” as what we sell? 4.7mn views were recorded of this segment over the first two weeks.
In a separate case, Zuboff describes a scientist saying that they — the scientists — write the music that people dance to, selling data to the highest bidder and turning people into what she calls "chattel for commerce." Attention mattered in the past, but it is intention that companies and governments want most today. Human futures, mined from intention, are sold on the market. Social media carved out a solipsistic black hole of attention; AI arrived to manufacture the intention that directs concentrated attention.
Consciousness and Computation
Before going further, let me distinguish several terms that are often being collapsed into one another. To give some structure to the terms, I'll use consciousness to mean the background of awareness, or "knowing-being," and awareness to mean directed attention that ordinarily requires effort. Consciousness is a unified continuum. The Self discerns inner and outer states and can direct the relational faculty, within which intention is fundamental to how we make decisions. The Self, clearly, is a relational dynamic rather than a fixed entity.
Intention directs attention, and relationality flows from it, alongside creativity, into discovery and innovation — creativity is simply applied imagination. This chain, attention following intention, is what we see as directly relational, and why we seek attention from others. We believe their intent based on our own subjective experience and language, and other empirical cues. Peter Hershock of the East-West Center at the University of Hawaii observes that intention and attention are the most basic characteristics we offer in a relationship. Consciousness, intention, attention, and language together form the theory of mind.
I argue here that people's relational and intentional capacity is being hijacked — first through attention-mining that had already isolated people as consumers, and through political echo chambers shaped by commercial and political ideology. This social breakdown, dystopia, well documented before 2022, was not unexpected. It was foreseen as early as 2007, when the iPhone hit the market — an authoritarian technics masked as democratization — making the Macintosh 1984 commercial, in 1984, all too prescient. Societies were not equipped for this technological onslaught, nor was any regulatory structure put in place for what amounts to another "cognitive revolution."
The smartphone became what is sometimes called a “glowing Skinner box.” To play on the popularization of chemical messengers, we could call it the Magical Screen of Dopamine, a highly sophisticated surveillance and behavioral-modification machine — the core enabler of the erosion of democracy, relationality, and subsequently individuality. If intention directs attention, then technologies capable of influencing intention are operating closer to the core of human agency than technologies that merely capture information. When intention-attention capacity is assumed or commandeered, freedom is curtailed and relational capacity atrophies.
A few more definitions can help at this point. Technology is applied knowledge and technics is the manner in which we apply it, carrying ethical, moral, and ecological dimensions. Democratic technics is the relational elaboration of applied knowledge in society, whereas authoritarian technics is top-down application of technology that is directed by a few and denied to many.
Intelligence is the functional ability to recognize patterns, solve problems, and engage in activities beyond and inclusive of simple impulse or reaction — beyond input and output. Computation, by contrast, is input/output rules and sequences of the kind seen in today's vast neural networks. There is a difference between performance and ontology.
For AI ethics, consciousness and intelligence belong to embodiment — hence Alan Turing's 1950 inquiry about whether machines can think, and the response from Edsger Dijkstra a few decades later - do we say submarines swim. Today, techno-utopians — as Anthropic's Constitutional AI Guidelines — claim that AIs are supposed to "embody the best of humanity." This is a dangerous conflation: AI is treated as intelligent, conscious, and embodied all at once. While AI computation can certainly facilitate scientific inquiry, whether by crunching data or predicting protein structures for use in medicine — already tremendous achievements — such technical feats are not evidence of actual humanity.
What I've described here is the beginning of the technification of human nature: the reduction of consciousness, intelligence, intention, and attention to functions that can be measured, modeled, predicted, and ultimately engineered. The danger is not simply that machines may become more like us, but that we may increasingly, erroneously, believe ourselves to be machines and that they are us. Once intention can be treated as something to be probabilistically, and then recursively, inferred and attention as something to be directed artificially, human agency itself becomes an object of technological management. Social media began by mining attention; AI promises to reach deeper, toward the modeling and manufacture of intention. This is how people become, quite literally, remotely controlled—not because a machine has acquired a mind, but because machines are believed to be conscious and human minds have become objects of computation.
The attempt to create the belief that AI is conscious is core to the ideology of an intellectual movement of techno-utopianism that makes consciousness itself computational—and once human beings are understood as computational systems, the technological management of human intention becomes both conceivable and commercially exploitable.
[Part 2 of this series, on AI and Society, begins with AI as a Cultural Mirror.]
This essay is adapted from talks delivered at the International Conference on Nation Building, AI and Blurred Morals, Bangkok, Thailand, and the 16th Social Ethics Society Conference, Davao, Philippines.
About the Author
Layne Hartsell, USA (雷恩∙哈特塞尔 / 마이클레인핫셀) — 3E: Energy, Economy, Environment. Research fellow, Center for Science, Technology, and Society, Department of Philosophy, Chulalongkorn University, Bangkok. Research fellow, 3E and Bioregional Letters, North Valley College, Mindanao, Philippines, and the Asia Institute, Tokyo. Past affiliations: Assistant professor, Convergence Studies, Sookmyung Women's University, and the Research Institute for Asian Women / Asia-Pacific Women's Information Network Center, Seoul; Research professor, Sungkyunkwan University and the Advanced Institute of Nanotechnology, Seoul/Suwon; Adjunct lecturer, Mahidol University, Siriraj Medical Center, Department of Molecular Biology and Bioinformatics, Bangkok; Researcher, University of Virginia College of Medicine, Charlottesville.
