Series Order
Part 1 | Is AI Conscious? The Risks of Technology Turning Human Nature into Data
Part 2 | Is AI Conscious? Part II | AI as a Cultural Mirror
By Layne Hartsell
What we see in AI is, in a real sense, us. AI reflects the culture that created it based on enormous amounts of training data. In fact, software, or machines, called “AI” are more precisely described as token-generating machines. Tokens are units of input — letters, punctuation, chunks of data — generated according to probability patterns. Trained on data at a large scale, this software produces output mimicking language as text.
This is not language. Biolinguistics holds that humans have an innate capacity for language, which is generative and creative. We also understand meaning. Basically, human language is not produced by probability matching from external inputs, and neither are intention and consciousness matters of computation. Token-generating machines lack innate faculties — and interestingly enough, the discussion today seems to reverse the mid-20th-century Chomsky–Skinner debate as alluded to above in the first essay of this series. At that time, humans were cast as blank slates; today, machines are treated as having innate faculties. It is an interesting convolution of cognitive and computer science.
Linguist Emily Bender calls AI systems “stochastic parrots” and, importantly, has cataloged harms that have already been done by AI: discriminatory outcomes in housing and criminal justice; denial of healthcare; hate speech along with misinformation and disinformation; deepfake pornography or false compromising photos and stories; and wage theft via algorithmic management. Bender, Alex Hanna and others compare AI to the children’s book Everything in the Whole Wide World, where Grover, a “muppet” from Sesame Street, visits a museum whose categories seem exhaustive but are arbitrary until he reaches the door to “Everything Else” — the outside world. Their point: AI benchmarks claim to measure general “understanding” but are mostly arbitrary — much like benchmarks claiming to measure general “understanding.” Opening out to the actual world is a different matter, that of fullness or integrality.
Artificial intelligence largely reflects the Western culture that produced it — an Echo of Narcissus, with recursion as both a technical and social danger. Technically, problems show up as “model collapse,” degradation of data and runaway feedback loops. Then there are cultural issues such as amplification of the general culture or subcultures. Add the erroneous belief that this software is conscious, and misinformation and disinformation are magnified rather than corrected — making the technology culturally insidious and thus a truly toxic mix, hence dystopia. We look into the mirror, the reflection comes back, more data is generated from that reflection, and the loop deepens into what I’d call “aperspectival psychosis”: a society-wide loss of the ability to distinguish and discern. This is a technological dystopia organized around market mechanics and profit with social isolation, rather than human relationality.
This dystopia shows up with particular force in the technical imposition of symptoms of autism, ADHD and psychosis — what some actually call “brain rot” — because the mind, or intention specifically, is targeted by highly sophisticated digital systems of companies and government. Around twenty years ago, marketers tried to use fMRI to peer directly into brains in order to influence people in the market and society; today, that goal is achieved by hijacking attention and commandeering intention. German-American philosopher Hannah Arendt (1906–1975) warned that when people lack the skills to distinguish accurate information from falsehood, they become easy subjects for totalitarianism. Unfortunately, we are orders of magnitude further into technological dystopia than when Arendt was writing, as algorithms for behavioral modification have become a business model monetizing attention and intention for near “pure profit,” to quote one of the major automotive companies. Yes, they realized that they could collect data from vehicles they sell.
What Computation Is — and Is Not
Computation today means multiple parallel input/output processes, highly complex neural networks, though they are still input/output processes. AI is a Cartesian Theatre of discrete, controllable, algorithmic computation trained on enormous datasets, producing sophisticated token-matching — what Lynn Andrea Stein at MIT calls “a sophisticated symbol-processing abacus.” However, there is no real scientific comparison between AI and embodiment and consciousness, as Nature is clearly “outrageously different,” as quantum physicist Roger Penrose points out. There seems to be no way to reduce consciousness to computation. The issue is that the conflation is dangerous: the belief that AI is conscious permits reduction and objectification in the first place.
Two beliefs circulate consistently: that AI understands language like a person, and that AI explains the human mind or is conscious. Neither is established. John Searle’s Chinese Room thought experiment remains useful: a basic description is that I could be in a room, receive a Chinese character through one door, match it to a character within the room, and then send it out the other door. While I might get the matching correct without error, I still would not understand Chinese, which could be easily tested. There is mimicked language from outside (matching), yet undeniable failure of understanding. Basically, with the machines today, we see data and output, but no theory of mind, that is, if we refer to mind as rooted in embodied contact with the natural world, developed over millennia through culture and the land. Consciousness and intelligence are embodied facts; for humans, they are further embedded in culture, history, emotion, various faculties, and then intention and attention.
Computronium and the Substrate Swap
The ultimate expression of computational reductionism is “computronium,” computer scientists Norman Margolus and Tommaso Toffoli’s term for a hypothetical substance optimized for maximum computational density. It is in the same area as the Matryoshka brain or Dyson swarm (cloud). Yet computronium fails even on materialist grounds. Consider the “substrate swap” thought experiment: scientists replace the neurons in a conscious brain, one by one, with computronium or optimized computational material — let’s call it a crystalline material — performing identical input/output functions, and ask, after each swap, whether the person is aware. If subjective experience vanishes while nothing changes functionally (computation), consciousness is expected to depend on a biological substrate, not computation.
If subjectivity persists, computation might suffice for consciousness. In the latter case, it would presumably be possible to upload into a computer. I have been highly skeptical that consciousness supervenes on computation, and have thought the case does not pass physicality. My view is, as mentioned, that consciousness and mind depend on embodiment and experience or a kind of pragmatic development.
Also, like consciousness, biology is not computational. For example, protein production operates through distributed cellular processes that do not fit into input/output models as was once thought. It is perfectly fine to abstract from nature to produce models, but the gross error is to believe the model. More important than the thought experiments is current anesthesiology research by Stuart Hameroff and Roger Penrose on microtubules, where consciousness is “turned on and off” when the patient goes to sleep. The reasoning in this physical theory of consciousness is that, while consciousness appears not to be computational, it seems to have something to do with quantum processes in microtubules — a biological theory of consciousness.
This “objective reduction” is speculated to be what constitutes conscious events and brings physics, chemistry and biology together. If accomplished, it would be quite amazing and important. While all of the above are interesting and should be studied, it is clear that what looks like a deep mystery to us seems to pose no difficulty for nature itself, which clearly “chooses” in various cases, such as between wave and particle duality, and seems to choose or discern with consciousness in other places — Penrose calls it “proto-consciousness.” If, somehow, it turns out that organic material is absolutely necessary, then the organoids (small brains) that are being produced might be far cheaper and easier to produce, with better computational performance, than silicon-based computation or other computronium.
Perhaps the substrate or brain and computronium are the wrong direction, though they might yield useful information. Perhaps consciousness is an attribute of Nature and we see varying degrees or aspects due to embodiment and experience, dependent on the dependent origination of embodiment and consciousness as in Buddhist philosophy; if so, this would indicate that actual understanding would be beyond our faculties. I am of this thinking.
Mathemythics and the Imitation Game
The Turing Test is a provisional, operational test for determining the ability of a computer to produce textual speech — measuring mimicry, not emotion, imagination, foresight or transcendence. It measures nothing in the realm of embodiment. While Turing thought successfully passing his test would be a milestone, or a canary in the coal mine — a point at which, if achieved, we should give the matter more social attention — he did not think computers were conscious beings. Later, Dutch mathematician Edsger Dijkstra, asking rhetorically, “Do we say submarines swim?”, thought the question uninteresting. Instead of Turing or Dijkstra, many want a mathemythics in AI, something akin to climate economist Steve Keen’s critique of mathematical formalism used to manufacture myths in economics. The patterns are applied widely at times.
What’s happening is a milieu of belief in which computer scientists and CEOs treat the reduction of consciousness to computation as already accomplished — taken up by credentialed people who state as settled fact what remains, according to physicists themselves, a live noncomputational question that apparently involves quantum mechanics. Sycophancy and sensation seem a better explanation for the current AI cultural phenomenon than physics and mathematics.
Historically, modernity’s default was that non-humans lack inner life; with AI, that default flipped. AI is assumed to understand until proven otherwise, a self-sealing loop.
Embodiment and the Social Fabric
British artist and writer John Ruskin observed that human consciousness arose through contact with nature; likewise, American wilderness guide Sigurd Olson recognized, after innumerable canoe trips into the wilderness, that the human psyche is “attuned to woods and fields and waters.” Poetics may provide a bit of help when it comes to the big questions, where the belief that AI possesses consciousness is a categorical mistake, and a mistake magnified into social consequences.
Relational instincts holding communities together — personhood, actual relationship, obligation, care — are threatened whenever mechanization enters our interactions.
When fluent text substitutes for evidence of inner life, criteria for personhood shift from intimacy and mutuality to superficial responsiveness and mindless interaction. However, relationships involve dynamic interaction between people whose interests conflict — that dynamism teaches patience, negotiation, new ideas and ways of being, and tolerance. A system optimized for mechanical operations with relationality stripped out has no such dynamism, and I suspect that skills and habits that have normally sustained reciprocity will atrophy. Therefore, while stupidity will spread, psychological distress would also ensue, since the Self can be defined as the discerning factor of relationality and, when systemically frustrated, will lead to psychological illness.
Researchers such as sociologist Jonathan Haidt are reporting on problems already concerning real relationships: inner life is attributed to the machine, which seems harmless and even “preferable to humans.” When widely believed, it creates structural erosion pushing society towards fragmentation, visible in the global pandemic of loneliness that was already an issue before AI due to social media and market society. The situation is also physically unhealthy: in a 2023 meta-analysis of more than two million people, Chinese epidemiologist Fan Wang found that social isolation and loneliness are each independently linked to higher mortality risk. Insidious and dystopia are appropriate terms to describe the conditions that are worsening.
Part 3 will follow with a discussion of the new technoutopianism and concluding comments.
About the Author
Layne Hartsell, USA (雷恩∙哈特塞尔 / 마이클레인핫셀) — 3E: Energy, Economy, Environment. Research fellow, Center for Science, Technology, and Society, Department of Philosophy, Chulalongkorn University, Bangkok. Research fellow, 3E and Bioregional Letters, North Valley College, Mindanao, Philippines, and the Asia Institute, Tokyo. Past affiliations: Assistant professor, Convergence Studies, Sookmyung Women's University, and the Research Institute for Asian Women / Asia-Pacific Women's Information Network Center, Seoul; Research professor, Sungkyunkwan University and the Advanced Institute of Nanotechnology, Seoul/Suwon; Adjunct lecturer, Mahidol University, Siriraj Medical Center, Department of Molecular Biology and Bioinformatics, Bangkok; Researcher, University of Virginia College of Medicine, Charlottesville.
