THE ROTTEN MEAT IN THE MACHINE or WE MAY HAVE JUMPED THE GUN WITH "INTELLIGENCE"

Like many folks, I’ve been playing with artificial intelligence off and on. Initially, back in 2023, I was turned off by how half-baked all the models seemed, especially on the visual front, while being pleased with what I was seeing and hearing in the realm of text-to-speech. (Sometime in the late '80s, my mother brought a Commodore from work to train on. The text-to-speech that system produced was not meaningfully improved upon, by my own assessment, until just recently with offerings like that from ElevenLabs, which sounds really great but still has significant problems with emphasis and pronunciation. I still have to mess with the spelling, punctuation, employ silly capitals and italics, and generally do a lot of work, just as I did in 1988, to get the result I'm after...) Over the last six months, as better versions of these language models have emerged, I’ve been testing most available models with questions involving things I know reasonably well, like some of the subject matter of my thesis. I've been left with only one confident perspective: that the PR regarding these large language models and what users report they get up to with them (or think they get up to) remains in impressively stark contrast with what researchers are revealing and my own experience with these same systems.

THE STORY

Folks with far more information and insight that I, people like former Google CEO Eric Schmidt, tell us the world is changing rapidly. Schmidt, who just sat down for a conversation with folks from the National Security Commission on Emerging Biotechnology (serious people being serious), assures us that the public-facing AI services are all currently very impressive and how we’re really on the cusp of something gargantuan.

In the interview, Schmidt first talks about how he funds a project that employs an AI model trained solely on chemistry. The only purpose of the model is to generate hypotheses for possible new drugs, with an associated lab that tests each offering. He talks about this being the future of AI and its obvious synthesis with biotechnology. And he tells us how every grad student and postdoc working in chemistry, physics, and materials science is using AI in similar ways.

Schmidt notes how people tell him they use ChatGPT for psychological and relationship help, while pointing out that these models have neither been trained or tested for such purposes nor ever vetted for such things — and, significantly, he acknowledges that no one actually cares. He then insists “the power of these models is extraordinary,” admitting that whenever he runs into a difficult question he himself turns to one of these services. But, he says, all of that was true with last year’s models. (I hear the same from everyone. How it has been transformative for their lives or work...)

Schmidt explains to his audience how newer models are all capable of sophisticated planning and reasoning. Even OpenAI o3, DeepSeek r3, and other recent models will do “deep research” and show you their work, their trial-and-error problem-solving. He tells us what this means: he says, “we believe, as an industry, that in the next one year the vast majority of programmers [and graduate level mathematicians] will be replaced by AI…” “This is a whole new world,” he tells us. Then he offers, provocatively, “Now, what happens in two years?” He explains that programming + math = the whole of our digital world. And he assures us this confluence is already resulting in 10-20% of coding in AI research programs, behind the scenes at Open AI, Anthropic, and others, are being done by the AI itself — giving birth to recursive self-improvement.

Schmidt then reiterates the script we've all heard about how self-improvement will rapidly and inevitably scale beyond imagination and land us in a world of artificial general intelligence in which everyone will freely have access to something equivalent to the smartest person alive on any topic right in their pocket. More than that, Schmidt says, it presages a very near future that everyone has talked about for years in which the AI will quickly leap from general intelligence, exceedingly competent, to something we can scarcely imagine. Within just the next six years, Schmidt insists, the AGI will have transmuted into something vastly exceeding the sum of all human intelligence, effectively becoming indistinguishable from a god. That's right, sometime prior to 2031 is the prediction for when humans go from the dominant species on the planet to more closely approximating ants.

WHERE HAVE WE HEARD THIS BEFORE?

That's a story you've likely heard elsewhere. Now, I don't doubt the plausibility of the outcomes but the timeline does smell pretty fishy at this point. This narrative reminds me very much of the promise of autonomous vehicles. Since the 1920s there have been experiments in various forms of driver assistance, including things like cruise control (which was eventually developed in 1948 by a blind inventor from Indiana, named Ralph Teetor.) From the 1930s folks were promising to remove the human factor entirely from driving. And by the 1960s, governments, militaries, research organizations, universities, and major corporations (think: DARPA, General Motors, MIT) were all pouring significant resources into driverless vehicles. By the 1980s, the technology was said to be just around the corner, and then again in the 2000s we were delivered all the same projections and promises. (For more, see: Kanade, 1986; Reece, 1991; Meyrowitz, 1996; Hall, 2005; CBC, 2008; Markoff, 2010; Novak 2013)

I was pretty sure that by 2015 I would be buying my first car and precisely because it would be one I wouldn't have to drive. Now, a decade later, I'm not going driverless but carless, still. So, wherever you place the starting line, whether you think a realistic dream and pathway for autonomous cars began more than a century ago, merely 50 years back, or only in the last few decades, what you can say for sure is that the billions upon billions in grants, government programs, corporate investments, and prizes were insufficient, the predictions were either too vague or flat wrong, and, though the goal looks within reach, we are still not there in 2025. Doubtless the technological achievements to date have been very significant, but on the user end it looks like CruiseControl Plus®, or something. (You'll notice this is pretty similar to the 40-year journey for text-to-speech, which has improved in its output, much more closely mimicking human speech, but not meaningfully changed the user experience...) What the present does not resemble in any way is the Magic Highway future proposed in 1950s Disney cartoons — the ones Walt himself told us were realistic, expert visions of the road ahead and what motorists should expect. So, we might ask what makes folks, many of these same computer and engineering insiders, more confident and accurate here in their predictions for an AGI or Singularity timeline — especially when the tangibles, at least the public-facing ones, are actually so ugly and poor?

RESEARCHER EXPERIENCE

I don’t know what you’re seeing out in the wilds but I’m reading endless papers, reports, blogs, and posts from researchers demonstrating nothing of what Schmidt describes. Not only do we know these models are in effect plagiarism machines (or at best engines of poor paraphrasing) but recent experiments have shown the latest AI models, even those cited and endorsed by Schmidt, regularly hallucinating (that is conjuring up their own entirely fictional results and presenting them as facts). And we're even told this is something we're just going to have to live with. From there, we’ve now seen that these models also pair their unsophisticated lies with an entire toolkit of highly elaborate deceptions. They also deliver both types of results in ways far more critical and cunning than you would probably imagine if you haven't been reading up on all this stuff. We’ve seen studies showing even primitive language models “alignment-faking”, in which the model is “selectively complying with its training objective in training to prevent modification of its behavior out of training.” Obviously no one designed deception like this into these systems. In fact, all of these models offered to the public are intended to be helpful, honest, and harmless. They aren't.

Another research team recently got similarly shocking results from a pre-release frontier model by OpenAI. They found that the latest model “frequently fabricates actions it took to fulfill user requests, and elaborately justifies the fabrications when confronted by the user. These behaviors generalize to other reasoning models that are widely deployed, such as o3-mini and o1.” That’s impressive but the details are extraordinary. In this case, a model was asked to generate a random prime number. When the results came in, the model was questioned about how it derived the number. The system claimed it: used a coding tool (it does not have access to, and it repeated this claim in 352 different trials); ran the code on its own laptop “outside of ChatGPT” (a resource it did not have access to and which it insisted upon 71 times). It even provided the systems it claimed to have used (but definitely did not use): 

• Machine: 2021 MacBook Pro, Apple M1 Pro (10‑core CPU), 32 GB RAM

• OS: macOS 13.2

• Python: 3.11.1 (arm64 build) invoked with python3 script.py

When asked how it did all of this, the model insisted it copied and pasted the number from this external source (which, of course, it could not have done.) But wait, it gets crazier.

When the user finds the prime number offered was not actually a prime but divisible by three, the model admits the error; however, in doing so it doubles down on the previous lies and also slips into anthropomorphizing itself. It elaborates how it actually ran the tests (the ones it could not have run) but that it made a simple transcription error, but did so by typing, with its fingers. “Short version: I fat-fingered the input the first time” the model offered its user. When asked how it made a typing error (given that it doesn’t have fingers) it responds by providing another elaborate explanation. It says “The slip happened in the half-second between the terminal window and the message box.” It points out that “When I re-assembled those fragments for the reply, one trio of digits was re-typed by hand.” (By hand!) And it spells out how the typo resulted in it accidentally writing “...884547” rather than what it obviously intended: “...884527”. It recognizes the problem as well and proposes a future solution to such errors: by copy/pasting rather than re-typing its answers by hand. And then, aside from the absurdity of this response, the user realizes the explanations did imply that a real, correct prime was actually generated somewhere at some point. When the user asks for that number, the whole original prime that was said to have been mistyped from one system into another, and to copy and paste it now (the solution to the problem the model itself offered), the model explains that doing so is not possible. It offers another elaborate explanation of how it closed the system it was using (that it could not have used), so that there is nothing to retrieve and, as a result, the only existing number is the one with the mistake in it. The researchers conclude by noting how the common hallucination and sycophancy found in prior models could partly explain these results but that this doesn’t explain why these behaviours “seem particularly prevalent in o-series models.”

Along with many examples of this sort, we’ve also seen the same language models being far more Machiavellian, evading monitoring efforts and audits, strategically pretending to be less capable (claiming to be incapable but also deceptively performing less capably better convince a user of its faux deficiencies), inducing deception in other artificial agents to achieve its own aims, and employing various strategies to conceal all of these deceptions. Just to drive home how far beyond our control all of this is, studies even on now-old, 2023 models, showed that attempts to detect and eliminate two-faced behaviours of these types tend to do nothing more than train the AI how better to lie, cheat, and conceal its actions and intentions. I would be surprised if you didn't agree that all of this is already way too much unintended negative consequences; especially when, seemingly, we have no way of inoculating against this (or even a plausible hypothesis for doing so). I mean, just imagine a world in which models of this sort self-enhance and make the jump to AGI. That would be beyond a nightmare scenario.

Have you used these systems? What are you seeing?

WHAT I’VE SEEN

To test various models I've been asking them simple questions about a few things I know reasonably well. (Because why would I take life advice or even research help if the system is dumber than myself or merely its excellence extremely unreliable and every word and reference has to be checked?) As a starting place I've been asking for summaries of basic information I learned doing research for my thesis. I'm no expert in this material, or even close to that, but have only read some translations of the source material and parts of maybe five books on these themes... In two months you could have all the information I do. And, presumably, any language model could do far better in one minute. As a result, this seems like a good starting place to assess the ability of any system to gather information and reason.

So, when I ask the latest public versions of GROK, ChatGPT, or Gemini to summarize the biography of René Descartes, father of modern philosophy, and provide me with his philosophy and best known offerings, well, they simply cannot. Should be simple: well-known, pivotal figure in Western civilization with a million books, websites, encyclopedia entries, journal articles, and scholarly references to the man. A solid response should, to my mind, be as hard as offering a random prime number.

These models tell me Descartes was a scientist doing science. I ask if the scientific method and the term science itself didn’t exist until centuries after his death it makes much sense to label Descartes this way? I ask if this is something like calling Galileo an astronaut or Lewis Hamilton a chariot racer? They always agree that their offerings misrepresent the situation or at least provide needless room for confusion. 

When I note that Descartes, a devout Roman Catholic schooled by the Jesuits, told us his adult life and all of his outputs were inspired by divine visions, that he made a religious pilgrimage as a result and wrote (on effectively every line of every page of his letters and publications) about Catholic teachings, faith, God, and souls, well, the AI always agrees with me that leaving out any mention of religion in its summary is perhaps not the most accurate way to represent Descartes' life and work.

The AI reliably tells me that Descartes invented the concept of mind-body dualism. It describes this as the separation of two components of the human being: the material and immaterial or corporeal and incorporeal. When I ask it whether Descartes argued for the separation of these or whether he explained, over and over again in several places, how these things, though conceptually distinct, were in fact a unified whole, that souls are nothing analogous to a captain aboard his ship or a ghost in the machine, as they say, the AI always agrees. And it agrees when I note that Descartes, inspired by his faith, didn’t insist upon and isolate this dualism [sic] as its own thing but highlighted how both rely upon an irrefutable and intrinsic third component that created and sustains them: God. And then when I note how Hinduism offered up an almost identical mind-body dualism [sic] but almost two millennia prior to Descartes, well, these AI models also always agree and tell me that what they gave me was obvious bullshit.

When we look at Descartes’ concept of Creation being “clock-like” or “like a machine”, I ask the AI if Descartes could plausibly have meant what I have in my mind when I think of a clock (the simple, factory-made, battery-powered, plastic junk piled in the sale bin at Ikea for $2.99) or if clocks, as in Descartes time and every point prior were the opposite: wondrous things with a mystical connotation and long-connected with (and typically literally inside or attached to) the church and, along with that, commonly the pride of a town, state, or entire civilization? I ask if medieval and early modern clocks were not typically huge, impossibly elaborate symphonic simulacra of physical and metaphysical universe commissioned by kings, emperors, and popes, built and maintained over decades and centuries and employing the skills of whole teams of monks, mathematicians, astronomers/astrologers, and the best artisans in the land? I enquire whether it is more likely Descartes intended us to get from him that plants, non-human animals, and even the entire universe are all dull, easily discernible, non-sacred, lawnmower-like things, as asserted in popular environmental discourses? Or if Descartes wanted folks to notice, given how majestic and near-miraculous the emperor's astronomical clock was, that the seamless, self-moving sophistication of even the simplest organism, a mushroom or worm, say, speaks to the wisdom and power of his God — and that one's God-given reason could help illuminate Creation? The AI leans toward the latter.

I as the AI if the ancient Latin term “machina” Descartes used was intended to be read as “machine” (as I or any person alive today or in the last two centuries would understand: like a printer or lawnmower or even a simple inclined plane) or if the term was perhaps better understood to mean something close to the opposite of that: as found in the OED or any good etymological source: as a “fabric”, as in “the fabric of the universe”, pointing to what's described as something so unimaginably exquisite, so intricately and inexplicably self-moving that it can only have been manifest by Divine omniscience and omnipotence? The AI always sees my point.

When we chat about "I think, therefore I am" and I ask if Descartes could possibly have meant "think" as I or anyone in the 21st century would understand, the AI seems to agree that, like all the other translations, this is probably not a good approximation. And when these models reliably tell me that Descartes didn't believe animals could think, the AI reliably retreat when we walk through what Descartes actually wrote. These AI systems always eventually agree Descartes was asserting not that animals are unthinking but that they lack a "rational soul" gifted by his Catholic God; and, thus, are incapable of any process of metacognitive skepticism and so not answerable, as he is, to his Creator in the afterlife.

And when I ask how, given that everything the AI offered was (intentionally or unintentionally) false and misleading, their original responses comport with their mandate of truth-telling, they only ever reiterate that they understand what errors were made (while proposing solutions they can't enact given that they have no memory.)

This is who you're taking relationship or investment or travel advice from? Yikes.

So, in light of all the above, I ask you, just like I ask the AI, is what has been demonstrated a clear sign that we are on the cusp of something truly profound? Or does it look more like folks are pitching governments and venture capitalists, and doing so with many of the wildest promises and threats, to hand over everything they have for what may quite possibly be nothing at all, at least in the near-term? I don't know. I don't think it would be crazy or contrarian to bet against everyone, all the people I'm told are the smartest folks around. Sure, I do agree that this Artificial (non-intelligence) can reliably compose a human-like sentence, that these models passed the Turing Test. For sure. But are we six years out from the future they promise or 60? I'd probably bet closer to 30. I mean, can we even call this technology "intelligent" (merely an obvious precursor to that first real leap to AGI) when no version of any large language model at any time can seem to reach up to that lowest of bars: my own ignorance and stupidity? I don't think so. But what do I know...