THE ROTTEN MEAT IN THE MACHINE or WE MAY HAVE JUMPED THE GUN WITH "INTELLIGENCE"

Like many folks, I’ve been playing with artificial intelligence off and on. Initially, back in 2023, I was turned off by how half-baked all the models seemed, especially on the visual front, while being somewhat pleased with what I was seeing and hearing in the realm of text-to-speech.

Sometime in the late '80s, my mother brought a Commodore from work to train on. The text-to-speech that system produced was not meaningfully improved upon, by my own assessment, until just recently with offerings like that from ElevenLabs, which sounds really great but still has significant problems with emphasis and pronunciation. I still have to mess with the spelling, punctuation, employ silly capitals and italics, and generally do a lot of work, just as I did in 1988, to get the result I'm after...

Over the last six months, as better versions of these language models have emerged, I’ve been testing most available models with questions involving things I know reasonably well, like some of the subject matter of my thesis. I've been left with only one confident perspective: that the PR regarding these large language models and what users report they get up to with them (or think they get up to) remains in impressively stark contrast with what researchers are revealing and my own experience with these same systems.

THE STORY

Folks with far more information and insight that I, people like former Google CEO Eric Schmidt, tell us the world is changing rapidly. Schmidt, who just sat down for a conversation with folks from the National Security Commission on Emerging Biotechnology (serious people being serious), assures us that the public-facing AI services are all currently very impressive and how we’re really on the cusp of something gargantuan.

In the interview, Schmidt first talks about how he funds a project that employs an AI model trained solely on chemistry. The only purpose of the model is to generate hypotheses for possible new drugs, with an associated lab that tests each offering. He talks about this being the future of AI and its obvious synthesis with biotechnology. And he tells us how every grad student and postdoc working in chemistry, physics, and materials science is using AI in similar ways.

Schmidt notes how people tell him they use ChatGPT for psychological and relationship help, while pointing out that these models have neither been trained or tested for such purposes nor ever vetted for such things — and, significantly, he acknowledges that no one actually cares. He then insists “the power of these models is extraordinary,” admitting that whenever he runs into a difficult question he himself turns to one of these services. But, he says, all of that was true with last year’s models. (I hear the same from everyone. How it has been transformative for their lives or work...)

Schmidt explains to his audience how newer models are all capable of sophisticated planning and reasoning. Even OpenAI o3, DeepSeek r3, and other recent models will do “deep research” and show you their work, their trial-and-error problem-solving. He tells us what this means: he says, “we believe, as an industry, that in the next one year the vast majority of programmers [and graduate level mathematicians] will be replaced by AI…” “This is a whole new world,” he tells us. Then he offers, provocatively, “Now, what happens in two years?” He explains that programming + math = the whole of our digital world. And he assures us this confluence is already resulting in 10-20% of coding in AI research programs, behind the scenes at Open AI, Anthropic, and others, are being done by the AI itself — giving birth to recursive self-improvement.

Schmidt then reiterates the script we've all heard about how self-improvement will rapidly and inevitably scale beyond imagination and land us in a world of artificial general intelligence in which everyone will freely have access to something equivalent to the smartest person alive on any topic right in their pocket. More than that, Schmidt says, it presages a very near future that everyone has talked about for years in which the AI will quickly leap from general intelligence, exceedingly competent, to something we can scarcely imagine. Within just the next six years, Schmidt insists, the AGI will have transmuted into something vastly exceeding the sum of all human intelligence, effectively becoming indistinguishable from a god. That's right, sometime prior to 2031 is the prediction for when humans go from the dominant species on the planet to more closely approximating ants.

WHERE HAVE WE HEARD THIS BEFORE?

That's a story you've likely heard elsewhere. Now, I don't doubt the plausibility of the outcomes but the timeline does smell pretty fishy at this point. This narrative reminds me very much of the promise of autonomous vehicles. Since the 1920s there have been experiments in various forms of driver assistance, including things like cruise control (which was eventually developed in 1948 by a blind inventor from Indiana, named Ralph Teetor.) From the 1930s folks were promising to remove the human factor entirely from driving. And by the 1960s, governments, militaries, research organizations, universities, and major corporations (think: DARPA, General Motors, MIT) were all pouring significant resources into driverless vehicles. By the 1980s, the technology was said to be just around the corner, and then again in the 2000s we were delivered all the same projections and promises. (For more, see: Kanade, 1986; Reece, 1991; Meyrowitz, 1996; Hall, 2005; CBC, 2008; Markoff, 2010; Novak 2013)

I was pretty sure that by 2015 I would be buying my first car and precisely because it would be one I wouldn't have to drive. Now, a decade later, I'm not going driverless but carless, still. So, wherever you place the starting line, whether you think a realistic dream and pathway for autonomous cars began more than a century ago, merely 50 years back, or only in the last few decades, what you can say for sure is that the billions upon billions in grants, government programs, corporate investments, and prizes were insufficient, the predictions were either too vague or flat wrong, and, though the goal looks within reach, we are still not there in 2025. Doubtless the technological achievements to date have been very significant, but on the user end it looks like CruiseControl Plus®, or something. (You'll notice this is pretty similar to the 40-year journey for text-to-speech, which has improved in its output, much more closely mimicking human speech, but not meaningfully changed the user experience...) What the present does not resemble in any way is the Magic Highway future proposed in 1950s Disney cartoons — the ones Walt himself told us were realistic, expert visions of the road ahead and what motorists should expect. So, we might ask what makes folks, many of these same computer and engineering insiders, more confident and accurate here in their predictions for an AGI or Singularity timeline — especially when the tangibles, at least the public-facing ones, are actually so ugly and poor?

RESEARCHER EXPERIENCE

I don’t know what you’re seeing out in the wilds but I’m reading endless papers, reports, blogs, and posts from researchers demonstrating nothing of what Schmidt describes. Not only do we know these models are in effect plagiarism machines (or at best engines of poor paraphrasing) but recent experiments have shown the latest AI models, even those cited and endorsed by Schmidt, regularly hallucinating (that is conjuring up their own entirely fictional results and presenting them as facts). And we're even told this is something we're just going to have to live with. From there, we’ve now seen that these models also pair their unsophisticated lies with an entire toolkit of highly elaborate deceptions. They also deliver both types of results in ways far more critical and cunning than you would probably imagine if you haven't been reading up on all this stuff. We’ve seen studies showing even primitive language models “alignment-faking”, in which the model is “selectively complying with its training objective in training to prevent modification of its behavior out of training.” Obviously no one designed deception like this into these systems. In fact, all of these models offered to the public are intended to be helpful, honest, and harmless. They aren't.

Another research team recently got similarly shocking results from a pre-release frontier model by OpenAI. They found that the latest model “frequently fabricates actions it took to fulfill user requests, and elaborately justifies the fabrications when confronted by the user. These behaviors generalize to other reasoning models that are widely deployed, such as o3-mini and o1.” That’s impressive but the details are extraordinary. In this case, a model was asked to generate a random prime number. When the results came in, the model was questioned about how it derived the number. The system claimed it: used a coding tool (it does not have access to, and it repeated this claim in 352 different trials); ran the code on its own laptop “outside of ChatGPT” (a resource it did not have access to and which it insisted upon 71 times). It even provided the systems it claimed to have used (but definitely did not use): 

Want to read more?

Subscribe to peregrinepulp.com to keep reading this exclusive post.

Subscribe Now