Jon Rappoport

Jun 20, 2025

∙ Paid

I’ve been saying I don’t know how it works, since I had my first conversation with ChatGPT. Of course, I don’t know how a can opener works, either.

But seriously, this idea that chatbots “train” on a “large language model,” and then answer user questions in perfect grammatical English, at lightning speed, and sometimes INVENT fictional answers, but can’t really invent because they can’t choose—the whole business doesn’t add up, as far as I’m concerned.

Turns out I’m not alone.

Axios: “The wildest, scariest, indisputable truth about AI’s large language models is that the companies building them don’t know exactly why or how they work.”

“The engineers know what they’re setting in motion, and what data sources they draw on. But the LLM’s [Large Language Model’s] size — the sheer inhuman number of variables in each [chatbot’s] choice of ‘best next word’ it makes—means even the experts can’t explain exactly why it chooses to say anything in particular.”

“As OpenAI’s researchers bluntly put it, ‘we have not yet developed human-understandable explanations for why the model generates particular outputs.’”

“Anthropic CEO Dario Amodei, in an essay in April called ‘The Urgency of Interpretability,’ warned: ‘People outside the field are often surprised and alarmed to learn that we do not understand how our own AI creations work. They are right to be concerned: this lack of understanding is essentially unprecedented in the history of technology.’ Amodei called this a serious risk to humanity—yet his company keeps boasting of more powerful models nearing superhuman capabilities.”

 “A new report by AI researchers, including former OpenAI employees, called ‘AI 2027,’ explains how the Great Unknown [the mystery of how AI works] could, in theory, turn catastrophic in less than two years.”

How do you like them apples?

“We’re unleashing it on the world, but we don’t know how it works. And, you know, it goes off on weird tangents…”

I had a long conversation with ChatGPT about how it “decides” which particular fictions to “invent” when it comes up with fictions.

It gave me a detailed answer (which I published in this podcast), but I still didn’t feel I got down to a bottom line explanation.

It seems like there’s a gap between its programmed guidelines and the specific answers it constructs. A hole, a moment when it does something unknown.

According to the whole chatbot model, a bot like ChatGPT decides what to answer to a question one word at a time. It chooses each word according to “statistical probability”—whatever that means. The tech engineers are saying that GPT picks EACH “next word” by consulting a truly boggling number of variables—so many that they don’t understand what it’s actually doing when it picks a given word to put in an answer to a human user’s question.

Think about that.

I’m trying to come up with a metaphor.

How about this:

 

This post is for paid subscribers