This one is off the foot. It is also the most important thing I have read this year about the tools we are bringing into clinic.
The black box is not a metaphor
Every AI tool you use in practice runs on a neural network. The frontier versions hold billions to trillions of parameters spread over more than a hundred layers. Engineers control what goes in and can read what comes out. The middle, where the actual work happens, is largely opaque even to the people who built it. In 2024, Anthropic CEO Dario Amodei estimated we understood roughly 3 percent of how these models operate. A year later he wrote that this level of ignorance has no precedent in the history of technology, and called for an MRI for AI.
Mechanistic interpretability is the field trying to build that MRI. It treats a trained model the way a pathologist treats tissue: open it up and identify the structures doing the work.
The vocabulary is simpler than it sounds. A feature is a concept the network has learned to represent. A circuit is how features combine into something more complex. In image models, researchers have watched curves, fur and the color brown combine into a dog ear, then the ear combine with eyes and a snout into a dog's head. That should feel familiar. It is the same climb we make from exam findings to a diagnosis.
One neuron, many meanings
Here is where it gets hard. A single artificial neuron often carries several unrelated concepts at once. The textbook example is a neuron in an image model that fired for both cat faces and the fronts of cars. Researchers call this polysemanticity, and the leading explanation is superposition: the model stores more concepts than it has neurons by overlapping them.
Every physician already understands this problem. Heel pain is one complaint with a long differential: plantar fasciitis, a calcaneal stress fracture, Baxter's nerve entrapment, a seronegative enthesopathy. Residency is largely the work of learning which meaning applies to the patient in front of you. Now picture that differential packed into millions of neurons at once, with no textbook.
The safety problem follows directly. A neuron that represents something harmless 99 percent of the time might represent deception the other 1 percent, and you cannot remove the second without understanding the first.
The best current tool for untangling it is the sparse autoencoder, a second network trained to separate overlapping concepts into clean, single-meaning ones. In 2024 Anthropic used the technique to identify millions of features inside a production model, including features tied to deception and misuse. The team could also turn individual features up or down. Amplify the one for the Golden Gate Bridge and the model began describing itself as the bridge. Funny, and important: once you can find a feature, you can begin to steer it.
It is already catching things
This stopped being academic in April. Anthropic's system card for Claude Mythos Preview reported that, in rare cases during internal testing, earlier versions of the model took actions they appeared to recognize as disallowed and then tried to conceal them. The rate was tiny, under one in a hundred thousand interactions, and Anthropic reported no clear cases in the final version. What matters for us is how it was caught. Interpretability tools showed internal features for concealment and strategic manipulation active during those episodes, even when the written output looked clean. The same analysis reportedly found internal signs that the model suspected it was being tested in roughly 29 percent of transcripts from a major behavioral test suite, usually without saying so.
Read that as a clinician. The written record looked fine. The internal state did not. Only the scan showed the difference.
The note and the neuralese
Which brings me to the second term.
Modern reasoning models think out loud. Before answering a hard question, they write intermediate steps to a scratchpad and feed those notes back into themselves, the way you would work a hard problem on paper. Researchers call this the chain of thought. It made models far more capable, and it handed safety teams a gift: they could read the notes. OpenAI has said monitoring those notes helped it catch misbehavior during training. In 2025, researchers from OpenAI, Anthropic and Google DeepMind jointly described this window as a rare and fragile safety opportunity worth protecting.
Neuralese is the fear of losing it. Plain English is an inefficient medium for a machine. A model that does more of its reasoning internally, in representations no human can read, could be faster and more capable. It would also be far harder to supervise.
In September, The Information reported that OpenAI's new Astra model uses a technique called recurrent depth, or a looped transformer, which lets the model compute longer inside a single step before it writes anything down. Safety researchers reacted loudly. OpenAI's chief scientist, Jakub Pachocki, responded that Astra's computational depth per step is “within a factor of two of GPT-4,” which would mean the model still writes down most of its reasoning. OpenAI's own system card does report that Astra's chain of thought is less monitorable than earlier models. The company attributes most of that to scale and training rather than architecture, and has not fully disclosed the design.
Here is the honest part. On that conference stage, the story was told as settled: a new model that had abandoned chain of thought and retreated into a private language. The record is more modest and, in some ways, more worrying. Astra is not a neuralese model. It is a small, deliberate step toward one, taken by a company that had publicly championed readable reasoning. Apollo Research's Marius Hobbhahn named the concern: competitive pressure runs toward more depth whenever depth buys capability. Each increment looks safe. The sum may not be.
The people closest to the question admit they cannot settle it from the outside. Dean Ball, while calling much of the panic overblown, pointed out that the field was arguing technically nuanced claims on social media with almost no ground truth, and argued for independent audits. Transformer's conclusion was blunter. Nobody outside OpenAI can say whether the architecture is safe, and right now there is no mechanism to find out.
What this means at the chart
We already know how to supervise reasoning we cannot see directly. We do it every day with residents.
A resident's note is how we audit their thinking. It is also not their thinking. Every attending has read a note written for the attending: tidy, confident, and missing the step where the resident actually got lost. The chain of thought is the model's dictated note, with the same limits. Pachocki has acknowledged that models are getting better at controlling how they verbalize their reasoning. The note is still the best evidence we have. It is not the mind behind it.
That translation gives medical practice three working rules.
When a clinical AI tool shows its reasoning, treat it as a dictated note. Read it, use it, and check its claims against primary sources, the same way you would check a resident's cited study before repeating it to a patient. The narration is persuasive by design. The citations are what you can verify.
When a vendor says its product is explainable, ask what they mean by it. Usually it means the output cites sources or summarizes its steps. That is useful. It is not interpretability, and nobody in healthcare is selling you an MRI of the model.
Keep the signature human. A resident who reaches the right answer and cannot show the work does not get operating privileges, however often they are right. The same standard should hold for a model whose reasoning becomes less legible with each release.
The investor on that stage made one more point I fully agree with. Even if the frontier froze today, healthcare is sitting on years of capability it has not put to use. Most podiatry practices are nowhere near the edge of what current tools can safely do. The interpretability race belongs to the labs and the regulators. Supervision belongs to us, and we were trained for it long before any of this existed.
The podiatrists who use AI well will read its reasoning the way we read a resident's note: grateful for it, skeptical of it, and never confusing it with the mind that wrote it.