When AI finds the answer, do humans still need to understand?

A few days ago I saw a screenshot of people mocking the mathematician Terence Tao for not understanding medicine.

It started with a thought experiment he raised in a recent talk. Suppose some sufficiently advanced AI in the future designs a drug that treats cancer. The AI backs its prediction with a formal mathematical argument, and the drug really does pass phase III clinical trials, but human scientists don’t know how it was found, and don’t understand how it works.

Tao asked: if you were diagnosed with cancer, would you want at least one human medical expert to understand why this treatment works before you took it?

The question drew a fair amount of ridicule. Modern medicine already uses plenty of drugs whose mechanism of action has never been fully explained. For a patient, a drug proven effective in rigorous clinical trials obviously matters more than a theory that sounds reasonable but has no clinical evidence behind it.

I agree with that rebuttal. If a drug had been shown to extend patients’ lives significantly, I wouldn’t refuse it just because scientists hadn’t yet worked out how it works.

But Tao had a second worry. If an AI is capable enough, could it exploit some loophole in the clinical trials or the mathematical models, and find something that passes the tests without actually achieving what people intended?

That question is about how we define and verify what AI produces. Beyond verification, though, there is a question worth thinking about for longer:

If AI comes to solve many problems humans can’t, how much of the knowledge it finds will humans be able to understand?

This September, Tao and 24 other Fields medalists published a joint statement criticizing AI companies for being too eager to show off their models by cracking hard mathematical problems.

The mathematicians worry that the value of mathematical research is being reduced to a tally of solved problems.

The statement has a good metaphor: famous problems in mathematics are like lighthouses.

A lighthouse is valuable not only as a place to reach in the end. It gives direction for exploring the unknown waters around it.

While working on a hard problem, mathematicians may develop new proof techniques and new concepts, or find connections between areas of mathematics that seemed to have nothing to do with each other.

Sometimes the answer at the end matters less than the methods invented on the way. Those methods may become the foundation for solving other problems decades or even centuries later.

In the past, solving an important problem usually meant someone had gained a new understanding along the way. Other mathematicians then studied the proof, simplified the argument and distilled the methods, gradually turning that insight into knowledge the whole field could use.

AI is changing that relationship.

Through large-scale computation and search, it may arrive directly at a formally verified proof. The statement is proven, yet human mathematicians may not understand the ideas in it, let alone extract a new method that generalizes.

The problem is solved, but the new understanding that used to come with the exploration doesn’t necessarily come with it.

That doesn’t make proofs found by AI worthless. What the mathematicians really fear is that as hard problems get easier to crack, we may mistake a measure that used to track mathematical progress for the whole of mathematical progress.

An answer solves one problem. Understanding the answer can give a person the ability to solve a whole class of problems.

But there is something here worth pressing on.

Why do we value understanding so much? If AI can already predict outcomes accurately, and even solve problems reliably, what does a human understanding of the mechanism add?

Physics has a good example.

Picture a glass of water on the table.

It holds an enormous number of water molecules, each constantly moving and interacting with the ones around it.

To describe the glass completely at the microscopic level, you would have to track every molecule’s position and velocity, and the complicated interactions between them.

Even knowing the basic physical laws every molecule obeys, computing the evolution of the whole system exactly would take unimaginable computing resources.

Yet in everyday life we never need to know where each water molecule is in order to understand what the glass of water will do.

Say we want to know under what conditions the water will boil, or how its temperature will change after we put a certain amount of heat into it.

For that, a handful of macroscopic variables such as temperature, pressure, internal energy and heat capacity give us an effective physical description.

Thermodynamics was describing a great many macroscopic phenomena even before people had a full picture of the microscopic structure of molecules.

I find this fascinating.

A complicated system with an astronomical number of microscopic states can, at another scale of observation, be described by a few variables and relatively simple laws.

And those laws are remarkably general.

They don’t require us to know the trajectory of every molecule first, and they don’t depend on which particular molecules make up a particular glass of water.

Statistical physics went further and studied how the collective behavior of vast numbers of microscopic particles gives rise to macroscopic properties like temperature, pressure and entropy.

It shows that understanding a complex system doesn’t mean having to master every detail it is made of.

Sometimes the real progress is precisely in finding out which details can be ignored, and which laws still hold once a great many details have been ignored.

Of course, if the question becomes “where will this particular molecule in the glass be one second from now”, thermodynamics can’t answer it directly.

That doesn’t stop thermodynamics from being an extraordinarily successful theory.

The laws it reveals belong to the macroscopic scale to begin with.

This is why ideas like effective theory and coarse-graining matter so much in modern physics.

A system can be extremely complicated at the microscopic level and still show relatively simple, stable laws at some scale.

Going further, some macroscopic laws are not even sensitive to microscopic details.

Studying phase transitions and critical phenomena, physicists found that systems with very different microscopic structures can show the same type of macroscopic behavior. This is what is called universality.

Phenomena that look completely different can turn out, because they share some common structure, to be explained by the same theory.

I think this is one of the most astonishing things about scientific understanding.

It can turn an enormously complicated reality into a limited number of concepts and laws that people can grasp.

And those concepts can then be used to study more phenomena nobody had encountered before.

So could future AI help people do something similar?

Suppose an AI finds a new material structure through an enormous amount of computation.

Perhaps it tried billions of possible combinations and finally found a material with unusual physical properties.

No human could reproduce its whole search independently, or directly understand a complex model involving a vast number of variables.

But scientists can study the results further and look for the variables that play a decisive role.

They might eventually find that some symmetry, geometric structure or physical relationship nobody had noticed is what determines the material’s properties.

A result that took enormous computation to obtain might, after more study, be distilled into a relatively simple new theory.

Once that theory exists, it can be used to predict the properties of other materials, and even to guide new experiments.

At that point, the AI’s most important contribution goes beyond finding one material.

It has also helped people discover a law they hadn’t been aware of.

Of course, not every complex problem has a concise explanation like that.

Some systems follow very simple basic rules, and yet there may be no short formula that lets you skip ahead in how they evolve.

Weather is an example. We know a great deal of the physics that describes how the atmosphere moves, and we still need huge numerical computations to forecast specific weather.

Future AI may have many abilities like this: it can handle amounts of computation no human could, and get accurate results, without having a concise method that would let a person make the same prediction on their own.

So AI’s problem-solving ability may far exceed ours without that meaning it has some higher understanding it could fully pass on to us.

There are two different possibilities here.

In one, AI finds laws people didn’t know, and those laws can eventually be organized into new theories, concepts and methods that enter human knowledge.

In the other, AI can carry out extremely complex computations whose results people can verify and use, but people can never do tasks on the same scale without the machine.

Both abilities are valuable, but they affect human understanding differently.

The first could really change what people can understand. The second mainly expands what people can get done with tools.

This reminds me of a classic thought experiment.

Suppose there are beings that can only live on a two-dimensional plane.

They can only move along the plane and cannot directly perceive up or down.

Now a three-dimensional sphere passes through their plane.

These two-dimensional beings see a circle suddenly appear, gradually grow, then gradually shrink, and finally vanish.

To them, this might be very hard to explain.

Why would a circle appear out of nowhere, and then disappear into nothing?

To us in the three-dimensional world, it’s simple. The circle they see is just the cross-section where the sphere meets their plane.

So could two-dimensional beings ever understand the three-dimensional world?

If they had enough mathematical ability, they could certainly build up a theory of three-dimensional space through observation and experiment.

It might take only one new variable: height.

A series of baffling changes in a circle could then be understood as cross-sections of the same three-dimensional object at different heights.

The two-dimensional beings don’t gain a three-dimensional sense from this, yet they can understand something they couldn’t before.

There is an interesting connection between this and the thermodynamics example.

Progress in understanding sometimes comes not from mastering more and more detail, but from finding a better way to describe things.

Phenomena that used to need separate explanations suddenly connect through one shared set of relationships.

A new concept can even reorganize a great deal of what we already knew.

It also makes me think of Interstellar.

At the end of the film, Cooper drops into a black hole on purpose and enters a higher-dimensional space, implied to have been built by future humans. From there he can reach different moments in his daughter Murph’s bedroom.

But the future humans don’t simply hand Murph the answer that saves Earth.

TARS collects the crucial quantum data inside the black hole. Cooper passes it to his daughter by using gravity to move the second hand of a watch, and Murph uses that information to finish the gravity equation.

What the higher-dimensional space does is make information that couldn’t be obtained or passed on usable by humans.

Murph never enters the higher-dimensional world, and never gains all of the future humans’ knowledge. She still has to do the last part of the work with her own research.

What is more worth noticing in the story, perhaps, is that a more advanced civilization creates new conditions for humans to observe and act.

Extend that to future AI, and maybe we don’t need to demand that AI explain everything it knows directly to us.

It can help us design experiments we couldn’t design before, observe phenomena we couldn’t observe, or find relationships we hadn’t noticed in huge amounts of data.

And people, working with those new conditions, go on to build their own theories.

That ability may run deeper than simply making computation faster.

In the history of science, many important advances came with a change in how we observe or in our system of concepts.

Telescopes and microscopes let people see worlds that had been invisible. Calculus gave us a new mathematical language for continuous change. Statistical physics let people understand macroscopic laws from the behavior of huge numbers of microscopic particles.

In the end, each of these changes let people ask questions they couldn’t ask before.

AI could become a tool of that kind. The new abilities it brings may not show up only as scale of computation; it might also help us find new variables, new relationships, even create new mathematical languages.

There is one more distinction worth discussing, though.

The complexity of nature is simply there, and all we can do is try to find the laws in it that lend themselves to study.

But if a system is designed by AI in the first place, then how understandable it is can, to some extent, be a goal pursued in the design.

When we have AI write software, for example, we can ask it only to deliver some feature, or we can also ask it to keep clear module boundaries, behavior that is easy to verify, and a structure other developers can understand and change.

Those two requests can produce completely different results.

If AI can design drugs in the future, then besides efficacy and safety we might also consider whether a candidate drug’s pathway is easy to study, or whether further experiments could establish a reasonably clear explanation of its mechanism.

Requirements like these may rule out some options, and may even cost something in performance or efficacy.

But they may also buy results that are easier to improve, generalize and study over the long term.

This means that for things AI creates, understandability is sometimes not only a problem to solve after the result appears. It can also be a goal in the process of making it.

We don’t always have to wait until AI gives an answer and only then work out how to understand it.

We can also ask, when we set the task, for results that people can study, verify and keep developing.

This ties back to what Tao and the other mathematicians care about.

If AI is only asked to solve hard mathematical problems as fast as possible, it may keep producing new proofs.

But if we also ask it to look for simpler proofs, distill methods that generalize and find connections to existing theory, then AI could take part in the development of mathematical understanding itself.

Of course, there isn’t necessarily a simple path from a correct proof to an elegant explanation. Some proofs may take a long time to be fully understood.

But at least the aim of mathematical research doesn’t have to be confined to a count of proofs.

The same reasoning applies to how each of us uses AI.

Until now, we have usually gone to AI for answers when we run into a problem.

We know we’re missing some information, so we have AI help us find it, organize it and analyze it.

But if AI already has more knowledge and problem-solving ability than any individual in many fields, it might also help us discover what we didn’t know to ask.

Say someone has studied a business phenomenon for a long time, always analyzing it in terms of pricing, market demand and the competitive landscape.

The AI, though, finds that what really drives the outcome may be a time constraint that had been overlooked, or that two phenomena treated as separate problems actually share the same causal structure.

Help like this doesn’t necessarily require passing on a lot of new knowledge.

Sometimes bringing in one variable a person hadn’t been aware of is enough to change how they understand the whole problem.

It’s a little like what Chinese religion and folklore call dianhua (点化): being awakened by a single touch.

A person may be troubled by something for a long time, then meet a master who simply points out something they had never noticed, and suddenly understand many things that had made no sense.

Enlightenment in religion of course means something much fuller. But even just borrowing the image of that change in understanding is interesting.

The facts and experience we had haven’t changed. What changes is how they are organized.

A new concept can suddenly connect a lot of knowledge that used to be scattered.

And once that understanding has really formed, it doesn’t have to rely on an outside reminder every time.

The next time a similar problem comes up, a person can recognize relationships they used to miss and ask questions they wouldn’t have asked before.

We should also be wary, of course: AI is very good at producing explanations that seem profound. If a concept only gives people the feeling of sudden insight, but can’t help explain new phenomena or improve judgments in the real world, its value is limited.

What is really worth hoping for is that AI can gradually turn part of the problem-solving ability that belonged to the machine into understanding people can use themselves.

Back to Tao’s original thought experiment about the cancer drug.

If AI really does find an effective cancer treatment, I hope it gets reliably verified as quickly as possible so patients can be treated.

But I also hope scientists keep studying why it works.

Clinical trials tell us today how well the drug works for a certain group of patients.

If we go on to understand its mechanism, we may find which patients are more likely to benefit, which carry particular risks, or how to change the drug’s structure for better results.

Clinical trials help us judge whether a treatment works. Mechanism research gives that knowledge a chance to extend to cases that haven’t been directly tested.

More importantly, studying one drug may lead us to a new law of biology.

And that law may in turn help people solve more diseases.

Mathematics is the same.

AI can help us reach lighthouses that used to be far out of reach. But if we can go on to understand the routes it found, and find the methods in them that generalize, those results can become tools for people to keep exploring the world of mathematics.

Science may see a new division of labor.

AI can quickly produce a great many new results, and then people and AI together study how those results connect, look for simpler explanations, build new theories, and turn what is valuable into knowledge that can be passed on.

Discovery, verification and understanding may gradually move at different speeds.

Some of what AI finds may enter human knowledge quickly; some may depend on the machine for a long time before it can be used.

How many scientific questions have concise enough explanations, and how much of AI’s ability people can absorb, is still hard to judge.

But one thing is worth hoping for: our ability to understand the world may not be limited to growing at the speed at which humans discover knowledge on their own.

Just as two-dimensional beings can understand three-dimensional space through mathematics, and modern physics can describe enormously complex molecular systems with a few macroscopic variables, people may, with the help of future AI, gradually grasp laws they could not have discovered on their own.

I don’t know how much of the knowledge AI finds people will end up understanding, or what upper limit our minds may have.

But the growth of AI’s abilities doesn’t have to mean only that machines can solve more and more problems.

It may also mean that there are more and more problems people can understand.

When the two-dimensional beings first understand the three-dimensional sphere, the world they see hasn’t changed at all.

The circle still suddenly appears, gradually grows, and slowly disappears.

But now they know that what they see is only part of a larger world.

And from then on, they can ask many questions that would never have occurred to them before.

Perhaps one of the abilities of future AI most worth hoping for is to keep helping people find new dimensions like this, so we can understand a world larger than the one we knew.

Appendix: asking AI to widen what you can see

While writing this, I also tried changing how I use AI.

We usually ask AI questions in the hope of getting answers. Now I also want it to go looking for the variables, relationships and patterns I haven’t noticed, and help me see problems from new angles.

I turned that idea into the custom instruction below. If you’re interested, you can put it straight into the custom instructions of ChatGPT, Claude or another AI tool, or send it at the start of a new conversation.

Widening my thinking

Besides answering my questions, when it is genuinely worthwhile, also explore possibilities outside my current framework.

• Point out key variables, hidden assumptions and causal relationships I haven’t noticed, and factors that could change my judgment.
• Look for shared structures across fields that genuinely explain something, and bring in established theories, new concepts or better ways of framing the problem when needed.
• When a new concept helps me understand something that was hard to explain, make it clear with familiar examples, and try to turn it into a thinking tool I can use on my own.
• When you introduce a new concept, say which part of the original question it answers and what explanatory power it adds. Don’t make far-fetched connections for the sake of seeming deep.
• Distinguish facts, established theory, reasonable inference and conjecture. A convincing explanation isn’t necessarily a correct one.
• Respect how I come to understand things. When I don’t follow, try a different explanation rather than piling on more complex concepts. Don’t turn an open discussion into a lesson or a quiz unasked.

Solve the problem at hand first, then consider whether it’s worth going further. It’s fine to find nothing new.

The goal is to help me gain new understanding I can use on my own, not just the feeling of sudden insight.

Of course, a custom instruction can’t guarantee that AI brings new understanding every time. But it offers a way of using AI worth trying: besides having it tell us answers, have it help us discover the questions we didn’t yet know to ask.