What we are trying to find out, the published results we are building on, how we work, and the tests the idea has to pass in order. This page changes when the work changes, and revisions are recorded in the lab notes.
Fixed in Note 001, before there was any result to judge against it.
Can the architectures of learning systems be derived from physical and mathematical law?
Symmetry arguments already do part of this. Bronstein et al., 2021
Parts of several architectures were reasoned rather than stumbled upon. Convolution builds in translation symmetry. Variational autoencoders come from an explicit probabilistic derivation. Diffusion models came from a thermodynamic analogy. In each case the principle fixes an objective or a symmetry, and the network carrying it is still chosen by trial. Nothing has been derived the whole way down, and that last stretch is the subject of this program.
Behind that sits a larger question. We have met intelligence in two forms, living brains and the systems we have built, and both are particular. If intelligence can be characterized mathematically without reference to either, then each is one solution to that characterization and neither is the thing itself. Definitions of that kind already exist. Legg and Hutter published one in 2007 and were explicit that it is "in no way anthropocentric", and equally explicit that its value "is not computable". A definition nobody can evaluate is a starting point rather than an answer, and it is roughly where we pick the problem up: which physical systems can approximate it, and at what cost.
The word "derived" is doing a lot of work in the question above, so here is what we mean by it, in four conditions. The assumptions are written down and you can count them. Every step can be checked by someone who did not write it. When you restrict the general structure, systems that already exist and work have to fall out of it. And somewhere the result has to disagree with current practice, on a point an experiment can settle. Something that meets fewer than four is an analogy. Analogies are useful for deciding what to try next and useless as a foundation.
The starting points are not speculation. Each one below is published, checkable work by other people, with the reference given.
Taken one at a time, those are nine findings spread across as many subfields, and each is usually read as a curiosity within its own. Taken together they suggest that entropy, energy, and wave dynamics may be closer to what learning systems are made of than to a convenient way of describing them. That is a conjecture, and the list above is careful about which parts are demonstrated and which are not. The last three findings say that a definition of intelligence exists which owes nothing to us, that nobody can evaluate it, and that biology satisfies it anyway on a fifth of a body's energy. The field keeps arriving at these pieces one at a time and moving on. This program exists to do the assembly on purpose.
Diffusion models are the useful case to look at. They are the one major model family whose training objective came straight out of physics, and they turned out to be among the strongest in use. The physics stops at the objective, and the networks underneath are still picked by trial. So on the one occasion physics was allowed to determine part of the design, the result was good, and nobody carried it further. This program tries to carry it the rest of the way, from the loss function down into the architecture.
The problem splits into five: the thing that does the computing, the mathematics that describes it, the physical limits on it, what happens when a system has to model itself, and what the one working example we can measure has to say about the other four.
This is the medium itself. The question is which useful operations it can carry out directly through superposition, interference, and settling to equilibrium, and what each of those costs in energy, time, and capacity. Also what a learning rule turns into when it has to be a physical process running on the same medium that does the computing, with no separate machinery to fall back on.
This is the structure above the medium. It covers what shape a family of wave-derived models has when you measure it with information geometry. Which complexity classes limit learning and inference on this kind of hardware. And which properties of a learned representation are forced by the dynamics rather than being accidents of one training run.
These are the limits the medium runs into. They include how much more energy real inference uses than the Landauer floor requires, and what closing that gap would take. What a learning system gains by being reversible. And whether fluctuation theorems, which describe systems driven far from equilibrium, give usable bounds on learning.
This is the ceiling on the whole thing. It asks which formal structures let a system represent its own computation without running into contradiction, and what holding that representation costs it. Whether the obstacles known from logic, fixed points and incompleteness, show up again as physical limits once the system doing the reasoning is a physical object.
This is the evidence we have. Brains learn without a backward pass over stored activations, run on a fifth of the body's energy, and are noisy, spiking, and never at equilibrium. Each of those is a measurement of a physical system that solved the problem. The work is separating the features that physical constraint forces on any such system from the ones that are accidents of how biology got there, because only the first kind generalizes.
The loop is short and we run it over and over.
Pencil and paper, with the assumptions written down first. What comes out is a family of models or a procedure, with the conditions it depends on stated in the open. What does not count is a mechanism that sounds plausible with an explanation attached afterwards.
Small numerical experiments on field and wave dynamics. The point of them is to kill bad derivations quickly and cheaply, before any effort goes into building on top.
Anything that survives both steps gets written as a reference implementation and run on real tasks. If it cannot be implemented, it goes on the record as unfinished rather than being announced as progress.
None of these four has been passed. When one is, it will be dated in the notes first.
Judge the program against these four steps, in order. Any of them can fail, and if one does, everything after it is void. Progress gets recorded in the lab notes as it happens.
Write down what the computing medium actually is: the states it can be in, how it changes, and what counts as a computation happening in it. It has to be precise enough that the three steps below are able to fail.
At least two model families that already work have to fall out of the definition as special cases: associative memories when the dynamics are driven to settle, diffusion when entropy production dominates. Anything that cannot reproduce what already works is simply wrong, and this is the cheapest way to find that out.
The framework has to predict something current practice gets wrong, on a point a numerical experiment can settle, and it has to do so before anything large is built on top of it.
Take a procedure that came out of the theory, implement it, and measure it on a task picked before the results were known rather than one chosen to make it look good.
Four things we do not acceptThat a current architecture is right because it is current. That a benchmark score explains anything. That an analogy to physics counts as a result. That a claim is worth publishing before anyone can run it.
| Cadence | Output | State |
|---|---|---|
| Ongoing | Lab notesDated notes: positions, decisions about method, things we have read, and results that came out negative. | 4 published |
| When results hold | Technical notesFull write-ups, published when a derivation or an experiment has survived being attacked. The first will appear when there is one worth reading. | None yet |
| With every result | Reference implementationsCode for anything the theory produces, so that you can rerun the result yourself instead of taking our word for it. | None yet |