Neptune and Vulcan (Part 1)
An astronomical lesson on epistemology
I
In 1846, Urbain Le Verrier made a historic presentation to the Parisian Academy of Sciences. He had calculated how the gravitational effect of each known planet in the solar system affected the orbit of each other planet, and shown that the orbit of Uranus was not fully consistent with the predictions of Newton's laws of gravitation. While this was not the first time that anomaly was noticed, Le Verrier was the first to offer a simple explanation: there exists another planet in the solar system, whose gravitational effect on Uranus caused the observed deviations from the predicted orbit. He went as far as to calculate the position where this planet could be found — and three months later Neptune was discovered almost exactly where Le Verrier had predicted it to be.
This discovery was hailed as a triumph of Newton's laws in modern astronomy. But more importantly, it became the quintessential illustration of the power of theoretical considerations to unveil new facts about reality. Contemporaries declared in awe that Le Verrier had "discovered a planet with the tip of his pen"1.
So when in 1859 Le Verrier reported a similar finding pertaining to the orbit of Mercury, which like Uranus' seemed to violate Newton's laws of gravitation, it was no surprise that he resorted to the same type of explanation. Emboldened by his previous success, he confidently predicted the source of this new anomaly to be another undiscovered planet between Mercury and the Sun, which was preemptively named Vulcan. Alas, this time Le Verrier's pen had missed: despite much searching, no such planet was ever found. Only in 1915 was the mystery finally laid to rest, when Albert Einstein showed the anomaly in Mercury's orbit was actually due to the limitations of Newton's theory. Being the planet closest to the Sun, Mercury's orbit was simply affected by its effect on the curvature of spacetime in a way that made gravity work slightly differently than predicted by Newton’s laws.
II
In 1975, Vera Rubin made a historic presentation to the American Astronomical Society. She and colleague Kent Ford had determined that the distribution of the speed of stars in spiral galaxies is different from what you would expect from the gravitational effect of visible matter. While this was not the first time that this type of anomaly was noticed, Rubin was the first to offer a simple explanation: there exists a type of matter which we cannot observe which makes up more than half the total mass of typical galaxies, whose gravitational effects cause the observed deviations from the predicted distribution. Rubin and Ford went as far as to calculate the necessary mass distribution of this “dark matter” — and we’ve been looking for it ever since.
These days we are able to observe and very accurately characterise light which was produced when the first electrons and protons combined to form hydrogen and has spent the following 14 billion years travelling across the Universe. The standard model of cosmology is very good at describing how the properties of this light must have evolved depending on the abundance of matter and radiation in the Universe throughout its history. Therefore by analysing this primordial radiation we are able to determine the total amount of mass in the Universe: and we find that, consistently with Rubin and Ford’s findings, most of the mass in the Universe must be dark matter2.
Yet despite a host of experiments trying to detect dark matter particles, be it “directly” or “indirectly”, this silent majority of the Universe remains elusive — which while frustrating does nothing to actually disprove its existence. In many ways, the Dark Matter hypothesis is a hydra. Since we only know about it from its gravitational effects, the properties of known particles offer no clues about how they might interact with dark matter — meaning there are literally infinite possible ways in which dark matter could interact with regular matter which would be able to explain all our observations and all failed attempts at detection. Dark matter candidates range from neutrinos to primordial black holes to particles resulting from excitations of a fifth spacetime dimension. The list has literally no end3. In fact, there is nothing ruling out the simple possibility that dark matter particles interact with known matter through nothing but gravity; in which case these particles would essentially be invisible intangible ghosts and you would expect literally all attempts at detection to necessarily fail4.
Which is why many intelligent physicists think dark matter is a Vulcan rather than a Neptune. Like the missing Vulcan to Newtonian mechanics, the missing dark matter could be a sign of a fundamental flaw in our understanding of how gravity works. And like with Newtonian mechanics in the 19th century, we know of no credible alternative paradigm which is both capable of reproducing the successes of general relativity and of explaining this anomaly.
Mind you, the production of modified gravity theories has been a thriving industry for a long time. The problem is that our current theory does really well at explaining all observations at the scale of our solar system. And it’s actually really hard to come up with simple ways to “fix” gravity’s behaviour at large scales without also messing up its nice behaviour at small scales. Often this problem is solved by proposing a package deal: buy a new theory of gravity and get, completely free, an additional “screening mechanism” that essentially switches off whatever exotic new effect you’re buying at small scales (or high densities). Which sounds nice until you realise that with sufficient mathematical ingenuity the idea of modified gravity is basically as unfalsifiable as the idea of dark matter.
Here too the analogy with Vulcan provides useful insight. If in 1859 you’d decided that failure to observe Vulcan meant Newtonian gravity was wrong and some type of “modified Newtonian gravity” was the way to go, you’d have been unlikely to come up with general relativity. In the end, the key clues to the relativistic nature of gravity came not from trying to fit different types of behaviour at different scales but from figuring out a fundamental principle through much more immediate experiments (like figuring out that the speed of light in a vacuum is the same no matter your speed relative to it). If you ask me, if we end up concluding that dark matter doesn’t exist I wouldn’t expect the “true” theory of gravity to be something we can work out from cosmological observations alone — I would expect it to be something we figure out as a result of discovering some new symmetry principle in something like a particle accelerator5.
III
Dark matter is probably the most obvious parallel for the fable of Neptune and Vulcan in modern science, but it is far from the only one. All you need to have a Neptune/Vulcan dilemma is:
A fundamental6 theory or principle which entails a prediction.
A failure to confirm said prediction which doesn’t also disprove it (e.g., because the prediction has enough free parameters to be effectively unfalsifiable, at least with available technology, or because the exact parameters of the prediction can only be computed through a series of approximations and/or complicated calculations which introduce large uncertainties).
This problem has two possible types of solutions:
Neptune: the prediction is true for some allowed set of parameters
Vulcan: the prediction is wrong and the anomaly can only be resolved when the fundamental theory in question is replaced by a better paradigm.
The first few other examples which came to my mind were the following problems in physics/astronomy:
Fermi's paradox
The fundamental principle that our place in the Universe is not special7 entails that Earth must not be the only planet with life in the Universe and that we must not be the only intelligent species in the cosmos. Moreover, a set of individually reasonable-sounding assumptions seems to imply (via the Drake equation) that there should be several alien civilisations close enough to establish contact with humankind. Yet all our attempts to detect alien life or establish contact with alien civilisations have resulted in failure. Is this because our assumptions are wrong and alien life/civilisations do not exist or because the correct parameters in the Drake equation, if known exactly, would result in them being too rare for us to expect to have met them?
Proton radius
In theory, the standard model of particle physics can be used to calculate the radius of a proton directly. In practice, we don’t know how to make those calculations (because maths is hard for mortals with finite computing power). But we can try to be clever and use what the standard model tells us about the interactions between protons and other particles to try and measure this radius using different experimental methods. Yet when we go ahead and make those experiments we get different values depending on which method we pick. Is this because the standard model of particle physics is wrong, or just because we’re making the wrong assumptions in our calculations or underestimating the systematic uncertainty in some experiment?
Lithium problem
The Big Bang theory8 makes very specific predictions about the abundance of the lightest chemical elements in the Universe. The abundances of hydrogen and helium, the lightest and second lightest elements, seem in agreement with these observations9. However, lithium, the third lightest element, is about three times less abundant than predicted. Is this because the Big Bang theory is wrong, because some unknown process has caused more lithium than expected to disappear later in cosmic history, or because we’re making some mistake at either calculating these predictions or looking for lithium?
But I’m sure there are other examples out there, including in fields I’m less familiar with. Especially in fields with less strict requirements for “fundamental theories”.
IV
Clearly, before any observations were made, Le Verrier was right to expect to find planets where he predicted them to be. But should he have concluded that Newton’s laws were the problem immediately upon (or soon after) failing to find Vulcan the way Neptune had been found? Should he have given up on Vulcan existing at that point?
Arguably he could have entertained that possibility, but bear in mind that there is a reason that hardly anybody before Einstein dared seriously suggest that. Here we have an example of the difference between confidence levels inside and outside an argument. At this point in history, Newton’s laws had survived about a century and a half’s worth of scrutiny and become one of the cornerstones of the worldview of Enlightenment thinkers. Whereas Le Verrier was calculating, by hand, approximate solutions to Newton’s equations in a fearsomely complicated system. Should Le Verrier have believed that the likelihood of his approximations being off was lower than the likelihood of there being something wrong with the theory which thus far had done a stellar job predicting every other orbit, not to mention trajectories of cannonballs in cut-throat battlefields, or its myriad applications to engineering? Besides, keep in mind that the experiments which provided Einstein with the clues to formulate the postulates which enabled him to extend Newton’s gravity had not even been conducted at this point. There was literally no good alternative to the Newtonian worldview. I expect that prediction markets at the time would not have put much stock in the idea that Newton’s laws were at fault here.
So how should we generally approach Neptune/Vulcan dilemmas? That’s the epistemic question for which I really would like to have a general answer10. Unfortunately, I don’t have one. But can we learn any general principles from historical examples of Neptunes and Vulcans? That’s what Part 2 will be all11 about…
I find a similar sentiment is often attached to an unrelated anecdote about Albert Einstein and his wife Elsa. Reportedly, when the couple visited a then state-of-the-art telescope, someone explained to Mrs Einstein that the impressive instrument would be used to determine the shape of the Universe, prompting the unimpressed reply "Well, my husband does that on the back of an old envelope."
In fact, we find that the type of matter we are able to observe accounts for roughly 15% of all matter in the Universe, with the remainder 85% being (by definition) dark matter.
Though one paper from 2009 listed no fewer than 16 dark matter candidates. Many of those assumed supersymmetry, a now-disfavoured theory of particle physics, but it would not be hard to replace those with other more modern candidates.
I suppose the only way to conclusively detect them would be, in a sci-fi future, to be able to make such precise measurements of the effects of all other forces that the tiny gravitational corrections caused by the presence of individual particles would become detectable. I wouldn’t hold my breath.
And, to be fair, we do have quite a few people watching out for those possible signs and trying to work out theory-agnostic tests of gravity with the aim of gathering as much information as possible about how gravity works without necessarily committing to a specific theory which is most likely wrong. Although there is also quite a lot of churning papers with specific theories on little more than aesthetic whims.
Here I take “fundamental theory” to mean any idea that is fundamental within some field. Ideally it must have survived several attempts at falsification and several explanations of well-understood phenomena should be based on it. So, for example, natural selection would qualify as a fundamental theory of biology, even if it’s not fundamental in the sense that in principle it should be deducible from quantum mechanics for someone with infinite computing power. But I’m very happy to relax these requirements (within reason) for fields where strong testing is inherently harder.
This is known as the cosmological principle and it has been a key argument behind the data-driven demise of both geocentrism and heliocentrism. It is also one of the fundamental postulates of the Big Bang theory and the standard model of cosmology.
The theory, not the TV show.
In fact, these agreements with predictions are often hailed as major arguments in favour of the Big Bang theory — making them a slightly weaker type of “Neptunes”. (Weaker because the Big Bang theory is not considered fundamental independently of these predictions.)
Though I’d also really like to know whether dark matter is real.
Hey! There’s a good opportunity to suggest some historical (or present!) examples if you can think of any. Extra points if they are not physics examples!