‘MICHAEL FISH WAS A GODSEND’. DISCUSS

You don’t need a noisy computer or an ensemble prediction to say that The Primacy of Doubt will make waves.

The author has seemingly exhaled just shy of 300-pages of extraordinary thoughts that had long lain dormant, maturing like a good wine and just waiting to be uncorked.

That moment was 2020 and the pandemic, explains Professor Palmer. ‘I was sitting at home. I said to myself, if I’m going to write this damn book. I'm going to write it now or never.’

It can be regarded as a summary of his life’s work with a broad introduction to the science of uncertainty, followed by how predicting our volatile world has been revolutionised by tools developed by Professor Palmer with colleagues at the Met office and in other universities, and how to apply these ideas to other areas of society like the economy and pandemics.

In a much more demanding Part III, the author presses on into Freewill, Consciousness and God, in a more freewheeling and speculative section that appears to shape the future and will upset the quantum mechanics community.

While The Primacy of Doubt is in part difficult, the style and tone are like walking by the sea in a stiff breeze. The air is bracing, the sentences are spare and luminous, and thrilling.

Even a non-specialist can come away with an armful, while not necessarily comprehending some of the theoretical backstories.

That is as one might expect given that Tim Palmer (Wolfson, 1974), Royal Society Research Professor in Climate Physics at Oxford, was all set once to take up a post-doctoral appointment with Stephen Hawking at Cambridge.

The reason he did not take it up is partly why he is so compelling now, and offers further insight into the manner in which unexpected ‘wrong turns’ in careers can be strokes of genius.

At the time, he notes, he had just completed his Oxford DPhil on general relativity. But he was very worried about two things.

First, he was worried that theoretical gravitational physics was of no consequence to anyone outside a tiny number of scientists, and second, and worse, he had grave doubts about the achievability of the Holy Grail of the 1970s in this field of inquiry, unifying quantum mechanics and general relativity.

He remains right about that, noting early in this book, ‘…it remains an unsolved problem.’

Following his instincts, he instead embarked on a remarkable career at the Meteorological Office and at the European Centre for Medium-Range Weather Forecasts.

The exceedingly useful case study that runs through the entire book is weather forecaster Michael Fish who famously got it wrong on October 15, 1987.

To rehearse, he delivered the weather forecast that night, and joked that a woman had rung up the BBC earlier that day and said that she’d heard there was a hurricane on the way. He noted, ‘Well, if you’re watching, don’t worry, there isn’t.’

By the next morning 15 million trees had blown down and 22 people had been killed.

Palmer is not ruling out similar mistakes today, but they have become far less likely owing to what is referred to as ensemble forecasting, which he pioneered and which is now ubiquitous in weather and climate modelling everywhere in the world, to the profound benefit of all and increasingly life and death situations where extreme weather is encountered.

An ensemble forecast is where you take a supercomputer model of the weather and run that model forwards numerous times, making tiny tweaks to input factors. If the many models stay convergent then the certainty of the weather forecast is high. If they rapidly diverge then it is volatile.

Palmer says that on the night of October 16, 1987, the models would have shown extraordinary divergence, ‘an explosively unpredictable situation.’

Much the same applied to the global economy in 2008, to the point where the Bank of England Chief Economist Andy Haldane actually compared economists to Michael Fish, in a speech in 2017.

In this sense, Haldane noted, the Global Financial Crash was their ‘Michael Fish moment’.

The underlying cause of such extraordinary outcomes, which can be potentially extended to include pandemics and invasions such as Russia’s assault on Ukraine, is an intermittent butterfly effect, says Palmer.

By this he is referring to chaos theory, ‘a type of fractal geometry discovered by meteorologist Ed Lorenz, which I rate as important as Einstein’s theories of relativity and Schrodinger and Heisenberg’s theory of quantum mechanics.’

American Edward Lorenz (1917-2008), who was at the Massachusetts Institute of Technology at the time of his breakthrough, wrote a paper in 1963 that distilled a core idea developed by Palmer, that within nature there is a science of chaos, not a linear perfection waiting to be unpacked one day by the right genius with the right equation.

Broadly stated, it was this starting point that later led Palmer later to be a lead author for the UN Climate Panel, the IPCC, and a renowned expert on weather and climate prediction.

In case it needs to be stated, the whole area of short, medium and long-term weather and climate prediction has grown into a subject of vast and immediate economic and human value, if you work in insurance or reinsurance; agriculture fisheries and food; the water industry or renewable energy, to name just the most obvious sectors.

Palmer mentions this because his professorial Chair at Oxford, which was created in 2010 at the 350th Anniversary of the Royal Society, involves teaching doctoral and post-doctoral students. In case you’re wondering they aren’t all disappearing into an academic black hole.

He says to the contrary they have options these days, and that increasingly some students who might once have been become purely theoretical physicists are now increasingly frontrunners in making sense of an increasingly turbulent global climate system

In this sense we might say that theoretical physics and the complex maths underlying it, have become intensely relevant to human society rather than being blue sky theory, although it necessarily remains that too.

The day we talk, in Tim’s office in the Atmospheric, Oceanic and Planetary Physics Building  part of the University of Oxford’s Department of Physics, Pakistan has just been inundated with terrible floods not one week after parts of the city of Dallas in the US state of Texas received an estimated 15 inches of rain in one 24-hour period.

This is one of Palmer’s frustrations, and why he is a vocal campaigner for a ‘CERN for climate studies’, referencing The European Organization for Nuclear Research, the largest particle physics laboratory in the world situated on the French-Swiss border.

‘The trouble with AI,’ he says, ‘is that it can only interpolate on the basis of previous data. If you show it a cat, it can detect a cat – say in your Apple photo album. If you show it a creature it’s never encountered before it is flummoxed.’

In weather predicting, he says that heavy rain is a cat but 15-inches in one day is the creature never before seen, and not something even current models of climate have been able to predict.

Completing the thought, he says we need a CERN-like effort to bring together bigger computers to run more sophisticated, collaborative modelling of weather and climate models, so that ‘the creature never seen before’ can be recognised and even predicted.

BREXIT scuppered UK involvement in the ‘CERN-like’ development even though Professor Palmer got quite far towards securing European Union support for developing the models that will one day be used in such a centre. He remains immensely frustrated.

Having said that, he says of his former Met colleagues, of July 19th, 2022, when temperatures smashed 40C in the UK for the first time in recorded history, ‘they did a pretty good job.’

Especially if you remember that lots of right-wing pundits were certain days before the heatwave that it was all a hoax, and those temperatures wouldn’t be reached.

‘Well it wasn’t [a hoax], and they were [reached].’

That is the power of ensemble forecasting in practice, because there was a strong convergence of models that said it really was going to be exceptionally hot on that day. If you can get such predictions right, they allow anticipatory actions such as getting beef herds into shady locations and turning up the refrigeration in supermarkets.

It’s no longer any exaggeration to say that lives are saved or lost by good weather forecasting.

To impart a flavour of the mind-boggling breadth of the rest of Palmer’s book, the Bank of England and the Great Financial Grash of 2008 occupies a chapter.

Palmer offers that one could be an economist with limited time, and jump straight into Chapter 8 in Part II, to see how Palmer’s embrace of chaos challenges an adjacent profession, in this case the (apparently still dismal) ‘dismal science.’

Here’s a taster that any economist will recognise and perhaps be angry about:

‘…the idea that we can model the economy based on the actions of representative agents who are maximising some specific utility function may not be a good assumption for a highly heterogeneous nonlinear system in which the individuals may be acting according to some individually tailored optimisation principles, but equally it is important to understand their shortcomings when making actual predictions of systems far from equilibrium.’

In other chapters he applies similar techniques to international conflict, such as Russia’s assault on Ukraine. There are models for predicting that too, drawn from exactly the same underlying insight.

He says at one point, in our discussion, ‘So, you know, those two topics of non-linearity and uncertainty have been kind of pervasive themes in my life…’

There’s a cultural, historical element to this insight that may point to a new strain of humility in the human pursuit of knowledge, which takes us back again to Michael Fish, and why he can be seen as a godsend.

‘Back then you weren’t allowed to show uncertainty. You forecast the weather and if you were wrong, you just moved on. It was a white, male model.’

The hurricane of 1987 was such a disaster for the Met Office that were calls for the then Director General, Sir John Houghton, to resign.

It was that crisis that allowed a rethink, a less ‘certain’ approach, the one developed by Palmer with colleagues like Michael McIntyre, Glenn Shutts and James Murphy.

‘The working premise [before the hurricane of 1987] was that it was entirely predictable, we just had to solve the theoretical physics behind it all.’

Now, says Palmer, and this is the core claim of the book, chaos is central to the universe, rendering quantum mechanics problematic at the deepest level.

By embracing chaos rather than seeing it as an error waiting to be ‘solved’, Palmer and a galaxy of colleagues have shed a lot of light on a world that becomes more predictable for being recognised as more ‘chaotic.’

He admits that the title of the book might seem a bit intimidating. An alternative title considered for a while was The Geometry of Chaos, he notes. But the title is a citation from James Gleick’s biography of American physicist Richard Feynman (1918-1988):

‘He believed in the primacy of doubt, not as a blemish on our ability to know, but as the essence of knowing.’

As soon as you have read this epigraph, it makes sense.

Equally, what the book does so well is to take the currently polarised view of climate change on offer in the world, that it is either incipient doom for humanity or that really it’s all an elaborate hoax, and offer a very skilful centre ground that tries to bring a better clarity to risk and its mitigation.

No plot spoiler but the verdict is not very optimistic if you were hoping that he’d say ‘don’t worry.’ What he does say is that our modelling is not perfect so we can’t know with complete certainty how this will unfold.

He tells the story about how a few years ago he attended a public debate organised by the Spectator magazine, with climate skeptic and former Chancellor Nigel Lawson. He appeared to ‘win’ in the sense that a vote taken at the beginning and at the end showed a lot of people coming over to Tim’s position, that climate change is without question real and it is of grave concern.

‘Our duty as scientists is to be honest, not to try and peddle some political agenda under the auspices of science.’

He adds that how to reduce carbon is immediately a value judgement and thus a political decision.

But asked how he would act if he were a politician elicits two views, one that he worries about emerging market access to abundant, cheap energy which is the only route to development and hence education.

And he worries also about the intermittency of renewables, noting that even utility sized batteries won’t be able to tide us over if the winds don’t blow for long periods of time.

As such he remains a fan of nuclear and just regrets that Britain threw away its knowledge base in the early 1960s, saying that most of the advanced (and exciting) work on molten salt reactors and their like is coming these days from China.

Could the July heatwave in the UK have been a ‘natural event’?

‘I was at home both days [July 18-19]. I went out into the garden for a few moments. You’d have to be an ostrich with your head in the sand not to think there is something unusual about 40 C temperatures in the UK.’