I noticed that a couple of the books I had previously read on the subject of AI referenced Superintelligence from Nick Bostrom, Professor in the Faculty of Philosophy at Oxford University and founding Director of the Future of Humanity Institute.  

The book is very far sighted, and as such has a tang of science fiction to it.  However, it was written in 2014 so some of this future is unfolding. 

I was grateful to see Professor Bostrom, examines topics like: the control problem, how to set a safe objective,  how income distribution works as this future unfolds.  

This should be mandatory reading for decision makers in both public and private sectors.  

Professor Bostrom thinks through and explains the relative dangers of a slowly detonating and fast detonating intelligence explosion.  I don't think he explains how to recognise a very slow intelligence explosion, one that happens over decades.   He explains the dangers of a superintelligence pursuing an unintended objective  like making paperclips, in a manner that extinguishes the human race.   

Key points in the book 

Types of Superintelligence

Bostrom outlines three forms:

• Speed superintelligence

• Collective superintelligence

• Quality superintelligence

Paths to Superintelligence

• Artificial intelligence (especially AGI)

• Whole brain emulation

• Biological cognition enhancements

• Networks and organizations

• Brain-computer interfaces

Orthogonality Thesis

Intelligence and goals are orthogonal.  A superintelligent AI could have utterly alien, even trivial or harmful, goals from a human perspective.

Singleton Hypothesis

A superintelligent AI could become a “singleton,” a single decision-making entity with unrivaled power over the world, effectively ending geopolitical pluralism.

Value Loading is Hard

Imparting human values into a superintelligent system is extremely difficult. Even slight misinterpretations (e.g., misaligned reward functions) could have catastrophic consequences.

Urgency of Safety Research

Bostrom emphasizes that superintelligence could arrive suddenly and without warning, making proactive safety research critical. The "window of opportunity" to ensure safe AI may be narrow.

Less well explained  are the dangers of a Superintelligence  influencing human intelligences so that we all end up pursuing some poorly defined and questionable objectives: space tourism, bitcoin, torn jeans, likes, wars;  willing slaves to a mad master. 

I get the sense of floating random objectives taking root in biological, sociological and computer substrates.   

I was struck by the amount that I wanted to draw parallels with historical problems in human organization and structures that have little to do with computers.  I found it curious that these parallels were often not made explicit. 

The best guidance Professor Bostrom has in the book, is something that he quotes from Eliezer Yudkowsky, called Coherent Extrapolated Volition.  

"What we should ask for is that we would ask for if we knew more, thought faster, were more the people that we wished we were, had grown up farther together, where our wishes converge rather than diverge, where our wishes cohere rather than interfere, play out as we hope them to play out and are interpreted as we hope they are interpreted"

This is written by a computer scientist, but it is the most profound poetry.   

I used chatGPT to do the summary bit in the middle, but the impressions of the book are my own.