Ben, thank you for sharing Andreessen's silly manifesto, I got a lot of laughs out of it. I don't think I've read anything this year with more meme potential. It's almost the Platonic ideal of tech CEO pomposity. There are too many highlights to list, but here are a few:
"We had a problem of isolation, so we invented the Internet"
"Love doesn’t scale"
Enlisting Nick Land, of all people, to argue that technology "is not anti-human – in fact, it may be the most pro-human thing there is".
"We believe in not Utopia, but also not Apocalypse"
(Halfway through I started mentally adding "In this house" before his endless "we believes")
"We believe in bravery, in courage" - those are the same thing mate
"We believe in pride, confidence, and self respect" - there's a pattern here
"We believe in the truth" - so do I as it happens
"We aspire to be… not that" - defining your aspirations in the negative, how Nietzschean
At the very bottom, other articles by him include "Disrupting the World’s Largest Asset Class with Adam Neumann".
'I can’t imagine anything with more potential for horror than eternal consciousness unmediated by the divine.'
Is bounded consciousness unmediated by the divine bearable only because we know it is finite? Sort of how you are able to sit through an orchestral piece by Schonberg because you know at some point it must surely end?
Yes, and because endless consciousness guided by man and his creations would be prey to the kind of flaws - deliberate and accidental - that could turn it into eternal torture.
I like your writing. You write like a normal bloke with normal feelings who has read enough about the modern world to explain to the rest of us normal but not so well-read people what the hell is going on.
Surely the answer is to eschew what Zvi Moshowicz calls the “Big Dial” model of progress -- whereby you’re either for or against *_Progress_* -- in favour of what I call the “Soundbwoy” model -- lots of little dials, you can twiddle them all at will; being a good soundbwoy, you probably want most of them set to High; but you have discretion to cut out GoF or AGI completely. (Incidentally, Big Dials are not restricted to tech progress discourse, and we could do with more soundbwoys across the board...)
Hi Jonathan! Yes, that's a nice image — clearly, it depends for optimists on whether the dials are powerful enough to produce sufficient outcomes, and to pessimists on whether they are powerful enough to constrain *excessive* outcomes but it's worth consideration.
> There is a good chance that we already live on the most beautiful, rich, poetic planet in the universe.
That's true! But it wouldn't be so unless the earliest lifeforms had decided to leave the most beautiful, rich, and poetic pocket of the ocean, and so on for their successors - leaving the garden is trad!
The trouble with bring ignorant about how technology is created, as opposed to how it is used, leads to sentences like "From artificial intelligence to synthetic bioweapons, I suspect there is a good chance that mankind will come to lament technological progress — if, that is, he even has the time to do so." If you've ever done work in actual AI, you'd know that AI can't on its own take over the world, but it can be used as a tool to do so. It's like having a pile of 2 x 4s, some plywood, and a hammer and a bag of nails - you can sit and watch them all day long, but the hammer will not on its own create a bookshelf or a house. The hammer lacks consciousness and volition, it's up to you - who has both of those needful things - to use the hammer and nails, and other tools, and create whatever you decide, within the limits imposed by the materials. AI cannot on its own take over the world, but it can be used by people with consciousness and volition to do so.
The trouble with most journalists is that they have a faint idea of how technology "works" and armed with that faint idea, believe every bit of the techno-hype and technobabble that comes their way, leading to risible headlines and clickbait stories that scare the hell out of people - and grab attention and page clicks. If you've ever actually had to *create* technology, you'd know instantly that that stuff was technohype and technobabble, stop screwing around, and get back to work. It's not the technology, it's the use to which it is put. We are told to pay no attention to the Man Behind The Curtain, but it is that man which we should be concerned about, not the means which he uses to realize his intent.
I assume you have a computer. It is possible for you to actually create technology which can be used on that computer. I strongly suggest that you take the time to install a program called Emacs on your computer - and build it from source code instead of just downloading a working black box binary. It will come with instructions for doing so. You most probably have an application called "terminal" in which you can accomplish this. Emacs comes with detailed manuals, and you can use it to create new functionalities using Emacs LISP. You can get it all from here: https://www.gnu.org/software/emacs/ I strongly suggest you do so, so you can emerge from the ranks of the techno-gullible, and recognize technobabble and technohype for the content-free nonsense that it really is.
(Also, even if concerns about misalignment *are* groundless, the potential for machines with immense destructive capacities to be misused by human actors might in and of itself become a reason for people to lament technological progress, inasmuch as the more dangerous the weapon the more powerful the villain.)
A nuclear weapon is actually a machine, the parts of the warhead which form the critical mass to create the explosion are precisely machined and have explosive charges behind them which must be fired in a precise time sequence in order to form that critical mass. This takes some pretty advanced electronics, and it's why nuclear warheads have a lifetime of about 8 years or so. The radiation from the uranium, and plutonium fries the electronics, corrodes the metal on the skin of the missile making it brittle and introducing metal fatigue, and distorts the precise geometry of the warhead parts which make the critical mass.
Incidentally, the most effective machine for mind control has been around since the 1930s, when it was developed and first used by the Nazis:
"Robert Kubey and Mihaly Csikszentmihalyi did a study which appeared in Scientific American in 2002[1]. Participants carried a beeper which beeped several times a day and when it did, they wrote down what they were doing and how they were feeling. When beeped while watching TV, people recorded feeling relaxed and passive. What was surprising was that the relaxation ended as soon as the TV was switched off, but the feelings of passivity and lowered alertness continued. Additionally, the participants had more trouble concentrating after viewing than before, and EEG studies showed less mental stimulation (identified by increased alpha brain wave production) while watching TV. Neither occurrences happened as a result of plain old reading. In other words, we associate “watching TV” with “being relaxed” (so we do relax), but after we finish watching we can’t concentrate, feel sluggish, and become as stressed (or more so) than before.
In order to understand television addiction, it’s important to note what is happening inside our brains. When you watch TV, brain activity switches from the left to the right hemisphere. How much? Research by Professor Herbert Krugman[3] showed that the right hemisphere becomes twice as active as the left, an extreme neurological anomaly. The crossover from left to right releases a surge of endorphins, which include beta-endorphins (pain numbing) and enkephalins. Endorphins are structurally identical to opium and its derivatives (morphine, codeine, heroin, etc.). Activities that release endorphins (also called opioid peptides) are usually habit-forming. External opiates act on the same receptor sites (opioid receptors) as endorphins, so there is little difference between the two. ...
There are further implications of the left-to-right hemisphere blood flow effect. Further research by Krugman revealed that our brain’s left hemisphere, which processes information logically and analytically, tunes out while we are watching television. The left hemisphere is the critical region for organizing, analyzing, and judging incoming data[4]. This tuning-out allows the right hemisphere of our brain, which processes information emotionally and uncritically, to function unimpeded.
In other words, we switch off our critical thinking abilities and just absorb anything thrown at us. We watch emotionally, not intelligently.
Further to this, psychophysiologist Thomas Mulholland found that after just 30 seconds of watching television the brain begins to produce alpha waves, which indicates torpid (almost comatose) rates of activity. Alpha brain waves are associated with unfocused, overly receptive states of consciousness (as with the left-to-right hemisphere shift). High frequency alpha waves do not normally occur when the eyes are open. In fact, Mulholland’s research implies that watching television is neurologically analogous to staring at a blank wall.[6] Production of alpha waves and the subsequent receptive state are also the goal of hypnotists. They’re both present during the “light hypnotic” state used by hypno-therapists for suggestion therapy. ... TV is everywhere these days: your phone; the internet; public spaces; download & watch it on your computer. The only real changes are the increased ease of time shifting (choosing when we watch), placeshifting (where we watch), and largely optional advertising." https://sidawson.org/2011/03/tv-is-heroin-crossed-with-hypnosis
With nuclear weapons being a lethal menace, and TV (and its descendants on the internet) being at best an attractive nuisance - and both of which were first used on a large scale nearly 80 years ago, lament of technology is nothing new - but people miss the mark when they lament the means for abuse, rather than take action against the person perpetrating the abuse. Guns don't fire themselves, after all... and cops act under orders in a command structure, where policy is set by those who pay them, it's why police forces tend to be composed of people with slightly below average intelligence - they follow orders instead of thinking about them.
I didn't suggest that AI is going to "take over the world" in some kind of colourful Terminator scenario. I'm certainly ignorant of how tech is made — and I *might* have been that naive the first time I heard the term "super intelligence" but I don't believe that. The potential problems, as I understand them (through reading more learned people, naturally), are malign use, which you allude to, and misaligned goals. The latter is something Russell, Sinclair and far greater scientists and inventors than I could be in a hundred lifetimes of dedicated study have been concerned about. I have no idea how much risk there is, of course, and I'm not going to pretend I ever will, but the fact that there *is* risk is not something one can dismiss with reference to credentialism.
What is a "misaligned goal"? Are we talking about unintended consequences here? "I'm sorry, sir, the computer won't let me do that" doesn't reflect sentience on the part of the computer, it reflects policy set by the person who ordered that the computer's output enforced that policy...
Yes, now I've misrepresented the film in an unfortunate way here because that’s more what I mean — it’s when the intended outcome of an instruction is misinterpreted in a destructive way. The most famous example is AI taking extreme measures to avoid being turned off because being turned on is essential to its goals — the second most famous, and far sillier, is an AI being asked to produce lots of paperclips and promptly turning valuable resources into paperclips (I should add that I don’t think *that* is going to happen).
"The most famous example is AI taking extreme measures to avoid being turned off because being turned on is essential to its goals" ... Space Odyssey 2001 - "Dave, do you really want to do this?" - turning off HAL, the AI computer - which resorted to extreme steps to "complete the mission.."
I was particularly struck by the comment concerning the seeming loss of the ability to comprehend the correct manner of risk assessment ie the dismissal of the potential for failure !!
All in all, an excellent exposé of the perils of eternal pessimism and the overwhelming requisite of not doing things just because we can.
1) It seems to me that material overabundance as much as (perhaps even more than) technological overreach is at the root of the dystopian aspects of life in the modern West. And tangentially related to this
2) it has always irritated me that most people take the technology they benefit from almost entirely for granted and show no interest in educating themselves about it. In fact it is a common chattering class pose to actually brag of one's 'technophobia'. Now no lay person could realistically do this in the case of modern electronics and computing but anyone is capable of understanding all the rest...irons, washing machines, cars (except electronics) power stations etc etc .. but how many do?
That is true....the 'happiness gene'. But I made a serious point - people who use their ingenuity to manage scarce resources, mend things etc get a lot of satisfaction from this. Our 'throw-away' culture is ultimately demoralising and infantilising. You can have my yacht...... if you can find it.
Don't fear technology, fear human nature.
Ben, thank you for sharing Andreessen's silly manifesto, I got a lot of laughs out of it. I don't think I've read anything this year with more meme potential. It's almost the Platonic ideal of tech CEO pomposity. There are too many highlights to list, but here are a few:
"We had a problem of isolation, so we invented the Internet"
"Love doesn’t scale"
Enlisting Nick Land, of all people, to argue that technology "is not anti-human – in fact, it may be the most pro-human thing there is".
"We believe in not Utopia, but also not Apocalypse"
(Halfway through I started mentally adding "In this house" before his endless "we believes")
"We believe in bravery, in courage" - those are the same thing mate
"We believe in pride, confidence, and self respect" - there's a pattern here
"We believe in the truth" - so do I as it happens
"We aspire to be… not that" - defining your aspirations in the negative, how Nietzschean
At the very bottom, other articles by him include "Disrupting the World’s Largest Asset Class with Adam Neumann".
'I can’t imagine anything with more potential for horror than eternal consciousness unmediated by the divine.'
Is bounded consciousness unmediated by the divine bearable only because we know it is finite? Sort of how you are able to sit through an orchestral piece by Schonberg because you know at some point it must surely end?
Yes, and because endless consciousness guided by man and his creations would be prey to the kind of flaws - deliberate and accidental - that could turn it into eternal torture.
I like your writing. You write like a normal bloke with normal feelings who has read enough about the modern world to explain to the rest of us normal but not so well-read people what the hell is going on.
Thank you so much!
Surely the answer is to eschew what Zvi Moshowicz calls the “Big Dial” model of progress -- whereby you’re either for or against *_Progress_* -- in favour of what I call the “Soundbwoy” model -- lots of little dials, you can twiddle them all at will; being a good soundbwoy, you probably want most of them set to High; but you have discretion to cut out GoF or AGI completely. (Incidentally, Big Dials are not restricted to tech progress discourse, and we could do with more soundbwoys across the board...)
Hi Jonathan! Yes, that's a nice image — clearly, it depends for optimists on whether the dials are powerful enough to produce sufficient outcomes, and to pessimists on whether they are powerful enough to constrain *excessive* outcomes but it's worth consideration.
> There is a good chance that we already live on the most beautiful, rich, poetic planet in the universe.
That's true! But it wouldn't be so unless the earliest lifeforms had decided to leave the most beautiful, rich, and poetic pocket of the ocean, and so on for their successors - leaving the garden is trad!
Good point!
The trouble with bring ignorant about how technology is created, as opposed to how it is used, leads to sentences like "From artificial intelligence to synthetic bioweapons, I suspect there is a good chance that mankind will come to lament technological progress — if, that is, he even has the time to do so." If you've ever done work in actual AI, you'd know that AI can't on its own take over the world, but it can be used as a tool to do so. It's like having a pile of 2 x 4s, some plywood, and a hammer and a bag of nails - you can sit and watch them all day long, but the hammer will not on its own create a bookshelf or a house. The hammer lacks consciousness and volition, it's up to you - who has both of those needful things - to use the hammer and nails, and other tools, and create whatever you decide, within the limits imposed by the materials. AI cannot on its own take over the world, but it can be used by people with consciousness and volition to do so.
The trouble with most journalists is that they have a faint idea of how technology "works" and armed with that faint idea, believe every bit of the techno-hype and technobabble that comes their way, leading to risible headlines and clickbait stories that scare the hell out of people - and grab attention and page clicks. If you've ever actually had to *create* technology, you'd know instantly that that stuff was technohype and technobabble, stop screwing around, and get back to work. It's not the technology, it's the use to which it is put. We are told to pay no attention to the Man Behind The Curtain, but it is that man which we should be concerned about, not the means which he uses to realize his intent.
I assume you have a computer. It is possible for you to actually create technology which can be used on that computer. I strongly suggest that you take the time to install a program called Emacs on your computer - and build it from source code instead of just downloading a working black box binary. It will come with instructions for doing so. You most probably have an application called "terminal" in which you can accomplish this. Emacs comes with detailed manuals, and you can use it to create new functionalities using Emacs LISP. You can get it all from here: https://www.gnu.org/software/emacs/ I strongly suggest you do so, so you can emerge from the ranks of the techno-gullible, and recognize technobabble and technohype for the content-free nonsense that it really is.
(Also, even if concerns about misalignment *are* groundless, the potential for machines with immense destructive capacities to be misused by human actors might in and of itself become a reason for people to lament technological progress, inasmuch as the more dangerous the weapon the more powerful the villain.)
A nuclear weapon is actually a machine, the parts of the warhead which form the critical mass to create the explosion are precisely machined and have explosive charges behind them which must be fired in a precise time sequence in order to form that critical mass. This takes some pretty advanced electronics, and it's why nuclear warheads have a lifetime of about 8 years or so. The radiation from the uranium, and plutonium fries the electronics, corrodes the metal on the skin of the missile making it brittle and introducing metal fatigue, and distorts the precise geometry of the warhead parts which make the critical mass.
Incidentally, the most effective machine for mind control has been around since the 1930s, when it was developed and first used by the Nazis:
"Robert Kubey and Mihaly Csikszentmihalyi did a study which appeared in Scientific American in 2002[1]. Participants carried a beeper which beeped several times a day and when it did, they wrote down what they were doing and how they were feeling. When beeped while watching TV, people recorded feeling relaxed and passive. What was surprising was that the relaxation ended as soon as the TV was switched off, but the feelings of passivity and lowered alertness continued. Additionally, the participants had more trouble concentrating after viewing than before, and EEG studies showed less mental stimulation (identified by increased alpha brain wave production) while watching TV. Neither occurrences happened as a result of plain old reading. In other words, we associate “watching TV” with “being relaxed” (so we do relax), but after we finish watching we can’t concentrate, feel sluggish, and become as stressed (or more so) than before.
In order to understand television addiction, it’s important to note what is happening inside our brains. When you watch TV, brain activity switches from the left to the right hemisphere. How much? Research by Professor Herbert Krugman[3] showed that the right hemisphere becomes twice as active as the left, an extreme neurological anomaly. The crossover from left to right releases a surge of endorphins, which include beta-endorphins (pain numbing) and enkephalins. Endorphins are structurally identical to opium and its derivatives (morphine, codeine, heroin, etc.). Activities that release endorphins (also called opioid peptides) are usually habit-forming. External opiates act on the same receptor sites (opioid receptors) as endorphins, so there is little difference between the two. ...
There are further implications of the left-to-right hemisphere blood flow effect. Further research by Krugman revealed that our brain’s left hemisphere, which processes information logically and analytically, tunes out while we are watching television. The left hemisphere is the critical region for organizing, analyzing, and judging incoming data[4]. This tuning-out allows the right hemisphere of our brain, which processes information emotionally and uncritically, to function unimpeded.
In other words, we switch off our critical thinking abilities and just absorb anything thrown at us. We watch emotionally, not intelligently.
Further to this, psychophysiologist Thomas Mulholland found that after just 30 seconds of watching television the brain begins to produce alpha waves, which indicates torpid (almost comatose) rates of activity. Alpha brain waves are associated with unfocused, overly receptive states of consciousness (as with the left-to-right hemisphere shift). High frequency alpha waves do not normally occur when the eyes are open. In fact, Mulholland’s research implies that watching television is neurologically analogous to staring at a blank wall.[6] Production of alpha waves and the subsequent receptive state are also the goal of hypnotists. They’re both present during the “light hypnotic” state used by hypno-therapists for suggestion therapy. ... TV is everywhere these days: your phone; the internet; public spaces; download & watch it on your computer. The only real changes are the increased ease of time shifting (choosing when we watch), placeshifting (where we watch), and largely optional advertising." https://sidawson.org/2011/03/tv-is-heroin-crossed-with-hypnosis
With nuclear weapons being a lethal menace, and TV (and its descendants on the internet) being at best an attractive nuisance - and both of which were first used on a large scale nearly 80 years ago, lament of technology is nothing new - but people miss the mark when they lament the means for abuse, rather than take action against the person perpetrating the abuse. Guns don't fire themselves, after all... and cops act under orders in a command structure, where policy is set by those who pay them, it's why police forces tend to be composed of people with slightly below average intelligence - they follow orders instead of thinking about them.
I didn't suggest that AI is going to "take over the world" in some kind of colourful Terminator scenario. I'm certainly ignorant of how tech is made — and I *might* have been that naive the first time I heard the term "super intelligence" but I don't believe that. The potential problems, as I understand them (through reading more learned people, naturally), are malign use, which you allude to, and misaligned goals. The latter is something Russell, Sinclair and far greater scientists and inventors than I could be in a hundred lifetimes of dedicated study have been concerned about. I have no idea how much risk there is, of course, and I'm not going to pretend I ever will, but the fact that there *is* risk is not something one can dismiss with reference to credentialism.
What is a "misaligned goal"? Are we talking about unintended consequences here? "I'm sorry, sir, the computer won't let me do that" doesn't reflect sentience on the part of the computer, it reflects policy set by the person who ordered that the computer's output enforced that policy...
Yes, now I've misrepresented the film in an unfortunate way here because that’s more what I mean — it’s when the intended outcome of an instruction is misinterpreted in a destructive way. The most famous example is AI taking extreme measures to avoid being turned off because being turned on is essential to its goals — the second most famous, and far sillier, is an AI being asked to produce lots of paperclips and promptly turning valuable resources into paperclips (I should add that I don’t think *that* is going to happen).
"The most famous example is AI taking extreme measures to avoid being turned off because being turned on is essential to its goals" ... Space Odyssey 2001 - "Dave, do you really want to do this?" - turning off HAL, the AI computer - which resorted to extreme steps to "complete the mission.."
I was particularly struck by the comment concerning the seeming loss of the ability to comprehend the correct manner of risk assessment ie the dismissal of the potential for failure !!
All in all, an excellent exposé of the perils of eternal pessimism and the overwhelming requisite of not doing things just because we can.
Thank you Stephen!
Two thoughts:
1) It seems to me that material overabundance as much as (perhaps even more than) technological overreach is at the root of the dystopian aspects of life in the modern West. And tangentially related to this
2) it has always irritated me that most people take the technology they benefit from almost entirely for granted and show no interest in educating themselves about it. In fact it is a common chattering class pose to actually brag of one's 'technophobia'. Now no lay person could realistically do this in the case of modern electronics and computing but anyone is capable of understanding all the rest...irons, washing machines, cars (except electronics) power stations etc etc .. but how many do?
A very interesting and enjoyable post though.
Any over abundance you have you can give to me.
Unfashionable I know but there is much to be said for the frugal mindset.
The guy with the Super Yacht that holds 200,000 gallons of fuel agrees with you.
Ah but is he 'happy'?
We both know happy don't happen.
You ethier got it or you don't.
That is true....the 'happiness gene'. But I made a serious point - people who use their ingenuity to manage scarce resources, mend things etc get a lot of satisfaction from this. Our 'throw-away' culture is ultimately demoralising and infantilising. You can have my yacht...... if you can find it.