Eliezer Yudkowsky
No content on this page itself — see linked references below.
5 linked references
June 13th, 2022
- reading Eliezer Yudkowsky on AGI Ruin
- > 22. There's a relatively simple core structure that explains why complicated cognitive machines work; which is why such a thing as general intelligence exists and not just a lot of unrelated special-purpose solutions; which is why capabilities generalize after outer optimization infuses them into something that has been optimized enough to become a powerful inner optimizer. The fact that this core structure is simple and relates generically to low-entropy high-structure environments is why humans can walk on the Moon. There is no analogous truth about there being a simple core of alignment, especially not one that is even easier for gradient descent to find than it would have been for natural selection to just find 'want inclusive reproductive fitness' as a well-generalizing solution within ancestral humans. Therefore, capabilities generalize further out-of-distribution than alignment, once they start to generalize at all.
- there's something clearly true about this, in a basic sense—that's core to the orthogonality thesis
- tho also evolution had no need to find such a thing
- but I do also find myself feeling like I have directly encountered some sense in which there is a truth about a simple core of alignment, that's something about omniwins / the bit in the middle / upward spiral etc
- this does at least seem true about human systems, in such a way that makes the human alignment problem (re ideologies & institutions) tractable tho difficult
- and if this were more fully understood, we might be able to see how it does (or could be made to) apply to AGI as well
- left hemispherey stuff seems relevant here, as does PCT
- ahh, relatedly to this, it's something like, okay, we have diverged in alignment from "inclusive reproductive fitness"
- but we haven't diverged from some more general meta evolutionary thing around optimizing for increased capacity for consciousness
- which in some sense is actually increased fitness, like it's an obvious attractor
- I feel like something is being slightly lost here in terms of the nature of evolution
- evolution doesn't care if there are fewer total humans or humans are having fewer babies
- there is simply a fact of the matter of what patterns persist at a given time
- some humans using birth control is IRRELEVANT to this
- humans coming to potentially destroy the biosphere via AGI or via climate change is much more relevant in some sense
- there's a figure-ground inversion needed here, which I feel like EY gets on one level but then is failing to make here
August 9th, 2021
- great post by Eliezer Yudkowsky on illusion of transparency and how he learned while writing HPMOR that his words weren't sufficient to pin down his meanings
- > So it taught me that Illusion of Transparency isn't about hint-missing, it's about there really actually respectably being multiple interpretations. And that was when I started to understand that my words were really actually not pinning down the meaning I had in my own mind, because I couldn't see all the possibilities I needed to exclude, because I was blinded by my own knowledge of the intended answer.
March 22nd, 2021
- I read some chunks of this huge mess of an exchange between David Chapman and Eliezer Yudkowsky on twitter, which is largely visible scrolling up from here:
- and at one point Eliezer Yudkowsky questions David Chapman's claim that people think bayes rules is central to probability, and some guy brilliantly manages to find this tweet from EY himself just 6 weeks prior saying:
- this is him utterly equating the two, as far as I can tell! (and, what's more, kind of implying that others are embarrassingly naive for not realizing they're the same)
February 13th, 2021
- Eliezer Yudkowsky's article Your Cheerful Price
- talks about the challenge of dividing gains from trade
- and the ongoing game of how that division takes place
- and how cheerful prices encourage ongoing positive vibes