June 13th, 2022
- "seems so" as an alternative to "I agree" or "that's true"
- it's like "I agree" in that it retains a subjective sense
- it's like "that's true" in that it doesn't emphasize the self
- (from convos with Daniel S)
- reading Eliezer Yudkowsky on AGI Ruin
- > 22. There's a relatively simple core structure that explains why complicated cognitive machines work; which is why such a thing as general intelligence exists and not just a lot of unrelated special-purpose solutions; which is why capabilities generalize after outer optimization infuses them into something that has been optimized enough to become a powerful inner optimizer. The fact that this core structure is simple and relates generically to low-entropy high-structure environments is why humans can walk on the Moon. There is no analogous truth about there being a simple core of alignment, especially not one that is even easier for gradient descent to find than it would have been for natural selection to just find 'want inclusive reproductive fitness' as a well-generalizing solution within ancestral humans. Therefore, capabilities generalize further out-of-distribution than alignment, once they start to generalize at all.
- there's something clearly true about this, in a basic sense—that's core to the orthogonality thesis
- tho also evolution had no need to find such a thing
- but I do also find myself feeling like I have directly encountered some sense in which there is a truth about a simple core of alignment, that's something about omniwins / the bit in the middle / upward spiral etc
- this does at least seem true about human systems, in such a way that makes the human alignment problem (re ideologies & institutions) tractable tho difficult
- and if this were more fully understood, we might be able to see how it does (or could be made to) apply to AGI as well
- left hemispherey stuff seems relevant here, as does PCT
- ahh, relatedly to this, it's something like, okay, we have diverged in alignment from "inclusive reproductive fitness"
- but we haven't diverged from some more general meta evolutionary thing around optimizing for increased capacity for consciousness
- which in some sense is actually increased fitness, like it's an obvious attractor
- I feel like something is being slightly lost here in terms of the nature of evolution
- evolution doesn't care if there are fewer total humans or humans are having fewer babies
- there is simply a fact of the matter of what patterns persist at a given time
- some humans using birth control is IRRELEVANT to this
- humans coming to potentially destroy the biosphere via AGI or via climate change is much more relevant in some sense
- there's a figure-ground inversion needed here, which I feel like EY gets on one level but then is failing to make here